Eric Day

New Gearman Server & Library in C, MySQL UDFs

We’re happy to announce the first version of the Gearman rewrite in C, along with some interesting new MySQL UDFs based on the C library. Check out the Gearman wiki for an overview of what this is, download details, and API documentation.

For the anxious, here is a quick how-to:

Download: http://launchpad.net/gearmand/trunk/0.1/+download/gearmand-0.1.tar.gz
tar xzf gearmand-0.1.tar.gz
cd gearmand-0.1/
./configure
make
make install

You should now have the job server (gearmand) installed in /usr/local/bin, along with headers in libraries in /usr/local/lib & include. You can now see it work by running some simple clients and workers in the examples/ directory:

gearmand &      (this is assuming /usr/local/bin is in your path)
cd examples
./reverse_worker &
./reverse_client "Hello, Gearman!"

If everything went well, your terminal should look something like this:

> gearmand &
[1] 2270
> ./reverse_worker &
[2] 2301
> ./reverse_client "Hello, Gearman!"
Job=H:lap:1 Workload=Hello, Gearman! Result=!namraeG ,olleH
Result=!namraeG ,olleH

What happened is the reverse_client program sent the string “Hello, Gearman!” to the job server (gearmand), telling it to run it with a function named “reverse”. The job server then found a worker is registered (out reverse_worker we started), and forwarded the job on to it. The reverse_worker program then reversed the string, and sent it back to the job server, which then sends it back to the original reverse_client. The “Job=…” line is the output from the reverse_worker, saying it received a job with the given workload, and sent a result back. The “Result=…” line is the final output from the reverse_client. You have now run your first Gearman job!

Before getting into the MySQL UDFs, lets look at how this could scale in a useful way. Imagine your reverse_client is actually an client inside of your PHP or Perl script running on your webserver. Next, your could have multiple job servers to spread the load and get redundancy, and then you could have a farm of machines running the workers. You now have a simple distributed computing framework to do time and/or resource intensive jobs like document or image conversion.

Now lets see how we can make MySQL run Gearman jobs:

Download: http://launchpad.net/gearman-mysql-udf/trunk/0.1/+download/gearman-mysql-udf-0.1.tar.gz
tar xzf gearman-mysql-udf-0.1.tar.gz
cd gearman-mysql-udf-0.1/
./configure --with-mysql=/usr/local/mysql/bin/mysql_config --libdir=/usr/local/mysql/lib/plugin/
make
make install

You may need to change your configure line depending on where you have MySQL installed (this is assuming /usr/local/mysql). Configure needs to know where the mysql_config tool is and where to install the plugin. The above assumes that your plugins should be in lib/plugin, but they may just be in lib. You will need to check your MySQL installation to see what paths you should use here.

Once you get the paths correct and everything installed, you can now load them through the MySQL command line tool:

CREATE FUNCTION gman_do RETURNS STRING
       SONAME "libgearman_mysql_udf.so";
CREATE FUNCTION gman_do_high RETURNS STRING
       SONAME "libgearman_mysql_udf.so";
CREATE FUNCTION gman_do_background RETURNS STRING
       SONAME "libgearman_mysql_udf.so";
CREATE AGGREGATE FUNCTION gman_sum RETURNS INTEGER
       SONAME "libgearman_mysql_udf.so";
CREATE FUNCTION gman_servers_set RETURNS STRING
       SONAME "libgearman_mysql_udf.so";

If you get errors, the most likely case is that MySQL cannot find the module in the library path. Try a different libdir in the configure above and reinstall. Once these are loaded, you can now tell it where to find a job server and then run a Gearman job from a query. This test assumes you still have gearmand and reverse_worker running in the terminal from the previous example.

mysql> SELECT gman_servers_set("127.0.0.1");

+-------------------------------+
| gman_servers_set("127.0.0.1") |
+-------------------------------+
| NULL                          |
+-------------------------------+
1 row in set (0.00 sec)

mysql> SELECT gman_do("reverse", Host) AS test FROM mysql.user;
+-----------+
| test      |
+-----------+
| 1.0.0.721 |
| pal       |
| pal       |
| tsohlacol |
| tsohlacol |
+-----------+
5 rows in set (0.00 sec)

Now you have a MySQL UDF that runs as a normal Gearman client! See the README on more information on how to run other types of jobs. I’ll be writing another blog entry shortly with a more interesting use case using these UDFs.

So what are the next steps for Gearman? We’ll be improving the server and clients a bit more, a new PHP extension based on the C library, more language wrappers using SWIG, persistent queues, queue replication, and also seeing how we’ll be able to plug this into other applications.

4 Responses to "New Gearman Server & Library in C, MySQL UDFs"

1. Gearmand Rewrite Released — g.raphaelli's weblog says:

   January 9, 2009 at 3:43 am

   [...] written by g, on Jan 9, 2009 5:43:40 PM. There is some big news for Gearman fans out today – the first release of the C-rewrite of the gearmand job server has been announced. [...]
2. Gilad Raphaelli says:

   January 9, 2009 at 4:03 am

   Congratulations and thank you very much!
3. Oddments » Blog Archive » Gearman MySQL UDFs: URL Processing says:

   January 9, 2009 at 2:08 pm

   [...] « New Gearman Server & Library in C, MySQL UDFs [...]
4. Oddments » Blog Archive » Gearman Persistent Queues & Replication says:

   January 14, 2009 at 5:41 am

   [...] that the core of Gearman is rewritten in C, I’ve started working on a new module to provide persistent queues and queue replication. The [...]

Leave a Reply