In PHP, sometimes you would just add something like this:

echo "I'm here!";
exit();

However, in Nginx configuration files, it isn’t as easy…

… or is it?

One thing that works well is the rewrite directive. You can append variables to the URL to be rewritten. Another great thing is a rewrite statement can go just about anywhere. So lets say we were trying to debug this location statement:

Now lets say its returning a 404, and I’m not 100% sure what the actual value of $fastcgi_script_name is. I can add this to it:

This will redirect your HTTP request to Google.com and put the value in the query textfield. Bingo, I can easily see the actual value! Pretty helpful when you have a large, complex server definition.

Related posts:

1. PHP, Nginx, and Output Flushing
2. Setting Up Nginx & PHP-FPM on Ubuntu 10.04
3. Simple Trick: History Command

Why Workers?

Sometimes there are situations when you want to parallel process things. Other times you might have a list of tasks to accomplish, and you don’t want to make the user wait after pressing a button. This is where “Workers” can come in. They are independent scripts that run along side of your application, performing tasks, or “jobs.”

An example is with Dating DNA and our score system. We generate scores between users to show how compatible they are with each other. When a user signs up, or makes a significant change to their profile questionnaire, we need to run a job to query our database, build a list of potential users, and generate scores. This takes 10-20 seconds, and while it is pretty fast, we don’t want to make the user wait for that. So we queue up a job for the user, divide up the work among several workers, and process the work.

General Concept

For this post, we’ll use the example of generating reports. Lets say on your internal website there is a button that you can click and it will email the user a report, and the report takes 2-3 minutes to generate. When the button is clicked, your code will insert the job into the queue. Meanwhile, workers are monitoring the queue. A worker script will pull the job off the queue, process the report, and send the email when its done.

For the queue management, we’ll use Redis. To let PHP read and write data to Redis, we’ll use the PHP Library predis. In our examples we’ll use PHP 5.3, however predis has a PHP 5.2 backport if you are not running 5.3.

Adding Jobs

To add jobs, we’ll need to connect to our Redis server:

/*
 * Connecting to Redis
 */

const REDIS_HOST = '127.0.0.1';
const REDIS_PORT = 6379;

$predis = new Predis\Client(array(
    'scheme' => 'tcp',
    'host'   => REDIS_HOST,
    'port'   => REDIS_PORT,
));

We’ll assume in all of our examples that we’ve done the following above & connected to Redis.

Now, to manage our queues we’ll use the Redis Datatype LIST. Whats awesome about lists is that regardless of size, adding or removing at the start or end of a list is extremely fast. So if your queue has 10 items, or 10,000,000 items, Redis wil be able to push and pop entries quickly.

We’ll have three queues, one for each priority: high, normal, and low. For the Redis key names, we’ll use queue.priority.high, queue.priority.normal, etc. When interacting with lists, you work with the ends, one called right, the other called left. So we’ll add items on the right with the RPUSH (Right Push) command, and we’ll pull items off the left with the BLPOP (Blocking Left Pop) command. We won’t worry about the pulling items just yet.

You store strings as the values for the list. My personal preference is to store JSON objects so you can easily pass variables needed to perform the job.

/*
 * Adding items to the queue
 */

$job = new stdClass();
$job->id = 1;
$job->report = 'general';
$job->email = 'test@example.com';

// Add the job to the high priority queue
$predis->rpush('queue.priority.high', json_encode($job));

// Or, you could add it to the normal or low priority queue.
$predis->rpush('queue.priority.normal', json_encode($job));
$predis->rpush('queue.priority.low', json_encode($job));

Simple enough! Having different queue priorities is very beneficial in managing which jobs should get done first. For example, you might have an Executive’s request go into the high priority queue so they get the report quickly. You might also have a weekly cron that queues up reports to be sent automatically, so those can go in the low priority as to not disrupt people trying to get a manual report.

Now, on to the worker’s code.

Processing Jobs

For now, lets say we have a script running in the PHP CLI (Command Line Interface) that you started by running this command on the server:

First thing is we want this worker to work continuously, so we can do a while loop:

/*
 * Simple Continuous While Loop
 */

// Always True
while(1)
{
	/* ... perform tasks here ...  */
}

We’ll worry about making them more intelligent later. Now, let’s have our worker check the queue. You can do so with the BLPOP command:

/*
 * Checking the Queue
 */
$job = $predis->blpop('queue.priority.high'
						, 'queue.priority.normal'
						, 'queue.priority.low'
						, 10);

What we’re telling PHP to do is to check each queue in order of priority: high, normal, and then low. If it finds an item, it will immediately return an array with the name of the queue it came from, and the string of data that was pulled.

The B in BLPOP is “blocking.” What that means is that Redis will wait until either an item enters one of the queues, or the timeout is reached. In this case, the timeout is 10 seconds. So instead of polling (checking every few seconds in a loop), we check and wait, and after 10 seconds it will return null and we can check again.

What this gives us is near instantaneous queues. As soon as something is available, it is passed to the workers that are listening. You can also have multiple workers, and it will pass jobs to the first listening worker, and the next job to the next worker, so you don’t have to worry about multiple workers getting the same queued item.

After $predis->blpop() returns, if it has an array, it returned an item. If not, the timeout had been reached. We can check to see if a Job was returned, and if so to process the job:

/*
 * Checking to see if a Job was returned
 */

if($job)
{
	// Index 0 of the array holds which queue was returned
	$queue_name = $job[0];
	// Index 1 of the array holds the string value of the job.
	// Since we are passing it JSON, we'll decode it:
	$details = json_decode($job[1]);

	/* ... do job work ... */
}

Now we can have multiple workers listening to the same queues and scale our workload. Redis is very fast & efficient, and you could have hundreds or even thousands of workers listening to a single redis server.

Continuously Running Workers

There are a lot of options when it comes to deploying these workers. You can use a framework like Gearman, but for simple things, I like very simple solutions. I came across a blog post by Joseph Scott about a little 10 line perl script called solo. What it does is it will run a command, and to ensure that no one else is running that same exact command, it will lock a configurable port. This is awesome because the you don’t have to work about lock files or filesystem tricks, the kernel handles it all.

So what you can do is create a cronjob using solo to execute your script. First copy solo somewhere, I put it in my /usr/local/bin on my linux server. Then add this to your cron job using the command “crontab -e -u (which user to use)”:

What this will do is try to run this command every minute. Solo will check to see if the port is already in use, and if it is, it will exit. Otherwise, it will lock the port and then execute the command. The port will stay locked as long as the command is executing. Once the command terminates, the port will unlock.

Now, PHP is a great language, but it has been known to have some memory leaks while running a long time in a single instance. So we can have our scripts exit periodically to be restarted by our cron job. So lets make our “while(1)” statement a little smarter:

/*
 * A Smarter While Statement
 */

// Set the time limit for php to 0 seconds
set_time_limit(0);

/*
 * We'll set our base time, which is one hour (in seconds).
 * Once we have our base time, we'll add anywhere between 0
 * to 10 minutes randomly, so all workers won't quick at the
 * same time.
 */
$time_limit = 60 * 60 * 1; // Minimum of 1 hour
$time_limit += rand(0, 60 * 10); // Adding additional time

// Set the start time
$start_time = time();

// Continue looping as long as we don't go past the time limit
while(time() < $start_time + $time_limit)
{
	/* ... perorm BLPOP command ... */
	/* ... process jobs when received ... */
}

/* ... will quit once the time limit has been reached ... */

One key thing to note is randomly shifting the time limit for the script. I like to do this because you don’t want your workers all stopping and starting at the same time. So if I have 8 workers, one might, but the 7 will continue until the 8th starts back up again via the cron job.

Bells & Whistles

After using workers for awhile, here are a couple of ideas to enhance your workers & system managing them. First off, you can add some monitoring for your queues. Using Redis a HASH, you can use them to store the state of your workers.

/*
 * Assigning Worker IDs & Monitoring
 *
 * Usage: php worker.php 1
 */

// Gets the worker ID from the command line argument
$worker_id = $argv[1];

// Setting the Worker's Status
$predis->hset('worker.status', $worker_id, 'Started');

// Set the last time this worker checked in, use this to
// help determine when scripts die
$predis->hset('worker.status.last_time', $worker_id, time());

Another problem with workers that run for a long time (several hours) is when you make a change to their code, they won’t reload that change until they exit. What I’ve found to successfully restart them is having a “version” number set in Redis that is checked at the end of every loop:

/*
 * Using Versions to Check for Reloads
 */

$version = $predis->get('worker.version'); // i.e. number: 6

while(time() < $start_time + $time_limit)
{
	/* ... check for jobs and process them ... */

	/* ... then, at the very end of the while ... */
	if($predis->get('worker.version') != $version)
	{
		echo "New Version Detected... \n";
		echo "Reloading... \n";
		exit();
	}
}

You would simply INCR (increment) worker.version and after finishing their last job, the worker would exit, and solo would start it up again.

You can also kill specific threads by having them check for their value in a hash:

/*
 * Using Kill Switches to Check for Reloads
 */

while(time() < $start_time + $time_limit)
{
	/* ... check for jobs and process them ... */

	/* ... then, at the very end of the while ... */
	// Check to see if a kill has been set.
	if($predis->hget('worker.kill', $worker_id))
	{
		// Make sure to unset the kill request before exiting, or
		// your worker will just keep restarting.
		$predis->hdel('worker.kill', $worker_id);

		echo "Kill Request Detected... \n";
		echo "Reloading... \n";
		exit();
	}
}

Tweak to Solo & Logging

I made one small tweak in my version of solo, and that was to help it enable logging. Lets say I had three workers in my crontab:

The “>> /tmp/worker.log.1″ tells solo I want to log it’s output to a tmp file that I can tail and monitor their progress. This is great for debugging problems. However, when I did this, solo would write to the tmp file, and not the output from my script. To overcome this I changed the last line of solo:

# old
exec @ARGV;
# new
exec "@ARGV";

This would ensure my script wrote out to the tmp file, and not just solo.

Examples

I’ve created an example on GitHub that you can clone on your own machine. All you will need is PHP 5.3 and Redis installed.

To install redis, simple run these commands on your unix based system:

It will copy the redis binaries to /usr/local/bin.

To get a copy of the code, you can download them here. HOWEVER, it doesn’t include predis! You’ll have to download and copy predis inside there via this link. It is much easier to clone it as so:

Then, using different terminal windows (or using screen), you can run different worker.php instances, use creator.php to insert jobs, and monitor.php to watch the progress. This is all done from the command line.

If you’re using windows, I suggest installed a VM of Ubuntu and using that. If you really want to use Redis on windows, there are some Windows Binaries you can google and download. Good luck!

Here is a video where I demo the example:

(sorry for the poor mic quality)

Final Thoughts

I’ll post here shortly about how to run Redis in production with the init.d scripts and configuration files. One caveat to using solo is if your server has an application that randomly selects ports to use (i.e. VoIP, FTP), it might select one of your worker’s ports. But on a production server, you should have a good feel for which ports are available for locking.

If you want to learn more about Redis, check out their website.

Hopefully this will be helpful for anyone looking to use PHP Workers in an easy, simple way.

Related posts:

1. MySQL, Redis, and a Billion Rows – A Love Story
2. Creating Chatroom / Walls with Redis & PHP
3. Debuging with PHP, Stack Traces, and Redis

Siege is an http load testing and benchmarking utility. It was designed to let web developers measure their code under duress, to see how it will stand up to load on the internet. Siege supports basic authentication, cookies, HTTP and HTTPS protocols. It lets its user hit a web server with a configurable number of simulated web browsers. Those browsers place the server “under siege.”

On Linux, you can install it easily via yum or apt. On OS X, you can install it easily via homebrew or macports. I personally prefer homebrew. In your terminal just type:

Now you can stress test a given app by simply using the siege command:

You can read the documentation and faq on their site to learn more. However, there is one question I get a lot when talking about siege: “How can I use siege to test users who are logged in?”

I would always recommend that they send a session header in the config, but today I learned there is an even easier way (although undocumented), thanks to this question in Server Fault. In your siege’s configuration you can add the login-url directive:

Awesome! Now each connection will authenticate before continuing their siege. A very handy tool, especially if your load is generated by user account pages and such.

Related posts:

1. Setting up SSH Key Authentication Between Servers
2. Local LAMP Developement & User Content
3. My PHP User Group Experience

Related posts:

1. SoapUI – a simple and raw Java SOAP Client
2. Memcached: Simple, Effective, and Powerful
3. ASP .NET Ajax – Straight forward and simple, the way it should be.

Problem

Our current backup system uses mysqldump. It dumps a 25 GB sql dump file, which compresses to about 2.5GB using gzip. The last time we needed to restore a backup it was only about 9GB, and it took several hours.

This time, I created the database, and from the mysql prompt I issued the following command:

It would run great until it got to about 10% of the way through the scores table. However, it would start to slow down. Because the rows were so small in our scores table, each INSERT statement had about 45,000-50,000 records. So each line had roughly 1MB of data.

At first it would insert a set of 50,000 in half a second or so. However, after a few million records, it would slow down to three second, and got to about 10 seconds per INSERT statement. This was a huge problem, given that I had roughly 18,000 INSERT statements, and at 10 seconds per INSERT, it would take 50 hours to restore. Our website was down during this restore, since it was our primary database. So being down for over two days was not an option.

While trying to diagnose the problem I noticed something. While using the MySQL command “show processlist” the thread for the Database Restore would be in the sleep state for 9-10 seconds, and then the query would execute in under 0.2 seconds. So it wasn’t a problem with MySQL storing the data, but a problem with reading the data from such a large database dump file.

So I tried from the server’s command line “mysql -u user_name -p database_name < /path/to/backup/database_dump.sql" with the same result. The longer into the file I got, the longer it was taking for MySQL to read the query.

Solution

So, after some thinking late at night at 3 AM, I came up with an idea. Why not split up the database sql dump into multiple files. So I used the linux “split” command like this:

This produced several dozen files in order, and it took about 10 minutes. The -n option told split to split each file up into 200 lines each. So the files were then named sql_aa, sql_ab, sql_ac all the way to sql_fg. Then, I did the following command using cat to pipe the files to mysql:

The only problem with this method is you don’t see a status report for each query executed, it just runs until you hit an error, displaying the error. If no errors occur, it will just return you to the prompt. So to monitor the progress I would execute a “show processlist;” command on mysql to see how far we were.

4 1/2 hours later, the entire database was restored. A few things to note, I didn’t try just using cat on the original file to see if it would read the file differently than the was mysql was trying. But the important thing is I got the database restored in a relatively timely manner.

Hopefully, in the very near future, we will have moved to a new score system that doesn’t have almost a billion rows in it.

Related posts:

1. MySQL, 40 Million Rows, MyISAM to InnoDB, 45 Minutes
2. Large MySQL Conversion & InnoDB
3. MySQL & PHP – SQL_CALC_FOUND_ROWS – An easy way to get the total number of rows regardless of LIMIT

ob_flush(); // Flush anything that might be in the header output buffer
flush(); // Send contents so far to the browser

Well, with Nginx, it will wait for an entire response from the PHP-FPM instance before sending data to the browser. This is because traditionally the time to generate the response is less than the time to send the response to the browser, so Nginx will let the CGI instance finish as quick as possible to free it up for other requests. Well, even if I called ob_flush() and flush(), Nginx would wait for the entire response before sending it to the client’s browser. So, for our staff panel, I had to disable this buffering. It took a lot of scouring the web, but I finally figured it out:

Nginx Configuration

You need to set a few variables. I couldn’t actually figure out how to disable the buffer for Nginx, but I could set it very low on a per location configuration.

So I have the following location configurations:

The important configurations are:

So I set the fastcgi_buffer_size and buffers to 1k. Then, you need to set the max temp file size to 0. Nginx by default will start to buffer on the disk. By setting it to 0 it will send it to the browser. The last piece of the puzzle I missed was turning gzip off, because it would try to buffer, even if it exceeded 1k, to compress the response.

Now, to get it to work, in my php script I echo out 1k worth of html commented out text to ensure everything else after it gets sent to the browser.

Now, I don’t recommend these settings for busy productions servers. However, only 3 people use this staff panel, so the performance differences are extremely low.

Related posts:

1. PHP Practices: Buffering your autoload
2. SwiftMailer – Sending Mail in PHP With Ease
3. Local LAMP Developement & User Content

function selectLocation()
{
	$('#modalSelectLocation').dialog({
			modal: true,
			dialogClass: 'modalSelectLocation'
			,buttons: {
				Cancel: function(){
					$(this).dialog('close');
				}
				,Continue: function() {
					alert('goto-buy');
				}

			}
		});
}

with the css of:

.modalSelectLocation .ui-dialog-titlebar { display:none; }

Tada! No more Title Bar. However, make sure to have some form of cancel button.

Related posts:

1. jQuery Tip: Better Toggle
2. Empowering JavaScript Through jQuery
3. CSS – ID vs Class

The Problem

When creating virtual machines, especially virtual linux servers, I want them to be able to do the following things:

1. Connect to the Internet
2. Connect over SSH from my Host Machine to the Guest Machine
3. Have the servers connect to each other on a private internal network

Out of the box, a VM is configured for #1, however it cannot do #2 and #3. There are some solutions that do some archaic port forwarding by issuing special commands from the Terminal. This is just painful, and if you just want one port to work, that’s manageable, but what if you want ssh, ftp, myql, http, https, etc? Any on multiple VMs? What a nightmare! What is painful is when googling for a solution, almost every “answer” is this crazy port forwarding method. There is a much cleaner, better way!

Solution – Host to Guest Connectivity

Go into your VM’s settings and go to “Network”, it will list options for 4 different adapters. Instead of hacking one adapter to do all the work, just setup more adapters for your configuration. For adapter #2, set it to Host-only. This enables an IP range for communicating from the Host to the Guest and vice versa. Start up your VM, and if you are using something like Ubuntu, edit your /etc/network/interfaces file so it looks like this:

After adding these lines, run the command:

Now VirtualBox is configured to host this traffic over the 192.168.56.x IP range (I’m not sure if/where you can change this). So do an ifconfig on your guest machine, and it should show you the IP it has been assigned. If you want, you can change your /etc/network/interfaces so it can be statically assigned. You should now be able to SSH into your box on it’s IP:

Solution – Guest to Guest Communication

Now, what if I have a Cassandra server, a Web Server, and a MySQL server each on their own VMs. What you do is create a third adapter set to “Internal Network”. Then in your /etc/network/interfaces file you set eth2 to a static IP of your choosing. I typically use the 10.0.0.x range. So each server is respectively 10.0.0.10, 10.0.0.20, and 10.0.0.30. However, over the 10.0.0.x range there is no DHCP, no Gateway, no nothing, just direct communications over these IPs.

Conclusion

So please, for the love of everything good, don’t resort to crazy port forwarding, complicated XML file editing, solutions for VirtualBox. Just use each type of interface configuration for what you need. Thank you!

Related posts:

1. My Computer Setup
2. Brother HL-2070N – Sharing an Vista Printer to a XP Machine
3. Local LAMP Developement & User Content

If you have any questions or comments, please feel free to leave them below.

Related posts:

1. Steamlined Web Development Presentation Video
2. 100th Blog Entry – First Year of Real Blogging
3. A New Stage in Life: My MacBook Pro

I decided to go with a 1 TB drive. Then I had to decide if I wanted to put it inside my PC (I can fit 4 HDDs, so no problem) or get an external. The biggest thing I loathe about external hard drives is the slow transfer speed over USB. I like the convenience of smaller external drives that don’t require an external power source. I own a few smaller external hard drives, but they aren’t good for my 60GB backups.

This is where my new toy comes in: the Thermaltake BlacX. Its basically a docking station for any 2.5″ or 3.5″ SATA hard drive. So now instead of swapping around cords and such, I have my docking station hooked up via an eSATA connection. So I can hot swap archive hard drives, keep a 300Gb/s transfer speed, and have the convenience of no swapping cables.

So far it as been really great. It also supports a USB 2.0 connection if your PC doesn’t support eSATA. I highly recommend it to anyone looking to be able store and archive data on multiple hard drives.

No related posts.