In PHP, sometimes you would just add something like this:

echo "I'm here!";
exit();

However, in Nginx configuration files, it isn’t as easy…

… or is it?

One thing that works well is the rewrite directive. You can append variables to the URL to be rewritten. Another great thing is a rewrite statement can go just about anywhere. So lets say we were trying to debug this location statement:

Now lets say its returning a 404, and I’m not 100% sure what the actual value of $fastcgi_script_name is. I can add this to it:

This will redirect your HTTP request to Google.com and put the value in the query textfield. Bingo, I can easily see the actual value! Pretty helpful when you have a large, complex server definition.

Related posts:

1. PHP, Nginx, and Output Flushing
2. Setting Up Nginx & PHP-FPM on Ubuntu 10.04
3. Simple Trick: History Command

Why Workers?

Sometimes there are situations when you want to parallel process things. Other times you might have a list of tasks to accomplish, and you don’t want to make the user wait after pressing a button. This is where “Workers” can come in. They are independent scripts that run along side of your application, performing tasks, or “jobs.”

An example is with Dating DNA and our score system. We generate scores between users to show how compatible they are with each other. When a user signs up, or makes a significant change to their profile questionnaire, we need to run a job to query our database, build a list of potential users, and generate scores. This takes 10-20 seconds, and while it is pretty fast, we don’t want to make the user wait for that. So we queue up a job for the user, divide up the work among several workers, and process the work.

General Concept

For this post, we’ll use the example of generating reports. Lets say on your internal website there is a button that you can click and it will email the user a report, and the report takes 2-3 minutes to generate. When the button is clicked, your code will insert the job into the queue. Meanwhile, workers are monitoring the queue. A worker script will pull the job off the queue, process the report, and send the email when its done.

For the queue management, we’ll use Redis. To let PHP read and write data to Redis, we’ll use the PHP Library predis. In our examples we’ll use PHP 5.3, however predis has a PHP 5.2 backport if you are not running 5.3.

Adding Jobs

To add jobs, we’ll need to connect to our Redis server:

/*
 * Connecting to Redis
 */

const REDIS_HOST = '127.0.0.1';
const REDIS_PORT = 6379;

$predis = new Predis\Client(array(
    'scheme' => 'tcp',
    'host'   => REDIS_HOST,
    'port'   => REDIS_PORT,
));

We’ll assume in all of our examples that we’ve done the following above & connected to Redis.

Now, to manage our queues we’ll use the Redis Datatype LIST. Whats awesome about lists is that regardless of size, adding or removing at the start or end of a list is extremely fast. So if your queue has 10 items, or 10,000,000 items, Redis wil be able to push and pop entries quickly.

We’ll have three queues, one for each priority: high, normal, and low. For the Redis key names, we’ll use queue.priority.high, queue.priority.normal, etc. When interacting with lists, you work with the ends, one called right, the other called left. So we’ll add items on the right with the RPUSH (Right Push) command, and we’ll pull items off the left with the BLPOP (Blocking Left Pop) command. We won’t worry about the pulling items just yet.

You store strings as the values for the list. My personal preference is to store JSON objects so you can easily pass variables needed to perform the job.

/*
 * Adding items to the queue
 */

$job = new stdClass();
$job->id = 1;
$job->report = 'general';
$job->email = 'test@example.com';

// Add the job to the high priority queue
$predis->rpush('queue.priority.high', json_encode($job));

// Or, you could add it to the normal or low priority queue.
$predis->rpush('queue.priority.normal', json_encode($job));
$predis->rpush('queue.priority.low', json_encode($job));

Simple enough! Having different queue priorities is very beneficial in managing which jobs should get done first. For example, you might have an Executive’s request go into the high priority queue so they get the report quickly. You might also have a weekly cron that queues up reports to be sent automatically, so those can go in the low priority as to not disrupt people trying to get a manual report.

Now, on to the worker’s code.

Processing Jobs

For now, lets say we have a script running in the PHP CLI (Command Line Interface) that you started by running this command on the server:

First thing is we want this worker to work continuously, so we can do a while loop:

/*
 * Simple Continuous While Loop
 */

// Always True
while(1)
{
	/* ... perform tasks here ...  */
}

We’ll worry about making them more intelligent later. Now, let’s have our worker check the queue. You can do so with the BLPOP command:

/*
 * Checking the Queue
 */
$job = $predis->blpop('queue.priority.high'
						, 'queue.priority.normal'
						, 'queue.priority.low'
						, 10);

What we’re telling PHP to do is to check each queue in order of priority: high, normal, and then low. If it finds an item, it will immediately return an array with the name of the queue it came from, and the string of data that was pulled.

The B in BLPOP is “blocking.” What that means is that Redis will wait until either an item enters one of the queues, or the timeout is reached. In this case, the timeout is 10 seconds. So instead of polling (checking every few seconds in a loop), we check and wait, and after 10 seconds it will return null and we can check again.

What this gives us is near instantaneous queues. As soon as something is available, it is passed to the workers that are listening. You can also have multiple workers, and it will pass jobs to the first listening worker, and the next job to the next worker, so you don’t have to worry about multiple workers getting the same queued item.

After $predis->blpop() returns, if it has an array, it returned an item. If not, the timeout had been reached. We can check to see if a Job was returned, and if so to process the job:

/*
 * Checking to see if a Job was returned
 */

if($job)
{
	// Index 0 of the array holds which queue was returned
	$queue_name = $job[0];
	// Index 1 of the array holds the string value of the job.
	// Since we are passing it JSON, we'll decode it:
	$details = json_decode($job[1]);

	/* ... do job work ... */
}

Now we can have multiple workers listening to the same queues and scale our workload. Redis is very fast & efficient, and you could have hundreds or even thousands of workers listening to a single redis server.

Continuously Running Workers

There are a lot of options when it comes to deploying these workers. You can use a framework like Gearman, but for simple things, I like very simple solutions. I came across a blog post by Joseph Scott about a little 10 line perl script called solo. What it does is it will run a command, and to ensure that no one else is running that same exact command, it will lock a configurable port. This is awesome because the you don’t have to work about lock files or filesystem tricks, the kernel handles it all.

So what you can do is create a cronjob using solo to execute your script. First copy solo somewhere, I put it in my /usr/local/bin on my linux server. Then add this to your cron job using the command “crontab -e -u (which user to use)”:

What this will do is try to run this command every minute. Solo will check to see if the port is already in use, and if it is, it will exit. Otherwise, it will lock the port and then execute the command. The port will stay locked as long as the command is executing. Once the command terminates, the port will unlock.

Now, PHP is a great language, but it has been known to have some memory leaks while running a long time in a single instance. So we can have our scripts exit periodically to be restarted by our cron job. So lets make our “while(1)” statement a little smarter:

/*
 * A Smarter While Statement
 */

// Set the time limit for php to 0 seconds
set_time_limit(0);

/*
 * We'll set our base time, which is one hour (in seconds).
 * Once we have our base time, we'll add anywhere between 0
 * to 10 minutes randomly, so all workers won't quick at the
 * same time.
 */
$time_limit = 60 * 60 * 1; // Minimum of 1 hour
$time_limit += rand(0, 60 * 10); // Adding additional time

// Set the start time
$start_time = time();

// Continue looping as long as we don't go past the time limit
while(time() < $start_time + $time_limit)
{
	/* ... perorm BLPOP command ... */
	/* ... process jobs when received ... */
}

/* ... will quit once the time limit has been reached ... */

One key thing to note is randomly shifting the time limit for the script. I like to do this because you don’t want your workers all stopping and starting at the same time. So if I have 8 workers, one might, but the 7 will continue until the 8th starts back up again via the cron job.

Bells & Whistles

After using workers for awhile, here are a couple of ideas to enhance your workers & system managing them. First off, you can add some monitoring for your queues. Using Redis a HASH, you can use them to store the state of your workers.

/*
 * Assigning Worker IDs & Monitoring
 *
 * Usage: php worker.php 1
 */

// Gets the worker ID from the command line argument
$worker_id = $argv[1];

// Setting the Worker's Status
$predis->hset('worker.status', $worker_id, 'Started');

// Set the last time this worker checked in, use this to
// help determine when scripts die
$predis->hset('worker.status.last_time', $worker_id, time());

Another problem with workers that run for a long time (several hours) is when you make a change to their code, they won’t reload that change until they exit. What I’ve found to successfully restart them is having a “version” number set in Redis that is checked at the end of every loop:

/*
 * Using Versions to Check for Reloads
 */

$version = $predis->get('worker.version'); // i.e. number: 6

while(time() < $start_time + $time_limit)
{
	/* ... check for jobs and process them ... */

	/* ... then, at the very end of the while ... */
	if($predis->get('worker.version') != $version)
	{
		echo "New Version Detected... \n";
		echo "Reloading... \n";
		exit();
	}
}

You would simply INCR (increment) worker.version and after finishing their last job, the worker would exit, and solo would start it up again.

You can also kill specific threads by having them check for their value in a hash:

/*
 * Using Kill Switches to Check for Reloads
 */

while(time() < $start_time + $time_limit)
{
	/* ... check for jobs and process them ... */

	/* ... then, at the very end of the while ... */
	// Check to see if a kill has been set.
	if($predis->hget('worker.kill', $worker_id))
	{
		// Make sure to unset the kill request before exiting, or
		// your worker will just keep restarting.
		$predis->hdel('worker.kill', $worker_id);

		echo "Kill Request Detected... \n";
		echo "Reloading... \n";
		exit();
	}
}

Tweak to Solo & Logging

I made one small tweak in my version of solo, and that was to help it enable logging. Lets say I had three workers in my crontab:

The “>> /tmp/worker.log.1″ tells solo I want to log it’s output to a tmp file that I can tail and monitor their progress. This is great for debugging problems. However, when I did this, solo would write to the tmp file, and not the output from my script. To overcome this I changed the last line of solo:

# old
exec @ARGV;
# new
exec "@ARGV";

This would ensure my script wrote out to the tmp file, and not just solo.

Examples

I’ve created an example on GitHub that you can clone on your own machine. All you will need is PHP 5.3 and Redis installed.

To install redis, simple run these commands on your unix based system:

It will copy the redis binaries to /usr/local/bin.

To get a copy of the code, you can download them here. HOWEVER, it doesn’t include predis! You’ll have to download and copy predis inside there via this link. It is much easier to clone it as so:

Then, using different terminal windows (or using screen), you can run different worker.php instances, use creator.php to insert jobs, and monitor.php to watch the progress. This is all done from the command line.

If you’re using windows, I suggest installed a VM of Ubuntu and using that. If you really want to use Redis on windows, there are some Windows Binaries you can google and download. Good luck!

Here is a video where I demo the example:

(sorry for the poor mic quality)

Final Thoughts

I’ll post here shortly about how to run Redis in production with the init.d scripts and configuration files. One caveat to using solo is if your server has an application that randomly selects ports to use (i.e. VoIP, FTP), it might select one of your worker’s ports. But on a production server, you should have a good feel for which ports are available for locking.

If you want to learn more about Redis, check out their website.

Hopefully this will be helpful for anyone looking to use PHP Workers in an easy, simple way.

Related posts:

1. MySQL, Redis, and a Billion Rows – A Love Story
2. Creating Chatroom / Walls with Redis & PHP
3. Debuging with PHP, Stack Traces, and Redis

Today, after seeing a tweet by Jeffery Zeldman that was retweeted by Andy Budd, I came across a new favorite website: Dark Patterns.

These patterns are not to be confused with Anti-Patterns. Anti-Patterns are commonly used techniques that are in-effective or counter-productive. They are consideres mistakes with unintentional consequences or pitfalls. Dark Patterns, however, are things web designers & developers do on purpose. From the home page of Dark Patterns website:

Normally when you think of “bad design”, you think of laziness or mistakes. These are known as design anti-patterns. Dark Patterns are different – they are not mistakes, they are carefully crafted with a solid understanding of human psychology, and they do not have the user’s interests in mind.

The website then names several of these “Dark Patterns”: Friend Spam, Privacy Zuckering, Misdirection, Bait and Switch, and many others.

I recommend checking out their website, and then self evaluating your own projects asking these questions:

• Do I use these Dark Patterns?
• Do any projects I work on use these Dark Patterns?
• If so, how can I fix them?

The Dark Patterns website has a 20 minute video which is a recorded presentation an event. What I best like about it is it shows a few examples of Dark Patterns, and then walks through the logic of avoiding them. I recommend giving it a view:

Related posts:

1. Five common PHP design patterns that aren’t used enough
2. Utah PHP Users Group
3. DRM vs. Users – The Good and The Bad

What is Nginx?

Nginx (pronounced engine-x) is a fast, powerful, lightweight web server. I won’t go into the theory under-the-hood, but it’s focus is high concurrency with low memory usage. So while Apache is more robust in supporting many different features, nginx focuses on handling the important features very quickly. I still use Apache internally for our SVN & Trac web server. Heck, even at this time I’m using Apache to host this blog. However, Dating DNA, Clipish, CEVO, Alienware Arena, and some other high traffic sites/apis use nginx.

Ngnix, unlike Apache, doesn’t actually load PHP. Instead, it hands it off as a proxy to a “php handler” which acts like an Application Server. So nginx by itself won’t serve PHP files, but just static files.

What is PHP-FPM?

In the past, when working with something like Nginx or lighttpd, you would use spawn-fcgi to host your PHP application. However, spawn-fcgi had some major drawbacks and problems. So a guy named Andrei Nigmatulin created PHP-FPM, which stands for “PHP FastCGI Process Manager.” Since then, several others have contributed and ultimately it was include into the PHP core in version 5.3.3.

So from a high level look, on every PHP request Apache will load the entire installed PHP environment each time. This is for every request, and while it has been optimized as much as it can, that is a lot of overhead! With PHP-FPM, it will spin up a configurable amount of children. Each load the PHP environment and then will serve as many requests as it can without having to reload the environment. This saves on a lot of overhead!

Why use Nginx & PHP-FPM?

I should note, it is possible to configure/compile Apache in such a way that it can have similar performance capabilities. However, it takes a ton of work. Meanwhile, Nginx & PHP-FPM are very fast from the start, so I prefer just using them. You do lose some features, like .htaccess files won’t work so you’ll have to do that configuration in your virtual hosts.

How to Setup Nginx & PHP-FPM

Nginx

First off, lets setup Nginx.

Thats it! If you go to your server’s IP Address or Domain Name you should see a “Welcome to Nginx!”

PHP-FPM

Because PHP-FPM is only included by default in PHP 5.3.3 and later, and Ubuntu 10.04 LTS only has PHP 5.2.3, we have two options. Either we can install by source, or we can add another repository to install PHP-FPM. The latter is much, much easier, and there is a good PHP-FPM Repo for Ubuntu 10.04. To add it, you just run the following commands:

Now that we have the new repository, we can install PHP5:

Excellent! Now, if you need to change some of PHP-FPM’s configurations, they are found in /etc/php5/fpm/. The file php5-fpm.conf configures how FPM will opporate, and the php.ini is the settings file that PHP will use while running in FPM.

A few settings I like to change in /etc/php5/fpm/php5-fpm.conf:

The php5-fpm.conf that comes is pretty well documented on the different settings. Once you make a change, make sure to restart php5-fpm.conf: /etc/init.d/php5-fpm restart

Configuring Nginx

Now, we have a few settings for Nginx. The configuration files are found in /etc/nginx/. First we’ll edit nginx.conf. Here are a few settings we’ll want to change:

Now, we need to add a VirtualHost! Nginx uses the same layout in Ubuntu as Apache, so we’ll add configurations for each site we want under /etc/nginx/sites-available/. So using vi, nano, or whichever editor you prefer, create a /etc/nginx/sites-available/www.example.com file:

Now, we need to create the symlink from sites-enabled to sites-available:

Restart nginx with “/etc/init.d/nginx restart”. Go ahead and put a test.php file in your directory with a Hello World example, and see if it works. It should work, and you should be good to go.

Related posts:

1. Setting Up Percona Server 5.5 on Ubuntu 10.04
2. Preparing a VMWare Ubuntu Guest OS
3. PHP, Nginx, and Output Flushing

What is Percona Server?

Percona Server is a fork of MySQL Server. So what does that mean? For the past few years, MySQL has had some “interesting” developments from the business stand point. Being bought by Sun, which was then sold to Oracle, and a lot of “drama” surrounding all of that. In turn, what seems to have happened is MySQL’s development stalled or slowed down as it took a back seat to the legal sides of things.

Percona took the GPL version of MySQL and created their own distribution of it called “Percona Server.” Their goal is for it to be “an enhanced drop-in replacement for MySQL.” So they switch out InnoDB for XtraDB under the hood (though it is still called InnoDB internally so there are not compatibility issues), fixed a bunch of bugs, added a lot more useful diagnostic and reporting data, and improved it’s performance vastly. Unlike some of the other MySQL Forks, I haven’t found a single compatibility issue between it and “stock” MySQL.

Why use Percona Server?

It is just a better version of MySQL that is completely compatible with a normal version of MySQL. The way I view Percona Server is it is what Sun & Oracle should have done with MySQL. I also think if you’re running MySQL with any real load at all, you should use Percona Server 5.5. I can’t find a single reason not to. It just performs so well.

It also comes with Yum & Apt repositories so you don’t have to worry if your Operating System is only running 5.1, you can run 5.5 easily.

Installing Percona Server 5.5 on Ubuntu

Following the instructions from their Apt & Yum installation directions:

Add the Percona Apt Signed Key:

Then, add the repositories too your /etc/apt/sources.list

Update your local repository cache:

Install Percona Server 5.5:

It’ll prompt for a root password, and you’ll be set! The great thing, is the binaries are all named the same. So you want to log in as root? You type “mysql -u root -p”. The files are all in the same place. The commands are all the same. Everything just works, only with better performance and additional tools. I haven’t had a single problem running it in production.

Related posts:

1. Sending Email from non-email Ubuntu Server
2. Setting up Nagios for Servers
3. Preparing a VMWare Ubuntu Guest OS

A lot of times VMWare will auto-install the VMWare tools for you when you first setup your VM. However, many times after setting it up I’ll do updates, and updates to the kernel will be applied, knocking out the VMWare Tools changes. These tools are used for things like file sharing between the guest and host.

So here are the steps to take to fully update and re-install the VMWare Tools. I got most of these from an article on Ubuntu’s website.

First, sudo aptitude update to make sure all my repository information is up-to-date.

Second, sudo aptitude safe-upgrade to get any and all updates for my newly installed OS.

Third, sudo apt-get install build-essential linux-headers-`uname -r` psmisc to install the linux headers for my kernel.

Fourth, copy and install the VMWare Tools:

# make a mount point if needed :
sudo mkdir /media/cdrom

# Install the tools
cd vmware-tools-distrib
sudo ./vmware-install.pl

Just follow the prompts and hit for all the default values.

Related posts:

1. Ubuntu Desktop Terminal – Su
2. Sending Email from non-email Ubuntu Server
3. My Honest Attempt With Linux Desktop

Hopefully at some point, your website gets a lot of traffic. Yay, you’ve reached your goal of getting good traffic, but it is soon followed by issues with performance and load. I like to call these the growing pains of a website. So as a web developer, I suddenly have the epiphany of “Hey, I need to scale my website!” What follows next is the biggest mistake a web developer can make:

They start looking at articles on how Google scales, or maybe how Facebook manages all of their traffic.

This is a mistake! To be brutally honest, you are not Google. You are not Facebook. You are not Twitter. You are a website that receives less than 0.000001% of the traffic that some major websites receive.

Why is this dangerous for web developers to do? Google, Twitter, Facebook, and others like them are solving complicated at a very large scale. I remember a presentation by a Twitter engineer who developed a program for a unique ID generator that can generate millions of IDs per second. The probability of you needing this type of solution is about the same as being struck by lightening. Applying these same practices at a much smaller scale are not realistic. If a locally owned grocery store wanted to open a second store, they would not adopt the same practices that Wal-mart use to manage their 8970 stores.

A Little Reality Check

I’m sure that most of my readers know of StackExchange.com. They power the popular website StackOverflow and several others. They have about two million visitors per day. That is a lot of traffic. StackOverflow is ranked #123 on Alexa. So you would imagine that they have a very large infrastructure serving all of this traffic?

Earlier this year, Stack Exchange wrote an article about their production environment. I was surprised on what exactly they were using. In paticular, the number of Production Servers*:

• 12 Web Servers (Windows Server 2008 R2)
• 2 Database Servers (Windows Server 2008 R2 and SQL Server 2008 R2)
• 2 Load Balancers (Ubuntu Server and HAProxy)
• 2 Caching Servers (Redis on CentOS)
• 1 Router / Firewall (Ubuntu Server)
• 3 DNS Servers (Bind on CentOS)

That is 22 servers for 2 Million Visits per day, serving 800 HTTP requests per second. Now, StackExchange did clarify that they did have other servers for management and fail over, but 22 servers handle their production load. This is a website that is ranked the 123rd most visited website in the world.

Honestly, most websites could be run on half a dozen servers if designed and configured correctly, including redundancy. Some really busy websites could run off a dozen servers. Unless you’re in the top 5,000 websites on the web, you really shouldn’t be worried about large-scale techniques.

So when you’re website is starting to grow, and you leave small scale, you’ll enter the phase of “Middle-Scale.”

What is Middle-Scale?

Middle-Scale is like being an awkward teenager:

You know that you can’t be the only one suffering through this, but you’re unsure how to proceed. It feels like you’re missing missing out on things everyone else must already know, but aren’t talking about. Like everyone else are awesome vampires or something:

But the reality is this: they don’t have some awesome secret! They are just normal teenagers.

This same idea applies to Middle-Scale websites.

Middle-Scale is when the most important things are still the best practices. Only now when you deviate from them you can feel those consequences. When you only had 100 users, a couple of nested queries and missing indexes didn’t cause that much of a problem. Your database is powerful enough to hide the inefficiencies. However, when you get to 10,000 users, your database can no longer hide the inefficiencies.

Middle-Scale is when simply separating your web server and database server isn’t enough. You’ll probably need to add some sort of cache like memcached. You’ll need to start tweaking your MySQL, Apache, and PHP configurations.

Then, after you’ve ironed out your inefficiencies, you’ll start to use multiple servers. You’ll probably add a Load Balancer with multiple web servers. After that, you’ll probably have some sort of Master-Slave replication for your Database for backups and fail-overs.

You start to leave this “Middle-Scale” classification when you move to multiple data centers, and start to do some load balancing at the the DNS layer. This is when you’ll start to have a dedicated sys-admin team.

Okay, I’m Middle-Scale! So what should I do? Where do I look?

First off, you must adhere to best practices. If you are working with PHP, research PHP performance and best practices. Do they same for each of your technologies, like Apache and MySQL. You will need to stop treating your application as one big app, and start to understand all of it’s moving parts.

Second, you must understand your specific problems. Scaling ins’t a problem, nor is it a solution. It is a generic term for many different types of solutions. Without understanding why your website is running slow, or why it cannot handle the load, you will not be able to create an effective solution.

So you don’t have a scaling problem. You have a MySQL performance issue, or a Apache problem, or a PHP problem. Most likely, it is something extremely specific. You have a high volume of MySQL write operations (i.e. UPDATE, INSERT, DELETE, REPLACE), or perhaps you are missing some indexes and have too many full table scans.

Third, Googling for help will only get you so far. You are starting to enter a phase when it is harder and harder to find answers to your broad issues. Talking with other experienced people who have gone through the Middle-Scale pains before will help immensely. I cannot recommend highly enough going to User groups. Being able to communicate with someone, either face to face, on the Phone, over IRC, etc. is invaluable. While I’ve learned a lot at conference and usergroup presentations, I’ve learned even more by just talking with the people attending and at the social gatherings.

Profile & Performance will Naturally Lead to Scaling

When you want to scale, it can feel like a very daunting task. It seems like this big unknown complicated solution. What in the world am I going to do? I remember feeling these worries when I first started to investigate load balancing and sharding for some websites I was working on.

The thing is, if you start to profile your application, you will discover it’s inefficiencies. I remember when I spent a sold week, working 12-16 hours a day profiling and optimizing Dating DNA’s database. I found a lot of bad queries, and I was able to cut our load times from 2-5 seconds to under 0.1 seconds. The CPU on the database server went from 80-90% CPU utilization to under 10%. It was incredible, and then I promptly took the entire next week off. When we migrated to new servers, I was able to move to less powerful database server and still have the same great performance. So by profiling and optimizing our database, I didn’t need to worry about spinning up multiple master databases and sharding our data.

With Clipish, we faced almost opposite scaling problems. The database was rarely an issue, but our web server CPU’s were. We do a lot of ImageMagick manipulations of images, and at high volumes on virtual servers this can be a big issue. So over the last year we’ve introduced some load balancing and CDN tools to help serve all 10 TB of bandwidth for Clipish.

The thing is, when you start to profile your application, you start to understand it’s low areas better, so you have a much better idea on what do to. Even if you don’t know your solution, it is much easier to find a solution with a sound understanding of the problem. For example “scaling mysql” yields much less helpful results than “mysql full table scans” in Google.

So should I ignore what Facebook and Google do for scaling?

Of course not! First off, they do cool stuff. Just because I’ll watch NASA launch a space shuttle doesn’t mean I’ll try to make a rocket system for my broken lawn mower. But you have to put what they are doing into context. People from large websites have published several good “best practices” articles on techniques that help any website. Especially things on the client/browser side of things. Just use caution. I cringe when I hear someone say “we’re trying to use Cassandra to solve XYZ problem at work” when it is severe overkill.

Final Thoughts

Most of the time when I talk about performance and scaling with other people, it is when they are in “critical mode.” Their website is down, slow, unusable, etc, and they are looking to fix the problem. I will say, it is much more difficult to profile in “critical mode” than profiling before hand. The reason is you are much more desperately focused on getting it working again instead of understanding the problem.

I’ll be giving a presentation this Thursday at UPHPU on Profiling PHP Applications. I’ll post the slides, and most likely write some articles on the subject afterwards. As always, feel free to email me or leave a comment.

Related posts:

1. Data Backups – There Are No Excuses
2. Speaking: Utah Open Source Conference 2009
3. Presentation: Real Life Scaling

The Problem

With Dating DNA, our goal was to display a compatibility score with every other user. This score is generated by taking two sets of answers to our 20 page survey, and running it through our algorithm. While it is super convenient for our users, this poses a problem. We wanted not only for people to be able to visit a profile and see a score, which is easy to generate a score on demand. We wanted our users to be able to browse other profiles sorted by their score with them. This requires us to pre-generate and store these scores, and then later query them.

So Ultimately, in theory, our “scores” data scaled at the following rate, with X as the number of registered users:

X^2 – X

That is at an exponential rate, which is practically impossible to scale at. The very first version of Dating DNA (before I took over the project) had about 1,500 users. The scores were stored in a single table. Every night, a “cron job” would run and get a list of every user, and loop through every possible iteration and re-generate each score. At 1,500 users that was 2,248,500 records. That is a lot for just 1,500 users. With our current user count, we would roughly have 359,999,400,000 score records. Thats 359 Billion records if you don’t want to count the commas.

This old system of daily cron jobs broke at about 2,000 users. We would have problems with the cron job taking over 48 hours to complete, and would end up with 3 scripts running at the say time. One for today, one for yesterday, and one for the day before that.

Smart Logic & Threading

We solved our first problem by using some common sense and smart logic. I won’t detail the lengthy measures we go through, but we can basically boil down our entire user base to an estimated top 5000 matches for any given user. If we have a heterosexual man named Joe, he doesn’t care about the hundreds of thousands of other heterosexual men who he scores a 2 or 3 with, but the heterosexual women he scores above a 6 with. So we don’t store the score for Frank, Jimmy, and Alan with Joe, but Sally, Rachael, and Tiffany.

The second part we solved was pre-generating scores for a user. After a User has reached a point in the survey where we have all the information we need to generate scores, and they are just filling in some miscellaneous, we put them in a queue. We then have a server process than is continuously running checking this queue, and spinning off multiple generation “threads” that crunch the data and store the score. We’ve spent a lot of time perfecting this system. Currently we typically can generate any given user’s matches in roughly 5 to 20 seconds, depending on how busy our website is.

Storing The Score in MySQL

The problem we now faced was the write through put of MySQL. Even through sharding and partitioning, we wanted to have a goal of sustaining 1,000 registrations per minute in a scalable and high performance manner. Which comes down to about 83,000 records per second that are either being inserted or updated. We then needed to be able to retrieve large volumes of scores just as fast.

I believe we could have bent MySQL to our will and got it to work, but it would be at a high cost of server power, and that cost wouldn’t scale well with our revenue stream. After we moved from the MySQL storage of the scores, I ran a query to see how many scores we were indeed storing. The final total was 950,363,992. Just 50 Million shy of one billion. It took 1 hour 49 min 38.27 sec to calculate that count. It is evidence that even though MySQL wasn’t the best choice for storing this data, it did it pretty well considering this single table was holding 90 times more data than any other table.

Picking Another Solution

In 2009 we started to throw around ideas for a new scoring system. I cannot stress enough when talking to others about “NoSQL” solutions the best solution for any given job is based on your data’s characteristics. User registration data needs to be treated differently than activity logging and basic stats. It might be okay to lose a few minutes of activity logging (depending on the app), but you definitely don’t want to be losing user accounts.

With Dating DNA’s scores, we had one great advantage. The data could be somewhat volatile, because we can always re-generate a set of matches for any given users. Of course, we didn’t want to lose all of it, because having to regenerate everyone’s scores is a major pain and extremely resource intensive. But if we lost a few minutes, anything lost could easily be regenerated. So when we started research for a solution, we were willing to sacrifice some persistance for performance. We wouldn’t be doing the same for our user registration data.

At first, I was contemplating building a completely in-house project to handle the data storage and retrieval of the scores. It would be a lot of work, and decided against. So I then thought about hacking together a custom solution with memcached. The idea would be a user’s set of matches would be stores in a variable in memcached. So the website and generation scripts would interface with memcached, and a server process would write inactive sets of scores (people who weren’t logged in) to a file on the disk. When they logged it and scores were being pulled and stored for that user, it would load the data from disk into memcached.

While the general concept was sound, the actual execution would be difficult. Memcached only supported strings for values, and we would still need some sort of database to manage which users had the data in memory vs disk, and the server process (probably just a php, python, or node.js script running continuously) would have to be running constantly, and if that broke things could get messy.

It boiled down too many points of failure and complexity. But it was a step in the right direction, so we kept looking for a better solution.

Redis, the Advanced Key-Store

I was talking with Joseph Scott, an employee of Automattic as a Bug Exorcist (not joking, his real title), and he mentioned I should look into Redis. He gave me a brief overview of what it was, and I shuffled that info back in my brain. I can’t remember how much longer it was before I checked out and compiled a copy of Redis, but I quickly discovered it could be a viable storage system for our scores.

So I spun up a virtual machine, installed Redis, and started to pound away at it. One of the things I wanted to test was the new feature (at the time) of Virtual Memory for redis. What this allowed was for Redis to make it’s own Virtual Memory on the server and store the lest recently used data to disk. When a Redis object was retrieved and it was in the VM, it would swap it back into memory, and swap older data to disk. This was just like the idea I had before with using an archaic system with memcached, but much more elegant.

The second thing was Redis’s support for multiple data types. So instead of having a json encoded string that held the scores and user ids for another user, we could have a hashtable or even a sorted set. It was a much more elegant solution than what we were thinking of before.

Some Limitations to Redis

However, there were a few limitations that we faced when implementing Redis. Redis works flawlessly with smaller sets of data. But the larger your data set, the more careful and aware you need to be about a few things that will kill your redis instance.

First off, with memcached, if you set a memory limit, it is a hard limit. I’ve never seen memcached use more memory than what you allow it to use. Redis, on the other hand, has soft memory limits. This is because of the way the Background Saves work so you can have persistant data. When a Background Save is issued, Redis will fork itself, and have one thread save a snapshot to the disk, and the other thread will continue to operate. In order to do this, Redis will exceed the standard memory limits, and your memory usage will go up much quicker. Once the background save is complete, it will close the forked backup process and sync the memory back to one data set. (I’m not a computer science guy, nor do I know a lot of lower level programming, so I might be describing this not 100% accurate, but this is how I envision it in my head).

Now, if you are not using the Virtual Memory, this isn’t that bad. However, when using Virtual Memory, the Background Saves take a great deal longer (from seconds to almost a minute or so), which isn’t too bad, but there is a catch. You will not be able to swap to the VM Disk until after the BG Save. This means that all Redis objects swaped to memory will stay in memory until after the BG Save is complete. This is because, like the memory from the fork, the Virtual Memory file is being used for the BG Save instead of the process handling requests.

So the one limitation we’ve encountered is we cannot run scripts that “query” large amounts of data from Redis. For example, it would be very simple to get a listing of users using the KEYS command, and then loop through the values using a HLEN to read the length. This will cause you to swap from and to the virtual memory a great deal. If a Background Save is occuring, Redis will not swap to disk until the BG Save is complete. This means if you have 10 GB of data in Virtual Memory, and you have a 1 GB instance of Redis, you will suddenly be reading gigs of data into Redis’s memory. If you are on a 2GB machine, you can easily use up all the memory on the server and then start using the System’s Swap.

Once you start using the Operating System’s Virtual Memory, it is game over. Your Redis instance’s performance will tank, and your Background Save might not finish, and you will need to restart redis.

There are some ways to give Redis a “Hard” limit on memory, but we opted to configure our servers in a way that doesn’t require this. If Redis hits the memory limit, it can start throwing write operation errors, which we didn’t want.

How We Configure Redis

After much trial and error and testing internally, we believe we found a sweet spot. We deploy a redis server, and spin up three redis instances on a different port each. Each is configured with 4 GB of Virtual Memory using 4096 size pages, and only 256 MB of “memory” using the vm-max-memory setting. While you would think this would mean a hard limit, it is a soft limit, and more of a goal “we’ll try to only use 256 MB of memory to store the data, if we’ll exceed it if needed.” Given our patterns of usage, Redis’s actually usage fluctuates (based on ps aux’s reporting) between 640 MB to 1100 MB of RAM, depending on if a background save is being executed or not. Redis is configured to perform background saves every 10 minutes, which take about a minute to perform.

So between the other admin services running on the box, and the three redis instances, we use just over 3GB RAM:

The amount of CPU required is extremely low, and almost 100% from writing the background saves to the disk:

So what do we get in return? We estimate each instance with 256 MB of data can hold roughly 2,000 active users. So with a single server we can support 6,000 users online at any given moment. We can store the scores for roughly 360,000 users on a single 4 GB box, which is about 1.8 Billion scores. Then, if we need more, we just provision another box, and our system will start assigning users to the instances on that machine.

Because of our ability to re-generate the scores, we decided to only convert the users who had logged in the past three months to the new system. If a user who hadn’t logged in since then logged in again, it would in the background assign them to a redis instance and rebuild their matches for them.

Using Redis with PHP

I recommend currently to use the PHP Redis client predis. I’ve used others like Rediska, but I prefer the straight forward approach of predis.

A high level view of how we use Redis with our PHP based website is we have a class called RedisManager than manages pretty much all the connections to Redis. It supports lazy connections (which is important to us, since we don’t want to have to connect to every instance of Redis we have), and I hope to open source it some day soon.

One key performance trick we’ve noticed is to use Pipelining to the redis instance. We don’t use this so much on the website, but our score generation “threads.” Writing thousands of scores one by one eats up a lot of network overhead versus sending them in batches (we send in batched of 500 or 1000, depending on the situation). Using pipelining is extremely fast for us, and I highly recommend it for any large batch of commands.

Using Redis Elsewhere in Dating DNA

Now, it might seem that we’ve put a lot of thought and effort into using Redis, and I want to make sure it was understood that Redis itself wasn’t difficult to use, but the volume of data were were dealing with. On Dating DNA, we also use Redis to power out in-app chat system (which I’ve written about previously), and it works great and is currently only using 146.70 MB of RAM, and serves thousands of requests per second.

The Future

I still have a lot of great ideas for Redis and Dating DNA, both with the score system, and outside of it. I plan on writing several reporting tools for Redis and hope to share them on github. I am currently working on the code and scripts for automatic deployment for Redis servers for Dating DNA, so we can scale easily with the push of a button. I’m excited for the work that is being done on Redis, and highly recommend it to anyone.

If there are details you would like to know more about, leave a comment and I’ll try to answer them. If you see me at tek11, feel free to ask me about this, and I can show you in detail how it works (internet permitting).

Related posts:

1. MySQL & PHP – SQL_CALC_FOUND_ROWS – An easy way to get the total number of rows regardless of LIMIT
2. MySQL, 40 Million Rows, MyISAM to InnoDB, 45 Minutes
3. Restoring Large MySQL Dump – 900 Million Rows

Problem

Our current backup system uses mysqldump. It dumps a 25 GB sql dump file, which compresses to about 2.5GB using gzip. The last time we needed to restore a backup it was only about 9GB, and it took several hours.

This time, I created the database, and from the mysql prompt I issued the following command:

It would run great until it got to about 10% of the way through the scores table. However, it would start to slow down. Because the rows were so small in our scores table, each INSERT statement had about 45,000-50,000 records. So each line had roughly 1MB of data.

At first it would insert a set of 50,000 in half a second or so. However, after a few million records, it would slow down to three second, and got to about 10 seconds per INSERT statement. This was a huge problem, given that I had roughly 18,000 INSERT statements, and at 10 seconds per INSERT, it would take 50 hours to restore. Our website was down during this restore, since it was our primary database. So being down for over two days was not an option.

While trying to diagnose the problem I noticed something. While using the MySQL command “show processlist” the thread for the Database Restore would be in the sleep state for 9-10 seconds, and then the query would execute in under 0.2 seconds. So it wasn’t a problem with MySQL storing the data, but a problem with reading the data from such a large database dump file.

So I tried from the server’s command line “mysql -u user_name -p database_name < /path/to/backup/database_dump.sql" with the same result. The longer into the file I got, the longer it was taking for MySQL to read the query.

Solution

So, after some thinking late at night at 3 AM, I came up with an idea. Why not split up the database sql dump into multiple files. So I used the linux “split” command like this:

This produced several dozen files in order, and it took about 10 minutes. The -n option told split to split each file up into 200 lines each. So the files were then named sql_aa, sql_ab, sql_ac all the way to sql_fg. Then, I did the following command using cat to pipe the files to mysql:

The only problem with this method is you don’t see a status report for each query executed, it just runs until you hit an error, displaying the error. If no errors occur, it will just return you to the prompt. So to monitor the progress I would execute a “show processlist;” command on mysql to see how far we were.

4 1/2 hours later, the entire database was restored. A few things to note, I didn’t try just using cat on the original file to see if it would read the file differently than the was mysql was trying. But the important thing is I got the database restored in a relatively timely manner.

Hopefully, in the very near future, we will have moved to a new score system that doesn’t have almost a billion rows in it.

Related posts:

1. MySQL, 40 Million Rows, MyISAM to InnoDB, 45 Minutes
2. Large MySQL Conversion & InnoDB
3. MySQL & PHP – SQL_CALC_FOUND_ROWS – An easy way to get the total number of rows regardless of LIMIT


aptitude install postfix #when prompted, select Internet Site
aptitude install mailutils


Then bingo, the mail command should work. You can test it by doing:


mail justin@example.com


It will ask for any Cc addresses, and a subject. Then, type your message, and when you are done hit “Ctrl-D” for done.

Related posts:

1. SwiftMailer – Sending Mail in PHP With Ease
2. Thunderbird Email Client: A Guide for Outlook Lovers (Part 1)
3. My iPhone Has Hindered My Email