In the past I would just use my iPhone’s email-to-txt email address. However, when I received the txt message, it wasn’t formated very pretty, and it would have a different “From Number.” So if we had a crazy day, I would have 20-30 message threads in my iPhone all about Nagios. I’ve been lazy in the past, and didn’t clear them out until I had about 200 of them, and it was a lot of “swipe, delete, swipe, delete, swipe, delete” to get rid of them.

What I like out this setup is with Twilio, I can buy a phone number for $1 a month. So all my notifications come through the same number. I’m also planning a way to call/text the number and it will read/send the current status and list the hosts that are down.

First thing is to get your Twilio account setup. Its pretty easy, and when I signed up (a long time ago) I got $30 in credit, not sure if they are doing that still for new customers. Either way you can add a credit card and say how much credit you want to add when your account dips below a certain amount.

Now, you can buy a number and set it up. Even if you’re only sending text messages, Twilio wants you to have a “SMS Request URL” before you can send any text messages. So I just put in a URL to my site that does nothing (yet). You’ll need to grab your Account SID and Token.

How it Works

I threw the code up on GitHub. There are a few things you’ll need to do:

First in sendTextMsg.php change your configurations to match your phone number, account SID and token. Here is the entire sendTextMsg.php file:

<?php

require('twilio-php/Services/Twilio.php');

/* Start Configs */

$sid = "A123.....";
$token = "29d6b9f.......";
$twilio_number = '4045550101';

/* End Configs */

# Get the Argvs
$phone  = $argv[1];
$msg    = $argv[2];
$msg    = str_replace('\n', "\n", $msg);

$client = new Services_Twilio($sid, $token);
try
{
    $message = $client->account->sms_messages->create(
        $twilio_number, // From a valid Twilio number
        $phone, // Text this number
        $msg
    );
} catch (Exception $ex)
{
    var_dump($ex);
}

As you can see, it is super simple. All you need to do is put this script somewhere with the Twilio PHP libraries (or just use my github code with all of it in there). Then, in your Nagios configurations add the commands:

Then you need to add it to your contacts:

Simple, easy, and works very well. You can test the script by simply executing the command:

Enjoy!

Related posts:

1. Setting up Nagios for Servers
2. Presentation: Real Life Scaling
3. BestBuy Gives Me $8,770 — Wait, Just $87.00

Why Workers?

Sometimes there are situations when you want to parallel process things. Other times you might have a list of tasks to accomplish, and you don’t want to make the user wait after pressing a button. This is where “Workers” can come in. They are independent scripts that run along side of your application, performing tasks, or “jobs.”

An example is with Dating DNA and our score system. We generate scores between users to show how compatible they are with each other. When a user signs up, or makes a significant change to their profile questionnaire, we need to run a job to query our database, build a list of potential users, and generate scores. This takes 10-20 seconds, and while it is pretty fast, we don’t want to make the user wait for that. So we queue up a job for the user, divide up the work among several workers, and process the work.

General Concept

For this post, we’ll use the example of generating reports. Lets say on your internal website there is a button that you can click and it will email the user a report, and the report takes 2-3 minutes to generate. When the button is clicked, your code will insert the job into the queue. Meanwhile, workers are monitoring the queue. A worker script will pull the job off the queue, process the report, and send the email when its done.

For the queue management, we’ll use Redis. To let PHP read and write data to Redis, we’ll use the PHP Library predis. In our examples we’ll use PHP 5.3, however predis has a PHP 5.2 backport if you are not running 5.3.

Adding Jobs

To add jobs, we’ll need to connect to our Redis server:

/*
 * Connecting to Redis
 */

const REDIS_HOST = '127.0.0.1';
const REDIS_PORT = 6379;

$predis = new Predis\Client(array(
    'scheme' => 'tcp',
    'host'   => REDIS_HOST,
    'port'   => REDIS_PORT,
));

We’ll assume in all of our examples that we’ve done the following above & connected to Redis.

Now, to manage our queues we’ll use the Redis Datatype LIST. Whats awesome about lists is that regardless of size, adding or removing at the start or end of a list is extremely fast. So if your queue has 10 items, or 10,000,000 items, Redis wil be able to push and pop entries quickly.

We’ll have three queues, one for each priority: high, normal, and low. For the Redis key names, we’ll use queue.priority.high, queue.priority.normal, etc. When interacting with lists, you work with the ends, one called right, the other called left. So we’ll add items on the right with the RPUSH (Right Push) command, and we’ll pull items off the left with the BLPOP (Blocking Left Pop) command. We won’t worry about the pulling items just yet.

You store strings as the values for the list. My personal preference is to store JSON objects so you can easily pass variables needed to perform the job.

/*
 * Adding items to the queue
 */

$job = new stdClass();
$job->id = 1;
$job->report = 'general';
$job->email = 'test@example.com';

// Add the job to the high priority queue
$predis->rpush('queue.priority.high', json_encode($job));

// Or, you could add it to the normal or low priority queue.
$predis->rpush('queue.priority.normal', json_encode($job));
$predis->rpush('queue.priority.low', json_encode($job));

Simple enough! Having different queue priorities is very beneficial in managing which jobs should get done first. For example, you might have an Executive’s request go into the high priority queue so they get the report quickly. You might also have a weekly cron that queues up reports to be sent automatically, so those can go in the low priority as to not disrupt people trying to get a manual report.

Now, on to the worker’s code.

Processing Jobs

For now, lets say we have a script running in the PHP CLI (Command Line Interface) that you started by running this command on the server:

First thing is we want this worker to work continuously, so we can do a while loop:

/*
 * Simple Continuous While Loop
 */

// Always True
while(1)
{
	/* ... perform tasks here ...  */
}

We’ll worry about making them more intelligent later. Now, let’s have our worker check the queue. You can do so with the BLPOP command:

/*
 * Checking the Queue
 */
$job = $predis->blpop('queue.priority.high'
						, 'queue.priority.normal'
						, 'queue.priority.low'
						, 10);

What we’re telling PHP to do is to check each queue in order of priority: high, normal, and then low. If it finds an item, it will immediately return an array with the name of the queue it came from, and the string of data that was pulled.

The B in BLPOP is “blocking.” What that means is that Redis will wait until either an item enters one of the queues, or the timeout is reached. In this case, the timeout is 10 seconds. So instead of polling (checking every few seconds in a loop), we check and wait, and after 10 seconds it will return null and we can check again.

What this gives us is near instantaneous queues. As soon as something is available, it is passed to the workers that are listening. You can also have multiple workers, and it will pass jobs to the first listening worker, and the next job to the next worker, so you don’t have to worry about multiple workers getting the same queued item.

After $predis->blpop() returns, if it has an array, it returned an item. If not, the timeout had been reached. We can check to see if a Job was returned, and if so to process the job:

/*
 * Checking to see if a Job was returned
 */

if($job)
{
	// Index 0 of the array holds which queue was returned
	$queue_name = $job[0];
	// Index 1 of the array holds the string value of the job.
	// Since we are passing it JSON, we'll decode it:
	$details = json_decode($job[1]);

	/* ... do job work ... */
}

Now we can have multiple workers listening to the same queues and scale our workload. Redis is very fast & efficient, and you could have hundreds or even thousands of workers listening to a single redis server.

Continuously Running Workers

There are a lot of options when it comes to deploying these workers. You can use a framework like Gearman, but for simple things, I like very simple solutions. I came across a blog post by Joseph Scott about a little 10 line perl script called solo. What it does is it will run a command, and to ensure that no one else is running that same exact command, it will lock a configurable port. This is awesome because the you don’t have to work about lock files or filesystem tricks, the kernel handles it all.

So what you can do is create a cronjob using solo to execute your script. First copy solo somewhere, I put it in my /usr/local/bin on my linux server. Then add this to your cron job using the command “crontab -e -u (which user to use)”:

What this will do is try to run this command every minute. Solo will check to see if the port is already in use, and if it is, it will exit. Otherwise, it will lock the port and then execute the command. The port will stay locked as long as the command is executing. Once the command terminates, the port will unlock.

Now, PHP is a great language, but it has been known to have some memory leaks while running a long time in a single instance. So we can have our scripts exit periodically to be restarted by our cron job. So lets make our “while(1)” statement a little smarter:

/*
 * A Smarter While Statement
 */

// Set the time limit for php to 0 seconds
set_time_limit(0);

/*
 * We'll set our base time, which is one hour (in seconds).
 * Once we have our base time, we'll add anywhere between 0
 * to 10 minutes randomly, so all workers won't quick at the
 * same time.
 */
$time_limit = 60 * 60 * 1; // Minimum of 1 hour
$time_limit += rand(0, 60 * 10); // Adding additional time

// Set the start time
$start_time = time();

// Continue looping as long as we don't go past the time limit
while(time() < $start_time + $time_limit)
{
	/* ... perorm BLPOP command ... */
	/* ... process jobs when received ... */
}

/* ... will quit once the time limit has been reached ... */

One key thing to note is randomly shifting the time limit for the script. I like to do this because you don’t want your workers all stopping and starting at the same time. So if I have 8 workers, one might, but the 7 will continue until the 8th starts back up again via the cron job.

Bells & Whistles

After using workers for awhile, here are a couple of ideas to enhance your workers & system managing them. First off, you can add some monitoring for your queues. Using Redis a HASH, you can use them to store the state of your workers.

/*
 * Assigning Worker IDs & Monitoring
 *
 * Usage: php worker.php 1
 */

// Gets the worker ID from the command line argument
$worker_id = $argv[1];

// Setting the Worker's Status
$predis->hset('worker.status', $worker_id, 'Started');

// Set the last time this worker checked in, use this to
// help determine when scripts die
$predis->hset('worker.status.last_time', $worker_id, time());

Another problem with workers that run for a long time (several hours) is when you make a change to their code, they won’t reload that change until they exit. What I’ve found to successfully restart them is having a “version” number set in Redis that is checked at the end of every loop:

/*
 * Using Versions to Check for Reloads
 */

$version = $predis->get('worker.version'); // i.e. number: 6

while(time() < $start_time + $time_limit)
{
	/* ... check for jobs and process them ... */

	/* ... then, at the very end of the while ... */
	if($predis->get('worker.version') != $version)
	{
		echo "New Version Detected... \n";
		echo "Reloading... \n";
		exit();
	}
}

You would simply INCR (increment) worker.version and after finishing their last job, the worker would exit, and solo would start it up again.

You can also kill specific threads by having them check for their value in a hash:

/*
 * Using Kill Switches to Check for Reloads
 */

while(time() < $start_time + $time_limit)
{
	/* ... check for jobs and process them ... */

	/* ... then, at the very end of the while ... */
	// Check to see if a kill has been set.
	if($predis->hget('worker.kill', $worker_id))
	{
		// Make sure to unset the kill request before exiting, or
		// your worker will just keep restarting.
		$predis->hdel('worker.kill', $worker_id);

		echo "Kill Request Detected... \n";
		echo "Reloading... \n";
		exit();
	}
}

Tweak to Solo & Logging

I made one small tweak in my version of solo, and that was to help it enable logging. Lets say I had three workers in my crontab:

The “>> /tmp/worker.log.1″ tells solo I want to log it’s output to a tmp file that I can tail and monitor their progress. This is great for debugging problems. However, when I did this, solo would write to the tmp file, and not the output from my script. To overcome this I changed the last line of solo:

# old
exec @ARGV;
# new
exec "@ARGV";

This would ensure my script wrote out to the tmp file, and not just solo.

Examples

I’ve created an example on GitHub that you can clone on your own machine. All you will need is PHP 5.3 and Redis installed.

To install redis, simple run these commands on your unix based system:

It will copy the redis binaries to /usr/local/bin.

To get a copy of the code, you can download them here. HOWEVER, it doesn’t include predis! You’ll have to download and copy predis inside there via this link. It is much easier to clone it as so:

Then, using different terminal windows (or using screen), you can run different worker.php instances, use creator.php to insert jobs, and monitor.php to watch the progress. This is all done from the command line.

If you’re using windows, I suggest installed a VM of Ubuntu and using that. If you really want to use Redis on windows, there are some Windows Binaries you can google and download. Good luck!

Here is a video where I demo the example:

(sorry for the poor mic quality)

Final Thoughts

I’ll post here shortly about how to run Redis in production with the init.d scripts and configuration files. One caveat to using solo is if your server has an application that randomly selects ports to use (i.e. VoIP, FTP), it might select one of your worker’s ports. But on a production server, you should have a good feel for which ports are available for locking.

If you want to learn more about Redis, check out their website.

Hopefully this will be helpful for anyone looking to use PHP Workers in an easy, simple way.

Related posts:

1. MySQL, Redis, and a Billion Rows – A Love Story
2. Creating Chatroom / Walls with Redis & PHP
3. Debuging with PHP, Stack Traces, and Redis

PHP

Its been a fun year with the Utah PHP Usergroup. I look forward to it each month, and if you’re in Utah, you definitely should come. We’ve had a lot of interesting talks, such as running your app in different clouds, several MySQL Forks and NoSQL stuff.

I also had a great time going to my first PHP specific conference. While I’ve attended and spoken at other conferences, Tek 11 was a lot of fun. Met a lot of smart and fun people.

As for work, I do almost exclusively PHP work. We power the Dating DNA website and APIs using PHP. All of the Clipish Apps’ APIs are powered by PHP. Alienware Arena, The GeForce StarCraft II Pro/Am Tournament, Thermaltake eSports, and a few other gaming tournaments sites, all powered by PHP.

I’m hoping to blog more about PHP this year. While I use it all over the place, it seems I write more about all the other pieces that interface with and support PHP.

New Technologies

There have been a handful of new technologies gaining traction. We’ve been using Redis in production for quite awhile now, and finding more and more uses for it. I’ve played around quite a bite with node.js, and I see some potential for augmenting PHP with it. MongoDB, CouchDB, and a few other data stores have shown a lot of potential in situations that play to their strength. I’m still a major fan of MySQL, but it’s nice to good options for things that MySQL isn’t as strong at.

New Goals

I’ve been working out, eating well, and such for almost 8 weeks now as well as my wife. The net effect is I’ve lost 16 pounds, and my wife has lost quite a bit too. This was done over the holidays even. We still have some ways to go to get to our goals, but we’re hoping to be successful in 2012 and hit our target goals. We’re even starting to enjoy lifting, cardio, and eating right.

I’m also setting the goal to blog once a week this year. I’ve tried daily, and that is just too much, but once a week is manageable. Hopefully it’ll be relating the technologies I use, to help share the cool stuff we get to use in-house.

Upcoming in 2012

We’re launching a new product in 2012 for Dating DNA, hopefully within a month or so. I’m also hoping to go to a few conferences this year, and perhaps speak at a few of them. Also, on the fun side of things, we’ll be going on a cruise in April, and perhaps a trip to China in September.

I hope everyone had an excellent 2011, and and amazing 2012!

Related posts:

1. My 2009 Technology Recap
2. Attending PHP Tek 2011
3. Goal for 2011: Learn C

What is Nginx?

Nginx (pronounced engine-x) is a fast, powerful, lightweight web server. I won’t go into the theory under-the-hood, but it’s focus is high concurrency with low memory usage. So while Apache is more robust in supporting many different features, nginx focuses on handling the important features very quickly. I still use Apache internally for our SVN & Trac web server. Heck, even at this time I’m using Apache to host this blog. However, Dating DNA, Clipish, CEVO, Alienware Arena, and some other high traffic sites/apis use nginx.

Ngnix, unlike Apache, doesn’t actually load PHP. Instead, it hands it off as a proxy to a “php handler” which acts like an Application Server. So nginx by itself won’t serve PHP files, but just static files.

What is PHP-FPM?

In the past, when working with something like Nginx or lighttpd, you would use spawn-fcgi to host your PHP application. However, spawn-fcgi had some major drawbacks and problems. So a guy named Andrei Nigmatulin created PHP-FPM, which stands for “PHP FastCGI Process Manager.” Since then, several others have contributed and ultimately it was include into the PHP core in version 5.3.3.

So from a high level look, on every PHP request Apache will load the entire installed PHP environment each time. This is for every request, and while it has been optimized as much as it can, that is a lot of overhead! With PHP-FPM, it will spin up a configurable amount of children. Each load the PHP environment and then will serve as many requests as it can without having to reload the environment. This saves on a lot of overhead!

Why use Nginx & PHP-FPM?

I should note, it is possible to configure/compile Apache in such a way that it can have similar performance capabilities. However, it takes a ton of work. Meanwhile, Nginx & PHP-FPM are very fast from the start, so I prefer just using them. You do lose some features, like .htaccess files won’t work so you’ll have to do that configuration in your virtual hosts.

How to Setup Nginx & PHP-FPM

Nginx

First off, lets setup Nginx.

Thats it! If you go to your server’s IP Address or Domain Name you should see a “Welcome to Nginx!”

PHP-FPM

Because PHP-FPM is only included by default in PHP 5.3.3 and later, and Ubuntu 10.04 LTS only has PHP 5.2.3, we have two options. Either we can install by source, or we can add another repository to install PHP-FPM. The latter is much, much easier, and there is a good PHP-FPM Repo for Ubuntu 10.04. To add it, you just run the following commands:

Now that we have the new repository, we can install PHP5:

Excellent! Now, if you need to change some of PHP-FPM’s configurations, they are found in /etc/php5/fpm/. The file php5-fpm.conf configures how FPM will opporate, and the php.ini is the settings file that PHP will use while running in FPM.

A few settings I like to change in /etc/php5/fpm/php5-fpm.conf:

The php5-fpm.conf that comes is pretty well documented on the different settings. Once you make a change, make sure to restart php5-fpm.conf: /etc/init.d/php5-fpm restart

Configuring Nginx

Now, we have a few settings for Nginx. The configuration files are found in /etc/nginx/. First we’ll edit nginx.conf. Here are a few settings we’ll want to change:

Now, we need to add a VirtualHost! Nginx uses the same layout in Ubuntu as Apache, so we’ll add configurations for each site we want under /etc/nginx/sites-available/. So using vi, nano, or whichever editor you prefer, create a /etc/nginx/sites-available/www.example.com file:

Now, we need to create the symlink from sites-enabled to sites-available:

Restart nginx with “/etc/init.d/nginx restart”. Go ahead and put a test.php file in your directory with a Hello World example, and see if it works. It should work, and you should be good to go.

Related posts:

1. Setting Up Percona Server 5.5 on Ubuntu 10.04
2. Preparing a VMWare Ubuntu Guest OS
3. PHP, Nginx, and Output Flushing

A lot of times VMWare will auto-install the VMWare tools for you when you first setup your VM. However, many times after setting it up I’ll do updates, and updates to the kernel will be applied, knocking out the VMWare Tools changes. These tools are used for things like file sharing between the guest and host.

So here are the steps to take to fully update and re-install the VMWare Tools. I got most of these from an article on Ubuntu’s website.

First, sudo aptitude update to make sure all my repository information is up-to-date.

Second, sudo aptitude safe-upgrade to get any and all updates for my newly installed OS.

Third, sudo apt-get install build-essential linux-headers-`uname -r` psmisc to install the linux headers for my kernel.

Fourth, copy and install the VMWare Tools:

# make a mount point if needed :
sudo mkdir /media/cdrom

# Install the tools
cd vmware-tools-distrib
sudo ./vmware-install.pl

Just follow the prompts and hit for all the default values.

Related posts:

1. Ubuntu Desktop Terminal – Su
2. Sending Email from non-email Ubuntu Server
3. My Honest Attempt With Linux Desktop

Hopefully at some point, your website gets a lot of traffic. Yay, you’ve reached your goal of getting good traffic, but it is soon followed by issues with performance and load. I like to call these the growing pains of a website. So as a web developer, I suddenly have the epiphany of “Hey, I need to scale my website!” What follows next is the biggest mistake a web developer can make:

They start looking at articles on how Google scales, or maybe how Facebook manages all of their traffic.

This is a mistake! To be brutally honest, you are not Google. You are not Facebook. You are not Twitter. You are a website that receives less than 0.000001% of the traffic that some major websites receive.

Why is this dangerous for web developers to do? Google, Twitter, Facebook, and others like them are solving complicated at a very large scale. I remember a presentation by a Twitter engineer who developed a program for a unique ID generator that can generate millions of IDs per second. The probability of you needing this type of solution is about the same as being struck by lightening. Applying these same practices at a much smaller scale are not realistic. If a locally owned grocery store wanted to open a second store, they would not adopt the same practices that Wal-mart use to manage their 8970 stores.

A Little Reality Check

I’m sure that most of my readers know of StackExchange.com. They power the popular website StackOverflow and several others. They have about two million visitors per day. That is a lot of traffic. StackOverflow is ranked #123 on Alexa. So you would imagine that they have a very large infrastructure serving all of this traffic?

Earlier this year, Stack Exchange wrote an article about their production environment. I was surprised on what exactly they were using. In paticular, the number of Production Servers*:

• 12 Web Servers (Windows Server 2008 R2)
• 2 Database Servers (Windows Server 2008 R2 and SQL Server 2008 R2)
• 2 Load Balancers (Ubuntu Server and HAProxy)
• 2 Caching Servers (Redis on CentOS)
• 1 Router / Firewall (Ubuntu Server)
• 3 DNS Servers (Bind on CentOS)

That is 22 servers for 2 Million Visits per day, serving 800 HTTP requests per second. Now, StackExchange did clarify that they did have other servers for management and fail over, but 22 servers handle their production load. This is a website that is ranked the 123rd most visited website in the world.

Honestly, most websites could be run on half a dozen servers if designed and configured correctly, including redundancy. Some really busy websites could run off a dozen servers. Unless you’re in the top 5,000 websites on the web, you really shouldn’t be worried about large-scale techniques.

So when you’re website is starting to grow, and you leave small scale, you’ll enter the phase of “Middle-Scale.”

What is Middle-Scale?

Middle-Scale is like being an awkward teenager:

You know that you can’t be the only one suffering through this, but you’re unsure how to proceed. It feels like you’re missing missing out on things everyone else must already know, but aren’t talking about. Like everyone else are awesome vampires or something:

But the reality is this: they don’t have some awesome secret! They are just normal teenagers.

This same idea applies to Middle-Scale websites.

Middle-Scale is when the most important things are still the best practices. Only now when you deviate from them you can feel those consequences. When you only had 100 users, a couple of nested queries and missing indexes didn’t cause that much of a problem. Your database is powerful enough to hide the inefficiencies. However, when you get to 10,000 users, your database can no longer hide the inefficiencies.

Middle-Scale is when simply separating your web server and database server isn’t enough. You’ll probably need to add some sort of cache like memcached. You’ll need to start tweaking your MySQL, Apache, and PHP configurations.

Then, after you’ve ironed out your inefficiencies, you’ll start to use multiple servers. You’ll probably add a Load Balancer with multiple web servers. After that, you’ll probably have some sort of Master-Slave replication for your Database for backups and fail-overs.

You start to leave this “Middle-Scale” classification when you move to multiple data centers, and start to do some load balancing at the the DNS layer. This is when you’ll start to have a dedicated sys-admin team.

Okay, I’m Middle-Scale! So what should I do? Where do I look?

First off, you must adhere to best practices. If you are working with PHP, research PHP performance and best practices. Do they same for each of your technologies, like Apache and MySQL. You will need to stop treating your application as one big app, and start to understand all of it’s moving parts.

Second, you must understand your specific problems. Scaling ins’t a problem, nor is it a solution. It is a generic term for many different types of solutions. Without understanding why your website is running slow, or why it cannot handle the load, you will not be able to create an effective solution.

So you don’t have a scaling problem. You have a MySQL performance issue, or a Apache problem, or a PHP problem. Most likely, it is something extremely specific. You have a high volume of MySQL write operations (i.e. UPDATE, INSERT, DELETE, REPLACE), or perhaps you are missing some indexes and have too many full table scans.

Third, Googling for help will only get you so far. You are starting to enter a phase when it is harder and harder to find answers to your broad issues. Talking with other experienced people who have gone through the Middle-Scale pains before will help immensely. I cannot recommend highly enough going to User groups. Being able to communicate with someone, either face to face, on the Phone, over IRC, etc. is invaluable. While I’ve learned a lot at conference and usergroup presentations, I’ve learned even more by just talking with the people attending and at the social gatherings.

Profile & Performance will Naturally Lead to Scaling

When you want to scale, it can feel like a very daunting task. It seems like this big unknown complicated solution. What in the world am I going to do? I remember feeling these worries when I first started to investigate load balancing and sharding for some websites I was working on.

The thing is, if you start to profile your application, you will discover it’s inefficiencies. I remember when I spent a sold week, working 12-16 hours a day profiling and optimizing Dating DNA’s database. I found a lot of bad queries, and I was able to cut our load times from 2-5 seconds to under 0.1 seconds. The CPU on the database server went from 80-90% CPU utilization to under 10%. It was incredible, and then I promptly took the entire next week off. When we migrated to new servers, I was able to move to less powerful database server and still have the same great performance. So by profiling and optimizing our database, I didn’t need to worry about spinning up multiple master databases and sharding our data.

With Clipish, we faced almost opposite scaling problems. The database was rarely an issue, but our web server CPU’s were. We do a lot of ImageMagick manipulations of images, and at high volumes on virtual servers this can be a big issue. So over the last year we’ve introduced some load balancing and CDN tools to help serve all 10 TB of bandwidth for Clipish.

The thing is, when you start to profile your application, you start to understand it’s low areas better, so you have a much better idea on what do to. Even if you don’t know your solution, it is much easier to find a solution with a sound understanding of the problem. For example “scaling mysql” yields much less helpful results than “mysql full table scans” in Google.

So should I ignore what Facebook and Google do for scaling?

Of course not! First off, they do cool stuff. Just because I’ll watch NASA launch a space shuttle doesn’t mean I’ll try to make a rocket system for my broken lawn mower. But you have to put what they are doing into context. People from large websites have published several good “best practices” articles on techniques that help any website. Especially things on the client/browser side of things. Just use caution. I cringe when I hear someone say “we’re trying to use Cassandra to solve XYZ problem at work” when it is severe overkill.

Final Thoughts

Most of the time when I talk about performance and scaling with other people, it is when they are in “critical mode.” Their website is down, slow, unusable, etc, and they are looking to fix the problem. I will say, it is much more difficult to profile in “critical mode” than profiling before hand. The reason is you are much more desperately focused on getting it working again instead of understanding the problem.

I’ll be giving a presentation this Thursday at UPHPU on Profiling PHP Applications. I’ll post the slides, and most likely write some articles on the subject afterwards. As always, feel free to email me or leave a comment.

Related posts:

1. Data Backups – There Are No Excuses
2. Speaking: Utah Open Source Conference 2009
3. Presentation: Real Life Scaling

I was reading The Restaurant at the End of the Universe by Douglas Adams, and I found a humorous example of event driven programming. So I thought I would share.

The Story

Arthur Dent was one of the last surviving humans after Earth had been destroyed to make way for a new hyperspace bypass. He was living on a spaceship known as the “Heart of Gold,” which onboard had an amazing piece of equipment called the “Nutri-Matic Drinks Synthesizer” which “claimed to produce the widest possible range of drinks personally matched to the tastes and metabolism of whoever cared to use it.”

The problem was that “when put to the test, however, it invariably produced a plastic cup filled with a liquid that was almost, but nit quite, entirely unlike tea.” So Arthur tried to reason with the machine with no success. He would ask for tea, and the machine would respond with a cheerful “Share and enjoy” and produce the same sickly liquid.

Finally, after all reason and attempts failed, he tried one last time:

“No,” he said, “look, it’s very, very simple … all I want … is a cup of tea. You are going to make one for me. Keep quiet and listen.”

And he sat. He told the Nutri-Matic about India, he told it about China, he told it about Ceylon. He told it about broad leaves drying in the sun. He told it about silver teapots. He told it about summer afternoons on the lawn. He told it about putting in the milk before the tea so it wouldn’t get scalded. He even told it (briefly) about the history of the East India Company.

“So that’s it, is it?” said the Nutri-Matic when he had finished.

“Yes,” said Arthur, “that is what I want.”

“You want the taste of dried leaves boiled in water?”

“Er, yes. With milk.”

“Squirted out of a cow?”

“Well, in a manner of speaking I suppose …”

“I’m going to need some help with this one,” said the machine tersely. All the cheerful burbling had dropped out of its voice and it now meant business.

“Well, anything I can do,” said Arthur.

“You’ve done quite enough,” the Nutri-Matic informed him.

So everything was grand? Well, not quite. There were two problems.

First, the Nutri-Matic needed help processing all this information. So, it asked the spaceship’s computer to help:

[The Nutri-Matic] summoned up the ship’s computer.

“Hi there!” said the ship’s computer.

The Nutri-Matic explained about tea to the ship’s computer. The computer boggled, linked logic circuits with the Nutri-Matic and together they lapsed into a grim silence.

Arthur watched and waited for a while, but nothing further happened.

He thumped it, but still nothing happened.

Eventually he gave up and wandered up to the bridge.

Now, having the Heart of Gold’s entire computer system help calculate how to generate a cow and grow a tree to produce leaves so Arthur could have his tea isn’t a terrible issue, except the second problem:

A Vogon ship approached and began to attack.

So when the rest of the crew rushed to the bridge, they were perplexed that their entire computer was jammed, and all of their controls, navigation, and weapons were unresponsive. However, the reason was soon discovered:

“What have you done to it, Monkeyman?” he breathed.

“Well,” said Arthur, “nothing in fact. It’s just that I think a short while ago it was trying to work out how to …”

“Yes?”

“Make me some tea.”

“That’s right guys,” the computer sang out suddenly, “just coping with that problem right now, and wow, it’s a biggy. Be with you in a while.” It lapsed back into a silence that was only matched for sheer intensity by the silence of the three people staring at Arthur Dent.

As if to relieve the tension, the Vogons chose that moment to start firing.

Don’t worry, fortunately the occupants of the ship were able to hold a seance, summon the great grandfather of the captain, and have the great grandfather’s spirit save them. (I know, such an awesome book)

Anyway, that is a long example, but it show a point:

Evented I/O

Basically Evented I/O is when your program does some sort of I/O (like querying a database, reading a file, etc) and continues on executing, not waiting (or blocking) for the response. So in our example above, the ship can be seen like a PHP script:

<?php

$ship = new HeartOfGold();
$nutrimatic = new NutriMatic();

$recipe = $ship->processQuery('How to make tea.');
$nutrimatic->makeDrink($recipe);

$enemies = $ship->detectEnemies();
if($enemies)
{
    $ship->evasiveManuvers();
}

The problem is the ship can’t detect enemies until it finishes processing Arther’s request on how to make tea. Now, in Node.js, you have functions that are callbacks. It is a way to say “Hey, when this request is done, do this afterwards.”

var ship = new HeartOfGold();
var nutrimatic = new NutriMatic();

ship.processQuery('How to make tea.', function(recipe){
   nutrimatic.makeDrink(recipe);
});

var enemies = ship.detectEnemies();
if(enemies.length > 0)
{
    ship.evasiveManuvers();
}

Now the function processQuery is non-blocking, meaning it will start the process of learning how to make tea and create a recipe, but continue execution. That way if it detects enemies, it can perform evasiveManuvers().

In a real life example, processQuery could be querying a database with a very slow query. In PHP, you can’t continue executing your code until that result is returned. However, in Node.js, if your SQL library takes advantage of Node.js’s evented abilities, you can send off a query and continue executing, and when the query returns you can do additional logic through your callback.

Hopefully that makes sense. I like finding programming analogies in books, especially with Douglas Adam’s series of Hitchhiker’s Guide to the Galaxy, since they are so fun to read.

Related posts:

1. Node.JS, LAMP, and The Future

The Problem

With Dating DNA, our goal was to display a compatibility score with every other user. This score is generated by taking two sets of answers to our 20 page survey, and running it through our algorithm. While it is super convenient for our users, this poses a problem. We wanted not only for people to be able to visit a profile and see a score, which is easy to generate a score on demand. We wanted our users to be able to browse other profiles sorted by their score with them. This requires us to pre-generate and store these scores, and then later query them.

So Ultimately, in theory, our “scores” data scaled at the following rate, with X as the number of registered users:

X^2 – X

That is at an exponential rate, which is practically impossible to scale at. The very first version of Dating DNA (before I took over the project) had about 1,500 users. The scores were stored in a single table. Every night, a “cron job” would run and get a list of every user, and loop through every possible iteration and re-generate each score. At 1,500 users that was 2,248,500 records. That is a lot for just 1,500 users. With our current user count, we would roughly have 359,999,400,000 score records. Thats 359 Billion records if you don’t want to count the commas.

This old system of daily cron jobs broke at about 2,000 users. We would have problems with the cron job taking over 48 hours to complete, and would end up with 3 scripts running at the say time. One for today, one for yesterday, and one for the day before that.

Smart Logic & Threading

We solved our first problem by using some common sense and smart logic. I won’t detail the lengthy measures we go through, but we can basically boil down our entire user base to an estimated top 5000 matches for any given user. If we have a heterosexual man named Joe, he doesn’t care about the hundreds of thousands of other heterosexual men who he scores a 2 or 3 with, but the heterosexual women he scores above a 6 with. So we don’t store the score for Frank, Jimmy, and Alan with Joe, but Sally, Rachael, and Tiffany.

The second part we solved was pre-generating scores for a user. After a User has reached a point in the survey where we have all the information we need to generate scores, and they are just filling in some miscellaneous, we put them in a queue. We then have a server process than is continuously running checking this queue, and spinning off multiple generation “threads” that crunch the data and store the score. We’ve spent a lot of time perfecting this system. Currently we typically can generate any given user’s matches in roughly 5 to 20 seconds, depending on how busy our website is.

Storing The Score in MySQL

The problem we now faced was the write through put of MySQL. Even through sharding and partitioning, we wanted to have a goal of sustaining 1,000 registrations per minute in a scalable and high performance manner. Which comes down to about 83,000 records per second that are either being inserted or updated. We then needed to be able to retrieve large volumes of scores just as fast.

I believe we could have bent MySQL to our will and got it to work, but it would be at a high cost of server power, and that cost wouldn’t scale well with our revenue stream. After we moved from the MySQL storage of the scores, I ran a query to see how many scores we were indeed storing. The final total was 950,363,992. Just 50 Million shy of one billion. It took 1 hour 49 min 38.27 sec to calculate that count. It is evidence that even though MySQL wasn’t the best choice for storing this data, it did it pretty well considering this single table was holding 90 times more data than any other table.

Picking Another Solution

In 2009 we started to throw around ideas for a new scoring system. I cannot stress enough when talking to others about “NoSQL” solutions the best solution for any given job is based on your data’s characteristics. User registration data needs to be treated differently than activity logging and basic stats. It might be okay to lose a few minutes of activity logging (depending on the app), but you definitely don’t want to be losing user accounts.

With Dating DNA’s scores, we had one great advantage. The data could be somewhat volatile, because we can always re-generate a set of matches for any given users. Of course, we didn’t want to lose all of it, because having to regenerate everyone’s scores is a major pain and extremely resource intensive. But if we lost a few minutes, anything lost could easily be regenerated. So when we started research for a solution, we were willing to sacrifice some persistance for performance. We wouldn’t be doing the same for our user registration data.

At first, I was contemplating building a completely in-house project to handle the data storage and retrieval of the scores. It would be a lot of work, and decided against. So I then thought about hacking together a custom solution with memcached. The idea would be a user’s set of matches would be stores in a variable in memcached. So the website and generation scripts would interface with memcached, and a server process would write inactive sets of scores (people who weren’t logged in) to a file on the disk. When they logged it and scores were being pulled and stored for that user, it would load the data from disk into memcached.

While the general concept was sound, the actual execution would be difficult. Memcached only supported strings for values, and we would still need some sort of database to manage which users had the data in memory vs disk, and the server process (probably just a php, python, or node.js script running continuously) would have to be running constantly, and if that broke things could get messy.

It boiled down too many points of failure and complexity. But it was a step in the right direction, so we kept looking for a better solution.

Redis, the Advanced Key-Store

I was talking with Joseph Scott, an employee of Automattic as a Bug Exorcist (not joking, his real title), and he mentioned I should look into Redis. He gave me a brief overview of what it was, and I shuffled that info back in my brain. I can’t remember how much longer it was before I checked out and compiled a copy of Redis, but I quickly discovered it could be a viable storage system for our scores.

So I spun up a virtual machine, installed Redis, and started to pound away at it. One of the things I wanted to test was the new feature (at the time) of Virtual Memory for redis. What this allowed was for Redis to make it’s own Virtual Memory on the server and store the lest recently used data to disk. When a Redis object was retrieved and it was in the VM, it would swap it back into memory, and swap older data to disk. This was just like the idea I had before with using an archaic system with memcached, but much more elegant.

The second thing was Redis’s support for multiple data types. So instead of having a json encoded string that held the scores and user ids for another user, we could have a hashtable or even a sorted set. It was a much more elegant solution than what we were thinking of before.

Some Limitations to Redis

However, there were a few limitations that we faced when implementing Redis. Redis works flawlessly with smaller sets of data. But the larger your data set, the more careful and aware you need to be about a few things that will kill your redis instance.

First off, with memcached, if you set a memory limit, it is a hard limit. I’ve never seen memcached use more memory than what you allow it to use. Redis, on the other hand, has soft memory limits. This is because of the way the Background Saves work so you can have persistant data. When a Background Save is issued, Redis will fork itself, and have one thread save a snapshot to the disk, and the other thread will continue to operate. In order to do this, Redis will exceed the standard memory limits, and your memory usage will go up much quicker. Once the background save is complete, it will close the forked backup process and sync the memory back to one data set. (I’m not a computer science guy, nor do I know a lot of lower level programming, so I might be describing this not 100% accurate, but this is how I envision it in my head).

Now, if you are not using the Virtual Memory, this isn’t that bad. However, when using Virtual Memory, the Background Saves take a great deal longer (from seconds to almost a minute or so), which isn’t too bad, but there is a catch. You will not be able to swap to the VM Disk until after the BG Save. This means that all Redis objects swaped to memory will stay in memory until after the BG Save is complete. This is because, like the memory from the fork, the Virtual Memory file is being used for the BG Save instead of the process handling requests.

So the one limitation we’ve encountered is we cannot run scripts that “query” large amounts of data from Redis. For example, it would be very simple to get a listing of users using the KEYS command, and then loop through the values using a HLEN to read the length. This will cause you to swap from and to the virtual memory a great deal. If a Background Save is occuring, Redis will not swap to disk until the BG Save is complete. This means if you have 10 GB of data in Virtual Memory, and you have a 1 GB instance of Redis, you will suddenly be reading gigs of data into Redis’s memory. If you are on a 2GB machine, you can easily use up all the memory on the server and then start using the System’s Swap.

Once you start using the Operating System’s Virtual Memory, it is game over. Your Redis instance’s performance will tank, and your Background Save might not finish, and you will need to restart redis.

There are some ways to give Redis a “Hard” limit on memory, but we opted to configure our servers in a way that doesn’t require this. If Redis hits the memory limit, it can start throwing write operation errors, which we didn’t want.

How We Configure Redis

After much trial and error and testing internally, we believe we found a sweet spot. We deploy a redis server, and spin up three redis instances on a different port each. Each is configured with 4 GB of Virtual Memory using 4096 size pages, and only 256 MB of “memory” using the vm-max-memory setting. While you would think this would mean a hard limit, it is a soft limit, and more of a goal “we’ll try to only use 256 MB of memory to store the data, if we’ll exceed it if needed.” Given our patterns of usage, Redis’s actually usage fluctuates (based on ps aux’s reporting) between 640 MB to 1100 MB of RAM, depending on if a background save is being executed or not. Redis is configured to perform background saves every 10 minutes, which take about a minute to perform.

So between the other admin services running on the box, and the three redis instances, we use just over 3GB RAM:

The amount of CPU required is extremely low, and almost 100% from writing the background saves to the disk:

So what do we get in return? We estimate each instance with 256 MB of data can hold roughly 2,000 active users. So with a single server we can support 6,000 users online at any given moment. We can store the scores for roughly 360,000 users on a single 4 GB box, which is about 1.8 Billion scores. Then, if we need more, we just provision another box, and our system will start assigning users to the instances on that machine.

Because of our ability to re-generate the scores, we decided to only convert the users who had logged in the past three months to the new system. If a user who hadn’t logged in since then logged in again, it would in the background assign them to a redis instance and rebuild their matches for them.

Using Redis with PHP

I recommend currently to use the PHP Redis client predis. I’ve used others like Rediska, but I prefer the straight forward approach of predis.

A high level view of how we use Redis with our PHP based website is we have a class called RedisManager than manages pretty much all the connections to Redis. It supports lazy connections (which is important to us, since we don’t want to have to connect to every instance of Redis we have), and I hope to open source it some day soon.

One key performance trick we’ve noticed is to use Pipelining to the redis instance. We don’t use this so much on the website, but our score generation “threads.” Writing thousands of scores one by one eats up a lot of network overhead versus sending them in batches (we send in batched of 500 or 1000, depending on the situation). Using pipelining is extremely fast for us, and I highly recommend it for any large batch of commands.

Using Redis Elsewhere in Dating DNA

Now, it might seem that we’ve put a lot of thought and effort into using Redis, and I want to make sure it was understood that Redis itself wasn’t difficult to use, but the volume of data were were dealing with. On Dating DNA, we also use Redis to power out in-app chat system (which I’ve written about previously), and it works great and is currently only using 146.70 MB of RAM, and serves thousands of requests per second.

The Future

I still have a lot of great ideas for Redis and Dating DNA, both with the score system, and outside of it. I plan on writing several reporting tools for Redis and hope to share them on github. I am currently working on the code and scripts for automatic deployment for Redis servers for Dating DNA, so we can scale easily with the push of a button. I’m excited for the work that is being done on Redis, and highly recommend it to anyone.

If there are details you would like to know more about, leave a comment and I’ll try to answer them. If you see me at tek11, feel free to ask me about this, and I can show you in detail how it works (internet permitting).

Related posts:

1. MySQL & PHP – SQL_CALC_FOUND_ROWS – An easy way to get the total number of rows regardless of LIMIT
2. MySQL, 40 Million Rows, MyISAM to InnoDB, 45 Minutes
3. Restoring Large MySQL Dump – 900 Million Rows

I’ve always wanted to attend a PHP conference for quite some time now, but it seemed for several years either it wasn’t in the budget, or I was just too busy. But I kept hearing great things about Tek, and it’s organizers, so when the Call for Papers opened up for Tek 11 this year, I submitted. Unfortunately I wasn’t able to make it on the presenter list, so I immediately registered and booked my flights so I wouldn’t find an excuse later on to not go.

For those in the Utah PHP Users Group who are not able to come out with me, and other friends of mine, let me know if there is anything interesting in the schedule, and I’ll see if I can attend and take some notes for you.

Related posts:

1. Goal for 2011: Learn C
2. Speaking: Utah Open Source Conference 2009
3. UTOSC – Day One

This decision was driven by a realization: the LAMP stack is dead. In the two decades since its birth, there have been fundamental shifts in the web’s make-up of content, protocols, servers, and clients. Together, these mark three ages of the web:

I. 1991-1999: The HTML Age.

The HTML Age was about documents, true to Tim Berners-Lee’s original vision of a “big, virtual documentation system in the sky.” The web was dominated by static, hand-coded files, which web clients crudely formatted (with defaults that offend even the mildest of typographiles). Static documents were served to static clients.

II. 2000-2009: The LAMP Age.

The LAMP Age was about databases. Rather than documents, the dominant web stacks were LAMP or LAMP-like. Whether CGI, PHP, Ruby on Rails, or Django, the dominant pattern was populating an HTML template with database values. Content was dynamic server-side, but still static client-side.

III. 2010-??: The Javascript Age.

The Javascript age is about event streams. Modern web pages are not pages, they are event-driven applications through which information moves. The core content vessel of the web — the document object model — still exists, but not as HTML markup. The DOM is an in-memory, efficiently-encoded data structure generated by Javascript.

LAMP architectures are dead because few web applications want to ship full payloads of markup to the client in response to a small event; they want to update just a fragment of the DOM, using Javascript. AJAX achieved this, but when your server-side LAMP templates are 10% HTML and 90% Javascript, it’s clear that you’re doing it wrong.

Claiming LAMP is Dead is Reaching

It goes on to explain how LAMP like architectures dead. I would think that is a very overzealous point of view, and to perhaps put it in some perspective:

During each of these “Ages” new technology was developed to solve problems. HTML was created to create inter-linking information. As the demand grew and became more difficult to manage, LAMP like tools like PHP, Django, Ruby on Rails, and others were developed to help make delivering this information easier. It wasn’t so much replacing the previous technology, but adding on to it. Sometimes you will replace older technology with newer ones, but we’re still using HTML & CSS, and augment it when needed.

But LAMP-like systems are deployed in production all over the place, powering some of the most visited websites on the net. While there are some people who are using Node.JS, like Yammer, but Node.JS isn’t serving nearly the volume of content like PHP, Ruby, Python, or event .NET.

What is Node.JS?

Basically, it is a lightweight framework wrapped around Google’s V8 JavaScript Engine. V8 is the JavaScript engine that powers Google Chrome, their open source Web Browser. Another part of Node.JS is that it is event based, which can make is very quick and efficient.

I’ve been tinkering with Node.JS for a couple of weeks, and it is very interesting and has a lot of potential. It’s unique scoping with JavaScript and event-based methodology makes certain things easy to accomplish that are more difficult in other traditional tools. There are some pretty cool libraries for Node.JS like Socket.IO, a library to support multiple transports using feature detection. So if your browser supports WebSocket or Adobe Flash Socket, it will use that, or AJAX long polling and multipart streaming, or even the Forever Iframe technique. So your application will use which ever type of communication is most efficient.

A Little Bit of a Reality Check

But lets be honest for a moment, Node.JS was created in 2010. It is still very, very young. In the video on the front page of the Node.JS is a video of it’s creator presenting to a PHP Users Group about Node, and even cautions using it in production. It still has a long way to go to maturing as a tool. Granted, being based off the V8 Engine brings a lot of maturity to the project.

As for replacing technologies like PHP, Ruby, and Django, I’m extremely skeptical that they are “dead.” Instead, I see Node.JS augmenting our existing technologies. Like how I’ve implemented Redis as a data store where MySQL was poorly suited, I see Node.JS functioning as a tool for real-time communication for the future of websites.

I hope to write more about Node.JS, and look forward to using it to solve new and unique problems. But I won’t be rewriting all my websites in Node.JS, it would be a nightmare. So like with NoSQL, and Cloud Computing, yes it is a new tool, but it won’t radically remove everything else up to this point. Like with Redis, I didn’t get rid of MySQL, I just use Redis when it is a better choice.

Related posts:

1. Local LAMP Developement & User Content
2. List of 50 PHP Tools
3. jQuery Tip: Better Toggle