No related posts.

Cal had sent me the questions before hand, and so I had jotted down a few things to mention during them. However, when we got to talking about the Utah Open Source Community, I forgot to mention one of the things I had written down to say, and that is this: I would not be where I am in my career, and would not have this great FOSS community in Utah, if it where not for the Shoulders of the Utah FOSS Giants that I stand on. (Standing on the Shoulders of Giants)

I had written three names down that I wanted to mention during the podcast. These three individuals were people who have had a great impact not only in the UTOS community, but in my own personal life and career as well. Because I was a bonehead and forgot then, I’d like to do it now.

Clint Savage (aka Herlo)

Clint is the founder of the Utah Open Source Foundation, he was also the guy who roped me into volunteering with UTOS for their conference. He was the guy who took the idea of a Utah Conference for Open Source and made it a reality. He spearheaded and helped organize the first three years, each growing larger than the year before, and getting better and better. He saw a vision inside his crazy head of a thriving community, and helped make it a reality.

Victor Villa (aka Mindjuju)

Victor was the President of the Utah PHP Usergroup (UPHPU) when I first started to attend in 2008. He was one busy cat with his work, and he had worked out a deal with his employer at the time to let the UPHPU group meet at their offices every month. Victor was always thinking of ways to make the meetings better, which were already great. He’d use his awesome A/V-fu to record the Audio, Video, and Screens of the presenters and then spliced them all together in a well produced video. He would ask, pester, and begged people to present so we always would have an awesome presentation every month. Every month, without fail, he was there. Come hail, snow, sickness, work emergencies, no matter what, he had those meetings month after month for several years.

Victor had the opportunity to change employers and get a better gig close to home, so as we lost our old venue he handed over the reigns of the Utah PHP Usergroup to another great guy, he didn’t take a break. He was soon after appointed the President of UTOS after Clint Savage needed to step down as he was traveling a lot.

Metaphorically speaking, if UTOS was a “child”, Clint had raised it up in it’s infancy to being a young adult ready to go to the next level. Victor was the guy to kick the “kid” in the butt and got it in a proper suit and tie ready to enter the business world. Victor worked tirelessly to get all the paperwork in order for its Non-Profit status (which is no small feat), and worked hard to make this upcoming UTOS Conference just as great as those before it. I know several of the “core-team” members of UTOS this last year had to step down due to lives being too busy, mine included, and he has had to do so much work. He has had a full-time job, gone back to school full-time, and also organizing Utah’s largest open source conference. Someone needs to give this man a gold medal! He is just such a great, strong pillar in the Open Source Community in Utah.

(A big thank you and shout-out needs to go to his wife, Tiff, as well. She puts up with us stealing her husband night after night, so we need to order two gold medals!)

Mac Newbold

The final person is Mac Newbold, the current president of the UPHPU group. He took over the reigns for Victor Villa, and orchestrated our current (and awesome) venue at C7′s Datacenter. I also believe he was the President of the UPHPU many years ago before Victor, so he has been involved with the UPHPU community for a very long time (the UPHPU group has been meeting since before 2004 I believe). He now has the great privilege of trying to cat-herd monthly presentations.

I know how busy all three of these guys are, and the sacrifices of time they give to the community. So I would like to sincerely say: thank you! Each one of you have had such a positive impact on my personal and professional life.

There are so many more great members in our local community, I wish I had time to write about every single one. If you live in Utah, I highly recommend to come out sometime to one of our local usergroup meetings, or even the yearly conference! And make sure to introduce yourself so we all can get to know you better!

No related posts.

Fatal error: Exception thrown without a stack frame in Unknown on line 0

Which is so much fun, because it doesn’t have a line number, so I had no direction as to what exactly was causing the problem. Some quick googling came up with this website stating the following error:

Most often, the error will appear if you use an exception handler combined with an error reporting to exception handler by converting it to an ErrorException, then there a suddenly a whole new magnitude of ways to throw errors within the exception handle, especially if E_NOTICE, E_STRICT and/or E_WARNING errors are converted. This form most often occurs when you use variables without first initializing them. This error may be preventable by wrapping the exception handler within a try/catch block.

A second form of this error occurs when you attempt to throw an exception in a destructor. This is well documented in the PHP manual, but this can still be triggered if you accidentally throw an exception:

Manually – calling “throw new Exception(‘hello world’)’ in a destructor

Implicitly – calling a function that throws an exception (e.g. calling function foo() which throws an exception)

Error handler (ErrorException) – instating a user-defined function as an error handler which throws an ErrorException (which is still an error)

The problem was that the code I was working with wasn’t using a registered exception handler, nor a destructor. So I went through the controller I was working with and commented out each line, and one by one brought each line back. After a few minutes I found the problem: I had a class that would save itself to the Session, and that class also had a __sleep method which is invoked on serialization (see PHP manual). Because I hadn’t manually closed my php session with session_write_close() (PHP Manual), PHP would close the session when cleaning up & closing the request, which is executed outside of the stack. My __sleep() function had an error and would throw an exception, leading to the Fatal Error.

It was a little tricky to track down, but made complete sense once I found it.

No related posts.

In the past I would just use my iPhone’s email-to-txt email address. However, when I received the txt message, it wasn’t formated very pretty, and it would have a different “From Number.” So if we had a crazy day, I would have 20-30 message threads in my iPhone all about Nagios. I’ve been lazy in the past, and didn’t clear them out until I had about 200 of them, and it was a lot of “swipe, delete, swipe, delete, swipe, delete” to get rid of them.

What I like out this setup is with Twilio, I can buy a phone number for $1 a month. So all my notifications come through the same number. I’m also planning a way to call/text the number and it will read/send the current status and list the hosts that are down.

First thing is to get your Twilio account setup. Its pretty easy, and when I signed up (a long time ago) I got $30 in credit, not sure if they are doing that still for new customers. Either way you can add a credit card and say how much credit you want to add when your account dips below a certain amount.

Now, you can buy a number and set it up. Even if you’re only sending text messages, Twilio wants you to have a “SMS Request URL” before you can send any text messages. So I just put in a URL to my site that does nothing (yet). You’ll need to grab your Account SID and Token.

How it Works

I threw the code up on GitHub. There are a few things you’ll need to do:

First in sendTextMsg.php change your configurations to match your phone number, account SID and token. Here is the entire sendTextMsg.php file:

<?php

require('twilio-php/Services/Twilio.php');

/* Start Configs */

$sid = "A123.....";
$token = "29d6b9f.......";
$twilio_number = '4045550101';

/* End Configs */

# Get the Argvs
$phone  = $argv[1];
$msg    = $argv[2];
$msg    = str_replace('\n', "\n", $msg);

$client = new Services_Twilio($sid, $token);
try
{
    $message = $client->account->sms_messages->create(
        $twilio_number, // From a valid Twilio number
        $phone, // Text this number
        $msg
    );
} catch (Exception $ex)
{
    var_dump($ex);
}

As you can see, it is super simple. All you need to do is put this script somewhere with the Twilio PHP libraries (or just use my github code with all of it in there). Then, in your Nagios configurations add the commands:

Then you need to add it to your contacts:

Simple, easy, and works very well. You can test the script by simply executing the command:

Enjoy!

No related posts.

Why Workers?

Sometimes there are situations when you want to parallel process things. Other times you might have a list of tasks to accomplish, and you don’t want to make the user wait after pressing a button. This is where “Workers” can come in. They are independent scripts that run along side of your application, performing tasks, or “jobs.”

An example is with Dating DNA and our score system. We generate scores between users to show how compatible they are with each other. When a user signs up, or makes a significant change to their profile questionnaire, we need to run a job to query our database, build a list of potential users, and generate scores. This takes 10-20 seconds, and while it is pretty fast, we don’t want to make the user wait for that. So we queue up a job for the user, divide up the work among several workers, and process the work.

General Concept

For this post, we’ll use the example of generating reports. Lets say on your internal website there is a button that you can click and it will email the user a report, and the report takes 2-3 minutes to generate. When the button is clicked, your code will insert the job into the queue. Meanwhile, workers are monitoring the queue. A worker script will pull the job off the queue, process the report, and send the email when its done.

For the queue management, we’ll use Redis. To let PHP read and write data to Redis, we’ll use the PHP Library predis. In our examples we’ll use PHP 5.3, however predis has a PHP 5.2 backport if you are not running 5.3.

Adding Jobs

To add jobs, we’ll need to connect to our Redis server:

/*
 * Connecting to Redis
 */

const REDIS_HOST = '127.0.0.1';
const REDIS_PORT = 6379;

$predis = new Predis\Client(array(
    'scheme' => 'tcp',
    'host'   => REDIS_HOST,
    'port'   => REDIS_PORT,
));

We’ll assume in all of our examples that we’ve done the following above & connected to Redis.

Now, to manage our queues we’ll use the Redis Datatype LIST. Whats awesome about lists is that regardless of size, adding or removing at the start or end of a list is extremely fast. So if your queue has 10 items, or 10,000,000 items, Redis wil be able to push and pop entries quickly.

We’ll have three queues, one for each priority: high, normal, and low. For the Redis key names, we’ll use queue.priority.high, queue.priority.normal, etc. When interacting with lists, you work with the ends, one called right, the other called left. So we’ll add items on the right with the RPUSH (Right Push) command, and we’ll pull items off the left with the BLPOP (Blocking Left Pop) command. We won’t worry about the pulling items just yet.

You store strings as the values for the list. My personal preference is to store JSON objects so you can easily pass variables needed to perform the job.

/*
 * Adding items to the queue
 */

$job = new stdClass();
$job->id = 1;
$job->report = 'general';
$job->email = 'test@example.com';

// Add the job to the high priority queue
$predis->rpush('queue.priority.high', json_encode($job));

// Or, you could add it to the normal or low priority queue.
$predis->rpush('queue.priority.normal', json_encode($job));
$predis->rpush('queue.priority.low', json_encode($job));

Simple enough! Having different queue priorities is very beneficial in managing which jobs should get done first. For example, you might have an Executive’s request go into the high priority queue so they get the report quickly. You might also have a weekly cron that queues up reports to be sent automatically, so those can go in the low priority as to not disrupt people trying to get a manual report.

Now, on to the worker’s code.

Processing Jobs

For now, lets say we have a script running in the PHP CLI (Command Line Interface) that you started by running this command on the server:

First thing is we want this worker to work continuously, so we can do a while loop:

/*
 * Simple Continuous While Loop
 */

// Always True
while(1)
{
	/* ... perform tasks here ...  */
}

We’ll worry about making them more intelligent later. Now, let’s have our worker check the queue. You can do so with the BLPOP command:

/*
 * Checking the Queue
 */
$job = $predis->blpop('queue.priority.high'
						, 'queue.priority.normal'
						, 'queue.priority.low'
						, 10);

What we’re telling PHP to do is to check each queue in order of priority: high, normal, and then low. If it finds an item, it will immediately return an array with the name of the queue it came from, and the string of data that was pulled.

The B in BLPOP is “blocking.” What that means is that Redis will wait until either an item enters one of the queues, or the timeout is reached. In this case, the timeout is 10 seconds. So instead of polling (checking every few seconds in a loop), we check and wait, and after 10 seconds it will return null and we can check again.

What this gives us is near instantaneous queues. As soon as something is available, it is passed to the workers that are listening. You can also have multiple workers, and it will pass jobs to the first listening worker, and the next job to the next worker, so you don’t have to worry about multiple workers getting the same queued item.

After $predis->blpop() returns, if it has an array, it returned an item. If not, the timeout had been reached. We can check to see if a Job was returned, and if so to process the job:

/*
 * Checking to see if a Job was returned
 */

if($job)
{
	// Index 0 of the array holds which queue was returned
	$queue_name = $job[0];
	// Index 1 of the array holds the string value of the job.
	// Since we are passing it JSON, we'll decode it:
	$details = json_decode($job[1]);

	/* ... do job work ... */
}

Now we can have multiple workers listening to the same queues and scale our workload. Redis is very fast & efficient, and you could have hundreds or even thousands of workers listening to a single redis server.

Continuously Running Workers

There are a lot of options when it comes to deploying these workers. You can use a framework like Gearman, but for simple things, I like very simple solutions. I came across a blog post by Joseph Scott about a little 10 line perl script called solo. What it does is it will run a command, and to ensure that no one else is running that same exact command, it will lock a configurable port. This is awesome because the you don’t have to work about lock files or filesystem tricks, the kernel handles it all.

So what you can do is create a cronjob using solo to execute your script. First copy solo somewhere, I put it in my /usr/local/bin on my linux server. Then add this to your cron job using the command “crontab -e -u (which user to use)”:

What this will do is try to run this command every minute. Solo will check to see if the port is already in use, and if it is, it will exit. Otherwise, it will lock the port and then execute the command. The port will stay locked as long as the command is executing. Once the command terminates, the port will unlock.

Now, PHP is a great language, but it has been known to have some memory leaks while running a long time in a single instance. So we can have our scripts exit periodically to be restarted by our cron job. So lets make our “while(1)” statement a little smarter:

/*
 * A Smarter While Statement
 */

// Set the time limit for php to 0 seconds
set_time_limit(0);

/*
 * We'll set our base time, which is one hour (in seconds).
 * Once we have our base time, we'll add anywhere between 0
 * to 10 minutes randomly, so all workers won't quick at the
 * same time.
 */
$time_limit = 60 * 60 * 1; // Minimum of 1 hour
$time_limit += rand(0, 60 * 10); // Adding additional time

// Set the start time
$start_time = time();

// Continue looping as long as we don't go past the time limit
while(time() < $start_time + $time_limit)
{
	/* ... perorm BLPOP command ... */
	/* ... process jobs when received ... */
}

/* ... will quit once the time limit has been reached ... */

One key thing to note is randomly shifting the time limit for the script. I like to do this because you don’t want your workers all stopping and starting at the same time. So if I have 8 workers, one might, but the 7 will continue until the 8th starts back up again via the cron job.

Bells & Whistles

After using workers for awhile, here are a couple of ideas to enhance your workers & system managing them. First off, you can add some monitoring for your queues. Using Redis a HASH, you can use them to store the state of your workers.

/*
 * Assigning Worker IDs & Monitoring
 *
 * Usage: php worker.php 1
 */

// Gets the worker ID from the command line argument
$worker_id = $argv[1];

// Setting the Worker's Status
$predis->hset('worker.status', $worker_id, 'Started');

// Set the last time this worker checked in, use this to
// help determine when scripts die
$predis->hset('worker.status.last_time', $worker_id, time());

Another problem with workers that run for a long time (several hours) is when you make a change to their code, they won’t reload that change until they exit. What I’ve found to successfully restart them is having a “version” number set in Redis that is checked at the end of every loop:

/*
 * Using Versions to Check for Reloads
 */

$version = $predis->get('worker.version'); // i.e. number: 6

while(time() < $start_time + $time_limit)
{
	/* ... check for jobs and process them ... */

	/* ... then, at the very end of the while ... */
	if($predis->get('worker.version') != $version)
	{
		echo "New Version Detected... \n";
		echo "Reloading... \n";
		exit();
	}
}

You would simply INCR (increment) worker.version and after finishing their last job, the worker would exit, and solo would start it up again.

You can also kill specific threads by having them check for their value in a hash:

/*
 * Using Kill Switches to Check for Reloads
 */

while(time() < $start_time + $time_limit)
{
	/* ... check for jobs and process them ... */

	/* ... then, at the very end of the while ... */
	// Check to see if a kill has been set.
	if($predis->hget('worker.kill', $worker_id))
	{
		// Make sure to unset the kill request before exiting, or
		// your worker will just keep restarting.
		$predis->hdel('worker.kill', $worker_id);

		echo "Kill Request Detected... \n";
		echo "Reloading... \n";
		exit();
	}
}

Tweak to Solo & Logging

I made one small tweak in my version of solo, and that was to help it enable logging. Lets say I had three workers in my crontab:

The “>> /tmp/worker.log.1″ tells solo I want to log it’s output to a tmp file that I can tail and monitor their progress. This is great for debugging problems. However, when I did this, solo would write to the tmp file, and not the output from my script. To overcome this I changed the last line of solo:

# old
exec @ARGV;
# new
exec "@ARGV";

This would ensure my script wrote out to the tmp file, and not just solo.

Examples

I’ve created an example on GitHub that you can clone on your own machine. All you will need is PHP 5.3 and Redis installed.

To install redis, simple run these commands on your unix based system:

It will copy the redis binaries to /usr/local/bin.

To get a copy of the code, you can download them here. HOWEVER, it doesn’t include predis! You’ll have to download and copy predis inside there via this link. It is much easier to clone it as so:

Then, using different terminal windows (or using screen), you can run different worker.php instances, use creator.php to insert jobs, and monitor.php to watch the progress. This is all done from the command line.

If you’re using windows, I suggest installed a VM of Ubuntu and using that. If you really want to use Redis on windows, there are some Windows Binaries you can google and download. Good luck!

Here is a video where I demo the example:

(sorry for the poor mic quality)

Final Thoughts

I’ll post here shortly about how to run Redis in production with the init.d scripts and configuration files. One caveat to using solo is if your server has an application that randomly selects ports to use (i.e. VoIP, FTP), it might select one of your worker’s ports. But on a production server, you should have a good feel for which ports are available for locking.

If you want to learn more about Redis, check out their website.

Hopefully this will be helpful for anyone looking to use PHP Workers in an easy, simple way.

No related posts.

PHP

Its been a fun year with the Utah PHP Usergroup. I look forward to it each month, and if you’re in Utah, you definitely should come. We’ve had a lot of interesting talks, such as running your app in different clouds, several MySQL Forks and NoSQL stuff.

I also had a great time going to my first PHP specific conference. While I’ve attended and spoken at other conferences, Tek 11 was a lot of fun. Met a lot of smart and fun people.

As for work, I do almost exclusively PHP work. We power the Dating DNA website and APIs using PHP. All of the Clipish Apps’ APIs are powered by PHP. Alienware Arena, The GeForce StarCraft II Pro/Am Tournament, Thermaltake eSports, and a few other gaming tournaments sites, all powered by PHP.

I’m hoping to blog more about PHP this year. While I use it all over the place, it seems I write more about all the other pieces that interface with and support PHP.

New Technologies

There have been a handful of new technologies gaining traction. We’ve been using Redis in production for quite awhile now, and finding more and more uses for it. I’ve played around quite a bite with node.js, and I see some potential for augmenting PHP with it. MongoDB, CouchDB, and a few other data stores have shown a lot of potential in situations that play to their strength. I’m still a major fan of MySQL, but it’s nice to good options for things that MySQL isn’t as strong at.

New Goals

I’ve been working out, eating well, and such for almost 8 weeks now as well as my wife. The net effect is I’ve lost 16 pounds, and my wife has lost quite a bit too. This was done over the holidays even. We still have some ways to go to get to our goals, but we’re hoping to be successful in 2012 and hit our target goals. We’re even starting to enjoy lifting, cardio, and eating right.

I’m also setting the goal to blog once a week this year. I’ve tried daily, and that is just too much, but once a week is manageable. Hopefully it’ll be relating the technologies I use, to help share the cool stuff we get to use in-house.

Upcoming in 2012

We’re launching a new product in 2012 for Dating DNA, hopefully within a month or so. I’m also hoping to go to a few conferences this year, and perhaps speak at a few of them. Also, on the fun side of things, we’ll be going on a cruise in April, and perhaps a trip to China in September.

I hope everyone had an excellent 2011, and and amazing 2012!

No related posts.

What is Nginx?

Nginx (pronounced engine-x) is a fast, powerful, lightweight web server. I won’t go into the theory under-the-hood, but it’s focus is high concurrency with low memory usage. So while Apache is more robust in supporting many different features, nginx focuses on handling the important features very quickly. I still use Apache internally for our SVN & Trac web server. Heck, even at this time I’m using Apache to host this blog. However, Dating DNA, Clipish, CEVO, Alienware Arena, and some other high traffic sites/apis use nginx.

Ngnix, unlike Apache, doesn’t actually load PHP. Instead, it hands it off as a proxy to a “php handler” which acts like an Application Server. So nginx by itself won’t serve PHP files, but just static files.

What is PHP-FPM?

In the past, when working with something like Nginx or lighttpd, you would use spawn-fcgi to host your PHP application. However, spawn-fcgi had some major drawbacks and problems. So a guy named Andrei Nigmatulin created PHP-FPM, which stands for “PHP FastCGI Process Manager.” Since then, several others have contributed and ultimately it was include into the PHP core in version 5.3.3.

So from a high level look, on every PHP request Apache will load the entire installed PHP environment each time. This is for every request, and while it has been optimized as much as it can, that is a lot of overhead! With PHP-FPM, it will spin up a configurable amount of children. Each load the PHP environment and then will serve as many requests as it can without having to reload the environment. This saves on a lot of overhead!

Why use Nginx & PHP-FPM?

I should note, it is possible to configure/compile Apache in such a way that it can have similar performance capabilities. However, it takes a ton of work. Meanwhile, Nginx & PHP-FPM are very fast from the start, so I prefer just using them. You do lose some features, like .htaccess files won’t work so you’ll have to do that configuration in your virtual hosts.

How to Setup Nginx & PHP-FPM

Nginx

First off, lets setup Nginx.

Thats it! If you go to your server’s IP Address or Domain Name you should see a “Welcome to Nginx!”

PHP-FPM

Because PHP-FPM is only included by default in PHP 5.3.3 and later, and Ubuntu 10.04 LTS only has PHP 5.2.3, we have two options. Either we can install by source, or we can add another repository to install PHP-FPM. The latter is much, much easier, and there is a good PHP-FPM Repo for Ubuntu 10.04. To add it, you just run the following commands:

Now that we have the new repository, we can install PHP5:

Excellent! Now, if you need to change some of PHP-FPM’s configurations, they are found in /etc/php5/fpm/. The file php5-fpm.conf configures how FPM will opporate, and the php.ini is the settings file that PHP will use while running in FPM.

A few settings I like to change in /etc/php5/fpm/php5-fpm.conf:

The php5-fpm.conf that comes is pretty well documented on the different settings. Once you make a change, make sure to restart php5-fpm.conf: /etc/init.d/php5-fpm restart

Configuring Nginx

Now, we have a few settings for Nginx. The configuration files are found in /etc/nginx/. First we’ll edit nginx.conf. Here are a few settings we’ll want to change:

Now, we need to add a VirtualHost! Nginx uses the same layout in Ubuntu as Apache, so we’ll add configurations for each site we want under /etc/nginx/sites-available/. So using vi, nano, or whichever editor you prefer, create a /etc/nginx/sites-available/www.example.com file:

Now, we need to create the symlink from sites-enabled to sites-available:

Restart nginx with “/etc/init.d/nginx restart”. Go ahead and put a test.php file in your directory with a Hello World example, and see if it works. It should work, and you should be good to go.

Related posts:

1. Setting Up Percona Server 5.5 on Ubuntu 10.04 I’m deploying some new servers today, and I realized I...
2. PHP, Nginx, and Output Flushing Alright, so one of the few hangups I’ve ran into...
3. Preparing a VMWare Ubuntu Guest OS I forget these steps all the time, so I figured...

A lot of times VMWare will auto-install the VMWare tools for you when you first setup your VM. However, many times after setting it up I’ll do updates, and updates to the kernel will be applied, knocking out the VMWare Tools changes. These tools are used for things like file sharing between the guest and host.

So here are the steps to take to fully update and re-install the VMWare Tools. I got most of these from an article on Ubuntu’s website.

First, sudo aptitude update to make sure all my repository information is up-to-date.

Second, sudo aptitude safe-upgrade to get any and all updates for my newly installed OS.

Third, sudo apt-get install build-essential linux-headers-`uname -r` psmisc to install the linux headers for my kernel.

Fourth, copy and install the VMWare Tools:

# make a mount point if needed :
sudo mkdir /media/cdrom

# Install the tools
cd vmware-tools-distrib
sudo ./vmware-install.pl

Just follow the prompts and hit for all the default values.

Related posts:

1. Ubuntu Desktop Terminal – Su This is just a quick reference for myself. When I...
2. Sending Email from non-email Ubuntu Server I have a couple of scripts that use the linux...
3. Sharing Sessions Across Multiple Servers With Memcache These last few days have been full of quickly implementing...

Hopefully at some point, your website gets a lot of traffic. Yay, you’ve reached your goal of getting good traffic, but it is soon followed by issues with performance and load. I like to call these the growing pains of a website. So as a web developer, I suddenly have the epiphany of “Hey, I need to scale my website!” What follows next is the biggest mistake a web developer can make:

They start looking at articles on how Google scales, or maybe how Facebook manages all of their traffic.

This is a mistake! To be brutally honest, you are not Google. You are not Facebook. You are not Twitter. You are a website that receives less than 0.000001% of the traffic that some major websites receive.

Why is this dangerous for web developers to do? Google, Twitter, Facebook, and others like them are solving complicated at a very large scale. I remember a presentation by a Twitter engineer who developed a program for a unique ID generator that can generate millions of IDs per second. The probability of you needing this type of solution is about the same as being struck by lightening. Applying these same practices at a much smaller scale are not realistic. If a locally owned grocery store wanted to open a second store, they would not adopt the same practices that Wal-mart use to manage their 8970 stores.

A Little Reality Check

I’m sure that most of my readers know of StackExchange.com. They power the popular website StackOverflow and several others. They have about two million visitors per day. That is a lot of traffic. StackOverflow is ranked #123 on Alexa. So you would imagine that they have a very large infrastructure serving all of this traffic?

Earlier this year, Stack Exchange wrote an article about their production environment. I was surprised on what exactly they were using. In paticular, the number of Production Servers*:

• 12 Web Servers (Windows Server 2008 R2)
• 2 Database Servers (Windows Server 2008 R2 and SQL Server 2008 R2)
• 2 Load Balancers (Ubuntu Server and HAProxy)
• 2 Caching Servers (Redis on CentOS)
• 1 Router / Firewall (Ubuntu Server)
• 3 DNS Servers (Bind on CentOS)

That is 22 servers for 2 Million Visits per day, serving 800 HTTP requests per second. Now, StackExchange did clarify that they did have other servers for management and fail over, but 22 servers handle their production load. This is a website that is ranked the 123rd most visited website in the world.

Honestly, most websites could be run on half a dozen servers if designed and configured correctly, including redundancy. Some really busy websites could run off a dozen servers. Unless you’re in the top 5,000 websites on the web, you really shouldn’t be worried about large-scale techniques.

So when you’re website is starting to grow, and you leave small scale, you’ll enter the phase of “Middle-Scale.”

What is Middle-Scale?

Middle-Scale is like being an awkward teenager:

You know that you can’t be the only one suffering through this, but you’re unsure how to proceed. It feels like you’re missing missing out on things everyone else must already know, but aren’t talking about. Like everyone else are awesome vampires or something:

But the reality is this: they don’t have some awesome secret! They are just normal teenagers.

This same idea applies to Middle-Scale websites.

Middle-Scale is when the most important things are still the best practices. Only now when you deviate from them you can feel those consequences. When you only had 100 users, a couple of nested queries and missing indexes didn’t cause that much of a problem. Your database is powerful enough to hide the inefficiencies. However, when you get to 10,000 users, your database can no longer hide the inefficiencies.

Middle-Scale is when simply separating your web server and database server isn’t enough. You’ll probably need to add some sort of cache like memcached. You’ll need to start tweaking your MySQL, Apache, and PHP configurations.

Then, after you’ve ironed out your inefficiencies, you’ll start to use multiple servers. You’ll probably add a Load Balancer with multiple web servers. After that, you’ll probably have some sort of Master-Slave replication for your Database for backups and fail-overs.

You start to leave this “Middle-Scale” classification when you move to multiple data centers, and start to do some load balancing at the the DNS layer. This is when you’ll start to have a dedicated sys-admin team.

Okay, I’m Middle-Scale! So what should I do? Where do I look?

First off, you must adhere to best practices. If you are working with PHP, research PHP performance and best practices. Do they same for each of your technologies, like Apache and MySQL. You will need to stop treating your application as one big app, and start to understand all of it’s moving parts.

Second, you must understand your specific problems. Scaling ins’t a problem, nor is it a solution. It is a generic term for many different types of solutions. Without understanding why your website is running slow, or why it cannot handle the load, you will not be able to create an effective solution.

So you don’t have a scaling problem. You have a MySQL performance issue, or a Apache problem, or a PHP problem. Most likely, it is something extremely specific. You have a high volume of MySQL write operations (i.e. UPDATE, INSERT, DELETE, REPLACE), or perhaps you are missing some indexes and have too many full table scans.

Third, Googling for help will only get you so far. You are starting to enter a phase when it is harder and harder to find answers to your broad issues. Talking with other experienced people who have gone through the Middle-Scale pains before will help immensely. I cannot recommend highly enough going to User groups. Being able to communicate with someone, either face to face, on the Phone, over IRC, etc. is invaluable. While I’ve learned a lot at conference and usergroup presentations, I’ve learned even more by just talking with the people attending and at the social gatherings.

Profile & Performance will Naturally Lead to Scaling

When you want to scale, it can feel like a very daunting task. It seems like this big unknown complicated solution. What in the world am I going to do? I remember feeling these worries when I first started to investigate load balancing and sharding for some websites I was working on.

The thing is, if you start to profile your application, you will discover it’s inefficiencies. I remember when I spent a sold week, working 12-16 hours a day profiling and optimizing Dating DNA’s database. I found a lot of bad queries, and I was able to cut our load times from 2-5 seconds to under 0.1 seconds. The CPU on the database server went from 80-90% CPU utilization to under 10%. It was incredible, and then I promptly took the entire next week off. When we migrated to new servers, I was able to move to less powerful database server and still have the same great performance. So by profiling and optimizing our database, I didn’t need to worry about spinning up multiple master databases and sharding our data.

With Clipish, we faced almost opposite scaling problems. The database was rarely an issue, but our web server CPU’s were. We do a lot of ImageMagick manipulations of images, and at high volumes on virtual servers this can be a big issue. So over the last year we’ve introduced some load balancing and CDN tools to help serve all 10 TB of bandwidth for Clipish.

The thing is, when you start to profile your application, you start to understand it’s low areas better, so you have a much better idea on what do to. Even if you don’t know your solution, it is much easier to find a solution with a sound understanding of the problem. For example “scaling mysql” yields much less helpful results than “mysql full table scans” in Google.

So should I ignore what Facebook and Google do for scaling?

Of course not! First off, they do cool stuff. Just because I’ll watch NASA launch a space shuttle doesn’t mean I’ll try to make a rocket system for my broken lawn mower. But you have to put what they are doing into context. People from large websites have published several good “best practices” articles on techniques that help any website. Especially things on the client/browser side of things. Just use caution. I cringe when I hear someone say “we’re trying to use Cassandra to solve XYZ problem at work” when it is severe overkill.

Final Thoughts

Most of the time when I talk about performance and scaling with other people, it is when they are in “critical mode.” Their website is down, slow, unusable, etc, and they are looking to fix the problem. I will say, it is much more difficult to profile in “critical mode” than profiling before hand. The reason is you are much more desperately focused on getting it working again instead of understanding the problem.

I’ll be giving a presentation this Thursday at UPHPU on Profiling PHP Applications. I’ll post the slides, and most likely write some articles on the subject afterwards. As always, feel free to email me or leave a comment.

Related posts:

1. PHP HipHop – What It Means There has been a lot of speculation about PHP, Facebook,...

I was reading The Restaurant at the End of the Universe by Douglas Adams, and I found a humorous example of event driven programming. So I thought I would share.

The Story

Arthur Dent was one of the last surviving humans after Earth had been destroyed to make way for a new hyperspace bypass. He was living on a spaceship known as the “Heart of Gold,” which onboard had an amazing piece of equipment called the “Nutri-Matic Drinks Synthesizer” which “claimed to produce the widest possible range of drinks personally matched to the tastes and metabolism of whoever cared to use it.”

The problem was that “when put to the test, however, it invariably produced a plastic cup filled with a liquid that was almost, but nit quite, entirely unlike tea.” So Arthur tried to reason with the machine with no success. He would ask for tea, and the machine would respond with a cheerful “Share and enjoy” and produce the same sickly liquid.

Finally, after all reason and attempts failed, he tried one last time:

“No,” he said, “look, it’s very, very simple … all I want … is a cup of tea. You are going to make one for me. Keep quiet and listen.”

And he sat. He told the Nutri-Matic about India, he told it about China, he told it about Ceylon. He told it about broad leaves drying in the sun. He told it about silver teapots. He told it about summer afternoons on the lawn. He told it about putting in the milk before the tea so it wouldn’t get scalded. He even told it (briefly) about the history of the East India Company.

“So that’s it, is it?” said the Nutri-Matic when he had finished.

“Yes,” said Arthur, “that is what I want.”

“You want the taste of dried leaves boiled in water?”

“Er, yes. With milk.”

“Squirted out of a cow?”

“Well, in a manner of speaking I suppose …”

“I’m going to need some help with this one,” said the machine tersely. All the cheerful burbling had dropped out of its voice and it now meant business.

“Well, anything I can do,” said Arthur.

“You’ve done quite enough,” the Nutri-Matic informed him.

So everything was grand? Well, not quite. There were two problems.

First, the Nutri-Matic needed help processing all this information. So, it asked the spaceship’s computer to help:

[The Nutri-Matic] summoned up the ship’s computer.

“Hi there!” said the ship’s computer.

The Nutri-Matic explained about tea to the ship’s computer. The computer boggled, linked logic circuits with the Nutri-Matic and together they lapsed into a grim silence.

Arthur watched and waited for a while, but nothing further happened.

He thumped it, but still nothing happened.

Eventually he gave up and wandered up to the bridge.

Now, having the Heart of Gold’s entire computer system help calculate how to generate a cow and grow a tree to produce leaves so Arthur could have his tea isn’t a terrible issue, except the second problem:

A Vogon ship approached and began to attack.

So when the rest of the crew rushed to the bridge, they were perplexed that their entire computer was jammed, and all of their controls, navigation, and weapons were unresponsive. However, the reason was soon discovered:

“What have you done to it, Monkeyman?” he breathed.

“Well,” said Arthur, “nothing in fact. It’s just that I think a short while ago it was trying to work out how to …”

“Yes?”

“Make me some tea.”

“That’s right guys,” the computer sang out suddenly, “just coping with that problem right now, and wow, it’s a biggy. Be with you in a while.” It lapsed back into a silence that was only matched for sheer intensity by the silence of the three people staring at Arthur Dent.

As if to relieve the tension, the Vogons chose that moment to start firing.

Don’t worry, fortunately the occupants of the ship were able to hold a seance, summon the great grandfather of the captain, and have the great grandfather’s spirit save them. (I know, such an awesome book)

Anyway, that is a long example, but it show a point:

Evented I/O

Basically Evented I/O is when your program does some sort of I/O (like querying a database, reading a file, etc) and continues on executing, not waiting (or blocking) for the response. So in our example above, the ship can be seen like a PHP script:

<?php

$ship = new HeartOfGold();
$nutrimatic = new NutriMatic();

$recipe = $ship->processQuery('How to make tea.');
$nutrimatic->makeDrink($recipe);

$enemies = $ship->detectEnemies();
if($enemies)
{
    $ship->evasiveManuvers();
}

The problem is the ship can’t detect enemies until it finishes processing Arther’s request on how to make tea. Now, in Node.js, you have functions that are callbacks. It is a way to say “Hey, when this request is done, do this afterwards.”

var ship = new HeartOfGold();
var nutrimatic = new NutriMatic();

ship.processQuery('How to make tea.', function(recipe){
   nutrimatic.makeDrink(recipe);
});

var enemies = ship.detectEnemies();
if(enemies.length > 0)
{
    ship.evasiveManuvers();
}

Now the function processQuery is non-blocking, meaning it will start the process of learning how to make tea and create a recipe, but continue execution. That way if it detects enemies, it can perform evasiveManuvers().

In a real life example, processQuery could be querying a database with a very slow query. In PHP, you can’t continue executing your code until that result is returned. However, in Node.js, if your SQL library takes advantage of Node.js’s evented abilities, you can send off a query and continue executing, and when the query returns you can do additional logic through your callback.

Hopefully that makes sense. I like finding programming analogies in books, especially with Douglas Adam’s series of Hitchhiker’s Guide to the Galaxy, since they are so fun to read.

Related posts:

1. Node.JS, LAMP, and The Future I just read an article about “Node.JS and the JavaScript...