I’ve met and talked with several companies that have had a great success with this approach. Toni Schneider, CEO at Automattic, has written about the 5 reasons why your company should be distributed.

But on the flip side, I’ve met a few developers, managers, and owners who have horror stories of how it “didn’t work out at all.” They talk about how these workers were too distant from their team, how they didn’t communicate well, and struggled to get them to produce.

These are legitimate concerns and challenges face businesses. However, I think there is one key thing businesses can do to help minimize these challenges and make their attempts to bring in a distributed workforce a reality. At to show how it can be done, we’ll use our fictional company, Muppets Software Inc:

Our Example: The Muppets

Muppets Software is a company that produces the finest websites. They have a small office out in California and are doing pretty well. Their team consists of:

Kermit – The Manager

Fozzie – Lead Developer

Rizzo – Junior Developer

Gonzo – Lead Sys-Admin

Miss Piggy – Sales

Things are going very well, but they are getting too busy, and to expand they need to bring on another developer. After a great deal of searching, they find an excellent candidate:

Beaker – Awesome Developer

There is only one problem, he lives in New York and can’t move to California. But he seems perfect, so Kermit decides to try having him be a remote worker. Fozzie and Rizzo think he’ll be good, so they hire him. It starts off pretty well at first, and they assign Beaker to help Fozzie with the new Sesame Street website. They decide to use a ticket system to help better manage what they need to get done, and set off to work.

After a time it seems Beaker would drop the ball and not get something done. It would be 4 PM in California, and Fozzie couldn’t get ahold of Beaker, so he would just have Rizzo do it. The next day Fozzie talked to Beaker, to find out Beaker hadn’t been in the impromtu meeting where this change was requested and didn’t know about it.

As time goes on, Gonzo starts to get annoyed with Beaker. It seems he would always come into the office with a list of things asking to be done. Since company policy says only Gonzo can make changes on production servers, he is suppose to be the only one to do it. Fozzie and Rizzo both have access to the production servers, and they have an unspoken agreement that the developers can make small changes to production servers as long as they stick their head in Gonzo’s cube and run it by him first. Of course, Beaker is unaware of this fact, so he continues to write requests to Gonzo.

Its coming down to crunch time with the Sesame project, and Miss Piggy is worried that they’ll miss their deadline. She keeps asking for last minute changes to make the site better for the client, so several nights she buys pizza for the office and she stays up late with the Developers to work overtime. She even calls Beaker to have him help out, but he is always the one to leave early, and seems the least effective out of everyone.

After the Sesame project, Gonzo, Fozzie, Rizzo and Miss Piggy talk with Kermit about their concerns with Beaker. They say he is a nice guy, but its been hard working with him. They say there isn’t one big incident, but rather it is just an overall vibe.

Kermit calls up Beaker and has a nice chat with him. Beaker says he feels bad the others feel this way, and shares some of his frustrations as well. He has a hard time staying up so late with the team on these late-night pushes, and many times tickets aren’t updated with the information he needs. Kermit sees his point, but doesn’t really know what to do to solve the problems. They’ve talked many times about keeping the ticket system up-to-date, but when things get busy, its so easy for the in-house employees to forget.

What to do?

I’ve seen and heard these problems, and I’ve even been in Beaker’s shoes in the past with a short-term project.

The bottom line is that it boils down to a problem with communication. It is so hard for remote developers to keep up with everything when there is a lot of communication that is happening outside of his reach. He isn’t at lunch with the other co-workers, or around the water cooler. He listens mostly to the meetings since it is so hard to get a word in without being there.

So what would be my recommendation? Turn your entire company into a distributed company.

Wait, what? I’m not suggesting a company immediately close it’s offices. But allow the rest of your employees telecommute as well. I’d say at least 3 out of the 5 days a week.

As a manager, this is a scary thing. But what this does is naturally change the communication channels. So instead of awkward channels of communication forced on the employees to use, these channels become the channels of communication that empower their own work. They get to enjoy the benefits of telecommuting, regardless of where they live, just like Beaker.

When everyone uses the same channels of communication, all of a sudden being 10 miles away or 1000 miles away does not matter. It will become a much more natural process to send emails, update tickets, and chat online. Even when people are in the office, these channels will tend to stay the primary methods of communication.

Almost all of the most successful distributed companies are 100% distributed, not just 10%. I really do think that is the key. Even with Dating DNA, where I lived in Utah and the rest of the team (all 3 of them) lived in San Diego, we were a very distributed team. Our iPhone Dev only met face-to-face with the owner/project manager of the company once or twice a year, even though they lived 15 minutes from each other. Even when we go to expand our team, I like knowing that if we found the perfect match for a new employee, we can hire them regardless of physical boundaries.

Now, this isn’t a silver bullet approach. There is a lot more to making a distributed workforce work. However, I think the key is for this paradigm shift in thought: we’re not just hiring a remote worker, we becoming a distributed company.

Related posts:

1. The Skill of Making Habits
2. Mac Keyboard Quirks – Making Mac more like Windows
3. Lifehacker – Making your Mac sing!

In the past I would just use my iPhone’s email-to-txt email address. However, when I received the txt message, it wasn’t formated very pretty, and it would have a different “From Number.” So if we had a crazy day, I would have 20-30 message threads in my iPhone all about Nagios. I’ve been lazy in the past, and didn’t clear them out until I had about 200 of them, and it was a lot of “swipe, delete, swipe, delete, swipe, delete” to get rid of them.

What I like out this setup is with Twilio, I can buy a phone number for $1 a month. So all my notifications come through the same number. I’m also planning a way to call/text the number and it will read/send the current status and list the hosts that are down.

First thing is to get your Twilio account setup. Its pretty easy, and when I signed up (a long time ago) I got $30 in credit, not sure if they are doing that still for new customers. Either way you can add a credit card and say how much credit you want to add when your account dips below a certain amount.

Now, you can buy a number and set it up. Even if you’re only sending text messages, Twilio wants you to have a “SMS Request URL” before you can send any text messages. So I just put in a URL to my site that does nothing (yet). You’ll need to grab your Account SID and Token.

How it Works

I threw the code up on GitHub. There are a few things you’ll need to do:

First in sendTextMsg.php change your configurations to match your phone number, account SID and token. Here is the entire sendTextMsg.php file:

<?php

require('twilio-php/Services/Twilio.php');

/* Start Configs */

$sid = "A123.....";
$token = "29d6b9f.......";
$twilio_number = '4045550101';

/* End Configs */

# Get the Argvs
$phone  = $argv[1];
$msg    = $argv[2];
$msg    = str_replace('\n', "\n", $msg);

$client = new Services_Twilio($sid, $token);
try
{
    $message = $client->account->sms_messages->create(
        $twilio_number, // From a valid Twilio number
        $phone, // Text this number
        $msg
    );
} catch (Exception $ex)
{
    var_dump($ex);
}

As you can see, it is super simple. All you need to do is put this script somewhere with the Twilio PHP libraries (or just use my github code with all of it in there). Then, in your Nagios configurations add the commands:

Then you need to add it to your contacts:

Simple, easy, and works very well. You can test the script by simply executing the command:

Enjoy!

Related posts:

1. Setting up Nagios for Servers
2. Presentation: Real Life Scaling
3. BestBuy Gives Me $8,770 — Wait, Just $87.00

In PHP, sometimes you would just add something like this:

echo "I'm here!";
exit();

However, in Nginx configuration files, it isn’t as easy…

… or is it?

One thing that works well is the rewrite directive. You can append variables to the URL to be rewritten. Another great thing is a rewrite statement can go just about anywhere. So lets say we were trying to debug this location statement:

Now lets say its returning a 404, and I’m not 100% sure what the actual value of $fastcgi_script_name is. I can add this to it:

This will redirect your HTTP request to Google.com and put the value in the query textfield. Bingo, I can easily see the actual value! Pretty helpful when you have a large, complex server definition.

Related posts:

1. PHP, Nginx, and Output Flushing
2. Setting Up Nginx & PHP-FPM on Ubuntu 10.04
3. Simple Trick: History Command

Senator Lee,

My name is Justin Carmony, and I am the C.T.O. for Dating DNA, LLC, a company that makes iPhone Apps. This is my first time writing a senator, but I feel it is important to convey my thoughts on SOPA and PIPA.

We have struggled with people downloading our Apps from 3rd Party Websites and not paying for them. We are able to monitor and track these “cracked” apps when they connect to our services, and these “cracked apps” number in the hundreds of thousands over time. That is potentially millions of dollars in lost revenue.

However, I do NOT support SOPA, PIPA, or bills like it that give “blacklist” or “takedown” authority without due process.

Even though we struggle with piracy, these bills will do very little in preventing unauthorized distribution of copyrighted materials. However, they are extremely dangerous in terms of abuse against legitimate businesses. Even though the cost of piracy can be great, we already have tools and laws to help us minimize piracy. We work hard at providing a great service, and we’ve even successfully turned “pirates” into “legal, legitimate customers.” SOPA & PIPA will do very little in helping us stop real piracy, but will also expose us to “takedowns claims” without any oversight or due process.

There is already testimonies on legal records of Media Companies such as Warner Music and Universal Music group admitting to using DMCA to take down content they had no copyright ownership to. Their motives instead were to silence groups being critical of them. (see “A Business Person’s SOPA Primer” — http://avoidingagoatrodeo.com/2011/12/26/a-businesspersons-sopa-primer/)

Please, as my elected representative, pull your support of SOPA. Our small business would not survive a “takedown” if a competitor used SOPA with false claims. We depend on the internet to keep us in business. I hope you can see, and understand, that due process and legal oversight should still apply to the internet.

Thank you,

Justin Carmony
Utah Citizen

Related posts:

1. Comcast Limit to 250GB a month for Residential
2. Google Apps Dropping IE 6
3. Yahoo vs. Rhapsody – Why Yahoo Lost

Why Workers?

Sometimes there are situations when you want to parallel process things. Other times you might have a list of tasks to accomplish, and you don’t want to make the user wait after pressing a button. This is where “Workers” can come in. They are independent scripts that run along side of your application, performing tasks, or “jobs.”

An example is with Dating DNA and our score system. We generate scores between users to show how compatible they are with each other. When a user signs up, or makes a significant change to their profile questionnaire, we need to run a job to query our database, build a list of potential users, and generate scores. This takes 10-20 seconds, and while it is pretty fast, we don’t want to make the user wait for that. So we queue up a job for the user, divide up the work among several workers, and process the work.

General Concept

For this post, we’ll use the example of generating reports. Lets say on your internal website there is a button that you can click and it will email the user a report, and the report takes 2-3 minutes to generate. When the button is clicked, your code will insert the job into the queue. Meanwhile, workers are monitoring the queue. A worker script will pull the job off the queue, process the report, and send the email when its done.

For the queue management, we’ll use Redis. To let PHP read and write data to Redis, we’ll use the PHP Library predis. In our examples we’ll use PHP 5.3, however predis has a PHP 5.2 backport if you are not running 5.3.

Adding Jobs

To add jobs, we’ll need to connect to our Redis server:

/*
 * Connecting to Redis
 */

const REDIS_HOST = '127.0.0.1';
const REDIS_PORT = 6379;

$predis = new Predis\Client(array(
    'scheme' => 'tcp',
    'host'   => REDIS_HOST,
    'port'   => REDIS_PORT,
));

We’ll assume in all of our examples that we’ve done the following above & connected to Redis.

Now, to manage our queues we’ll use the Redis Datatype LIST. Whats awesome about lists is that regardless of size, adding or removing at the start or end of a list is extremely fast. So if your queue has 10 items, or 10,000,000 items, Redis wil be able to push and pop entries quickly.

We’ll have three queues, one for each priority: high, normal, and low. For the Redis key names, we’ll use queue.priority.high, queue.priority.normal, etc. When interacting with lists, you work with the ends, one called right, the other called left. So we’ll add items on the right with the RPUSH (Right Push) command, and we’ll pull items off the left with the BLPOP (Blocking Left Pop) command. We won’t worry about the pulling items just yet.

You store strings as the values for the list. My personal preference is to store JSON objects so you can easily pass variables needed to perform the job.

/*
 * Adding items to the queue
 */

$job = new stdClass();
$job->id = 1;
$job->report = 'general';
$job->email = 'test@example.com';

// Add the job to the high priority queue
$predis->rpush('queue.priority.high', json_encode($job));

// Or, you could add it to the normal or low priority queue.
$predis->rpush('queue.priority.normal', json_encode($job));
$predis->rpush('queue.priority.low', json_encode($job));

Simple enough! Having different queue priorities is very beneficial in managing which jobs should get done first. For example, you might have an Executive’s request go into the high priority queue so they get the report quickly. You might also have a weekly cron that queues up reports to be sent automatically, so those can go in the low priority as to not disrupt people trying to get a manual report.

Now, on to the worker’s code.

Processing Jobs

For now, lets say we have a script running in the PHP CLI (Command Line Interface) that you started by running this command on the server:

First thing is we want this worker to work continuously, so we can do a while loop:

/*
 * Simple Continuous While Loop
 */

// Always True
while(1)
{
	/* ... perform tasks here ...  */
}

We’ll worry about making them more intelligent later. Now, let’s have our worker check the queue. You can do so with the BLPOP command:

/*
 * Checking the Queue
 */
$job = $predis->blpop('queue.priority.high'
						, 'queue.priority.normal'
						, 'queue.priority.low'
						, 10);

What we’re telling PHP to do is to check each queue in order of priority: high, normal, and then low. If it finds an item, it will immediately return an array with the name of the queue it came from, and the string of data that was pulled.

The B in BLPOP is “blocking.” What that means is that Redis will wait until either an item enters one of the queues, or the timeout is reached. In this case, the timeout is 10 seconds. So instead of polling (checking every few seconds in a loop), we check and wait, and after 10 seconds it will return null and we can check again.

What this gives us is near instantaneous queues. As soon as something is available, it is passed to the workers that are listening. You can also have multiple workers, and it will pass jobs to the first listening worker, and the next job to the next worker, so you don’t have to worry about multiple workers getting the same queued item.

After $predis->blpop() returns, if it has an array, it returned an item. If not, the timeout had been reached. We can check to see if a Job was returned, and if so to process the job:

/*
 * Checking to see if a Job was returned
 */

if($job)
{
	// Index 0 of the array holds which queue was returned
	$queue_name = $job[0];
	// Index 1 of the array holds the string value of the job.
	// Since we are passing it JSON, we'll decode it:
	$details = json_decode($job[1]);

	/* ... do job work ... */
}

Now we can have multiple workers listening to the same queues and scale our workload. Redis is very fast & efficient, and you could have hundreds or even thousands of workers listening to a single redis server.

Continuously Running Workers

There are a lot of options when it comes to deploying these workers. You can use a framework like Gearman, but for simple things, I like very simple solutions. I came across a blog post by Joseph Scott about a little 10 line perl script called solo. What it does is it will run a command, and to ensure that no one else is running that same exact command, it will lock a configurable port. This is awesome because the you don’t have to work about lock files or filesystem tricks, the kernel handles it all.

So what you can do is create a cronjob using solo to execute your script. First copy solo somewhere, I put it in my /usr/local/bin on my linux server. Then add this to your cron job using the command “crontab -e -u (which user to use)”:

What this will do is try to run this command every minute. Solo will check to see if the port is already in use, and if it is, it will exit. Otherwise, it will lock the port and then execute the command. The port will stay locked as long as the command is executing. Once the command terminates, the port will unlock.

Now, PHP is a great language, but it has been known to have some memory leaks while running a long time in a single instance. So we can have our scripts exit periodically to be restarted by our cron job. So lets make our “while(1)” statement a little smarter:

/*
 * A Smarter While Statement
 */

// Set the time limit for php to 0 seconds
set_time_limit(0);

/*
 * We'll set our base time, which is one hour (in seconds).
 * Once we have our base time, we'll add anywhere between 0
 * to 10 minutes randomly, so all workers won't quick at the
 * same time.
 */
$time_limit = 60 * 60 * 1; // Minimum of 1 hour
$time_limit += rand(0, 60 * 10); // Adding additional time

// Set the start time
$start_time = time();

// Continue looping as long as we don't go past the time limit
while(time() < $start_time + $time_limit)
{
	/* ... perorm BLPOP command ... */
	/* ... process jobs when received ... */
}

/* ... will quit once the time limit has been reached ... */

One key thing to note is randomly shifting the time limit for the script. I like to do this because you don’t want your workers all stopping and starting at the same time. So if I have 8 workers, one might, but the 7 will continue until the 8th starts back up again via the cron job.

Bells & Whistles

After using workers for awhile, here are a couple of ideas to enhance your workers & system managing them. First off, you can add some monitoring for your queues. Using Redis a HASH, you can use them to store the state of your workers.

/*
 * Assigning Worker IDs & Monitoring
 *
 * Usage: php worker.php 1
 */

// Gets the worker ID from the command line argument
$worker_id = $argv[1];

// Setting the Worker's Status
$predis->hset('worker.status', $worker_id, 'Started');

// Set the last time this worker checked in, use this to
// help determine when scripts die
$predis->hset('worker.status.last_time', $worker_id, time());

Another problem with workers that run for a long time (several hours) is when you make a change to their code, they won’t reload that change until they exit. What I’ve found to successfully restart them is having a “version” number set in Redis that is checked at the end of every loop:

/*
 * Using Versions to Check for Reloads
 */

$version = $predis->get('worker.version'); // i.e. number: 6

while(time() < $start_time + $time_limit)
{
	/* ... check for jobs and process them ... */

	/* ... then, at the very end of the while ... */
	if($predis->get('worker.version') != $version)
	{
		echo "New Version Detected... \n";
		echo "Reloading... \n";
		exit();
	}
}

You would simply INCR (increment) worker.version and after finishing their last job, the worker would exit, and solo would start it up again.

You can also kill specific threads by having them check for their value in a hash:

/*
 * Using Kill Switches to Check for Reloads
 */

while(time() < $start_time + $time_limit)
{
	/* ... check for jobs and process them ... */

	/* ... then, at the very end of the while ... */
	// Check to see if a kill has been set.
	if($predis->hget('worker.kill', $worker_id))
	{
		// Make sure to unset the kill request before exiting, or
		// your worker will just keep restarting.
		$predis->hdel('worker.kill', $worker_id);

		echo "Kill Request Detected... \n";
		echo "Reloading... \n";
		exit();
	}
}

Tweak to Solo & Logging

I made one small tweak in my version of solo, and that was to help it enable logging. Lets say I had three workers in my crontab:

The “>> /tmp/worker.log.1″ tells solo I want to log it’s output to a tmp file that I can tail and monitor their progress. This is great for debugging problems. However, when I did this, solo would write to the tmp file, and not the output from my script. To overcome this I changed the last line of solo:

# old
exec @ARGV;
# new
exec "@ARGV";

This would ensure my script wrote out to the tmp file, and not just solo.

Examples

I’ve created an example on GitHub that you can clone on your own machine. All you will need is PHP 5.3 and Redis installed.

To install redis, simple run these commands on your unix based system:

It will copy the redis binaries to /usr/local/bin.

To get a copy of the code, you can download them here. HOWEVER, it doesn’t include predis! You’ll have to download and copy predis inside there via this link. It is much easier to clone it as so:

Then, using different terminal windows (or using screen), you can run different worker.php instances, use creator.php to insert jobs, and monitor.php to watch the progress. This is all done from the command line.

If you’re using windows, I suggest installed a VM of Ubuntu and using that. If you really want to use Redis on windows, there are some Windows Binaries you can google and download. Good luck!

Here is a video where I demo the example:

(sorry for the poor mic quality)

Final Thoughts

I’ll post here shortly about how to run Redis in production with the init.d scripts and configuration files. One caveat to using solo is if your server has an application that randomly selects ports to use (i.e. VoIP, FTP), it might select one of your worker’s ports. But on a production server, you should have a good feel for which ports are available for locking.

If you want to learn more about Redis, check out their website.

Hopefully this will be helpful for anyone looking to use PHP Workers in an easy, simple way.

Related posts:

1. MySQL, Redis, and a Billion Rows – A Love Story
2. Creating Chatroom / Walls with Redis & PHP
3. Debuging with PHP, Stack Traces, and Redis

PHP

Its been a fun year with the Utah PHP Usergroup. I look forward to it each month, and if you’re in Utah, you definitely should come. We’ve had a lot of interesting talks, such as running your app in different clouds, several MySQL Forks and NoSQL stuff.

I also had a great time going to my first PHP specific conference. While I’ve attended and spoken at other conferences, Tek 11 was a lot of fun. Met a lot of smart and fun people.

As for work, I do almost exclusively PHP work. We power the Dating DNA website and APIs using PHP. All of the Clipish Apps’ APIs are powered by PHP. Alienware Arena, The GeForce StarCraft II Pro/Am Tournament, Thermaltake eSports, and a few other gaming tournaments sites, all powered by PHP.

I’m hoping to blog more about PHP this year. While I use it all over the place, it seems I write more about all the other pieces that interface with and support PHP.

New Technologies

There have been a handful of new technologies gaining traction. We’ve been using Redis in production for quite awhile now, and finding more and more uses for it. I’ve played around quite a bite with node.js, and I see some potential for augmenting PHP with it. MongoDB, CouchDB, and a few other data stores have shown a lot of potential in situations that play to their strength. I’m still a major fan of MySQL, but it’s nice to good options for things that MySQL isn’t as strong at.

New Goals

I’ve been working out, eating well, and such for almost 8 weeks now as well as my wife. The net effect is I’ve lost 16 pounds, and my wife has lost quite a bit too. This was done over the holidays even. We still have some ways to go to get to our goals, but we’re hoping to be successful in 2012 and hit our target goals. We’re even starting to enjoy lifting, cardio, and eating right.

I’m also setting the goal to blog once a week this year. I’ve tried daily, and that is just too much, but once a week is manageable. Hopefully it’ll be relating the technologies I use, to help share the cool stuff we get to use in-house.

Upcoming in 2012

We’re launching a new product in 2012 for Dating DNA, hopefully within a month or so. I’m also hoping to go to a few conferences this year, and perhaps speak at a few of them. Also, on the fun side of things, we’ll be going on a cruise in April, and perhaps a trip to China in September.

I hope everyone had an excellent 2011, and and amazing 2012!

Related posts:

1. My 2009 Technology Recap
2. Attending PHP Tek 2011
3. Goal for 2011: Learn C

Well, this year for Christmas, Daniel wanted to help raise up $250 for… a water buffalo. Yup, thats right, a Water Buffalo.

What in the world for? To donate for a family in a rural. Why? From the Heifer International website:

A water buffalo can lead a hungry family out of poverty and give them a chance for a bright future filled with hope and free from hunger.

In poor Filipino villages, water buffalo from Heifer provide draft power for planting rice and potatoes, milk for protein and manure for fertilizer and fuel. A farmer can plant four times more rice with a buffalo than by hand.

Water buffalo haul heavy loads to the market, where the sale of extra produce brings in vital income for clothing, medicine and school. By renting their buffalo to neighbors, Heifer partner families can earn money for home improvements. And one day, those same neighbors might receive a water buffalo of their own as recipients pass on the gifts of animals and training.

Protein-rich milk, strength to till soil, manure to enrich the land … so many benefits. And, in turn, water buffalo are happy just to graze on coarse grasses and other plants not suitable for harvesting.

So Daniel wants to raise the funds for one!

I need your help.

Well actually I need your help to help me to give someone a Water Buffalo.

Think about it for a second. You can say you gave someone a Water Buffalo for Christmas. You know Water Buffaloes right?

Like this one:

While we were in Cambodia I noticed how much of the work is still done manually. Also in the few “Real China Adventures” we have had I have been amazed at what life is like outside the cities. Don’t get me wrong the people seem happy and content. But I am sure there are places they are not. And I think we should help them.

So I have started a Heifer International Team, Daniel’s Shoe Factory, and I would like you to help me raise $250 to give someone in South East Asia a water buffalo.

Want more information: Click Here

How to help me: Click Here and join or donate.

I am hoping to raise $250 by Jan 1st 2012.

So step up. Unless you yourself are in need of a water buffalo, then no worries.

So lets help Daniel buy a Water Buffalo for a family in need for this Christmas!

Related posts:

1. Bad PR, The Internet, and Censorship – Just Take It
2. Merry Christmas & Happy Holidays

Related posts:

1. Demystifying CSS and WordPress
2. WordPress 2.7 – I Love It!
3. WordPress & OOP

Ultimately, I’m approaching it with a great deal of hesitation. Here is why:

When you start up a Video Game, much like watching a Movie, you will see some credits. In Modern Warfare 2, you’ll see Activision’s name, which is the Publisher, and Infinity Ward, which is the Game Development Studio. While the publisher can be involved in making (or ruining) a game, ultimately its up to the studio to produce a high quality game.

However, the Infinity Ward who made Modern Warfare 2 is not the same Infinity Ward who made Modern Warfare 3. Activision fired the lead game designer and a co-founder of Infinity Ward, and about half the staff resigned soon after that. They are now in a lawsuit against Activision who tried to withhold 60% of the staff’s bonuses.

As a Developer myself, there are two things I think are critical when working on a product, especially a video game: Design Direction and Talented Developers. From what I’ve read on the several news stories about staff leaving Infinity Ward, the majority were Leads and Seniors developers, animators, etc. So I would say the bulk of Infinity Ward’s talent left. Why does this matter?

Call of Duty, Call of Duty 2, Call of Duty 4: Modern Warfare, and Call of Duty: Modern Warfare 2 were are great games. Each added on top of the other, trying new things and bringing noticeable improvements to each installment. Also, while Infinity Ward grew, many of the core team members were the same throughout the series. So you had a great team, with a lot of talent, and experience working together to make great games.

Meanwhile, other Call of Duty games by other studios like Treyarch, released games like Call of Duty: World at War and Call of Duty: Black Ops. While these games weren’t bad games, they weren’t ground breaking either. They felt like minor updates instead of fresh, new material.

So now we have an Infinity Ward with half of their staff brand new, and trying to release a game on a very tight deadline. So I wouldn’t be surprised if there are technical issues at release, especially for the PC. From all the previews I’ve ready about Modern Warfare 3, it sounds like it really is just an updated version of MW2 with new maps and a new game play mode.

So while the videos and images of Call of Duty: Modern Warfare 3 look impressive, and the marketing behind it is great, I’m not holding my breath. But this will be the first time I won’t be purchasing Call of Duty: Modern Warfare 3 for PC, my personal choice for playing FPS games. I’ll likely pickup a copy of it for XBox 360 so I can play with friends, but here is my prediction:

Critics will give it high ratings (9+), actual players will give it okay ratings (7+), but Activision and Infinity Ward will have made their biggest mistake. They will have moved from being innovators to just producers. While they might sell millions of copies of this game, and they might pump out a few more titles, they open themselves to being displaced as the leader in the future. I’m hoping that future games will be able to bring in a breath of fresh air while Activision milks their franchise into the ground.

Who knows, maybe I’ll be proven wrong and Modern Warfare 3 will be great. I just have my doubts, especially when you have an exodus of senior talent from a company. I’ve seen that happen before, and rarely does the company produce the same quality as before.

Related posts:

1. Ghostbusters – The Video Game
2. I’m Boycotting Electronic Arts (EA) Games
3. EA – Lack of Quality & Out of Touch

Siege is an http load testing and benchmarking utility. It was designed to let web developers measure their code under duress, to see how it will stand up to load on the internet. Siege supports basic authentication, cookies, HTTP and HTTPS protocols. It lets its user hit a web server with a configurable number of simulated web browsers. Those browsers place the server “under siege.”

On Linux, you can install it easily via yum or apt. On OS X, you can install it easily via homebrew or macports. I personally prefer homebrew. In your terminal just type:

Now you can stress test a given app by simply using the siege command:

You can read the documentation and faq on their site to learn more. However, there is one question I get a lot when talking about siege: “How can I use siege to test users who are logged in?”

I would always recommend that they send a session header in the config, but today I learned there is an even easier way (although undocumented), thanks to this question in Server Fault. In your siege’s configuration you can add the login-url directive:

Awesome! Now each connection will authenticate before continuing their siege. A very handy tool, especially if your load is generated by user account pages and such.

Related posts:

1. Setting up SSH Key Authentication Between Servers
2. Local LAMP Developement & User Content
3. My PHP User Group Experience