I’ve met and talked with several companies that have had a great success with this approach. Toni Schneider, CEO at Automattic, has written about the 5 reasons why your company should be distributed.

But on the flip side, I’ve met a few developers, managers, and owners who have horror stories of how it “didn’t work out at all.” They talk about how these workers were too distant from their team, how they didn’t communicate well, and struggled to get them to produce.

These are legitimate concerns and challenges face businesses. However, I think there is one key thing businesses can do to help minimize these challenges and make their attempts to bring in a distributed workforce a reality. At to show how it can be done, we’ll use our fictional company, Muppets Software Inc:

Our Example: The Muppets

Muppets Software is a company that produces the finest websites. They have a small office out in California and are doing pretty well. Their team consists of:

Kermit – The Manager

Fozzie – Lead Developer

Rizzo – Junior Developer

Gonzo – Lead Sys-Admin

Miss Piggy – Sales

Things are going very well, but they are getting too busy, and to expand they need to bring on another developer. After a great deal of searching, they find an excellent candidate:

Beaker – Awesome Developer

There is only one problem, he lives in New York and can’t move to California. But he seems perfect, so Kermit decides to try having him be a remote worker. Fozzie and Rizzo think he’ll be good, so they hire him. It starts off pretty well at first, and they assign Beaker to help Fozzie with the new Sesame Street website. They decide to use a ticket system to help better manage what they need to get done, and set off to work.

After a time it seems Beaker would drop the ball and not get something done. It would be 4 PM in California, and Fozzie couldn’t get ahold of Beaker, so he would just have Rizzo do it. The next day Fozzie talked to Beaker, to find out Beaker hadn’t been in the impromtu meeting where this change was requested and didn’t know about it.

As time goes on, Gonzo starts to get annoyed with Beaker. It seems he would always come into the office with a list of things asking to be done. Since company policy says only Gonzo can make changes on production servers, he is suppose to be the only one to do it. Fozzie and Rizzo both have access to the production servers, and they have an unspoken agreement that the developers can make small changes to production servers as long as they stick their head in Gonzo’s cube and run it by him first. Of course, Beaker is unaware of this fact, so he continues to write requests to Gonzo.

Its coming down to crunch time with the Sesame project, and Miss Piggy is worried that they’ll miss their deadline. She keeps asking for last minute changes to make the site better for the client, so several nights she buys pizza for the office and she stays up late with the Developers to work overtime. She even calls Beaker to have him help out, but he is always the one to leave early, and seems the least effective out of everyone.

After the Sesame project, Gonzo, Fozzie, Rizzo and Miss Piggy talk with Kermit about their concerns with Beaker. They say he is a nice guy, but its been hard working with him. They say there isn’t one big incident, but rather it is just an overall vibe.

Kermit calls up Beaker and has a nice chat with him. Beaker says he feels bad the others feel this way, and shares some of his frustrations as well. He has a hard time staying up so late with the team on these late-night pushes, and many times tickets aren’t updated with the information he needs. Kermit sees his point, but doesn’t really know what to do to solve the problems. They’ve talked many times about keeping the ticket system up-to-date, but when things get busy, its so easy for the in-house employees to forget.

What to do?

I’ve seen and heard these problems, and I’ve even been in Beaker’s shoes in the past with a short-term project.

The bottom line is that it boils down to a problem with communication. It is so hard for remote developers to keep up with everything when there is a lot of communication that is happening outside of his reach. He isn’t at lunch with the other co-workers, or around the water cooler. He listens mostly to the meetings since it is so hard to get a word in without being there.

So what would be my recommendation? Turn your entire company into a distributed company.

Wait, what? I’m not suggesting a company immediately close it’s offices. But allow the rest of your employees telecommute as well. I’d say at least 3 out of the 5 days a week.

As a manager, this is a scary thing. But what this does is naturally change the communication channels. So instead of awkward channels of communication forced on the employees to use, these channels become the channels of communication that empower their own work. They get to enjoy the benefits of telecommuting, regardless of where they live, just like Beaker.

When everyone uses the same channels of communication, all of a sudden being 10 miles away or 1000 miles away does not matter. It will become a much more natural process to send emails, update tickets, and chat online. Even when people are in the office, these channels will tend to stay the primary methods of communication.

Almost all of the most successful distributed companies are 100% distributed, not just 10%. I really do think that is the key. Even with Dating DNA, where I lived in Utah and the rest of the team (all 3 of them) lived in San Diego, we were a very distributed team. Our iPhone Dev only met face-to-face with the owner/project manager of the company once or twice a year, even though they lived 15 minutes from each other. Even when we go to expand our team, I like knowing that if we found the perfect match for a new employee, we can hire them regardless of physical boundaries.

Now, this isn’t a silver bullet approach. There is a lot more to making a distributed workforce work. However, I think the key is for this paradigm shift in thought: we’re not just hiring a remote worker, we becoming a distributed company.

Related posts:

1. The Skill of Making Habits
2. Mac Keyboard Quirks – Making Mac more like Windows
3. Lifehacker – Making your Mac sing!

In the past I would just use my iPhone’s email-to-txt email address. However, when I received the txt message, it wasn’t formated very pretty, and it would have a different “From Number.” So if we had a crazy day, I would have 20-30 message threads in my iPhone all about Nagios. I’ve been lazy in the past, and didn’t clear them out until I had about 200 of them, and it was a lot of “swipe, delete, swipe, delete, swipe, delete” to get rid of them.

What I like out this setup is with Twilio, I can buy a phone number for $1 a month. So all my notifications come through the same number. I’m also planning a way to call/text the number and it will read/send the current status and list the hosts that are down.

First thing is to get your Twilio account setup. Its pretty easy, and when I signed up (a long time ago) I got $30 in credit, not sure if they are doing that still for new customers. Either way you can add a credit card and say how much credit you want to add when your account dips below a certain amount.

Now, you can buy a number and set it up. Even if you’re only sending text messages, Twilio wants you to have a “SMS Request URL” before you can send any text messages. So I just put in a URL to my site that does nothing (yet). You’ll need to grab your Account SID and Token.

How it Works

I threw the code up on GitHub. There are a few things you’ll need to do:

First in sendTextMsg.php change your configurations to match your phone number, account SID and token. Here is the entire sendTextMsg.php file:

<?php

require('twilio-php/Services/Twilio.php');

/* Start Configs */

$sid = "A123.....";
$token = "29d6b9f.......";
$twilio_number = '4045550101';

/* End Configs */

# Get the Argvs
$phone  = $argv[1];
$msg    = $argv[2];
$msg    = str_replace('\n', "\n", $msg);

$client = new Services_Twilio($sid, $token);
try
{
    $message = $client->account->sms_messages->create(
        $twilio_number, // From a valid Twilio number
        $phone, // Text this number
        $msg
    );
} catch (Exception $ex)
{
    var_dump($ex);
}

As you can see, it is super simple. All you need to do is put this script somewhere with the Twilio PHP libraries (or just use my github code with all of it in there). Then, in your Nagios configurations add the commands:

Then you need to add it to your contacts:

Simple, easy, and works very well. You can test the script by simply executing the command:

Enjoy!

Related posts:

1. Setting up Nagios for Servers
2. Presentation: Real Life Scaling
3. BestBuy Gives Me $8,770 — Wait, Just $87.00

Why Workers?

Sometimes there are situations when you want to parallel process things. Other times you might have a list of tasks to accomplish, and you don’t want to make the user wait after pressing a button. This is where “Workers” can come in. They are independent scripts that run along side of your application, performing tasks, or “jobs.”

An example is with Dating DNA and our score system. We generate scores between users to show how compatible they are with each other. When a user signs up, or makes a significant change to their profile questionnaire, we need to run a job to query our database, build a list of potential users, and generate scores. This takes 10-20 seconds, and while it is pretty fast, we don’t want to make the user wait for that. So we queue up a job for the user, divide up the work among several workers, and process the work.

General Concept

For this post, we’ll use the example of generating reports. Lets say on your internal website there is a button that you can click and it will email the user a report, and the report takes 2-3 minutes to generate. When the button is clicked, your code will insert the job into the queue. Meanwhile, workers are monitoring the queue. A worker script will pull the job off the queue, process the report, and send the email when its done.

For the queue management, we’ll use Redis. To let PHP read and write data to Redis, we’ll use the PHP Library predis. In our examples we’ll use PHP 5.3, however predis has a PHP 5.2 backport if you are not running 5.3.

Adding Jobs

To add jobs, we’ll need to connect to our Redis server:

/*
 * Connecting to Redis
 */

const REDIS_HOST = '127.0.0.1';
const REDIS_PORT = 6379;

$predis = new Predis\Client(array(
    'scheme' => 'tcp',
    'host'   => REDIS_HOST,
    'port'   => REDIS_PORT,
));

We’ll assume in all of our examples that we’ve done the following above & connected to Redis.

Now, to manage our queues we’ll use the Redis Datatype LIST. Whats awesome about lists is that regardless of size, adding or removing at the start or end of a list is extremely fast. So if your queue has 10 items, or 10,000,000 items, Redis wil be able to push and pop entries quickly.

We’ll have three queues, one for each priority: high, normal, and low. For the Redis key names, we’ll use queue.priority.high, queue.priority.normal, etc. When interacting with lists, you work with the ends, one called right, the other called left. So we’ll add items on the right with the RPUSH (Right Push) command, and we’ll pull items off the left with the BLPOP (Blocking Left Pop) command. We won’t worry about the pulling items just yet.

You store strings as the values for the list. My personal preference is to store JSON objects so you can easily pass variables needed to perform the job.

/*
 * Adding items to the queue
 */

$job = new stdClass();
$job->id = 1;
$job->report = 'general';
$job->email = 'test@example.com';

// Add the job to the high priority queue
$predis->rpush('queue.priority.high', json_encode($job));

// Or, you could add it to the normal or low priority queue.
$predis->rpush('queue.priority.normal', json_encode($job));
$predis->rpush('queue.priority.low', json_encode($job));

Simple enough! Having different queue priorities is very beneficial in managing which jobs should get done first. For example, you might have an Executive’s request go into the high priority queue so they get the report quickly. You might also have a weekly cron that queues up reports to be sent automatically, so those can go in the low priority as to not disrupt people trying to get a manual report.

Now, on to the worker’s code.

Processing Jobs

For now, lets say we have a script running in the PHP CLI (Command Line Interface) that you started by running this command on the server:

First thing is we want this worker to work continuously, so we can do a while loop:

/*
 * Simple Continuous While Loop
 */

// Always True
while(1)
{
	/* ... perform tasks here ...  */
}

We’ll worry about making them more intelligent later. Now, let’s have our worker check the queue. You can do so with the BLPOP command:

/*
 * Checking the Queue
 */
$job = $predis->blpop('queue.priority.high'
						, 'queue.priority.normal'
						, 'queue.priority.low'
						, 10);

What we’re telling PHP to do is to check each queue in order of priority: high, normal, and then low. If it finds an item, it will immediately return an array with the name of the queue it came from, and the string of data that was pulled.

The B in BLPOP is “blocking.” What that means is that Redis will wait until either an item enters one of the queues, or the timeout is reached. In this case, the timeout is 10 seconds. So instead of polling (checking every few seconds in a loop), we check and wait, and after 10 seconds it will return null and we can check again.

What this gives us is near instantaneous queues. As soon as something is available, it is passed to the workers that are listening. You can also have multiple workers, and it will pass jobs to the first listening worker, and the next job to the next worker, so you don’t have to worry about multiple workers getting the same queued item.

After $predis->blpop() returns, if it has an array, it returned an item. If not, the timeout had been reached. We can check to see if a Job was returned, and if so to process the job:

/*
 * Checking to see if a Job was returned
 */

if($job)
{
	// Index 0 of the array holds which queue was returned
	$queue_name = $job[0];
	// Index 1 of the array holds the string value of the job.
	// Since we are passing it JSON, we'll decode it:
	$details = json_decode($job[1]);

	/* ... do job work ... */
}

Now we can have multiple workers listening to the same queues and scale our workload. Redis is very fast & efficient, and you could have hundreds or even thousands of workers listening to a single redis server.

Continuously Running Workers

There are a lot of options when it comes to deploying these workers. You can use a framework like Gearman, but for simple things, I like very simple solutions. I came across a blog post by Joseph Scott about a little 10 line perl script called solo. What it does is it will run a command, and to ensure that no one else is running that same exact command, it will lock a configurable port. This is awesome because the you don’t have to work about lock files or filesystem tricks, the kernel handles it all.

So what you can do is create a cronjob using solo to execute your script. First copy solo somewhere, I put it in my /usr/local/bin on my linux server. Then add this to your cron job using the command “crontab -e -u (which user to use)”:

What this will do is try to run this command every minute. Solo will check to see if the port is already in use, and if it is, it will exit. Otherwise, it will lock the port and then execute the command. The port will stay locked as long as the command is executing. Once the command terminates, the port will unlock.

Now, PHP is a great language, but it has been known to have some memory leaks while running a long time in a single instance. So we can have our scripts exit periodically to be restarted by our cron job. So lets make our “while(1)” statement a little smarter:

/*
 * A Smarter While Statement
 */

// Set the time limit for php to 0 seconds
set_time_limit(0);

/*
 * We'll set our base time, which is one hour (in seconds).
 * Once we have our base time, we'll add anywhere between 0
 * to 10 minutes randomly, so all workers won't quick at the
 * same time.
 */
$time_limit = 60 * 60 * 1; // Minimum of 1 hour
$time_limit += rand(0, 60 * 10); // Adding additional time

// Set the start time
$start_time = time();

// Continue looping as long as we don't go past the time limit
while(time() < $start_time + $time_limit)
{
	/* ... perorm BLPOP command ... */
	/* ... process jobs when received ... */
}

/* ... will quit once the time limit has been reached ... */

One key thing to note is randomly shifting the time limit for the script. I like to do this because you don’t want your workers all stopping and starting at the same time. So if I have 8 workers, one might, but the 7 will continue until the 8th starts back up again via the cron job.

Bells & Whistles

After using workers for awhile, here are a couple of ideas to enhance your workers & system managing them. First off, you can add some monitoring for your queues. Using Redis a HASH, you can use them to store the state of your workers.

/*
 * Assigning Worker IDs & Monitoring
 *
 * Usage: php worker.php 1
 */

// Gets the worker ID from the command line argument
$worker_id = $argv[1];

// Setting the Worker's Status
$predis->hset('worker.status', $worker_id, 'Started');

// Set the last time this worker checked in, use this to
// help determine when scripts die
$predis->hset('worker.status.last_time', $worker_id, time());

Another problem with workers that run for a long time (several hours) is when you make a change to their code, they won’t reload that change until they exit. What I’ve found to successfully restart them is having a “version” number set in Redis that is checked at the end of every loop:

/*
 * Using Versions to Check for Reloads
 */

$version = $predis->get('worker.version'); // i.e. number: 6

while(time() < $start_time + $time_limit)
{
	/* ... check for jobs and process them ... */

	/* ... then, at the very end of the while ... */
	if($predis->get('worker.version') != $version)
	{
		echo "New Version Detected... \n";
		echo "Reloading... \n";
		exit();
	}
}

You would simply INCR (increment) worker.version and after finishing their last job, the worker would exit, and solo would start it up again.

You can also kill specific threads by having them check for their value in a hash:

/*
 * Using Kill Switches to Check for Reloads
 */

while(time() < $start_time + $time_limit)
{
	/* ... check for jobs and process them ... */

	/* ... then, at the very end of the while ... */
	// Check to see if a kill has been set.
	if($predis->hget('worker.kill', $worker_id))
	{
		// Make sure to unset the kill request before exiting, or
		// your worker will just keep restarting.
		$predis->hdel('worker.kill', $worker_id);

		echo "Kill Request Detected... \n";
		echo "Reloading... \n";
		exit();
	}
}

Tweak to Solo & Logging

I made one small tweak in my version of solo, and that was to help it enable logging. Lets say I had three workers in my crontab:

The “>> /tmp/worker.log.1″ tells solo I want to log it’s output to a tmp file that I can tail and monitor their progress. This is great for debugging problems. However, when I did this, solo would write to the tmp file, and not the output from my script. To overcome this I changed the last line of solo:

# old
exec @ARGV;
# new
exec "@ARGV";

This would ensure my script wrote out to the tmp file, and not just solo.

Examples

I’ve created an example on GitHub that you can clone on your own machine. All you will need is PHP 5.3 and Redis installed.

To install redis, simple run these commands on your unix based system:

It will copy the redis binaries to /usr/local/bin.

To get a copy of the code, you can download them here. HOWEVER, it doesn’t include predis! You’ll have to download and copy predis inside there via this link. It is much easier to clone it as so:

Then, using different terminal windows (or using screen), you can run different worker.php instances, use creator.php to insert jobs, and monitor.php to watch the progress. This is all done from the command line.

If you’re using windows, I suggest installed a VM of Ubuntu and using that. If you really want to use Redis on windows, there are some Windows Binaries you can google and download. Good luck!

Here is a video where I demo the example:

(sorry for the poor mic quality)

Final Thoughts

I’ll post here shortly about how to run Redis in production with the init.d scripts and configuration files. One caveat to using solo is if your server has an application that randomly selects ports to use (i.e. VoIP, FTP), it might select one of your worker’s ports. But on a production server, you should have a good feel for which ports are available for locking.

If you want to learn more about Redis, check out their website.

Hopefully this will be helpful for anyone looking to use PHP Workers in an easy, simple way.

Related posts:

1. MySQL, Redis, and a Billion Rows – A Love Story
2. Creating Chatroom / Walls with Redis & PHP
3. Debuging with PHP, Stack Traces, and Redis

Siege is an http load testing and benchmarking utility. It was designed to let web developers measure their code under duress, to see how it will stand up to load on the internet. Siege supports basic authentication, cookies, HTTP and HTTPS protocols. It lets its user hit a web server with a configurable number of simulated web browsers. Those browsers place the server “under siege.”

On Linux, you can install it easily via yum or apt. On OS X, you can install it easily via homebrew or macports. I personally prefer homebrew. In your terminal just type:

Now you can stress test a given app by simply using the siege command:

You can read the documentation and faq on their site to learn more. However, there is one question I get a lot when talking about siege: “How can I use siege to test users who are logged in?”

I would always recommend that they send a session header in the config, but today I learned there is an even easier way (although undocumented), thanks to this question in Server Fault. In your siege’s configuration you can add the login-url directive:

Awesome! Now each connection will authenticate before continuing their siege. A very handy tool, especially if your load is generated by user account pages and such.

Related posts:

1. Setting up SSH Key Authentication Between Servers
2. Local LAMP Developement & User Content
3. My PHP User Group Experience

Update: I was still having problems with using .local for my development sites. After some extensive debugging and research I’ve found out some interesting things. It appears that Multicast DNS (mDNS), which is used in Apple’s Bonjuor for network discovery without a tradition DNS Server, has the Top Level Domain .local reserved for special mDNS functionality. Mac OS X will treat domains ending with .local differently! Use another TLD, like .dev, and it will work correctly.

This article will still help you fix problems with /etc/hosts abnormalities with it not having the correct priority.

Update 2: Thanks for all the comments! A few updates: First off, per the comments, if you declare both a IPv4 and IPv6 address for a .local domain like so:

Then it will work correctly. Second, I ended up doing a fresh install of Lion and this problem has gone away for me. So it appears that this issue would affect some people who upgraded from Snow Leopard. So as Chris suggested below, I don’t think it has to do with the resolve order. I haven’t been able to figure out the difference between Lion installs that work correctly, and those that don’t. So if you are have the following problems, the suggestions below will work to fix it.

This morning I was up a few hours earlier than normal today, and was looking forward to getting in some solid programming hours earlier so I wouldn’t work late tonight. Instead, I spent about four hours fighting a new change (I’d call it a bug) in Mac OS X Lion.

The /etc/hosts File

Now, instead of running a full blown DNS server on my machine, which is overkill and more work to edit and maintain, I would make manual adjustments to my /etc/hosts file. For those not familiar with the /etc/hosts file, it allows you to add entries that traditionally your computer will look up first before trying your server DNS. This is where your computer would set “localhost” is the IP “127.0.0.1″.

I use this a lot with local development. For example, when I work on joind.in, I use MAMP and create a VirtualHost for the code under the server name local.joind.in (i.e. http://local.joind.in/). I don’t have access to the joind.in DNS, so I create an entry in my /etc/hosts file. You have to have admin permissions to edit the file, so I normally run the command “sudo nano /etc/hosts” and make my changes and then save.

The /etc/hosts file looks something like this:

Now, my actual /etc/hosts file contains 83 entries. Some are just short cuts to machines I often connect to. Sometimes during a migration when I’m making DNS entries, I can make the change in my /etc/hosts file as well, so I don’t have to wait for the DNS changes to propagate to test. All in all, it is a very handle little file that I use a great deal. Even windows has their own hosts file that works well.

Problem #1: Bugs with Multiple Domains per Line

With the new Lion Update comes better support for IPv6, however this seems to have broken the /etc/hosts file from allowing multiple domains per line. Before you could do something like this, which is works in several other operating systems:

However, this now seems to break and not work. You must keep each line to one ip and one domain. So your configuration should look like this:

There was also reports about the order of IPv4 and IPv6 entries (blog post about it). I personally did not have this problem, but others seem to have had it. The problem was that OS X Lion seemed to ignore IPv4 entries after a IPv6 entry was made.

Problem #2: DNS Resolution Orders has DNS Servers Before the Hosts File

This is the problem that really, really ticked me off and caused me to waste hours and hours trying to debug and find a work-around. I was working on my local copy of Clipish, and because we have different domains for it that work together, I replace .com with .local. So I would have something like this in my /etc/hosts file:

However, anytime I made a request in Chrome to any of those domains, it would take 5.01 seconds. At first I thought it might be a bug with the new version of MAMP for OS X Lion. But I found using Chrome’s & FireFox’s developer tools that it was taking 5 seconds for the DNS lookup, and 0.01 seconds for the rest. So I would go into my terminal and ping the domain. It worked just fine without the delay.

Apparently, the terminal and some of the BSD Unix tools correctly use /etc/resolv.conf and the correct order of /etc/hosts first and then DNS servers. However, everything else on OS X Lion, including all of your Applications, do it backwards! This wasn’t the case in Tiger, Leopard, or Snow Leopard.

You can test this by running the command “scutil –dns”:

I spent hours and hours googling, finding a bunch of people with this same problem, but no solution on how to fix the ordering. I have my suspicions why Apple made this change, which I’ll address later on. But after I couldn’t figure out a way to change the resolve order myself, I had to find another alternative. (Of course, OS X removed /etc/nsswitch.conf so I can’t change it there, grrr!)

Solution: DNSMasq

The only solution we have, that I could find, is to use your own DNS Server locally on your own machine. I’ve heard of others setting up BIND on OS X. Now, the problem with this solution is that it isn’t nearly as quick and easy to manage DNS entries.

Then, I found out about DNSMasq. It is a lightweight DNS server that is easy to configure. However, it had one key feature we needed: it will read the /etc/hosts file and serve it’s entries. This means we can have our OS check the DNSMasq instance first, which will check the /etc/hosts file, restoring the proper order.

Installing DNSMasq

After some Googling around, I think installing dnsmasq using MacPorts is the easiest. First, you’ll need to install Xcode so your machine can compile with the tools. It is free on the Mac App Store (warning, it is a kind of big download). Then, you’ll need to download and install the MacPorts for Lion. If you already have MacPorts installed, make sure it is the current version of Lion.

Once it is installed, open up the terminal and then execute these commands. First, run selfupdate to make sure your MacPorts repositories are up to date:

Then, install dnsmasq:

That should execute with no problems. Now, it is installed, but we need to have OS X run it:

Now, if you want to make some changes to your dnsmasq configurations, you can do that in this file:

/opt/local/etc/dnsmasq.conf

One thing I like to do is control which DNS Server’s dnsmasq will use. By default it will use /etc/resolv.conf, but OS X will overwrite that file on reboot and anytime you change internet connections (like switching wifi hotspots). So I would copy /etc/resolv.conf to /etc/resolv.dnsmasq.conf. When dealing with files in /etc/ and /opt/ you’ll need to run these commands using sudo.

Then I set dnsmasq to read my new resolve.dnsmasq.conf by setting “resolv-file=/etc/resolv.dnsmasq.conf” in /opt/local/etc/dnsmasq.conf.

Now, we need to configure OS X to add 127.0.0.1 to our DNS Servers. Go to System Preferences > Network > (Select Connection, i.e. Wi-Fi) > Advanced > DNS.

You can have other DNS entries, like Google’s DNS Entries (8.8.8.8, 8.8.4.4). Just make sure 127.0.0.1 is on top.

One thing I haven’t been able to figure out is how to reboot dnsmasq, but if you kill the process in the Activity Monitor OS X will restart it:

This is a lot of work to setup (especially if you don’t have Xcode install). However, it will work, and give you a lot more control over your DNS. So in a way it is an improvement! Just wish it wasn’t necessary.

If you would like to do wildcard dns entries, something you couldn’t do in /etc/hosts, then you can add this to your dnsmasq config:

address=/.local/127.0.0.1

Now any domains ending in .local will route to 127.0.0.1

Why is this necessary Apple?

What I don’t get, is why this change Apple? No other operating system that I know if, including Windows for crying out loud, does it this way. This had to be a deliberate change by Apple. I can only see one of the following reasons:

• Security – Maybe, just maybe, it was for security. Technically, a virus can overwrite the /etc/hosts file if it the virus can prompt the user for their password to run a command as root. So you could maybe have a user visit www.google.com when in reality it isn’t a google server. But even then, HTTPS wouldn’t work without ugly errors. So this might be a reason, but out of all the things that can go wrong with a virus, it is a poor excuse. The virus could just hack mDNSresponder to use /etc/hosts.
• Prevent User Hacks – Most likely, I think this is why Apple made this change. I know with jailbreaking iOS devices and getting hacked apps, iTunes will try to validate an App by “phoning home” to Apple. However, if you change the entries in your /etc/hosts file so iTunes can’t reach the correct Apple Servers. I know people who have circumvented other software activation systems do the same technique. I think this is why Apple made the change, though at the expensive of all the professionals who actually use /etc/hosts for work.
• An Honest Bug – It is possible this was a bug, but seeing as how you would have to deliberately change the resolve orders for the operating system, I doubt this is the case.

If this was an intended change by Apple, it is a frustrating one. One of my favorite things is that underneath the OS X’s nice UI and Applications is a BSD Unix system. I love how I can compile and install so many things just like I would on Ubuntu. But if they continue to change that Unix underbelly and make it less and less like it is everywhere else, then I think they’ll be frustrating many, many professionals.

Hopefully in the future I’ll find a way to change the resolve order in Mac OS X, but until this, this is how I worked around the problem.

Related posts:

1. iPhone SDK 2.1 – Another NDA
2. Local LAMP Developement & User Content
3. Is Apple Blinded by Style?

I was reading The Restaurant at the End of the Universe by Douglas Adams, and I found a humorous example of event driven programming. So I thought I would share.

The Story

Arthur Dent was one of the last surviving humans after Earth had been destroyed to make way for a new hyperspace bypass. He was living on a spaceship known as the “Heart of Gold,” which onboard had an amazing piece of equipment called the “Nutri-Matic Drinks Synthesizer” which “claimed to produce the widest possible range of drinks personally matched to the tastes and metabolism of whoever cared to use it.”

The problem was that “when put to the test, however, it invariably produced a plastic cup filled with a liquid that was almost, but nit quite, entirely unlike tea.” So Arthur tried to reason with the machine with no success. He would ask for tea, and the machine would respond with a cheerful “Share and enjoy” and produce the same sickly liquid.

Finally, after all reason and attempts failed, he tried one last time:

“No,” he said, “look, it’s very, very simple … all I want … is a cup of tea. You are going to make one for me. Keep quiet and listen.”

And he sat. He told the Nutri-Matic about India, he told it about China, he told it about Ceylon. He told it about broad leaves drying in the sun. He told it about silver teapots. He told it about summer afternoons on the lawn. He told it about putting in the milk before the tea so it wouldn’t get scalded. He even told it (briefly) about the history of the East India Company.

“So that’s it, is it?” said the Nutri-Matic when he had finished.

“Yes,” said Arthur, “that is what I want.”

“You want the taste of dried leaves boiled in water?”

“Er, yes. With milk.”

“Squirted out of a cow?”

“Well, in a manner of speaking I suppose …”

“I’m going to need some help with this one,” said the machine tersely. All the cheerful burbling had dropped out of its voice and it now meant business.

“Well, anything I can do,” said Arthur.

“You’ve done quite enough,” the Nutri-Matic informed him.

So everything was grand? Well, not quite. There were two problems.

First, the Nutri-Matic needed help processing all this information. So, it asked the spaceship’s computer to help:

[The Nutri-Matic] summoned up the ship’s computer.

“Hi there!” said the ship’s computer.

The Nutri-Matic explained about tea to the ship’s computer. The computer boggled, linked logic circuits with the Nutri-Matic and together they lapsed into a grim silence.

Arthur watched and waited for a while, but nothing further happened.

He thumped it, but still nothing happened.

Eventually he gave up and wandered up to the bridge.

Now, having the Heart of Gold’s entire computer system help calculate how to generate a cow and grow a tree to produce leaves so Arthur could have his tea isn’t a terrible issue, except the second problem:

A Vogon ship approached and began to attack.

So when the rest of the crew rushed to the bridge, they were perplexed that their entire computer was jammed, and all of their controls, navigation, and weapons were unresponsive. However, the reason was soon discovered:

“What have you done to it, Monkeyman?” he breathed.

“Well,” said Arthur, “nothing in fact. It’s just that I think a short while ago it was trying to work out how to …”

“Yes?”

“Make me some tea.”

“That’s right guys,” the computer sang out suddenly, “just coping with that problem right now, and wow, it’s a biggy. Be with you in a while.” It lapsed back into a silence that was only matched for sheer intensity by the silence of the three people staring at Arthur Dent.

As if to relieve the tension, the Vogons chose that moment to start firing.

Don’t worry, fortunately the occupants of the ship were able to hold a seance, summon the great grandfather of the captain, and have the great grandfather’s spirit save them. (I know, such an awesome book)

Anyway, that is a long example, but it show a point:

Evented I/O

Basically Evented I/O is when your program does some sort of I/O (like querying a database, reading a file, etc) and continues on executing, not waiting (or blocking) for the response. So in our example above, the ship can be seen like a PHP script:

<?php

$ship = new HeartOfGold();
$nutrimatic = new NutriMatic();

$recipe = $ship->processQuery('How to make tea.');
$nutrimatic->makeDrink($recipe);

$enemies = $ship->detectEnemies();
if($enemies)
{
    $ship->evasiveManuvers();
}

The problem is the ship can’t detect enemies until it finishes processing Arther’s request on how to make tea. Now, in Node.js, you have functions that are callbacks. It is a way to say “Hey, when this request is done, do this afterwards.”

var ship = new HeartOfGold();
var nutrimatic = new NutriMatic();

ship.processQuery('How to make tea.', function(recipe){
   nutrimatic.makeDrink(recipe);
});

var enemies = ship.detectEnemies();
if(enemies.length > 0)
{
    ship.evasiveManuvers();
}

Now the function processQuery is non-blocking, meaning it will start the process of learning how to make tea and create a recipe, but continue execution. That way if it detects enemies, it can perform evasiveManuvers().

In a real life example, processQuery could be querying a database with a very slow query. In PHP, you can’t continue executing your code until that result is returned. However, in Node.js, if your SQL library takes advantage of Node.js’s evented abilities, you can send off a query and continue executing, and when the query returns you can do additional logic through your callback.

Hopefully that makes sense. I like finding programming analogies in books, especially with Douglas Adam’s series of Hitchhiker’s Guide to the Galaxy, since they are so fun to read.

Related posts:

1. Node.JS, LAMP, and The Future

Related posts:

1. SoapUI – a simple and raw Java SOAP Client
2. Memcached: Simple, Effective, and Powerful
3. ASP .NET Ajax – Straight forward and simple, the way it should be.

Solution, open up the terminal and run the command:

sudo chown <username> /dev/bpf*

After a reboot, these permissions get reset, so you need to do it after each reset (and hence why I need this every now and then but can never remember the command).

Related posts:

1. XAMPP for Mac – My Frustrations & Solutions
2. Ubuntu Desktop Terminal – Su
3. MySQL Stored Procedure, NAME_CONST, and Character Sets

This decision was driven by a realization: the LAMP stack is dead. In the two decades since its birth, there have been fundamental shifts in the web’s make-up of content, protocols, servers, and clients. Together, these mark three ages of the web:

I. 1991-1999: The HTML Age.

The HTML Age was about documents, true to Tim Berners-Lee’s original vision of a “big, virtual documentation system in the sky.” The web was dominated by static, hand-coded files, which web clients crudely formatted (with defaults that offend even the mildest of typographiles). Static documents were served to static clients.

II. 2000-2009: The LAMP Age.

The LAMP Age was about databases. Rather than documents, the dominant web stacks were LAMP or LAMP-like. Whether CGI, PHP, Ruby on Rails, or Django, the dominant pattern was populating an HTML template with database values. Content was dynamic server-side, but still static client-side.

III. 2010-??: The Javascript Age.

The Javascript age is about event streams. Modern web pages are not pages, they are event-driven applications through which information moves. The core content vessel of the web — the document object model — still exists, but not as HTML markup. The DOM is an in-memory, efficiently-encoded data structure generated by Javascript.

LAMP architectures are dead because few web applications want to ship full payloads of markup to the client in response to a small event; they want to update just a fragment of the DOM, using Javascript. AJAX achieved this, but when your server-side LAMP templates are 10% HTML and 90% Javascript, it’s clear that you’re doing it wrong.

Claiming LAMP is Dead is Reaching

It goes on to explain how LAMP like architectures dead. I would think that is a very overzealous point of view, and to perhaps put it in some perspective:

During each of these “Ages” new technology was developed to solve problems. HTML was created to create inter-linking information. As the demand grew and became more difficult to manage, LAMP like tools like PHP, Django, Ruby on Rails, and others were developed to help make delivering this information easier. It wasn’t so much replacing the previous technology, but adding on to it. Sometimes you will replace older technology with newer ones, but we’re still using HTML & CSS, and augment it when needed.

But LAMP-like systems are deployed in production all over the place, powering some of the most visited websites on the net. While there are some people who are using Node.JS, like Yammer, but Node.JS isn’t serving nearly the volume of content like PHP, Ruby, Python, or event .NET.

What is Node.JS?

Basically, it is a lightweight framework wrapped around Google’s V8 JavaScript Engine. V8 is the JavaScript engine that powers Google Chrome, their open source Web Browser. Another part of Node.JS is that it is event based, which can make is very quick and efficient.

I’ve been tinkering with Node.JS for a couple of weeks, and it is very interesting and has a lot of potential. It’s unique scoping with JavaScript and event-based methodology makes certain things easy to accomplish that are more difficult in other traditional tools. There are some pretty cool libraries for Node.JS like Socket.IO, a library to support multiple transports using feature detection. So if your browser supports WebSocket or Adobe Flash Socket, it will use that, or AJAX long polling and multipart streaming, or even the Forever Iframe technique. So your application will use which ever type of communication is most efficient.

A Little Bit of a Reality Check

But lets be honest for a moment, Node.JS was created in 2010. It is still very, very young. In the video on the front page of the Node.JS is a video of it’s creator presenting to a PHP Users Group about Node, and even cautions using it in production. It still has a long way to go to maturing as a tool. Granted, being based off the V8 Engine brings a lot of maturity to the project.

As for replacing technologies like PHP, Ruby, and Django, I’m extremely skeptical that they are “dead.” Instead, I see Node.JS augmenting our existing technologies. Like how I’ve implemented Redis as a data store where MySQL was poorly suited, I see Node.JS functioning as a tool for real-time communication for the future of websites.

I hope to write more about Node.JS, and look forward to using it to solve new and unique problems. But I won’t be rewriting all my websites in Node.JS, it would be a nightmare. So like with NoSQL, and Cloud Computing, yes it is a new tool, but it won’t radically remove everything else up to this point. Like with Redis, I didn’t get rid of MySQL, I just use Redis when it is a better choice.

Related posts:

1. Local LAMP Developement & User Content
2. List of 50 PHP Tools
3. jQuery Tip: Better Toggle

It basically is a system of managing a tournament bracket. Here is an example:

Example Tournament Brackets

It basically is a PHP script that pulls from a database with key-value pairs. In a template file, in which you would create the bracket’s HTML, you would insert tags with an id. Then, depending on if you are viewing or editing it, it with either insert the value, or a input box used for changing the value.

There was a basic admin listing of the brackets that are active, as well as access to archived brackets:

Admin Panel

You could add a new Bracket and select a template to use. When you select a template, it will show a preview of the template below.

Create a Bracket

Once you create a bracket, you can edit it. It isn’t pretty, but it gets the job done:

Editing a Bracket

Here is an example of the finished product that I just created. If you want a better example, here is the recent League of Legends tournament bracket. While it isn’t pretty, it works. Some of the biggest problems with it is the editing and managing of the templates. You can only use some sort of WYSIWYG editor to be able to make sense of them, since they are a giant table. What that ends up leading to is template files with 2,000+ lines. It still amazes me this chunk of PHP code has been used over 1,000 times for different tournaments around the world.

One day we are going to rewrite the system to allow the end user to make their own templates, and more interaction & connectivity between the bracket and CTE (CEVO Tournament Engine). But until, my first real PHP project can continue to chug along just fine.

Related posts:

1. WhySecurity.org – A New Mini-Project
2. Project Management & Entrepreneurship
3. Web Development 10-Years Ago & Now