Hopefully at some point, your website gets a lot of traffic. Yay, you’ve reached your goal of getting good traffic, but it is soon followed by issues with performance and load. I like to call these the growing pains of a website. So as a web developer, I suddenly have the epiphany of “Hey, I need to scale my website!” What follows next is the biggest mistake a web developer can make:

They start looking at articles on how Google scales, or maybe how Facebook manages all of their traffic.

This is a mistake! To be brutally honest, you are not Google. You are not Facebook. You are not Twitter. You are a website that receives less than 0.000001% of the traffic that some major websites receive.

Why is this dangerous for web developers to do? Google, Twitter, Facebook, and others like them are solving complicated at a very large scale. I remember a presentation by a Twitter engineer who developed a program for a unique ID generator that can generate millions of IDs per second. The probability of you needing this type of solution is about the same as being struck by lightening. Applying these same practices at a much smaller scale are not realistic. If a locally owned grocery store wanted to open a second store, they would not adopt the same practices that Wal-mart use to manage their 8970 stores.

A Little Reality Check

I’m sure that most of my readers know of StackExchange.com. They power the popular website StackOverflow and several others. They have about two million visitors per day. That is a lot of traffic. StackOverflow is ranked #123 on Alexa. So you would imagine that they have a very large infrastructure serving all of this traffic?

Earlier this year, Stack Exchange wrote an article about their production environment. I was surprised on what exactly they were using. In paticular, the number of Production Servers*:

• 12 Web Servers (Windows Server 2008 R2)
• 2 Database Servers (Windows Server 2008 R2 and SQL Server 2008 R2)
• 2 Load Balancers (Ubuntu Server and HAProxy)
• 2 Caching Servers (Redis on CentOS)
• 1 Router / Firewall (Ubuntu Server)
• 3 DNS Servers (Bind on CentOS)

That is 22 servers for 2 Million Visits per day, serving 800 HTTP requests per second. Now, StackExchange did clarify that they did have other servers for management and fail over, but 22 servers handle their production load. This is a website that is ranked the 123rd most visited website in the world.

Honestly, most websites could be run on half a dozen servers if designed and configured correctly, including redundancy. Some really busy websites could run off a dozen servers. Unless you’re in the top 5,000 websites on the web, you really shouldn’t be worried about large-scale techniques.

So when you’re website is starting to grow, and you leave small scale, you’ll enter the phase of “Middle-Scale.”

What is Middle-Scale?

Middle-Scale is like being an awkward teenager:

You know that you can’t be the only one suffering through this, but you’re unsure how to proceed. It feels like you’re missing missing out on things everyone else must already know, but aren’t talking about. Like everyone else are awesome vampires or something:

But the reality is this: they don’t have some awesome secret! They are just normal teenagers.

This same idea applies to Middle-Scale websites.

Middle-Scale is when the most important things are still the best practices. Only now when you deviate from them you can feel those consequences. When you only had 100 users, a couple of nested queries and missing indexes didn’t cause that much of a problem. Your database is powerful enough to hide the inefficiencies. However, when you get to 10,000 users, your database can no longer hide the inefficiencies.

Middle-Scale is when simply separating your web server and database server isn’t enough. You’ll probably need to add some sort of cache like memcached. You’ll need to start tweaking your MySQL, Apache, and PHP configurations.

Then, after you’ve ironed out your inefficiencies, you’ll start to use multiple servers. You’ll probably add a Load Balancer with multiple web servers. After that, you’ll probably have some sort of Master-Slave replication for your Database for backups and fail-overs.

You start to leave this “Middle-Scale” classification when you move to multiple data centers, and start to do some load balancing at the the DNS layer. This is when you’ll start to have a dedicated sys-admin team.

Okay, I’m Middle-Scale! So what should I do? Where do I look?

First off, you must adhere to best practices. If you are working with PHP, research PHP performance and best practices. Do they same for each of your technologies, like Apache and MySQL. You will need to stop treating your application as one big app, and start to understand all of it’s moving parts.

Second, you must understand your specific problems. Scaling ins’t a problem, nor is it a solution. It is a generic term for many different types of solutions. Without understanding why your website is running slow, or why it cannot handle the load, you will not be able to create an effective solution.

So you don’t have a scaling problem. You have a MySQL performance issue, or a Apache problem, or a PHP problem. Most likely, it is something extremely specific. You have a high volume of MySQL write operations (i.e. UPDATE, INSERT, DELETE, REPLACE), or perhaps you are missing some indexes and have too many full table scans.

Third, Googling for help will only get you so far. You are starting to enter a phase when it is harder and harder to find answers to your broad issues. Talking with other experienced people who have gone through the Middle-Scale pains before will help immensely. I cannot recommend highly enough going to User groups. Being able to communicate with someone, either face to face, on the Phone, over IRC, etc. is invaluable. While I’ve learned a lot at conference and usergroup presentations, I’ve learned even more by just talking with the people attending and at the social gatherings.

Profile & Performance will Naturally Lead to Scaling

When you want to scale, it can feel like a very daunting task. It seems like this big unknown complicated solution. What in the world am I going to do? I remember feeling these worries when I first started to investigate load balancing and sharding for some websites I was working on.

The thing is, if you start to profile your application, you will discover it’s inefficiencies. I remember when I spent a sold week, working 12-16 hours a day profiling and optimizing Dating DNA’s database. I found a lot of bad queries, and I was able to cut our load times from 2-5 seconds to under 0.1 seconds. The CPU on the database server went from 80-90% CPU utilization to under 10%. It was incredible, and then I promptly took the entire next week off. When we migrated to new servers, I was able to move to less powerful database server and still have the same great performance. So by profiling and optimizing our database, I didn’t need to worry about spinning up multiple master databases and sharding our data.

With Clipish, we faced almost opposite scaling problems. The database was rarely an issue, but our web server CPU’s were. We do a lot of ImageMagick manipulations of images, and at high volumes on virtual servers this can be a big issue. So over the last year we’ve introduced some load balancing and CDN tools to help serve all 10 TB of bandwidth for Clipish.

The thing is, when you start to profile your application, you start to understand it’s low areas better, so you have a much better idea on what do to. Even if you don’t know your solution, it is much easier to find a solution with a sound understanding of the problem. For example “scaling mysql” yields much less helpful results than “mysql full table scans” in Google.

So should I ignore what Facebook and Google do for scaling?

Of course not! First off, they do cool stuff. Just because I’ll watch NASA launch a space shuttle doesn’t mean I’ll try to make a rocket system for my broken lawn mower. But you have to put what they are doing into context. People from large websites have published several good “best practices” articles on techniques that help any website. Especially things on the client/browser side of things. Just use caution. I cringe when I hear someone say “we’re trying to use Cassandra to solve XYZ problem at work” when it is severe overkill.

Final Thoughts

Most of the time when I talk about performance and scaling with other people, it is when they are in “critical mode.” Their website is down, slow, unusable, etc, and they are looking to fix the problem. I will say, it is much more difficult to profile in “critical mode” than profiling before hand. The reason is you are much more desperately focused on getting it working again instead of understanding the problem.

I’ll be giving a presentation this Thursday at UPHPU on Profiling PHP Applications. I’ll post the slides, and most likely write some articles on the subject afterwards. As always, feel free to email me or leave a comment.

Related posts:

1. Data Backups – There Are No Excuses
2. Speaking: Utah Open Source Conference 2009
3. Presentation: Real Life Scaling

The Problem

With Dating DNA, our goal was to display a compatibility score with every other user. This score is generated by taking two sets of answers to our 20 page survey, and running it through our algorithm. While it is super convenient for our users, this poses a problem. We wanted not only for people to be able to visit a profile and see a score, which is easy to generate a score on demand. We wanted our users to be able to browse other profiles sorted by their score with them. This requires us to pre-generate and store these scores, and then later query them.

So Ultimately, in theory, our “scores” data scaled at the following rate, with X as the number of registered users:

X^2 – X

That is at an exponential rate, which is practically impossible to scale at. The very first version of Dating DNA (before I took over the project) had about 1,500 users. The scores were stored in a single table. Every night, a “cron job” would run and get a list of every user, and loop through every possible iteration and re-generate each score. At 1,500 users that was 2,248,500 records. That is a lot for just 1,500 users. With our current user count, we would roughly have 359,999,400,000 score records. Thats 359 Billion records if you don’t want to count the commas.

This old system of daily cron jobs broke at about 2,000 users. We would have problems with the cron job taking over 48 hours to complete, and would end up with 3 scripts running at the say time. One for today, one for yesterday, and one for the day before that.

Smart Logic & Threading

We solved our first problem by using some common sense and smart logic. I won’t detail the lengthy measures we go through, but we can basically boil down our entire user base to an estimated top 5000 matches for any given user. If we have a heterosexual man named Joe, he doesn’t care about the hundreds of thousands of other heterosexual men who he scores a 2 or 3 with, but the heterosexual women he scores above a 6 with. So we don’t store the score for Frank, Jimmy, and Alan with Joe, but Sally, Rachael, and Tiffany.

The second part we solved was pre-generating scores for a user. After a User has reached a point in the survey where we have all the information we need to generate scores, and they are just filling in some miscellaneous, we put them in a queue. We then have a server process than is continuously running checking this queue, and spinning off multiple generation “threads” that crunch the data and store the score. We’ve spent a lot of time perfecting this system. Currently we typically can generate any given user’s matches in roughly 5 to 20 seconds, depending on how busy our website is.

Storing The Score in MySQL

The problem we now faced was the write through put of MySQL. Even through sharding and partitioning, we wanted to have a goal of sustaining 1,000 registrations per minute in a scalable and high performance manner. Which comes down to about 83,000 records per second that are either being inserted or updated. We then needed to be able to retrieve large volumes of scores just as fast.

I believe we could have bent MySQL to our will and got it to work, but it would be at a high cost of server power, and that cost wouldn’t scale well with our revenue stream. After we moved from the MySQL storage of the scores, I ran a query to see how many scores we were indeed storing. The final total was 950,363,992. Just 50 Million shy of one billion. It took 1 hour 49 min 38.27 sec to calculate that count. It is evidence that even though MySQL wasn’t the best choice for storing this data, it did it pretty well considering this single table was holding 90 times more data than any other table.

Picking Another Solution

In 2009 we started to throw around ideas for a new scoring system. I cannot stress enough when talking to others about “NoSQL” solutions the best solution for any given job is based on your data’s characteristics. User registration data needs to be treated differently than activity logging and basic stats. It might be okay to lose a few minutes of activity logging (depending on the app), but you definitely don’t want to be losing user accounts.

With Dating DNA’s scores, we had one great advantage. The data could be somewhat volatile, because we can always re-generate a set of matches for any given users. Of course, we didn’t want to lose all of it, because having to regenerate everyone’s scores is a major pain and extremely resource intensive. But if we lost a few minutes, anything lost could easily be regenerated. So when we started research for a solution, we were willing to sacrifice some persistance for performance. We wouldn’t be doing the same for our user registration data.

At first, I was contemplating building a completely in-house project to handle the data storage and retrieval of the scores. It would be a lot of work, and decided against. So I then thought about hacking together a custom solution with memcached. The idea would be a user’s set of matches would be stores in a variable in memcached. So the website and generation scripts would interface with memcached, and a server process would write inactive sets of scores (people who weren’t logged in) to a file on the disk. When they logged it and scores were being pulled and stored for that user, it would load the data from disk into memcached.

While the general concept was sound, the actual execution would be difficult. Memcached only supported strings for values, and we would still need some sort of database to manage which users had the data in memory vs disk, and the server process (probably just a php, python, or node.js script running continuously) would have to be running constantly, and if that broke things could get messy.

It boiled down too many points of failure and complexity. But it was a step in the right direction, so we kept looking for a better solution.

Redis, the Advanced Key-Store

I was talking with Joseph Scott, an employee of Automattic as a Bug Exorcist (not joking, his real title), and he mentioned I should look into Redis. He gave me a brief overview of what it was, and I shuffled that info back in my brain. I can’t remember how much longer it was before I checked out and compiled a copy of Redis, but I quickly discovered it could be a viable storage system for our scores.

So I spun up a virtual machine, installed Redis, and started to pound away at it. One of the things I wanted to test was the new feature (at the time) of Virtual Memory for redis. What this allowed was for Redis to make it’s own Virtual Memory on the server and store the lest recently used data to disk. When a Redis object was retrieved and it was in the VM, it would swap it back into memory, and swap older data to disk. This was just like the idea I had before with using an archaic system with memcached, but much more elegant.

The second thing was Redis’s support for multiple data types. So instead of having a json encoded string that held the scores and user ids for another user, we could have a hashtable or even a sorted set. It was a much more elegant solution than what we were thinking of before.

Some Limitations to Redis

However, there were a few limitations that we faced when implementing Redis. Redis works flawlessly with smaller sets of data. But the larger your data set, the more careful and aware you need to be about a few things that will kill your redis instance.

First off, with memcached, if you set a memory limit, it is a hard limit. I’ve never seen memcached use more memory than what you allow it to use. Redis, on the other hand, has soft memory limits. This is because of the way the Background Saves work so you can have persistant data. When a Background Save is issued, Redis will fork itself, and have one thread save a snapshot to the disk, and the other thread will continue to operate. In order to do this, Redis will exceed the standard memory limits, and your memory usage will go up much quicker. Once the background save is complete, it will close the forked backup process and sync the memory back to one data set. (I’m not a computer science guy, nor do I know a lot of lower level programming, so I might be describing this not 100% accurate, but this is how I envision it in my head).

Now, if you are not using the Virtual Memory, this isn’t that bad. However, when using Virtual Memory, the Background Saves take a great deal longer (from seconds to almost a minute or so), which isn’t too bad, but there is a catch. You will not be able to swap to the VM Disk until after the BG Save. This means that all Redis objects swaped to memory will stay in memory until after the BG Save is complete. This is because, like the memory from the fork, the Virtual Memory file is being used for the BG Save instead of the process handling requests.

So the one limitation we’ve encountered is we cannot run scripts that “query” large amounts of data from Redis. For example, it would be very simple to get a listing of users using the KEYS command, and then loop through the values using a HLEN to read the length. This will cause you to swap from and to the virtual memory a great deal. If a Background Save is occuring, Redis will not swap to disk until the BG Save is complete. This means if you have 10 GB of data in Virtual Memory, and you have a 1 GB instance of Redis, you will suddenly be reading gigs of data into Redis’s memory. If you are on a 2GB machine, you can easily use up all the memory on the server and then start using the System’s Swap.

Once you start using the Operating System’s Virtual Memory, it is game over. Your Redis instance’s performance will tank, and your Background Save might not finish, and you will need to restart redis.

There are some ways to give Redis a “Hard” limit on memory, but we opted to configure our servers in a way that doesn’t require this. If Redis hits the memory limit, it can start throwing write operation errors, which we didn’t want.

How We Configure Redis

After much trial and error and testing internally, we believe we found a sweet spot. We deploy a redis server, and spin up three redis instances on a different port each. Each is configured with 4 GB of Virtual Memory using 4096 size pages, and only 256 MB of “memory” using the vm-max-memory setting. While you would think this would mean a hard limit, it is a soft limit, and more of a goal “we’ll try to only use 256 MB of memory to store the data, if we’ll exceed it if needed.” Given our patterns of usage, Redis’s actually usage fluctuates (based on ps aux’s reporting) between 640 MB to 1100 MB of RAM, depending on if a background save is being executed or not. Redis is configured to perform background saves every 10 minutes, which take about a minute to perform.

So between the other admin services running on the box, and the three redis instances, we use just over 3GB RAM:

The amount of CPU required is extremely low, and almost 100% from writing the background saves to the disk:

So what do we get in return? We estimate each instance with 256 MB of data can hold roughly 2,000 active users. So with a single server we can support 6,000 users online at any given moment. We can store the scores for roughly 360,000 users on a single 4 GB box, which is about 1.8 Billion scores. Then, if we need more, we just provision another box, and our system will start assigning users to the instances on that machine.

Because of our ability to re-generate the scores, we decided to only convert the users who had logged in the past three months to the new system. If a user who hadn’t logged in since then logged in again, it would in the background assign them to a redis instance and rebuild their matches for them.

Using Redis with PHP

I recommend currently to use the PHP Redis client predis. I’ve used others like Rediska, but I prefer the straight forward approach of predis.

A high level view of how we use Redis with our PHP based website is we have a class called RedisManager than manages pretty much all the connections to Redis. It supports lazy connections (which is important to us, since we don’t want to have to connect to every instance of Redis we have), and I hope to open source it some day soon.

One key performance trick we’ve noticed is to use Pipelining to the redis instance. We don’t use this so much on the website, but our score generation “threads.” Writing thousands of scores one by one eats up a lot of network overhead versus sending them in batches (we send in batched of 500 or 1000, depending on the situation). Using pipelining is extremely fast for us, and I highly recommend it for any large batch of commands.

Using Redis Elsewhere in Dating DNA

Now, it might seem that we’ve put a lot of thought and effort into using Redis, and I want to make sure it was understood that Redis itself wasn’t difficult to use, but the volume of data were were dealing with. On Dating DNA, we also use Redis to power out in-app chat system (which I’ve written about previously), and it works great and is currently only using 146.70 MB of RAM, and serves thousands of requests per second.

The Future

I still have a lot of great ideas for Redis and Dating DNA, both with the score system, and outside of it. I plan on writing several reporting tools for Redis and hope to share them on github. I am currently working on the code and scripts for automatic deployment for Redis servers for Dating DNA, so we can scale easily with the push of a button. I’m excited for the work that is being done on Redis, and highly recommend it to anyone.

If there are details you would like to know more about, leave a comment and I’ll try to answer them. If you see me at tek11, feel free to ask me about this, and I can show you in detail how it works (internet permitting).

Related posts:

1. MySQL & PHP – SQL_CALC_FOUND_ROWS – An easy way to get the total number of rows regardless of LIMIT
2. MySQL, 40 Million Rows, MyISAM to InnoDB, 45 Minutes
3. Restoring Large MySQL Dump – 900 Million Rows

Today some memories came from when I first started with web development. So before I forget anymore of the details, I thought I would share my unique way I became a web developer. I never went to school to learn Computer Science or Information Systems. I’ve had a total of 2 programming classes, which were my in sophomore and junior years of High School, one of which we lovingly nicknamed Warcraft 101, because we would spend the entire time getting out butts kicked in Warcraft 2 by my classmate Daren (We all got A’s, we had just finished the entire coursework in 2 and a half weeks).

The reason I share this is perhaps others who hope to become a web developer can learn some of the valuable lessons I picked up along the way. That, and that my close programming friends and colleagues can get a kick out of my humble roots.

Heritage

How I ended up in Technology to begin with is probably because part of it is in my blood. My Father, Kevin Carmony, has been the owner of several technology businesses. So I was exposed frequently to computers and other technologies my whole life. I remember some time when I was about five or six, in the late 1980s, playing against my younger brother on a multiplayer ASCII LAN game at the old Streamlined Information Systems office. It was wandering around a 2D maze hunting these 8-bit monsters, and each other, and it was amazing fun.

I also had access to a lot of expensive software that were hand-me-downs. One of them I remember is Photoshop 3 and 4, spending hours trying to design websites with it. To give you an idea how old that is, the current version is CS5 (aka version 12). The first program I really used to make a website was (and get ready to gasp) Microsoft FrontPage in 1997. There were also a couple of website books laying around that I would read, or at least try to.

On the other side of the equation was my mother, while herself wasn’t extremely technical, encouraged and “sponsored” my learning when I was young. By sponsoring I mean she many times bought computers, digital piano keyboards, DSL (we were some of the very first people to have “high” speed internet in our town), and paid for many other expenses. I remember a several hundred dollar long distance bill because I would call the “Provo” dial-up connection instead of the broken “Ogden” phone number. I even once ordered a “temporary” AOL dial-up account, with her credit card and without permission, because our ISP was down, and I needed to check my website. I forgot to cancel it afterward, and it racked up a few months of fees before my mother realized what had happened. She never yelled or mad over these “expenses”, that I realize now as an adult, were not cheap. She just told me in the future what to do to avoid causing them again.

She was also very patient with a son whose grades weren’t the best, and who would rather work on a website than do his homework. I know if she would have came down hard on me, my GPA would probably be higher (its wasn’t bad, just not great), but I wouldn’t have learned what I did, nor make the connections that eventually fast tracked my career. I owe her a great deal (I love you mom!)

Humble Beginnings

I first learned about websites, and how I could make them, while sitting in my 7th Grade Band Class. My fellow clarinet player Kenny Cottrell explained to me in between songs about HTML, Notepad, and how I could learn to make my own website. So I set off to learn HTML. In my excitement, I found an online book on HTML, and printed the entire 400 page book on my mother’s home printer. Single page, of course, because I couldn’t figure out how to easily do double page, and it was on her expensive laser printer (this was 1997) with expensive cartridges.

My very first website was hosted on one of my father’s web servers, and using FrontPage, I made a website about a game called The Realm, one of the very first graphical Massive Multiplayer Online games (MMO, think World of Warcraft). It was bad, really bad. I can’t find any pictures or old files from it, but you can take my word for it. When spending weekends at my Grandma’s house with my Dad, I would spend hours designing and writing websites, or at least try to. They were all bad, but I learned a lot by trying over and over again. I would design a site, look a a professional site, and try to see why mine stunk while the professional ones were so much nicer. I remember looking at Amazon‘s rounded corner tabs for hours trying to get mine to look just as nice.

I then got my first domain, RPGLegacy.com, in 1998 and started a website with game reviews and walkthroughs for PlayStation RPG games. I remember writing reviews and information for games like Final Fantasy VII, Suikoden I & II, and even more obscure (and terrible) titles like The Granstream Saga. I started to get perhaps 100 visitors a month, and I thought that was great. I even got emails from people asking me for help. One subject in particular was in Breath of Fire 3 dozens had emailed me about getting stuck in a castle. I myself had gotten stuck in the same place for hours, and posted an in-depth solution for finding Honey the Robot in the castle. For being a 15 year old kid in junior high, it was a lot of fun, and I learned a lot. I had moved to Dreamweaver for making the website, and the designs weren’t half bad. However, I don’t have a screenshot of how the old site used to look.

Counter-Hack

I was 16 and I in high school. My classmate Daren (the same who whooped me at Warcraft 2) introduced me to a game called Counter-Strike. It was a First Person Shooter, and it was a lot of fun. However, at the same time, there were a lot of these “cheats” and “hacks” programs that were coming out. My friends and I would be accused of “cheating” and “hacking” when in reality we weren’t. Also, some of the claims on how we were cheating were rather absurd. So I started to investigate the truth and fiction behind these programs. I learned a lot, and thought I would share that knowledge, so I started Counter-Hack (http://www.counter-hack.net) in the summer of 2001.

Little did I know how extremely popular this website would become. Within a week or two I had a website wtih decent content up using Dreamweaver. This was 100% HTML based, no CSS, trying to use Dreamweaver’s Template system. What I would have given for some PHP or even WordPress. While with RPGLegacy, I had maybe a max of 5,000 visitors over it’s entire lifespan, within a month I had 30,000 visitors to Counter-Hack. Soon I was getting thousands of visitors per day. About the same time I met Anthony Ouwehand (nicknamed H3X), who had ran another popular website about video game hacks. He graciously helped me with his PHP/MySQL skills. He designed, and developed, the Counter-Hack website that was launched in 2002 and ran until 2008 when everyone involved with the project just were too busy, and the site had ran it’s course and purpose. Two years of that time the project ran with the rest of the volunteers while I served an LDS Mission in Torreon Mexico.

During the years with Counter-Hack, a few highlights were interviews with Wired for news stories, working with Valve Software to help recover the HL2 Leak, and an interview and article with Rolling Stone Magazine. During it’s height, Counter-Hack was covering dozens of games with hundreds of thousands of visitors a month. During it’s later years, Counter-Hack implemented a Wiki system for much of it’s content, something that was pretty new at the time. All and all, it was a great experience with dozens of volunteers and great memories. For a hobby during High School and the year after graduating, I couldn’t have asked for a better experience. I learned a lot before even starting college.

Developing Out of Necessity for CEVO

The reason why for the background story is for two reasons: I had started practicing web design and development from a very early time, and I gained a unique background and knowledge with my work with Counter-Hack. So in March 2005, being home only a few weeks from Mexico and still with a heavy mexican accent, I was approached to work with a company called CEVO. It was an online video game league that was emerging, and their first game was Counter-Strike. They needed someone to help consult with preventing cheating and “hacking” in their matches, and I was a perfect fit. So I joined as a consultant, and quickly became the Operations Director within a few months. CEVO also had brought on Eric Ping to be the CTO, and the company started to grow.

One of the challenges CEVO faced was it was a completely distributed team. Charlie Plitt, the owner, lived in Baltimore, MD, Eric lived in Ephrata, WS, and I lived in Ogden, UT. We had staff and volunteers that literally lived in all 50 states except Hawaii and Alaska, as well as some across Canada. Our customers also lived all over North America, and we relied on the Website, Email, and VoIP to run the entire company. It was insane and awesome at the same time. This put a ton of demand on Eric, our only developer for the website, to get new features implemented quickly. During the first two years, we had re-designed the website a total of 6 times before finally settling down on the current design, which is being redesigned now as well. Eric couldn’t handle it all, and since we didn’t have the funds to hire another developer, I thought since I knew something about web design and programming, I could help out.

Eric took me under his wing and basically mentored me along as I started to take on project after project. I had become familiar with PHP and MySQL when I was tinkering with Counter-Hack’s code, but now I was really learning. I was also learning extremely quickly because I had no other option. It wasn’t like working at another company where I had a team of Senior Developers that could bail me out. Eric worked such late hours that normally if I was awake, he was asleep, and we meet for a few hours in the afternoon and evenings. But if I had a bug, or a problem, there was only one person who could solve the problems: myself. Working on a team of two developers for a start-up company is extremely demanding, and I was constantly looking for ways to work more efficiently. It was the only way to meet CEVO’s growing demands, by learning how to be a quicker and better developer. So while I had a day-job of doing Tech Support for a local company, I spent every other free moment working for CEVO. As CEVO grew, we ran into scaling problems and performance problems, things a normal “Jr. Developer” wouldn’t have hands-on experience dealing with.

Eventually, I had learned so much working for CEVO, that I was able to quit my day-job doing tech support, and did contract work on the side. I helped launch some e-commerce websites, some basic business websites, and spent the rest of the time working on CEVO. I had started to go back to school for my Business Administration degree at WSU, and I was pretty much busy non-stop. But it was a lot of fun, and allowed me a lot of freedom not having a normal 9 to 5 job.

Ambient Partners, LLC

In 2006, I had the most amazing thing happen to me: I met my wife. We had dated for almost a year and we decided to get married. So by the end of 2006, I had a serious realization: I needed a steady income. While contract work paid really well, I could go a month or two without a check from clients. When living at home with relatively no expenses with the exception of my car, I could get away with this. But health insurance, rent, groceries, etc. I couldn’t live that way. So I decided to find a full-time programming job. Fortunately, the company I had worked tech support at had split with their California office and changed names. They were a company primarily doing software for DVD Rental Kiosks, and needed a Web Developer to do work on their web technologies. So by January 1st, 2007, I became a full-time employee for Ambient Partners. Our development team consisted of myself, a Senior Developer, and the CTO. As I look back at what the three of us accomplished as a development team, it amazes me.

When I joined, our main client had about 100 kiosks in the field. After doing the company website, I was given a very big project: RBO, Rent-Buy-Online. We wanted to provide clients with “white-label” solution to allow people to reserve and buy DVDs on a website, and then go pick them up at a Kiosk. There were two parts to this solution. The website itself, and the web services to supply the website with data. I had never written web services before, so I had a lot to learn very quickly. Also, it was to be written completely in ASP .NET, a framework, and C#, a language, I had never worked with before. So I started to learn how to build this project in .NET. It was very difficult on multiple fronts, but on in particular that I had underestimated was going from a loosely-typed language (PHP) to a strongly-typed language (C# .NET). So I tried several different methods, started a few different projects, and after a three months I got an email from my boss. It was short and sweet: “Justin, money is tight and we can’t afford to keep you on staff if you’re unable to make real progress on RBO. Either we need to see some real progress very soon, or staffing changes will be made.”

I learned very quickly that spinning your wheels trying to do something perfect, but never getting done, is an excellent way to stay unemployeed. I immediately shifted gears after the thought of explaining to my new wife that I had lost my job. Fortunately, after the fear of doing something wrong was overcome by my fear of not finishing anything, I completed RBO to version 1 in record time. I had made some ugly coding decision that we refractored out later, or scratched completely, but it was a working prototype. My boss was happy, and I was happy and still employed. While the front website has been redone for Blockbuster Express, it is powered by the same web services I built in 2007.

After building RBO, I was tasked with building a reporting & support suite that would manage millions of transactions. This second project went much smoother, and I put a lot of effort into it. I knew the people making the decision whether or not to buy our multi-million dollar software suite would be personally using this piece of our solution, so I wanted it to make them really smile. I was told after the demo of our new software suite, the executives all mentioned they were looking forward to using their “executive reporting tool.” Several months later, after successfully building some pretty slick software with the rest of the team (of which other things were even cooler then what I made, like Chris’s auto-updater system), Ambient Partners was purchased by NRC.

Dating DNA

By 2008 when the NCR deal was underway, I had a choice. I could stay, take a very nice raise, and work for a very large corporation. But, deep down inside, I had a bad feeling about working for NCR. As a small team, we were very effective and there was almost zero political or bureaucratic non-sense in Ambient. As for NCR, I would go from being a developer in a company of three developers to a company of hundreds, if not thousands, of developers. I was also tired of working in .NET and Web Services, as I was moving away from what I loved to do: building cool websites with cool technology.

My father at the same time was working on his new business: Dating DNA. They had a web developer in San Diego, but he was expensive because living in San Diego is expensive. So Kevin asked me if I would be interested in working for Dating DNA. We tested the waters by having myself build a Web-based iPhone App for Dating DNA (the App SDK hadn’t been released yet.) It worked really well, so I turned down the offer to become an NCR employee, and returned to my roots as a developer: working from home working with code I love.

Through 2008 and 2009 I worked as a full-time developer for Dating DNA. I took over all responsibilities for all their technology. In those two years, we did a lot. I’ll have to write a new blog post to completely cover everything we did that was awesome, but here were a few highlights: Built a real-time score generation system that could calculate hundreds of scores per second. Built the iPhone’s first Dating App, and to this day is a top ranking App in the charts and highest rated dating app. We scaled from 3,000 users to hundreds of thousands of users. We built a handful of new iPhone Apps, the main one being Clipish. We built custom chat rooms using Ajax and Comet, and a bunch of other stuff.

Alienware & CEVO

All this time, I was still doing work in the evenings with CEVO. In 2009, we were approached with the opportunity to do something we hadn’t done before in CEVO. Dell’s brand Alienware wanted us to make them a website like CEVO’s, only completely branded for Alienware. We built, from the ground up, and custom solution for Dell and Alienware, and Alienware Arena was born. This was a great project to work on, and we were able to get it done on an extremely tight timeline and a strict budget. I did 100% of all the graphical design, following Alienware’s look and feel, and I’m very proud with the result. It was built by myself, Eric Ping, and our new talented developer Mike Stevens. While I can’t say how many members Alienware Arena now has, it is a lot. Building such a successful website for a large company like Dell doesn’t come around very often. Each year Dell has us add more and more features to it, and it has been a great project for CEVO.

Utah Open Source

One thing that happened in November 2008 that I would consider one of the crucial events that “fast-tracked” my education as a web developer was being introduced to the local Open Source groups in Utah. I spoke at the Utah PHP Usergroup and was introduced to the Utah Open Source Foundation. Through these groups, I met dozens, and eventually hundreds, of talented, passionate people. While through CEVO I fast-tracked my web design and PHP development, through the Open Source groups I broadened my knowledge of so many more technologies. I learned about nginx, nagios, redis, memcached, php apc, linux server administration, and Git just to name a few off the top of my head. What is great is not only did I learn about these things, but I met people who know a lot about them. So when I ran into problems, I already knew a solution that could work, and knew people I could ask questions too. That, and I’ve made a lot of great friends through the different meetings, lunches, and conferences. I’ve picked up some contract work through my connections with these groups, and overall they have been extremely beneficial and great.

Chief Technology Officer

All of these different things played a part in me having the job I have today, CTO of Dating DNA. I’ve written already about my new responsibilities as our new CTO, and what it means for the company. These few short months as CTO we’ve made a lot of changes to handle even more scaling (especially with holiday surges and such). We’re in the process of improving our already fast score generation system, and moving our user photos to a more scalable solution in the near future. We’re evaluating our usability and such for our website, and seeing if a redesign on certain areas would be beneficial. There is a lot of work to be done, but I truly enjoy it.

Advice & Lessons Learned

After reading and thinking about the different things I’ve experienced and gone through to get me to this point, I’ve had a few thoughts that I think can help anyone in our field, and other fields of work:

• Surround yourself by people who help each other learn – Looking back, all of the people who have really helped me along with my career and education (not just schooling) have been people who help everyone learn. They each others things, and then learn from others. They harbor a culture of continual learning, and being in the tech industry which is always changing, this is critical.
• Always be learning something new, always – Trust me, there is always something to learn in this industry. Even if you’re learning something that isn’t directly involved, you never know what it might lead to. My work with Counter-Hack lead me to CEVO, which in turn lead to an accelerated web development “course” of “holy crap, we need to get this done and working or we crash and burn.” Even to this day, a lot of the things I learned about how hacks work I apply in other areas of computer science.
• Don’t let formal education be your only source of knowledge – Those that know me know I can be a little “harsh” when talking about formal educational institutions. Especially my frustrations with certain types of developers that are produced from these institutions. But the bottom line is this: they can be a great source of knowledge, and you can learn a lot from them. However, if you don’t learn additional information outside of the classroom, you are going to be sorely disappointed at how much you know when you graduate. Fundamentals are crucial, but practical application is just as important.
• Apply and build something important to you – There isn’t a better teacher than experience, and getting is as soon as possible, and as frequent as possible, will help a lot. I’ve spent more years unprofessional doing my job than professionally (at least for another year or two). Working on something meaningful to you, not just going through the motions of tutorials, really teach someone what it is like to do this kind of work.
• Networking and getting to know people is crucial – There is such an important emphasis on skill in the technical world, that knowing people and their actual abilities is vital. Also being know for your set of knowledge is important. That way when you want to learn something new, you know who to seek out. Before becoming Dating DNA’s CTO, and the rumor went out I was considering a new job, I had a lot of people contact me to see if I was interested in certain positions. There is no down side to being a “social” developer. Just because this isn’t “Marketing” doesn’t mean social networking isn’t important. I personally don’t like the term “networking” since it makes it seem like a chore. Make lots of friends in the programming groups and circles in your area and community, and it will be beneficial.
• Love what you do – If you don’t love what you do, then there is a good chance you will not go far in this industry. It doesn’t mean this is the only thing you do. Other hobbies and activities are important. But if you dread going to work, and do your work, every day, then it’s time to find something else.

This went a lot later in the evening than I thought, so I hope my thoughts are coherent, and if nothing else, entertaining. Its been a great deal of fun since those first days with Photoshop 3 trying to design something that didn’t look terrible. I look forward to the next few decades to see where web technology takes us.

Related posts:

1. So You Want To Be A Web Developer?
2. Web Development 10-Years Ago & Now
3. Being a Productive Developer

A Little Background

On January 2nd, we started to run into some serious performance problems with Dating DNA, so we began the process of going and deciding what were our problems, and how to optimize them. At the beginning, the Database was the bottleneck. It would get flooded with requests, and unable to handle them all, each query would slow down. We were using a tool called Jet Profiler (which I will post about more in detail later), but here is a graph it would output before we started out optimizations:

The light blue is total threads connected. The dark blue is thread running a query. The red are threads that are taking 2 seconds or longer to run, which are slow queries. The red is bad, very bad. So we were getting in bad shape. It was lovingly nick named “The Red Zone” while we were working on optimizations. Now, granted, we weren’t in “The Red Zone” the whole time, but when things got busy, things would slow down.

But optimization after optimization, we started to get things more and more under control:

After about three days of optimizations, we get back down to a manageable load:

The Problem

However, when the website was having high traffic, we noticed an anomaly, which looked like “blue waves.” We lovingly gave them the nickname of “blue meanies” from the Beatles’ Yellow Submarine Cartoon. On Jet Profiler, here is what it would report:

It would get worse and worse, these big blue waves of connections reporting as “sleeping.” At first we didn’t think it would be a problem. However, when ever we had “blue meanies” the site and iPhone app felt really slow. So, I won’t cover all the things I tested and tried that didn’t work, but he is ultimately how we figured out our problem.

The Solution

At first, I thought it was an issue with Garbage Collection with PHP. So I set the wait_timeout on MySQL to something really low, like 5 seconds. We then started to get errors all over the website, so we knew that they were legitimate connections from PHP. The only thing that made sense is that PHP & Apache now had become the bottle neck, that MySQL was returning requests so quickly that the threads were almost always sleeping, waiting for the PHP to finish. We slowly started to disable different functions on the website, trying to narrow down if there was a particular part of the website that was causing it. After a few hours, we figured out the feature: the ChatWalls. So we started to investigate why turning off the ChatWalls would make MySQL run faster, since we had moved the ChatWalls completely off MySQL and to run on Redis.

What we found is one particular function had a typo, that would cause PHP to iterate over an array not 10 to 20 times, but 1,000-2,000 times or more. This function was also called a lot by several Ajax calls. So, I fixed the typo, and the blue waves went away.

What was happening is Apache & PHP were spending so much time processing the buggy function, that it would cause the rest of the web requests to slow down greatly. That would keep open may too many MySQL connections, causing the blue ways, and slowing down the website even more.

So in reality, the blue waves were a symptom of the problem, not the cause. It is the whole Correlation vs Causation situation (which I probably should blog about in more detail when it comes to finding performance issues).

So if you have a lot of sleeping connections, but MySQL is performing well, most likely it is PHP or Apache slowing things down. I hope this can help those having a similar problem. As for our Database, its working well now. A few more problems to iron out, but it is running really fast. The few red spikes are from the score generation system that are doing bulk inserts, and do not slow down the end user experience:

Related posts:

1. MySQL & PHP – SQL_CALC_FOUND_ROWS – An easy way to get the total number of rows regardless of LIMIT
2. MySQL – Does Table Exist w/o Throwing Errors
3. MySQL, 40 Million Rows, MyISAM to InnoDB, 45 Minutes

This last week has been a roller coaster for us at Dating DNA. We had an excellent holiday season, but with mass volume of new sign ups our servers started to slow down. Severely. Something you never want to have happen. So over this last week I’ve worked about 80 hours (though I’m a salary employee, woohoo… ) and implemented a lot of new performance boots, and I have about four or five blog posts worth of discoveries I need to write. So before I forget them, I’m going to write them.

The first change I’d like to talk about is moving our ChatWalls (kind of like a Chatroom and Forums/Walls mashed up together) from a MySQL backend to a Redis backend (http://redis.io/). This initial change actually only took about 4-5 hours. Working out some additional kinks we found with our ChatWalls took another day or so, but the Migration to Redis was extremely smooth.

First, to give you an idea about our ChatWalls, here is a video of myself using them:

Redis, in a very simplified explanation, is like Memcache as a Key Value storage in memory. However, it is persistent (data not lost after restart), has more advanced data types (hashes, sorted sets), has built in virtual memory options (entire dataset doesn’t have to always be in memory), and you have a lot of cool operations beyond what memcache has (return sorted sets by score, purge out old records from a set, etc).

Now, I’ll walk through the basics of installing Redis, getting it up and running, reading and setting data in Redis, and migrating our old content.

Install Redis

Ok, compiling and installing Redis is a dream. It is very easy. On this server it is running Ubuntu 8.04 LTS. You go to the Redis Website and click on downloads. Get the latest stable version, and download it to the server (I used wget). Untar it (tar -zxvf /path/to/file.tgz), and go into the source code’s directory. Now, Redis uses a noticeably less amount of memory when compiled for 32bit versus 64bit due to pointer sizes (see FAQ). So to compile on a 64bit linux machine, you need to install libc6-dev-i386, which on ubuntu was “aptitude install libc6-dev-i386″.

Then, it was a simple “make 32bit” and “make install” and it compiled and installed redis in /usr/local/bin for me. Now, I wanted it to start up for me automatically when I rebooted the server, so I saved this init.d script as /etc/init.d/redis. Then I copied the sample configuration file out of the source file to /etc/redis/redis.conf. I ran “sysv-rc-conf” to set the Runlevels for it to execute the script at (2, 3, 4, and 5). If you don’t see “redis” listed, make sure you chmod +x /etc/init.d/redis so that it is executable.

Then I ran the command “/etc/init.d/redis start” and it was running.

Redis & PHP – Rediska

Now, while looking for a PHP Redis library I found Rediska. The documentation was pretty basic and straight forward, and I was able to get it installed pretty easy into my PHP App. Now, this library isn’t perfect, I found a few bugs while implementing it, especially with some values legal values for ZRANGEBYSCORE as defined by the Redis documentation, but Rediska would throw exceptions because I was passing non-int values. I’ll submit some bugs with patches to them next week so they can update Rediska.

A few things to look over in the Rediska library: first off, read how they manage multiple instances of their Rediska class. Its a little odd, but it works. I just don’t like passing an array with options in every command I have to make. So I would try something like $rediska->getHash(‘key_value’); and expect it to return a Rediska_Key_Hash object, but it would just return an array. So, in the end, I ended up using very little of the other Rediska classes, and stuck to just using the Rediska class and it’s methods. The good thing is they have good phpDocs, so if you have a good IDE, it will be easy to know which method does what.

Designing for a Key-Value Database Store

Now, I highly recommend reading this article on the different Redis Data Types. Also, give a look over their PHP Twitter Close with Redis, since they show some of the techniques for using a Key-Value as a data store.

So the very, very first thing you must do, and we did, is map out how you will design your application for Redis. You don’t have tables and columns any more, just a really big array with some cool tricks. So unlike MySQL if you have a typo in your table or field name, you won’t get errors. It will just run, but you application will have some serious bugs. So pick a naming convention, and document exactly the “tables” and “keys” you’ll be using, with what data types, and stick to it. If you don’t, you will be hating life. >/soap_box<

I like the following naming convention. Separate word groups by periods (.) and preface changing variables (like an ID) with a colon (:). So, here some map definitions for our keys:

chatwall.viewers > Hash[user_id] viewer_json – A hashtable with a key of the user id and a value of the member’s json object. This will hold the entire list of all users.

chatwall.wall:{wall_id}.viewers > Hash[user_id] viewer_json – A Hashtable that will hold a list of all the viewers for a given room. Example key: chatwall.wall:382.viewers

chatwall.nextPostId > Int – A int we will auto increment to get unique post IDs

chatwall.posts > Hash[post_id] post_json – The data for a post in a json object.

chatwall.wall:{wall_id}.posts > SortedSet We will store the score AND member as the Post ID. This will allow us to quickly get the last 50 posts in a set, and even pass a minimum Post ID to only get new ones.

[...]

cache.chatwall:{wall_id}.information > Json String – This will hold a json string of the cached wall information from MySQL.

These were just some of the definitions. Now, these definitions are for documentation only. You can use whatever naming conventions on the fly and Redis will use it. But it will be extremely useful if you have a list of what the key names are.

We also prefaced any “cached” data, so data that we treat like it was in memcached, with the string “cache.” The reason being if we wanted to clear any cached data, we easily could do so with a custom script. But we don’t want to confuse any of our good data we want to stay persistent being deleted on accident.

Refactoring the Code

Now, one thing that I was extremely grateful for when I designed the ChatWall system is each type of data had a PHP class with members, and functions that would perform the CRUD operations. So it was extremely easy to change the CRUD functions to read/write to Redis, and instead of writing out a query, I generated a key and passed for the value the variable $this. Super easy. Having all the data access code encapsulated correctly made it easy to simply go down my class methods and re-write them.

Now, there were a few caveats. One was moving posts, because an admin could move a post to a different forum or chatroom. Because we depended on Indexes from MySQL to catch the move, on our update commands we had to check the old record first to see if the room moved. If it did, we would have to remove an entry from the old chatwall.wall:{id}.posts and put it in the new one.

Migrating the Data

I simply wrote a script to select the data from our MySQL tables, set them to the new Classes, and called the “InsertRecord” function. In 10 seconds the script had migrated all of our data to Redis, which was taking up 25M of data in Redis. Pretty awesome.

Performance

I couldn’t be happier. It is really, really fast now. We had to double check our php code to prevent any calls to the Database, and make sure Redis had all the information it needed. Once we fixed a few areas, and our ChatWalls calls were being all 100% being executed against Redis, it was blazing fast. I am really, really happy with the change.

I would recommend getting and installing the redis-tools for your server. They make it really easy to see what your Redis server is doing. Our server is currently serving up 200-500 requests per second, and using up about 2-5% of our CPU with 20-30MB of ram. The background saves take 2-3 seconds to perform.

Bottom line, I highly recommend checking out Redis. I wouldn’t recommend replacing MySQL with it, but using it along side MySQL to handle cached data and highly accessed/changed data. It has been a huge performance boost for Dating DNA. We are in the process of finishing migrating our Match Score System to Redis, which is quite a bigger project. 500 million rows big. But it is going great, and looking to finishing with that project soon.

Related posts:

1. Large MySQL Conversion & InnoDB
2. MySQL & PHP – SQL_CALC_FOUND_ROWS – An easy way to get the total number of rows regardless of LIMIT
3. Memcached: Simple, Effective, and Powerful

First off, converting from one large set of InnoDB tables to a new structure. I’ve done something similar in the past, and I think I’ve honed in a little bit more the technique. So here are some thoughts regarding doing this process.

• Run Conversion on Separate Server – Looking back, I wish I had just spun up a beefy server at Rackspace Cloud, transferred the data, and gone to town running my conversion scripts. There were a few times where I tanked our Database server to the point of noticing slower load times on our website.
• Write to SQL Script Instead Directly – At first when I wrote my conversion scripts, it would read a dataset for a user, and the do inserts for each row into the Database to the new tables. If you’re doing less than 200,000 rows, this probably is fine. But doing 5 million or so takes too long.
• Insert Data By Primary Key Order – You’d think I would have learned my lesson from the last time I did this, but I found out even for a smaller data set of a few million (instead of tens of millions before). Inserting by the correct order saves a ton of time. The easiest way to think of it is if you had to put files into a filing cabinet. It would be much faster to just move them in pre-sorted, instead of sorting them as you add them. Also, InnoDB doesn’t know whether you are inserting 10 rows, or 10 million. So when you insert unsorted, it will spend a lot of resources shuffling around pages.
• Write Your Conversions Tools to be Robust – What you don’t want is for your script to only be able to run start to finish, and if it breaks, you have to start all over. What you want is to be able to run sections, test, rerun sections, convert large all sections, then right before you change to the new script, run the conversion of the newer entries.
• Monitor Status, Progress, & Time Left – One thing you’ll definitely want to know is how long your script will take to run. You will get to be like Windows and try to guess how long something will take (and hopefully be more accurate). I like to keep it simple. I’m converting X1 million rows, I started a Y1 time, and now I’ve completed X2 conversions, and the current Time is Y2. So I figure out how long its taken me to do the ones I’ve finished, and calculate how long it will take to do the rest at the same rate.

Hopefully this will help out someone else who is making a large scale conversion from one table structure to another.

Related posts:

1. MySQL, 40 Million Rows, MyISAM to InnoDB, 45 Minutes
2. MySQL – Does Table Exist w/o Throwing Errors
3. Digital Conversion Video

Well, Facebook has made their announcement, and it is called HipHop. Here is an excerpt:

Today I’m excited to share the project a small team of amazing people and I have been working on for the past two years; HipHop for PHP. With HipHop we’ve reduced the CPU usage on our Web servers on average by about fifty percent, depending on the page. Less CPU means fewer servers, which means less overhead. This project has had a tremendous impact on Facebook. We feel the Web at large can benefit from HipHop, so we are releasing it as open source this evening in hope that it brings a new focus toward scaling large complex websites with PHP. While HipHop has shown us incredible results, it’s certainly not complete and you should be comfortable with beta software before trying it out.

HipHop for PHP isn’t technically a compiler itself. Rather it is a source code transformer. HipHop programmatically transforms your PHP source code into highly optimized C++ and then uses g++ to compile it. HipHop executes the source code in a semantically equivalent manner and sacrifices some rarely used features — such as eval() — in exchange for improved performance. HipHop includes a code transformer, a reimplementation of PHP’s runtime system, and a rewrite of many common PHP Extensions to take advantage of these performance optimizations.

Marco Tabini has a nice summary on what PHP need to know about HipHop. However, people I’ve talked to have had several opinions on HipHop, ranging from pure delight to doubts to indifference. Personally I think this is a great thing for PHP, and I’ll tell you why.

HipHop will enabled business to pick a web scripting language and use it from start-up to internet-giant. While HipHop isn’t the first tool used to speed up PHP, such as APC, Memcached, and others; it is more of a current reminder: PHP is serious about the web. For people already using PHP, this is just a nice feature. For those who aren’t using PHP, it shows that it is a very strong candidate.

I’ve worked in a non LAMP shop before. To non-PHP people, it has the stigma, regardless of actual performance, of being hard to scale. It is a “scripted language”, and in the Java / .NET world where I was working, it was looked at as being an absurd choice. Now, PHP can make a strong case of being able to mature with your application and scale appropriately.

One fear that I have as a Project Manager, and anyone has who decided on the technology that they will implement, is “picking the wrong team.” During my .NET days, Microsoft’s team announced a new technology for .NET applications. It showed a lot of promise and simplified some things that were a pain in .NET. Our team adopted the new technology and ran with it. It wasn’t perfect, but we liked it. Less than a year later, Microsoft announced that it would be moving the project to a drastically different team, which basically meant “there wasn’t enough adoption, so we’re putting this project on the back burner.” The new team was basically to maintain the code forward, but not develop new features.

What happened is we invested on this new technology, and we saw the potential it would have in the future. We lost our investment, since that future potential would never be realized. Now our projects had this new technology embedded in many spots. Re-writing those places would take a great deal of time. From a Project Manager perspective, it was a poor choice.

For PHP, the emergence of HipHop, a concept that is working across thousands of production servers, it sends a strong message to the web development community as a whole. Not only is PHP widely adopted, flexible yet powerful, and easy to implement. PHP is competitive, and is continually evolving to adapt to web developer’s needs. It is a safe bet for many projects, and it isn’t going anywhere but up.

Will I be implementing HipHop right now? Probably not. Ilia Alshanetsky has a very healthy reality check on what HipHop is for. But the bottom line is I don’t need what HipHop is offering yet. I can see very great uses in the future for Dating DNA’s compatibility generation, as well as certain pieces of other PHP projects that could use a serious performance boost. But for now, my Blog will not be HipHoping along. I’m excited for not only what HipHop does, but what it’s greater underlining meaning as a whole with PHP.

Related posts:

1. Characteristics of Good PHP Code
2. So You Want To Be A Web Developer?
3. Web Development 10-Years Ago & Now

I’ve gone back and looked at this year of 2009, and the things I’ve blogged about, and I thought I’d give a little recap.

My CSS Conversion

At the beginning of the year I realized I had 100% percent to using CSS. I’m still a little in awe that being such a table junkie I finally did it. There were a few key things that I did to help me learn all the difficult things with CSS. I hope I can help some poor web developer out there understand their CSS a little better.

A Year of PHP IDEs

I really invested some time in trying out and experimenting with several PHP Integrated Development Environments (IDEs). I had been using Zend Studio for several years, however more and more I found myself frustrated with it’s quirks. However, I haven’t had time yet to blog about the new IDE I am using, NetBeans. I’ve made the switch from Zend Studio to NetBeans, and while I’m not completely sold, I’m still giving it some time. Maybe over christmas break I’ll blog some about it.

Interesting Challenges w/ MySQL

While I haven’t blogged about most of them, I have had some interesting challenges w/ MySQL over the last year. One was converting 40 Million rows from a MyISAM engine to an InnoDB Engine. I’ve been having some serious heart-to-heart conversations with MySQL (metaphorically of course) on how I’m going to scale it to millions and billions of rows. I think I’ve mapped out in my head how we’re going to handle it for Dating DNA. That will be one challenge I’ll be addressing in 2010.

New Blog Design

This year I got around to re-designing my blog using the Carrington Theme as a base. I kinda like it, but it still could use a little more polish. One day I’ll find time to give it that extra polish.

Memcached

This year I really got into the theory on how to design an application with Memcached. I’ve decreased the load on several websites by using it, and wrote an in-depth article on the theory of caching, and how to implement it effectively. I’ve been amazed at how well it works, and that I hadn’t used it before.

A Year of Presentations

I’ve given two presentations this year, my Memcached presentation at UPHPU, and my “Real Life Scaling” presentation at the Utah Open Source Conference. I got great feedback from both presentations, and I look forward to what I’ll present on this upcoming year. I’m thinking about putting together a presentation called “Being a Web Ninja with jQuery,” and showing of the dozen of awesome applications I’ve built using jQuery.

Utah Open Source Conference 2009

This year I helped organize and throw together the Utah Open Source Conference. I was made the Sponsorship Manager, and it was interesting trying to talk to companies in a recession about donating to a non-profit conference. I haven’t had a chance to blog about different techniques on how to professional ask for money for something like this, but hopefully in 2010 I’ll be able to share some insights.

Things I Didn’t Blog About

There were a lot of things I didn’t get around to blogging about this year. I’ll quickly touch on them here, and hopefully in 2010 I’ll be able to blog about them more.

Web Services – I’ve done a LOT of web service work. Web services provide a unique challenge, since many times its hard to debug them. At my old job, one of the biggest challenges is we really couldn’t debug our web services until all of the other pieces of the puzzle were done. Then we spent long hours with trial and error, debugging via dumping logs. I’ve learned a lot of tricks with PHP and web services. Hopefully I’ll be blog more about those soon.

Performance Tuning LAMP – I’ve had to do a lot of tuning to keep the Dating DNA website afloat. Because of the awesomeness of our iPhone apps, we’ve had a lot of load on our web servers. There are a lot of things you can do to get the LAMP stack to run well.

Alienware Arena – Here was a fun little project I was able to crank out for probably the largest client I’ve done work for: Dell. We made them a website for their Alienware brand called “Alienware Arena.”

There were a lot of challenges to this website, mainly timeline and budget. But we were able to walk away happy with our work, and the Dell / Alienware team were really happy. It was interesting going through and trying to copy / mimic the look and feel of the Alienware website. I’m really proud of what we made.

Never Ending iPhone App Store Saga – Anyone who knows me has heard of the bloody wars that my current employer has had with Apple, their iPhone, and the App Store. The ridiculous approval process, the extremely long wait periods, and Apple’s OCD when it comes to their phone. I won’t say much anything else on the topic, but boy it has been frustrating.

What to Expect with 2010

Boy, 2010 will be interesting. What to expect? I’ll be doing some pretty sick programming in jQuery, as I’ll be adding integrating chat to the Dating DNA website. What else? A lot more when it comes to integrating websites into other social network platforms.

From a business platform, 2010 will be a lot of work spent on taking some great ideas and trying to monetize them. While working on awesome stuff is great, its important to pay the bills with them. I will have an entire year to sign up Sponsors for the 2010 UTOS Conference instead of 2 short months.

Hopefully I’ll get in a better habit of documenting my discoveries and solutions here in my blog. Sharing information is the best way to help everyone out.

I hope everyone has a happy holiday and good luck in the new year! 2010, here we come!

Related posts:

1. Speaking: Utah Open Source Conference 2009
2. So You Want To Be A Web Developer?
3. Presentation: Real Life Scaling

If you have any questions or comments, please feel free to leave them below.

Related posts:

1. Steamlined Web Development Presentation Video
2. 100th Blog Entry – First Year of Real Blogging
3. A New Stage in Life: My MacBook Pro

Scaling is a real issue for many websites. However, many developers try to implement solutions used by the giants of the internet: Google, Facebook, Twitter, WordPress, etc. However, many of these techniques are designed for very unique and high demand situations. Implementing these techniques prematurely to smaller websites can lead to overly-complex solutions that can be difficult to manage, and even hinder progress.

See real life examples of from two popular websites: cevo.com and datingdna.com. Learn what challenges and bottlenecks they faced, and the solutions they created to overcome them. Find out what optimizations and implementations returned the greatest ROI for scaling with the project.

CEVO is an online gaming league that hosts thousands of matches for hundreds of video game events. They have contracted with DirectTV, Dell, and other companies to offer branded gaming events. Because the company is run by a staff spread across the USA and Canada, the website runs 100% of their business. Hundreds of gaming servers and player clients communicate with CEVO’s APIs to get real-time information. Learn how they’ve handled the ever increasing load on their servers.

Dating DNA is an online dating community that integrates with today’s popular social networking sites. The site has exploded since it released its iPhone App, which is currently the #1 iPhone Dating App. Learn how Dating DNA has scaled from having daily user sign-ups increase 1000% in a matter of weeks.

This presentation will be given by Justin Carmony, a lead developer for both projects.

I will be speaking on Friday, Oct. 9th at 7:00 PM. I’m looking forward to presenting and have my slides done early for once. Hopefully you all will enjoy it! If you aren’t signed up to attend this year, really consider! It is going to be a very awesome, there are a lot of great of local, and national, presenters. Use the coupon UPHPU to get 50% off attendance! See you there!

Related posts:

1. Adeona – Open Source LoJack for Laptops
2. Why Are Some Open Source Advocates Hypocrites?
3. Speaking: Utah PHP Usergroup – Streamlined Web Development