Hopefully at some point, your website gets a lot of traffic. Yay, you’ve reached your goal of getting good traffic, but it is soon followed by issues with performance and load. I like to call these the growing pains of a website. So as a web developer, I suddenly have the epiphany of “Hey, I need to scale my website!” What follows next is the biggest mistake a web developer can make:

They start looking at articles on how Google scales, or maybe how Facebook manages all of their traffic.

This is a mistake! To be brutally honest, you are not Google. You are not Facebook. You are not Twitter. You are a website that receives less than 0.000001% of the traffic that some major websites receive.

Why is this dangerous for web developers to do? Google, Twitter, Facebook, and others like them are solving complicated at a very large scale. I remember a presentation by a Twitter engineer who developed a program for a unique ID generator that can generate millions of IDs per second. The probability of you needing this type of solution is about the same as being struck by lightening. Applying these same practices at a much smaller scale are not realistic. If a locally owned grocery store wanted to open a second store, they would not adopt the same practices that Wal-mart use to manage their 8970 stores.

A Little Reality Check

I’m sure that most of my readers know of StackExchange.com. They power the popular website StackOverflow and several others. They have about two million visitors per day. That is a lot of traffic. StackOverflow is ranked #123 on Alexa. So you would imagine that they have a very large infrastructure serving all of this traffic?

Earlier this year, Stack Exchange wrote an article about their production environment. I was surprised on what exactly they were using. In paticular, the number of Production Servers*:

• 12 Web Servers (Windows Server 2008 R2)
• 2 Database Servers (Windows Server 2008 R2 and SQL Server 2008 R2)
• 2 Load Balancers (Ubuntu Server and HAProxy)
• 2 Caching Servers (Redis on CentOS)
• 1 Router / Firewall (Ubuntu Server)
• 3 DNS Servers (Bind on CentOS)

That is 22 servers for 2 Million Visits per day, serving 800 HTTP requests per second. Now, StackExchange did clarify that they did have other servers for management and fail over, but 22 servers handle their production load. This is a website that is ranked the 123rd most visited website in the world.

Honestly, most websites could be run on half a dozen servers if designed and configured correctly, including redundancy. Some really busy websites could run off a dozen servers. Unless you’re in the top 5,000 websites on the web, you really shouldn’t be worried about large-scale techniques.

So when you’re website is starting to grow, and you leave small scale, you’ll enter the phase of “Middle-Scale.”

What is Middle-Scale?

Middle-Scale is like being an awkward teenager:

You know that you can’t be the only one suffering through this, but you’re unsure how to proceed. It feels like you’re missing missing out on things everyone else must already know, but aren’t talking about. Like everyone else are awesome vampires or something:

But the reality is this: they don’t have some awesome secret! They are just normal teenagers.

This same idea applies to Middle-Scale websites.

Middle-Scale is when the most important things are still the best practices. Only now when you deviate from them you can feel those consequences. When you only had 100 users, a couple of nested queries and missing indexes didn’t cause that much of a problem. Your database is powerful enough to hide the inefficiencies. However, when you get to 10,000 users, your database can no longer hide the inefficiencies.

Middle-Scale is when simply separating your web server and database server isn’t enough. You’ll probably need to add some sort of cache like memcached. You’ll need to start tweaking your MySQL, Apache, and PHP configurations.

Then, after you’ve ironed out your inefficiencies, you’ll start to use multiple servers. You’ll probably add a Load Balancer with multiple web servers. After that, you’ll probably have some sort of Master-Slave replication for your Database for backups and fail-overs.

You start to leave this “Middle-Scale” classification when you move to multiple data centers, and start to do some load balancing at the the DNS layer. This is when you’ll start to have a dedicated sys-admin team.

Okay, I’m Middle-Scale! So what should I do? Where do I look?

First off, you must adhere to best practices. If you are working with PHP, research PHP performance and best practices. Do they same for each of your technologies, like Apache and MySQL. You will need to stop treating your application as one big app, and start to understand all of it’s moving parts.

Second, you must understand your specific problems. Scaling ins’t a problem, nor is it a solution. It is a generic term for many different types of solutions. Without understanding why your website is running slow, or why it cannot handle the load, you will not be able to create an effective solution.

So you don’t have a scaling problem. You have a MySQL performance issue, or a Apache problem, or a PHP problem. Most likely, it is something extremely specific. You have a high volume of MySQL write operations (i.e. UPDATE, INSERT, DELETE, REPLACE), or perhaps you are missing some indexes and have too many full table scans.

Third, Googling for help will only get you so far. You are starting to enter a phase when it is harder and harder to find answers to your broad issues. Talking with other experienced people who have gone through the Middle-Scale pains before will help immensely. I cannot recommend highly enough going to User groups. Being able to communicate with someone, either face to face, on the Phone, over IRC, etc. is invaluable. While I’ve learned a lot at conference and usergroup presentations, I’ve learned even more by just talking with the people attending and at the social gatherings.

Profile & Performance will Naturally Lead to Scaling

When you want to scale, it can feel like a very daunting task. It seems like this big unknown complicated solution. What in the world am I going to do? I remember feeling these worries when I first started to investigate load balancing and sharding for some websites I was working on.

The thing is, if you start to profile your application, you will discover it’s inefficiencies. I remember when I spent a sold week, working 12-16 hours a day profiling and optimizing Dating DNA’s database. I found a lot of bad queries, and I was able to cut our load times from 2-5 seconds to under 0.1 seconds. The CPU on the database server went from 80-90% CPU utilization to under 10%. It was incredible, and then I promptly took the entire next week off. When we migrated to new servers, I was able to move to less powerful database server and still have the same great performance. So by profiling and optimizing our database, I didn’t need to worry about spinning up multiple master databases and sharding our data.

With Clipish, we faced almost opposite scaling problems. The database was rarely an issue, but our web server CPU’s were. We do a lot of ImageMagick manipulations of images, and at high volumes on virtual servers this can be a big issue. So over the last year we’ve introduced some load balancing and CDN tools to help serve all 10 TB of bandwidth for Clipish.

The thing is, when you start to profile your application, you start to understand it’s low areas better, so you have a much better idea on what do to. Even if you don’t know your solution, it is much easier to find a solution with a sound understanding of the problem. For example “scaling mysql” yields much less helpful results than “mysql full table scans” in Google.

So should I ignore what Facebook and Google do for scaling?

Of course not! First off, they do cool stuff. Just because I’ll watch NASA launch a space shuttle doesn’t mean I’ll try to make a rocket system for my broken lawn mower. But you have to put what they are doing into context. People from large websites have published several good “best practices” articles on techniques that help any website. Especially things on the client/browser side of things. Just use caution. I cringe when I hear someone say “we’re trying to use Cassandra to solve XYZ problem at work” when it is severe overkill.

Final Thoughts

Most of the time when I talk about performance and scaling with other people, it is when they are in “critical mode.” Their website is down, slow, unusable, etc, and they are looking to fix the problem. I will say, it is much more difficult to profile in “critical mode” than profiling before hand. The reason is you are much more desperately focused on getting it working again instead of understanding the problem.

I’ll be giving a presentation this Thursday at UPHPU on Profiling PHP Applications. I’ll post the slides, and most likely write some articles on the subject afterwards. As always, feel free to email me or leave a comment.

Related posts:

1. Data Backups – There Are No Excuses
2. Speaking: Utah Open Source Conference 2009
3. Presentation: Real Life Scaling

Today some memories came from when I first started with web development. So before I forget anymore of the details, I thought I would share my unique way I became a web developer. I never went to school to learn Computer Science or Information Systems. I’ve had a total of 2 programming classes, which were my in sophomore and junior years of High School, one of which we lovingly nicknamed Warcraft 101, because we would spend the entire time getting out butts kicked in Warcraft 2 by my classmate Daren (We all got A’s, we had just finished the entire coursework in 2 and a half weeks).

The reason I share this is perhaps others who hope to become a web developer can learn some of the valuable lessons I picked up along the way. That, and that my close programming friends and colleagues can get a kick out of my humble roots.

Heritage

How I ended up in Technology to begin with is probably because part of it is in my blood. My Father, Kevin Carmony, has been the owner of several technology businesses. So I was exposed frequently to computers and other technologies my whole life. I remember some time when I was about five or six, in the late 1980s, playing against my younger brother on a multiplayer ASCII LAN game at the old Streamlined Information Systems office. It was wandering around a 2D maze hunting these 8-bit monsters, and each other, and it was amazing fun.

I also had access to a lot of expensive software that were hand-me-downs. One of them I remember is Photoshop 3 and 4, spending hours trying to design websites with it. To give you an idea how old that is, the current version is CS5 (aka version 12). The first program I really used to make a website was (and get ready to gasp) Microsoft FrontPage in 1997. There were also a couple of website books laying around that I would read, or at least try to.

On the other side of the equation was my mother, while herself wasn’t extremely technical, encouraged and “sponsored” my learning when I was young. By sponsoring I mean she many times bought computers, digital piano keyboards, DSL (we were some of the very first people to have “high” speed internet in our town), and paid for many other expenses. I remember a several hundred dollar long distance bill because I would call the “Provo” dial-up connection instead of the broken “Ogden” phone number. I even once ordered a “temporary” AOL dial-up account, with her credit card and without permission, because our ISP was down, and I needed to check my website. I forgot to cancel it afterward, and it racked up a few months of fees before my mother realized what had happened. She never yelled or mad over these “expenses”, that I realize now as an adult, were not cheap. She just told me in the future what to do to avoid causing them again.

She was also very patient with a son whose grades weren’t the best, and who would rather work on a website than do his homework. I know if she would have came down hard on me, my GPA would probably be higher (its wasn’t bad, just not great), but I wouldn’t have learned what I did, nor make the connections that eventually fast tracked my career. I owe her a great deal (I love you mom!)

Humble Beginnings

I first learned about websites, and how I could make them, while sitting in my 7th Grade Band Class. My fellow clarinet player Kenny Cottrell explained to me in between songs about HTML, Notepad, and how I could learn to make my own website. So I set off to learn HTML. In my excitement, I found an online book on HTML, and printed the entire 400 page book on my mother’s home printer. Single page, of course, because I couldn’t figure out how to easily do double page, and it was on her expensive laser printer (this was 1997) with expensive cartridges.

My very first website was hosted on one of my father’s web servers, and using FrontPage, I made a website about a game called The Realm, one of the very first graphical Massive Multiplayer Online games (MMO, think World of Warcraft). It was bad, really bad. I can’t find any pictures or old files from it, but you can take my word for it. When spending weekends at my Grandma’s house with my Dad, I would spend hours designing and writing websites, or at least try to. They were all bad, but I learned a lot by trying over and over again. I would design a site, look a a professional site, and try to see why mine stunk while the professional ones were so much nicer. I remember looking at Amazon‘s rounded corner tabs for hours trying to get mine to look just as nice.

I then got my first domain, RPGLegacy.com, in 1998 and started a website with game reviews and walkthroughs for PlayStation RPG games. I remember writing reviews and information for games like Final Fantasy VII, Suikoden I & II, and even more obscure (and terrible) titles like The Granstream Saga. I started to get perhaps 100 visitors a month, and I thought that was great. I even got emails from people asking me for help. One subject in particular was in Breath of Fire 3 dozens had emailed me about getting stuck in a castle. I myself had gotten stuck in the same place for hours, and posted an in-depth solution for finding Honey the Robot in the castle. For being a 15 year old kid in junior high, it was a lot of fun, and I learned a lot. I had moved to Dreamweaver for making the website, and the designs weren’t half bad. However, I don’t have a screenshot of how the old site used to look.

Counter-Hack

I was 16 and I in high school. My classmate Daren (the same who whooped me at Warcraft 2) introduced me to a game called Counter-Strike. It was a First Person Shooter, and it was a lot of fun. However, at the same time, there were a lot of these “cheats” and “hacks” programs that were coming out. My friends and I would be accused of “cheating” and “hacking” when in reality we weren’t. Also, some of the claims on how we were cheating were rather absurd. So I started to investigate the truth and fiction behind these programs. I learned a lot, and thought I would share that knowledge, so I started Counter-Hack (http://www.counter-hack.net) in the summer of 2001.

Little did I know how extremely popular this website would become. Within a week or two I had a website wtih decent content up using Dreamweaver. This was 100% HTML based, no CSS, trying to use Dreamweaver’s Template system. What I would have given for some PHP or even WordPress. While with RPGLegacy, I had maybe a max of 5,000 visitors over it’s entire lifespan, within a month I had 30,000 visitors to Counter-Hack. Soon I was getting thousands of visitors per day. About the same time I met Anthony Ouwehand (nicknamed H3X), who had ran another popular website about video game hacks. He graciously helped me with his PHP/MySQL skills. He designed, and developed, the Counter-Hack website that was launched in 2002 and ran until 2008 when everyone involved with the project just were too busy, and the site had ran it’s course and purpose. Two years of that time the project ran with the rest of the volunteers while I served an LDS Mission in Torreon Mexico.

During the years with Counter-Hack, a few highlights were interviews with Wired for news stories, working with Valve Software to help recover the HL2 Leak, and an interview and article with Rolling Stone Magazine. During it’s height, Counter-Hack was covering dozens of games with hundreds of thousands of visitors a month. During it’s later years, Counter-Hack implemented a Wiki system for much of it’s content, something that was pretty new at the time. All and all, it was a great experience with dozens of volunteers and great memories. For a hobby during High School and the year after graduating, I couldn’t have asked for a better experience. I learned a lot before even starting college.

Developing Out of Necessity for CEVO

The reason why for the background story is for two reasons: I had started practicing web design and development from a very early time, and I gained a unique background and knowledge with my work with Counter-Hack. So in March 2005, being home only a few weeks from Mexico and still with a heavy mexican accent, I was approached to work with a company called CEVO. It was an online video game league that was emerging, and their first game was Counter-Strike. They needed someone to help consult with preventing cheating and “hacking” in their matches, and I was a perfect fit. So I joined as a consultant, and quickly became the Operations Director within a few months. CEVO also had brought on Eric Ping to be the CTO, and the company started to grow.

One of the challenges CEVO faced was it was a completely distributed team. Charlie Plitt, the owner, lived in Baltimore, MD, Eric lived in Ephrata, WS, and I lived in Ogden, UT. We had staff and volunteers that literally lived in all 50 states except Hawaii and Alaska, as well as some across Canada. Our customers also lived all over North America, and we relied on the Website, Email, and VoIP to run the entire company. It was insane and awesome at the same time. This put a ton of demand on Eric, our only developer for the website, to get new features implemented quickly. During the first two years, we had re-designed the website a total of 6 times before finally settling down on the current design, which is being redesigned now as well. Eric couldn’t handle it all, and since we didn’t have the funds to hire another developer, I thought since I knew something about web design and programming, I could help out.

Eric took me under his wing and basically mentored me along as I started to take on project after project. I had become familiar with PHP and MySQL when I was tinkering with Counter-Hack’s code, but now I was really learning. I was also learning extremely quickly because I had no other option. It wasn’t like working at another company where I had a team of Senior Developers that could bail me out. Eric worked such late hours that normally if I was awake, he was asleep, and we meet for a few hours in the afternoon and evenings. But if I had a bug, or a problem, there was only one person who could solve the problems: myself. Working on a team of two developers for a start-up company is extremely demanding, and I was constantly looking for ways to work more efficiently. It was the only way to meet CEVO’s growing demands, by learning how to be a quicker and better developer. So while I had a day-job of doing Tech Support for a local company, I spent every other free moment working for CEVO. As CEVO grew, we ran into scaling problems and performance problems, things a normal “Jr. Developer” wouldn’t have hands-on experience dealing with.

Eventually, I had learned so much working for CEVO, that I was able to quit my day-job doing tech support, and did contract work on the side. I helped launch some e-commerce websites, some basic business websites, and spent the rest of the time working on CEVO. I had started to go back to school for my Business Administration degree at WSU, and I was pretty much busy non-stop. But it was a lot of fun, and allowed me a lot of freedom not having a normal 9 to 5 job.

Ambient Partners, LLC

In 2006, I had the most amazing thing happen to me: I met my wife. We had dated for almost a year and we decided to get married. So by the end of 2006, I had a serious realization: I needed a steady income. While contract work paid really well, I could go a month or two without a check from clients. When living at home with relatively no expenses with the exception of my car, I could get away with this. But health insurance, rent, groceries, etc. I couldn’t live that way. So I decided to find a full-time programming job. Fortunately, the company I had worked tech support at had split with their California office and changed names. They were a company primarily doing software for DVD Rental Kiosks, and needed a Web Developer to do work on their web technologies. So by January 1st, 2007, I became a full-time employee for Ambient Partners. Our development team consisted of myself, a Senior Developer, and the CTO. As I look back at what the three of us accomplished as a development team, it amazes me.

When I joined, our main client had about 100 kiosks in the field. After doing the company website, I was given a very big project: RBO, Rent-Buy-Online. We wanted to provide clients with “white-label” solution to allow people to reserve and buy DVDs on a website, and then go pick them up at a Kiosk. There were two parts to this solution. The website itself, and the web services to supply the website with data. I had never written web services before, so I had a lot to learn very quickly. Also, it was to be written completely in ASP .NET, a framework, and C#, a language, I had never worked with before. So I started to learn how to build this project in .NET. It was very difficult on multiple fronts, but on in particular that I had underestimated was going from a loosely-typed language (PHP) to a strongly-typed language (C# .NET). So I tried several different methods, started a few different projects, and after a three months I got an email from my boss. It was short and sweet: “Justin, money is tight and we can’t afford to keep you on staff if you’re unable to make real progress on RBO. Either we need to see some real progress very soon, or staffing changes will be made.”

I learned very quickly that spinning your wheels trying to do something perfect, but never getting done, is an excellent way to stay unemployeed. I immediately shifted gears after the thought of explaining to my new wife that I had lost my job. Fortunately, after the fear of doing something wrong was overcome by my fear of not finishing anything, I completed RBO to version 1 in record time. I had made some ugly coding decision that we refractored out later, or scratched completely, but it was a working prototype. My boss was happy, and I was happy and still employed. While the front website has been redone for Blockbuster Express, it is powered by the same web services I built in 2007.

After building RBO, I was tasked with building a reporting & support suite that would manage millions of transactions. This second project went much smoother, and I put a lot of effort into it. I knew the people making the decision whether or not to buy our multi-million dollar software suite would be personally using this piece of our solution, so I wanted it to make them really smile. I was told after the demo of our new software suite, the executives all mentioned they were looking forward to using their “executive reporting tool.” Several months later, after successfully building some pretty slick software with the rest of the team (of which other things were even cooler then what I made, like Chris’s auto-updater system), Ambient Partners was purchased by NRC.

Dating DNA

By 2008 when the NCR deal was underway, I had a choice. I could stay, take a very nice raise, and work for a very large corporation. But, deep down inside, I had a bad feeling about working for NCR. As a small team, we were very effective and there was almost zero political or bureaucratic non-sense in Ambient. As for NCR, I would go from being a developer in a company of three developers to a company of hundreds, if not thousands, of developers. I was also tired of working in .NET and Web Services, as I was moving away from what I loved to do: building cool websites with cool technology.

My father at the same time was working on his new business: Dating DNA. They had a web developer in San Diego, but he was expensive because living in San Diego is expensive. So Kevin asked me if I would be interested in working for Dating DNA. We tested the waters by having myself build a Web-based iPhone App for Dating DNA (the App SDK hadn’t been released yet.) It worked really well, so I turned down the offer to become an NCR employee, and returned to my roots as a developer: working from home working with code I love.

Through 2008 and 2009 I worked as a full-time developer for Dating DNA. I took over all responsibilities for all their technology. In those two years, we did a lot. I’ll have to write a new blog post to completely cover everything we did that was awesome, but here were a few highlights: Built a real-time score generation system that could calculate hundreds of scores per second. Built the iPhone’s first Dating App, and to this day is a top ranking App in the charts and highest rated dating app. We scaled from 3,000 users to hundreds of thousands of users. We built a handful of new iPhone Apps, the main one being Clipish. We built custom chat rooms using Ajax and Comet, and a bunch of other stuff.

Alienware & CEVO

All this time, I was still doing work in the evenings with CEVO. In 2009, we were approached with the opportunity to do something we hadn’t done before in CEVO. Dell’s brand Alienware wanted us to make them a website like CEVO’s, only completely branded for Alienware. We built, from the ground up, and custom solution for Dell and Alienware, and Alienware Arena was born. This was a great project to work on, and we were able to get it done on an extremely tight timeline and a strict budget. I did 100% of all the graphical design, following Alienware’s look and feel, and I’m very proud with the result. It was built by myself, Eric Ping, and our new talented developer Mike Stevens. While I can’t say how many members Alienware Arena now has, it is a lot. Building such a successful website for a large company like Dell doesn’t come around very often. Each year Dell has us add more and more features to it, and it has been a great project for CEVO.

Utah Open Source

One thing that happened in November 2008 that I would consider one of the crucial events that “fast-tracked” my education as a web developer was being introduced to the local Open Source groups in Utah. I spoke at the Utah PHP Usergroup and was introduced to the Utah Open Source Foundation. Through these groups, I met dozens, and eventually hundreds, of talented, passionate people. While through CEVO I fast-tracked my web design and PHP development, through the Open Source groups I broadened my knowledge of so many more technologies. I learned about nginx, nagios, redis, memcached, php apc, linux server administration, and Git just to name a few off the top of my head. What is great is not only did I learn about these things, but I met people who know a lot about them. So when I ran into problems, I already knew a solution that could work, and knew people I could ask questions too. That, and I’ve made a lot of great friends through the different meetings, lunches, and conferences. I’ve picked up some contract work through my connections with these groups, and overall they have been extremely beneficial and great.

Chief Technology Officer

All of these different things played a part in me having the job I have today, CTO of Dating DNA. I’ve written already about my new responsibilities as our new CTO, and what it means for the company. These few short months as CTO we’ve made a lot of changes to handle even more scaling (especially with holiday surges and such). We’re in the process of improving our already fast score generation system, and moving our user photos to a more scalable solution in the near future. We’re evaluating our usability and such for our website, and seeing if a redesign on certain areas would be beneficial. There is a lot of work to be done, but I truly enjoy it.

Advice & Lessons Learned

After reading and thinking about the different things I’ve experienced and gone through to get me to this point, I’ve had a few thoughts that I think can help anyone in our field, and other fields of work:

• Surround yourself by people who help each other learn – Looking back, all of the people who have really helped me along with my career and education (not just schooling) have been people who help everyone learn. They each others things, and then learn from others. They harbor a culture of continual learning, and being in the tech industry which is always changing, this is critical.
• Always be learning something new, always – Trust me, there is always something to learn in this industry. Even if you’re learning something that isn’t directly involved, you never know what it might lead to. My work with Counter-Hack lead me to CEVO, which in turn lead to an accelerated web development “course” of “holy crap, we need to get this done and working or we crash and burn.” Even to this day, a lot of the things I learned about how hacks work I apply in other areas of computer science.
• Don’t let formal education be your only source of knowledge – Those that know me know I can be a little “harsh” when talking about formal educational institutions. Especially my frustrations with certain types of developers that are produced from these institutions. But the bottom line is this: they can be a great source of knowledge, and you can learn a lot from them. However, if you don’t learn additional information outside of the classroom, you are going to be sorely disappointed at how much you know when you graduate. Fundamentals are crucial, but practical application is just as important.
• Apply and build something important to you – There isn’t a better teacher than experience, and getting is as soon as possible, and as frequent as possible, will help a lot. I’ve spent more years unprofessional doing my job than professionally (at least for another year or two). Working on something meaningful to you, not just going through the motions of tutorials, really teach someone what it is like to do this kind of work.
• Networking and getting to know people is crucial – There is such an important emphasis on skill in the technical world, that knowing people and their actual abilities is vital. Also being know for your set of knowledge is important. That way when you want to learn something new, you know who to seek out. Before becoming Dating DNA’s CTO, and the rumor went out I was considering a new job, I had a lot of people contact me to see if I was interested in certain positions. There is no down side to being a “social” developer. Just because this isn’t “Marketing” doesn’t mean social networking isn’t important. I personally don’t like the term “networking” since it makes it seem like a chore. Make lots of friends in the programming groups and circles in your area and community, and it will be beneficial.
• Love what you do – If you don’t love what you do, then there is a good chance you will not go far in this industry. It doesn’t mean this is the only thing you do. Other hobbies and activities are important. But if you dread going to work, and do your work, every day, then it’s time to find something else.

This went a lot later in the evening than I thought, so I hope my thoughts are coherent, and if nothing else, entertaining. Its been a great deal of fun since those first days with Photoshop 3 trying to design something that didn’t look terrible. I look forward to the next few decades to see where web technology takes us.

Related posts:

1. So You Want To Be A Web Developer?
2. Web Development 10-Years Ago & Now
3. Being a Productive Developer

A Little Background

On January 2nd, we started to run into some serious performance problems with Dating DNA, so we began the process of going and deciding what were our problems, and how to optimize them. At the beginning, the Database was the bottleneck. It would get flooded with requests, and unable to handle them all, each query would slow down. We were using a tool called Jet Profiler (which I will post about more in detail later), but here is a graph it would output before we started out optimizations:

The light blue is total threads connected. The dark blue is thread running a query. The red are threads that are taking 2 seconds or longer to run, which are slow queries. The red is bad, very bad. So we were getting in bad shape. It was lovingly nick named “The Red Zone” while we were working on optimizations. Now, granted, we weren’t in “The Red Zone” the whole time, but when things got busy, things would slow down.

But optimization after optimization, we started to get things more and more under control:

After about three days of optimizations, we get back down to a manageable load:

The Problem

However, when the website was having high traffic, we noticed an anomaly, which looked like “blue waves.” We lovingly gave them the nickname of “blue meanies” from the Beatles’ Yellow Submarine Cartoon. On Jet Profiler, here is what it would report:

It would get worse and worse, these big blue waves of connections reporting as “sleeping.” At first we didn’t think it would be a problem. However, when ever we had “blue meanies” the site and iPhone app felt really slow. So, I won’t cover all the things I tested and tried that didn’t work, but he is ultimately how we figured out our problem.

The Solution

At first, I thought it was an issue with Garbage Collection with PHP. So I set the wait_timeout on MySQL to something really low, like 5 seconds. We then started to get errors all over the website, so we knew that they were legitimate connections from PHP. The only thing that made sense is that PHP & Apache now had become the bottle neck, that MySQL was returning requests so quickly that the threads were almost always sleeping, waiting for the PHP to finish. We slowly started to disable different functions on the website, trying to narrow down if there was a particular part of the website that was causing it. After a few hours, we figured out the feature: the ChatWalls. So we started to investigate why turning off the ChatWalls would make MySQL run faster, since we had moved the ChatWalls completely off MySQL and to run on Redis.

What we found is one particular function had a typo, that would cause PHP to iterate over an array not 10 to 20 times, but 1,000-2,000 times or more. This function was also called a lot by several Ajax calls. So, I fixed the typo, and the blue waves went away.

What was happening is Apache & PHP were spending so much time processing the buggy function, that it would cause the rest of the web requests to slow down greatly. That would keep open may too many MySQL connections, causing the blue ways, and slowing down the website even more.

So in reality, the blue waves were a symptom of the problem, not the cause. It is the whole Correlation vs Causation situation (which I probably should blog about in more detail when it comes to finding performance issues).

So if you have a lot of sleeping connections, but MySQL is performing well, most likely it is PHP or Apache slowing things down. I hope this can help those having a similar problem. As for our Database, its working well now. A few more problems to iron out, but it is running really fast. The few red spikes are from the score generation system that are doing bulk inserts, and do not slow down the end user experience:

Related posts:

1. MySQL & PHP – SQL_CALC_FOUND_ROWS – An easy way to get the total number of rows regardless of LIMIT
2. MySQL – Does Table Exist w/o Throwing Errors
3. MySQL, 40 Million Rows, MyISAM to InnoDB, 45 Minutes

This last week has been a roller coaster for us at Dating DNA. We had an excellent holiday season, but with mass volume of new sign ups our servers started to slow down. Severely. Something you never want to have happen. So over this last week I’ve worked about 80 hours (though I’m a salary employee, woohoo… ) and implemented a lot of new performance boots, and I have about four or five blog posts worth of discoveries I need to write. So before I forget them, I’m going to write them.

The first change I’d like to talk about is moving our ChatWalls (kind of like a Chatroom and Forums/Walls mashed up together) from a MySQL backend to a Redis backend (http://redis.io/). This initial change actually only took about 4-5 hours. Working out some additional kinks we found with our ChatWalls took another day or so, but the Migration to Redis was extremely smooth.

First, to give you an idea about our ChatWalls, here is a video of myself using them:

Redis, in a very simplified explanation, is like Memcache as a Key Value storage in memory. However, it is persistent (data not lost after restart), has more advanced data types (hashes, sorted sets), has built in virtual memory options (entire dataset doesn’t have to always be in memory), and you have a lot of cool operations beyond what memcache has (return sorted sets by score, purge out old records from a set, etc).

Now, I’ll walk through the basics of installing Redis, getting it up and running, reading and setting data in Redis, and migrating our old content.

Install Redis

Ok, compiling and installing Redis is a dream. It is very easy. On this server it is running Ubuntu 8.04 LTS. You go to the Redis Website and click on downloads. Get the latest stable version, and download it to the server (I used wget). Untar it (tar -zxvf /path/to/file.tgz), and go into the source code’s directory. Now, Redis uses a noticeably less amount of memory when compiled for 32bit versus 64bit due to pointer sizes (see FAQ). So to compile on a 64bit linux machine, you need to install libc6-dev-i386, which on ubuntu was “aptitude install libc6-dev-i386″.

Then, it was a simple “make 32bit” and “make install” and it compiled and installed redis in /usr/local/bin for me. Now, I wanted it to start up for me automatically when I rebooted the server, so I saved this init.d script as /etc/init.d/redis. Then I copied the sample configuration file out of the source file to /etc/redis/redis.conf. I ran “sysv-rc-conf” to set the Runlevels for it to execute the script at (2, 3, 4, and 5). If you don’t see “redis” listed, make sure you chmod +x /etc/init.d/redis so that it is executable.

Then I ran the command “/etc/init.d/redis start” and it was running.

Redis & PHP – Rediska

Now, while looking for a PHP Redis library I found Rediska. The documentation was pretty basic and straight forward, and I was able to get it installed pretty easy into my PHP App. Now, this library isn’t perfect, I found a few bugs while implementing it, especially with some values legal values for ZRANGEBYSCORE as defined by the Redis documentation, but Rediska would throw exceptions because I was passing non-int values. I’ll submit some bugs with patches to them next week so they can update Rediska.

A few things to look over in the Rediska library: first off, read how they manage multiple instances of their Rediska class. Its a little odd, but it works. I just don’t like passing an array with options in every command I have to make. So I would try something like $rediska->getHash(‘key_value’); and expect it to return a Rediska_Key_Hash object, but it would just return an array. So, in the end, I ended up using very little of the other Rediska classes, and stuck to just using the Rediska class and it’s methods. The good thing is they have good phpDocs, so if you have a good IDE, it will be easy to know which method does what.

Designing for a Key-Value Database Store

Now, I highly recommend reading this article on the different Redis Data Types. Also, give a look over their PHP Twitter Close with Redis, since they show some of the techniques for using a Key-Value as a data store.

So the very, very first thing you must do, and we did, is map out how you will design your application for Redis. You don’t have tables and columns any more, just a really big array with some cool tricks. So unlike MySQL if you have a typo in your table or field name, you won’t get errors. It will just run, but you application will have some serious bugs. So pick a naming convention, and document exactly the “tables” and “keys” you’ll be using, with what data types, and stick to it. If you don’t, you will be hating life. >/soap_box<

I like the following naming convention. Separate word groups by periods (.) and preface changing variables (like an ID) with a colon (:). So, here some map definitions for our keys:

chatwall.viewers > Hash[user_id] viewer_json – A hashtable with a key of the user id and a value of the member’s json object. This will hold the entire list of all users.

chatwall.wall:{wall_id}.viewers > Hash[user_id] viewer_json – A Hashtable that will hold a list of all the viewers for a given room. Example key: chatwall.wall:382.viewers

chatwall.nextPostId > Int – A int we will auto increment to get unique post IDs

chatwall.posts > Hash[post_id] post_json – The data for a post in a json object.

chatwall.wall:{wall_id}.posts > SortedSet We will store the score AND member as the Post ID. This will allow us to quickly get the last 50 posts in a set, and even pass a minimum Post ID to only get new ones.

[...]

cache.chatwall:{wall_id}.information > Json String – This will hold a json string of the cached wall information from MySQL.

These were just some of the definitions. Now, these definitions are for documentation only. You can use whatever naming conventions on the fly and Redis will use it. But it will be extremely useful if you have a list of what the key names are.

We also prefaced any “cached” data, so data that we treat like it was in memcached, with the string “cache.” The reason being if we wanted to clear any cached data, we easily could do so with a custom script. But we don’t want to confuse any of our good data we want to stay persistent being deleted on accident.

Refactoring the Code

Now, one thing that I was extremely grateful for when I designed the ChatWall system is each type of data had a PHP class with members, and functions that would perform the CRUD operations. So it was extremely easy to change the CRUD functions to read/write to Redis, and instead of writing out a query, I generated a key and passed for the value the variable $this. Super easy. Having all the data access code encapsulated correctly made it easy to simply go down my class methods and re-write them.

Now, there were a few caveats. One was moving posts, because an admin could move a post to a different forum or chatroom. Because we depended on Indexes from MySQL to catch the move, on our update commands we had to check the old record first to see if the room moved. If it did, we would have to remove an entry from the old chatwall.wall:{id}.posts and put it in the new one.

Migrating the Data

I simply wrote a script to select the data from our MySQL tables, set them to the new Classes, and called the “InsertRecord” function. In 10 seconds the script had migrated all of our data to Redis, which was taking up 25M of data in Redis. Pretty awesome.

Performance

I couldn’t be happier. It is really, really fast now. We had to double check our php code to prevent any calls to the Database, and make sure Redis had all the information it needed. Once we fixed a few areas, and our ChatWalls calls were being all 100% being executed against Redis, it was blazing fast. I am really, really happy with the change.

I would recommend getting and installing the redis-tools for your server. They make it really easy to see what your Redis server is doing. Our server is currently serving up 200-500 requests per second, and using up about 2-5% of our CPU with 20-30MB of ram. The background saves take 2-3 seconds to perform.

Bottom line, I highly recommend checking out Redis. I wouldn’t recommend replacing MySQL with it, but using it along side MySQL to handle cached data and highly accessed/changed data. It has been a huge performance boost for Dating DNA. We are in the process of finishing migrating our Match Score System to Redis, which is quite a bigger project. 500 million rows big. But it is going great, and looking to finishing with that project soon.

Related posts:

1. Large MySQL Conversion & InnoDB
2. MySQL & PHP – SQL_CALC_FOUND_ROWS – An easy way to get the total number of rows regardless of LIMIT
3. Memcached: Simple, Effective, and Powerful

In web development, one big area where I don’t see us using a lot is branching, tagging, and merging. However, with our iPhone Apps, we use tagging and branching a lot. As I started to think why, one of the biggest things was the environment. Client development, especially with the iPhone, is double clicking on a project file for Xcode, and I’m ready to go. There is almost no hassle running trunk versus running a tag or branch. It is all the same.

Building against a website, however, is different. Your environment requires a lot more information and pieces. For me, I need to make changes to my host file (for custom domains), Apache/Nginx configuration, MySQL Connection information, etc. There is no file i “double click” and go. It is a pain anytime I need to change my local (or staging or even production) paths and settings.

This is where the svn switch command comes in handy, and is extremely important. It allows you to change your checked out working copy’s source (or url), while maintaining any local changes you may have made. You can tell the actual path, regardless of file or directory names, by doing an svn info command. The best way to understand is through examples.

Arg, should have made this a branch.

You’re working on a big change, such as completely rewriting your internal message system. Suddenly, your boss walks in saying “There is a bug we need you to fix and push live now!” It is a simple bug, but your trunk is only half way through a major change. What do you do?

1. New Branch on Server – You need to make a new branch on the server based off the server’s version of trunk. It’s a simple copy function, while having it execute on the server:

   svn copy -m “Branching for new email system” http://svn.example.com/trunk/ http://svn.example.com/branches/new_email_system/

2. Switch your trunk working copy to new branch – This allows you to keep all of your current changes locally, but switches you to the branch version:

   cd /path/to/trunk/
   svn switch http://svn.example.com/branches/new_email_system/

3. Commit changes to branch – Commit your changes to the repository which will be applied to the branch:

   svn commit -m “changes to new message system.”

4. Switch back to trunk. – Make sure you are still at your trunk. If you did an svn info, it would show the URL location at the branch. Execute the switch again:

   svn switch http://svn.example.com/trunk/

5. Switch between trunk & branch as needed – When you need to work on the trunk, switch to the trunk. When you need to work on the branch, switch to the branch. The great thing is you won’t need to change any URLs in your configs in apache and such, it will just work.
6. When your branch is ready, merge into trunk – Perform a merge since the starting of the branch to the last revision in the branch, applying it to the trunk:

   svn merge -r 100:125 path/to/branches/new_email_system path/to/trunk

7. Test changes made by merge, then commit. – Test the changes locally, and when you are satisfied things went well, commit the merged changes to trunk: svn commit -m “new message”
8. Continue Development – you are safe to proceed as usual.

Over the new few weeks I plan on posting more little tutorials and tricks of some more advanved SVN commands, and hopefully it can help some of my fellow team members. Let me know if you have any questions, or want to know how to do other things. But the bottom line is if you haven’t used the svn switch command, especially in web development, you are sorely missing out.

Related posts:

1. Zend Studio vs PHP Development Tools
2. Speaking: Utah PHP Usergroup – Streamlined Web Development
3. Streamlined Web Development: In-Depth

Well, Facebook has made their announcement, and it is called HipHop. Here is an excerpt:

Today I’m excited to share the project a small team of amazing people and I have been working on for the past two years; HipHop for PHP. With HipHop we’ve reduced the CPU usage on our Web servers on average by about fifty percent, depending on the page. Less CPU means fewer servers, which means less overhead. This project has had a tremendous impact on Facebook. We feel the Web at large can benefit from HipHop, so we are releasing it as open source this evening in hope that it brings a new focus toward scaling large complex websites with PHP. While HipHop has shown us incredible results, it’s certainly not complete and you should be comfortable with beta software before trying it out.

HipHop for PHP isn’t technically a compiler itself. Rather it is a source code transformer. HipHop programmatically transforms your PHP source code into highly optimized C++ and then uses g++ to compile it. HipHop executes the source code in a semantically equivalent manner and sacrifices some rarely used features — such as eval() — in exchange for improved performance. HipHop includes a code transformer, a reimplementation of PHP’s runtime system, and a rewrite of many common PHP Extensions to take advantage of these performance optimizations.

Marco Tabini has a nice summary on what PHP need to know about HipHop. However, people I’ve talked to have had several opinions on HipHop, ranging from pure delight to doubts to indifference. Personally I think this is a great thing for PHP, and I’ll tell you why.

HipHop will enabled business to pick a web scripting language and use it from start-up to internet-giant. While HipHop isn’t the first tool used to speed up PHP, such as APC, Memcached, and others; it is more of a current reminder: PHP is serious about the web. For people already using PHP, this is just a nice feature. For those who aren’t using PHP, it shows that it is a very strong candidate.

I’ve worked in a non LAMP shop before. To non-PHP people, it has the stigma, regardless of actual performance, of being hard to scale. It is a “scripted language”, and in the Java / .NET world where I was working, it was looked at as being an absurd choice. Now, PHP can make a strong case of being able to mature with your application and scale appropriately.

One fear that I have as a Project Manager, and anyone has who decided on the technology that they will implement, is “picking the wrong team.” During my .NET days, Microsoft’s team announced a new technology for .NET applications. It showed a lot of promise and simplified some things that were a pain in .NET. Our team adopted the new technology and ran with it. It wasn’t perfect, but we liked it. Less than a year later, Microsoft announced that it would be moving the project to a drastically different team, which basically meant “there wasn’t enough adoption, so we’re putting this project on the back burner.” The new team was basically to maintain the code forward, but not develop new features.

What happened is we invested on this new technology, and we saw the potential it would have in the future. We lost our investment, since that future potential would never be realized. Now our projects had this new technology embedded in many spots. Re-writing those places would take a great deal of time. From a Project Manager perspective, it was a poor choice.

For PHP, the emergence of HipHop, a concept that is working across thousands of production servers, it sends a strong message to the web development community as a whole. Not only is PHP widely adopted, flexible yet powerful, and easy to implement. PHP is competitive, and is continually evolving to adapt to web developer’s needs. It is a safe bet for many projects, and it isn’t going anywhere but up.

Will I be implementing HipHop right now? Probably not. Ilia Alshanetsky has a very healthy reality check on what HipHop is for. But the bottom line is I don’t need what HipHop is offering yet. I can see very great uses in the future for Dating DNA’s compatibility generation, as well as certain pieces of other PHP projects that could use a serious performance boost. But for now, my Blog will not be HipHoping along. I’m excited for not only what HipHop does, but what it’s greater underlining meaning as a whole with PHP.

Related posts:

1. Characteristics of Good PHP Code
2. So You Want To Be A Web Developer?
3. Web Development 10-Years Ago & Now

Dear Google Apps admin,​

In order to continue to improve our products and deliver more sophisticated features and performance, we are harnessing some of the latest improvements in web browser technology. This includes faster JavaScript processing and new standards like HTML5. As a result, over the course of 2010, we will be phasing out support for Microsoft Internet Explorer 6.0 ​as well as other older browsers that are not supported by their own manufacturers.

We plan to begin phasing out support of these older browsers on the Google Docs suite and the Google Sites editor on March 1, 2010. After that point, certain functionality within these applications may have higher latency and may not work correctly in these older browsers. Later in 2010, we will start to phase out support for these browsers for Google Mail and Google Calendar.

Google Apps will continue to support Internet Explorer 7.0 and above, Firefox 3.0 and above, Google Chrome 4.0 and above, and Safari 3.0 and above.

Starting this week, users on these older browsers will see a message in Google Docs and the Google Sites editor explaining this change and asking them to upgrade their browser. We will also alert you again closer to March 1 to remind you of this change.

In 2009, the Google Apps team delivered more than 100 improvements to enhance your product experience. We are aiming to beat that in 2010 and continue to deliver the best and most innovative collaboration products for businesses.

Thank you for your continued support!

Sincerely,

The Google Apps team

For all Web Developers around the world, we should all be rejoicing! Having a large internet giant telling its consumers to “drop IE 6″ and other older browsers will make all our lives easy. I remember while working on a very large site, we used PNG images sparely for some background images. The deadline was extremely tight, but we were (barely) on schedule. Out of the blue I get an email forwarded to me from the marketing director for the client, and it went something like this:

Why are all the backgrounds looking nothing like the mock? Its all a funny blue.

Of course, the Marketing Director had IE 6. Whats worse, is that he was working for a very large computer technology business. We had to drop everything and apply ugly IE 6 hacks to enable PNG support. At the time, I tried explaining that the browser he was using was released while I was still in highschool, but he didn’t understand. Hopefully now the big internet giants will start telling people “Get off IE 6.” Once people adapt better browsers, the more we can do with the web.

Related posts:

1. Why Google Chrome is Good for FireFox
2. Is Google following Microsoft’s footsteps?
3. Google Wave Jargon

First, everyone does everything. Check your ego at the door. In the first startup I was with, I was in charge of adding reporting to the eCommerce application and taking the garbage out. Was it an effective use of my time? Not at all. Did it have to be done? Yes. Did we have the money to hire someone else? Nope. If your startup is renting office space, you probably won’t have to worry about this… but who signs for packages? Who re-orders coffee? On the flip side, you could be the one taking technical questions at the next investor pitch or invited to the big customer Christmas party.

I thought he hit the nail on the head with the article. Over the last few years I’ve worked with three startups, one of which was successfully bought by a large corporation. The other two have successfully gained some good revenue, but are still in the “Looking for More Money” phase, and I continue to do work for them on a contractual basis.

My only concern was that at the end of the article, he only mentions that he is still okay with working for a startup. He didn’t mention some of the biggest “perks” I consider working for a startup. So here are some of the items that, if you can handle the “draw backs”, are some great things. These are not “guaranteed”, as each startup is different, but typically you’ll find some of these.

• Accelerated Learning – I’ve felt that I’ve learned at such a faster rate while working in a startup. In large development teams, you typically work on very specific things. Your team only works on one part of a larger project or product. However, with the startup, you typically have to do a lot more varied work. At Ambient, the company that was sold, I was a junior developer of a company of three developers. Here is the list of some of stuff I did in roughly one year:
  • Company Website
  • All Web Services
  • All Web Designs
  • Website Demos & Prototypes
  • Data Mining Services
  • Executive Dashboard
  • Sales & Inventory Reporting Website
  • Customizable Online DVD Rental System

  Not only did I work almost exclusively on these projects, they integrated with the pieces the other two developers worked on. For three developers, we cranked out a lot of great software. Since I was the junior developer, when ever we had a tech support issue that was a little too complicated for our two tech support guys, I had to diagnose and solve a lot of complicated issues. I got experience with VPNs, LDAP and Active Directory, very large databases (100,000,000+ rows), VNC & RDP, and automated deployments across thousands of remote locations, to name a few.

  If you enjoy getting your hands on a lot of different technologies, and do a lot more than in a normal company, you can really learn a lot at a startup.

• Less Politics, More Work – While I know this doesn’t apply to all startups, the ones I’ve worked in seem to have a lot less politics than other companies. I believe its because, first of all, there are less people to have politics with. Second, there is way to much work to get done to spend time on playing politics. Everyone seems to just be focused on getting work done.
• More Input, More Direction – While working in a startup, I’ve found that I have a lot more say in how we do things compared to other companies. Creativity isn’t just a bonus, it is a requirement. Many times you are working on an idea, not a detailed specification. Your creativity is required to make the idea fully functional and work properly. I’ve found many times my ideas are implemented, and while looking at the final product, I can see my influences all over the place.
• Closer Team, Fewer Annoying Co-workers – Some may think this is a myth, however what I have experienced is startups don’t have time to put up with teams that don’t work well together. For the exception of one guy who was really nice, but a major pain to work with, I’ve enjoyed working with all my co-workers and (I hope they can say the same about me). Typically people who don’t enjoy the fast pace leave after a short while. If there is an employee who doesn’t get along with the others, typically they will go find work elsewhere because the stress of the pace and not getting along well with others compound out of control. Even if they don’t leave on their own, management will usually find some solution, because a team that doesn’t work well together won’t produce the needed products to make a success. Unlike a large corporation, there isn’t anywhere to hide bad workers. If you’re not carrying your own weight, you’re gone.
• No Boredom, High Adrenaline - The pace can be stressful, but I found it most of the time exhilarating. There is never, ever a lack of work. Currently for the two startups I’m doing work for, there are years and years worth of work I can do. Sometimes the deadlines can be a killer, but meeting them is very rewarding. Its an awesome feeling when your team of two developers and one project manager create a product that out performs & demolishes competing products from large, well established companies.
• High Levels of Satisfaction – I personally feel I get more out of working for a successful startup. Since developers will produce more code and applications than in other companies, there is more to be proud of. If you’re team is really tight, and a few of you create an amazing product, it is an amazing feeling.

So while startups are not for everyone, and they have their drawbacks, I believe certain types of developers can find rewarding work in them. I know I’ve been lucky to work on some excellent products, and I haven’t had an experience (yet) where I’ve gotten burned. Maybe the things I’ve mentioned aren’t as common as I think, but I truly believe startups offer opportunities for developers to really accelerate their learning and excel at rewards jobs.

Related posts:

1. Software Development With Clients In Mind
2. Article: Why Developers Get Fired
3. OS X and Tabs – Skipping Drop Down Controls

If you have any questions or comments, please feel free to leave them below.

Related posts:

1. Steamlined Web Development Presentation Video
2. 100th Blog Entry – First Year of Real Blogging
3. A New Stage in Life: My MacBook Pro

I have learned the hard way over the years that it doesn’t matter how clever you think you are or how much everyone loves you on your team. The fact is, there are always potential circumstances that can come to a head, resulting in the ax coming down on your valuable head.

Sometimes these doomsday series of events are in your control. Sometimes it’s like being blindsided by a bus.

…

We might as well start with the obvious. If your job consistently isn’t getting done, then you will eventually be toast. All it takes is a few missed deadlines and your manager will have no choice because “you know what” rolls downhill. A manager can only absorb so many blows from unhappy end users or their own boss.

Why wouldn’t you see this one coming? Well, you might be thinking the missed deadlines are not your fault. Your excuses may include “the design was bad” or “the deadlines are not realistic” or “they are making me code in Java and I am a .NET expert.”

Guess what? Excuses don’t matter. Results matter.

While I hope I never get fired from a job, hopefully these tips can help any developer out.

Related posts:

1. Great PHP Article: Leadership in Software Development
2. PHP Article: 10 Principles of the PHP Masters
3. Article: 5 Things Your Clients Should Know