ob_flush(); // Flush anything that might be in the header output buffer
flush(); // Send contents so far to the browser

Well, with Nginx, it will wait for an entire response from the PHP-FPM instance before sending data to the browser. This is because traditionally the time to generate the response is less than the time to send the response to the browser, so Nginx will let the CGI instance finish as quick as possible to free it up for other requests. Well, even if I called ob_flush() and flush(), Nginx would wait for the entire response before sending it to the client’s browser. So, for our staff panel, I had to disable this buffering. It took a lot of scouring the web, but I finally figured it out:

Nginx Configuration

You need to set a few variables. I couldn’t actually figure out how to disable the buffer for Nginx, but I could set it very low on a per location configuration.

So I have the following location configurations:

The important configurations are:

So I set the fastcgi_buffer_size and buffers to 1k. Then, you need to set the max temp file size to 0. Nginx by default will start to buffer on the disk. By setting it to 0 it will send it to the browser. The last piece of the puzzle I missed was turning gzip off, because it would try to buffer, even if it exceeded 1k, to compress the response.

Now, to get it to work, in my php script I echo out 1k worth of html commented out text to ensure everything else after it gets sent to the browser.

Now, I don’t recommend these settings for busy productions servers. However, only 3 people use this staff panel, so the performance differences are extremely low.

Related posts:

1. PHP Practices: Buffering your autoload
2. SwiftMailer – Sending Mail in PHP With Ease
3. Local LAMP Developement & User Content

A Little Background

On January 2nd, we started to run into some serious performance problems with Dating DNA, so we began the process of going and deciding what were our problems, and how to optimize them. At the beginning, the Database was the bottleneck. It would get flooded with requests, and unable to handle them all, each query would slow down. We were using a tool called Jet Profiler (which I will post about more in detail later), but here is a graph it would output before we started out optimizations:

The light blue is total threads connected. The dark blue is thread running a query. The red are threads that are taking 2 seconds or longer to run, which are slow queries. The red is bad, very bad. So we were getting in bad shape. It was lovingly nick named “The Red Zone” while we were working on optimizations. Now, granted, we weren’t in “The Red Zone” the whole time, but when things got busy, things would slow down.

But optimization after optimization, we started to get things more and more under control:

After about three days of optimizations, we get back down to a manageable load:

The Problem

However, when the website was having high traffic, we noticed an anomaly, which looked like “blue waves.” We lovingly gave them the nickname of “blue meanies” from the Beatles’ Yellow Submarine Cartoon. On Jet Profiler, here is what it would report:

It would get worse and worse, these big blue waves of connections reporting as “sleeping.” At first we didn’t think it would be a problem. However, when ever we had “blue meanies” the site and iPhone app felt really slow. So, I won’t cover all the things I tested and tried that didn’t work, but he is ultimately how we figured out our problem.

The Solution

At first, I thought it was an issue with Garbage Collection with PHP. So I set the wait_timeout on MySQL to something really low, like 5 seconds. We then started to get errors all over the website, so we knew that they were legitimate connections from PHP. The only thing that made sense is that PHP & Apache now had become the bottle neck, that MySQL was returning requests so quickly that the threads were almost always sleeping, waiting for the PHP to finish. We slowly started to disable different functions on the website, trying to narrow down if there was a particular part of the website that was causing it. After a few hours, we figured out the feature: the ChatWalls. So we started to investigate why turning off the ChatWalls would make MySQL run faster, since we had moved the ChatWalls completely off MySQL and to run on Redis.

What we found is one particular function had a typo, that would cause PHP to iterate over an array not 10 to 20 times, but 1,000-2,000 times or more. This function was also called a lot by several Ajax calls. So, I fixed the typo, and the blue waves went away.

What was happening is Apache & PHP were spending so much time processing the buggy function, that it would cause the rest of the web requests to slow down greatly. That would keep open may too many MySQL connections, causing the blue ways, and slowing down the website even more.

So in reality, the blue waves were a symptom of the problem, not the cause. It is the whole Correlation vs Causation situation (which I probably should blog about in more detail when it comes to finding performance issues).

So if you have a lot of sleeping connections, but MySQL is performing well, most likely it is PHP or Apache slowing things down. I hope this can help those having a similar problem. As for our Database, its working well now. A few more problems to iron out, but it is running really fast. The few red spikes are from the score generation system that are doing bulk inserts, and do not slow down the end user experience:

Related posts:

1. MySQL & PHP – SQL_CALC_FOUND_ROWS – An easy way to get the total number of rows regardless of LIMIT
2. MySQL – Does Table Exist w/o Throwing Errors
3. MySQL, 40 Million Rows, MyISAM to InnoDB, 45 Minutes

In web development, one big area where I don’t see us using a lot is branching, tagging, and merging. However, with our iPhone Apps, we use tagging and branching a lot. As I started to think why, one of the biggest things was the environment. Client development, especially with the iPhone, is double clicking on a project file for Xcode, and I’m ready to go. There is almost no hassle running trunk versus running a tag or branch. It is all the same.

Building against a website, however, is different. Your environment requires a lot more information and pieces. For me, I need to make changes to my host file (for custom domains), Apache/Nginx configuration, MySQL Connection information, etc. There is no file i “double click” and go. It is a pain anytime I need to change my local (or staging or even production) paths and settings.

This is where the svn switch command comes in handy, and is extremely important. It allows you to change your checked out working copy’s source (or url), while maintaining any local changes you may have made. You can tell the actual path, regardless of file or directory names, by doing an svn info command. The best way to understand is through examples.

Arg, should have made this a branch.

You’re working on a big change, such as completely rewriting your internal message system. Suddenly, your boss walks in saying “There is a bug we need you to fix and push live now!” It is a simple bug, but your trunk is only half way through a major change. What do you do?

1. New Branch on Server – You need to make a new branch on the server based off the server’s version of trunk. It’s a simple copy function, while having it execute on the server:

   svn copy -m “Branching for new email system” http://svn.example.com/trunk/ http://svn.example.com/branches/new_email_system/

2. Switch your trunk working copy to new branch – This allows you to keep all of your current changes locally, but switches you to the branch version:

   cd /path/to/trunk/
   svn switch http://svn.example.com/branches/new_email_system/

3. Commit changes to branch – Commit your changes to the repository which will be applied to the branch:

   svn commit -m “changes to new message system.”

4. Switch back to trunk. – Make sure you are still at your trunk. If you did an svn info, it would show the URL location at the branch. Execute the switch again:

   svn switch http://svn.example.com/trunk/

5. Switch between trunk & branch as needed – When you need to work on the trunk, switch to the trunk. When you need to work on the branch, switch to the branch. The great thing is you won’t need to change any URLs in your configs in apache and such, it will just work.
6. When your branch is ready, merge into trunk – Perform a merge since the starting of the branch to the last revision in the branch, applying it to the trunk:

   svn merge -r 100:125 path/to/branches/new_email_system path/to/trunk

7. Test changes made by merge, then commit. – Test the changes locally, and when you are satisfied things went well, commit the merged changes to trunk: svn commit -m “new message”
8. Continue Development – you are safe to proceed as usual.

Over the new few weeks I plan on posting more little tutorials and tricks of some more advanved SVN commands, and hopefully it can help some of my fellow team members. Let me know if you have any questions, or want to know how to do other things. But the bottom line is if you haven’t used the svn switch command, especially in web development, you are sorely missing out.

Related posts:

1. Zend Studio vs PHP Development Tools
2. Speaking: Utah PHP Usergroup – Streamlined Web Development
3. Streamlined Web Development: In-Depth

Installing just XAMPP and getting it to run with the Control Panel was easy. All my headaches started when I tried to get my vhosts installed and working. The problem I ran into getting XAMPP to run on my MacBook Pro were the following:

1. XAMPP for Mac is different from XAMPP for Windows
2. Max OS X comes with a Version of Apache Installed
   
3. Mac OS X has additional permissions that are different from Linux

XAMPP for Mac != XAMPP for Windows

My first mistake was that I assumed that XAMPP for Mac is the same as XAMPP for Windows. After digging into it, the file structure and config files are pretty different. The php.ini file is different, and all the binary files are in /xampp/xamppfiles/bin/. On windows the binary files are in their own folders. It took me awhile to figure out where everything was at. This led me into my next problem:

Mac OS X comes with a Version of Apache Installed

When I was trying to debug why stuff wasn’t working, I opened up the terminal and started running commands. There is just one problem, OS X already has a version of Apache installed. So when I cd’d into the bin directory, I would execute this command:

apachectrl -t -D DUMP_VHOSTS

There was just one problem, it was executing the apachectl that was installed by Mac OS X, not the one for XAMPP. So it all my debugging with the config files, etc. wasn’t working. I was going insane. Finally I figured out what was going on and I had to execute this command instead:

./apachectl -t -D DUMP_VHOSTS

It started behaving like I expected! This lead my to my third hang up.

Mac OS X has additional permissions that are different from Linux

This threw me for a loop to. I don’t understand exactly why this was happening, but I kept gettinga  Permission Denied when trying to view the vhost I was trying to setup. The way I used to do it on Windows is I have a partition for all of my development files. I would have a folder at the root of that drive called SVN_Repositories that holds all the SVN Repositories I use for work. So on my MacBook Pro at / I did the same, I had my SVN_Repositories folder. So my vhost would point to a directory like with a name something like this: “/SVN_Repositories/ExampleRepo/trunk/httpdocs/”

So my linux side clicked in and thought “Hrm, it looks like there is something wrong with the permissions. Let me change the folder permissions to 777 just to test.” So I did that and it still didn’t work. This it made me think it still was a vhost configuration issue. So I spent hours testing all sorts of stuff and just getting more frustrating. Then I read somewhere that XAMPP can have problems with OS X’s unique permissions on addition to the FreeBSD stuff it does. It said to host the vhost content in either the Applications folder or Users folder. So I moved my SVN_Repositories folder to my Users folder so it was like this: “/Users/username/SVN_Repositories/ExampleRepo/trunk/httpdocs/”

It finally worked! So the lessons I learned were:

• Put your vhost content in the Users folder.
• If you try things from the terminal, make sure you do ./apachectl so you execute the correct binary file.
• Most of your config files are in the /Applications/xampp/etc/ folder.

As always, hopefully this helps someone. Feel free to leave any questions or comments.

Related posts:

1. PHP Video Tutorial – Getting Started – Installing XAMPP
2. Page.FindControl() Returning Null Issues and Solutions Within Another Control
3. Local LAMP Developement & User Content

When working with social networking sites, many times they allow for users to upload their own videos, pictures, etc. My current project wasn’t different, but I didn’t want to download 80GB of user content every week to store on my PC. I wanted a way to point my browser to the live server to get recent images, etc.

My solution was simple: .htaccess file using a 301 redirect. I basically told my local apache server to return a HTTP status of 301 (Moved Permanently) and point the browser to the live content. I created a .htaccess file inside the “uploads” directory and added one line of code:

I restarted apache and now when I work locally, I can still see all of the user’s photos and uploaded content. Works like a champ!

Related posts:

1. ASP .NET & System.Threading.ThreadAbortException Error
2. OS X and Tabs – Skipping Drop Down Controls