MySQL on Justin Carmony

Setting Up Percona Server 5.5 on Ubuntu 10.04

Mon, 24 Oct 2011 18:43:25 +0000

I’m deploying some new servers today, and I realized I hadn’t documented my steps on my blog. So here are a few posts detailing my setup process.

What is Percona Server?

Percona Server is a fork of MySQL Server. So what does that mean? For the past few years, MySQL has had some “interesting” developments from the business stand point. Being bought by Sun, which was then sold to Oracle, and a lot of “drama” surrounding all of that. In turn, what seems to have happened is MySQL’s development stalled or slowed down as it took a back seat to the legal sides of things.

MySQL, Redis, and a Billion Rows - A Love Story

Mon, 23 May 2011 15:20:35 +0000

This last week we pushed live a very large architecture change for Dating DNA. For those who know me, and have heard me talk about the Dating DNA Scoring System, they know how big of a problem we faced. For those who don’t know, let me give some background:

The Problem

With Dating DNA, our goal was to display a compatibility score with every other user. This score is generated by taking two sets of answers to our 20 page survey, and running it through our algorithm. While it is super convenient for our users, this poses a problem. We wanted not only for people to be able to visit a profile and see a score, which is easy to generate a score on demand. We wanted our users to be able to browse other profiles sorted by their score with them. This requires us to pre-generate and store these scores, and then later query them.

First PHP Project - Tournament Brackets

Tue, 12 Apr 2011 20:47:01 +0000

I mentioned on my twitter account that my very first project in PHP is used in production today, and I had a few people ask me what it was. So I thought I would show some examples of it in action. While I had done other things in PHP, this was the first time I had sat down and built someone by myself instead of just modifying someone else’s code. I built it for the very first time back in 2005, and since then have had to update it a few times. But it is still used regularly to host CEVO’s tournaments.

Restoring Large MySQL Dump - 900 Million Rows

Thu, 07 Apr 2011 03:05:12 +0000

This last weekend I had a fun opportunity of restoring roughly 912 Million Rows to a database. 902 Million belonged to one single table (902,966,645 rows, to be exact). To give an idea of growth, the last time I blogged about this database we had about 40 million rows. This giant table, the “Scores” table, is has a very small schema: two ints, a tiny int, and a DECIMAL(5,2).

Problem

Our current backup system uses mysqldump. It dumps a 25 GB sql dump file, which compresses to about 2.5GB using gzip. The last time we needed to restore a backup it was only about 9GB, and it took several hours.

How I Became a Web Developer & CTO

Wed, 23 Feb 2011 11:57:56 +0000

Note: many of the screenshots of websites I had done in the past are from the Internet Archive and many of the images are missing. But it can give an idea on the projects I had worked on. One day I’ll try and track down the original files and update the screenshots.

Today some memories came from when I first started with web development. So before I forget anymore of the details, I thought I would share my unique way I became a web developer. I never went to school to learn Computer Science or Information Systems. I’ve had a total of 2 programming classes, which were my in sophomore and junior years of High School, one of which we lovingly nicknamed Warcraft 101, because we would spend the entire time getting out butts kicked in Warcraft 2 by my classmate Daren (We all got A’s, we had just finished the entire coursework in 2 and a half weeks).

MySQL Stored Procedure, NAME_CONST, and Character Sets

Wed, 02 Feb 2011 07:12:35 +0000

For a Project I had helped work on, they recently needed to move servers and in the process upgraded from MySQL 5.0 to MySQL 5.1.41. They ran into some quirks with 5.1 and I thought I would document our work-a-rounds for them.

We were having serious performance problems with a Stored Procedure that would be called by a Windows Client on thousands of machines. Before on 5.0 it would take under 0.02 seconds, and now it was taking 25-35 seconds. In this stored procedure we would call a query something like this:

MySQL Sleeping Connections & PHP

Sat, 08 Jan 2011 20:51:48 +0000

If you want to skip to the explanation, just read below. But before then, here is a little background.

A Little Background

On January 2nd, we started to run into some serious performance problems with Dating DNA, so we began the process of going and deciding what were our problems, and how to optimize them. At the beginning, the Database was the bottleneck. It would get flooded with requests, and unable to handle them all, each query would slow down. We were using a tool called Jet Profiler (which I will post about more in detail later), but here is a graph it would output before we started out optimizations:

Creating Chatroom / Walls with Redis & PHP

Sat, 08 Jan 2011 00:15:16 +0000

Preface: This is not a step-by-step tutorial, but more of an outline on what I did. Also, I wrote this really quickly before heading off to dinner, so if there are parts that are unclear, or you have questions about, please leave a comment!

This last week has been a roller coaster for us at Dating DNA. We had an excellent holiday season, but with mass volume of new sign ups our servers started to slow down. Severely. Something you never want to have happen. So over this last week I’ve worked about 80 hours (though I’m a salary employee, woohoo… :P) and implemented a lot of new performance boots, and I have about four or five blog posts worth of discoveries I need to write. So before I forget them, I’m going to write them.

Large MySQL Conversion & InnoDB

Sat, 16 Oct 2010 03:01:32 +0000

Over the last week at Dating DNA, we’ve re-engineered our internal mail system that users use to send messages to each other. It took a few days longer to convert and re-engineer everything, but we finally made the change. With the exception of two small bugs, it all went off without a hitch. But before all the different things I learned doing it fade, I wanted to document them here.

First off, converting from one large set of InnoDB tables to a new structure. I’ve done something similar in the past, and I think I’ve honed in a little bit more the technique. So here are some thoughts regarding doing this process.

My 2009 Technology Recap

Wed, 23 Dec 2009 18:55:31 +0000

Its been two years now that I’ve been a more or less “serious blogger.” I had using the term blogger, since when people hear about blogs, they think of people either detailing their entire lives, or pumping some angel. For me, blogging has just been a way to share information with people I know locally, and those I don’t even know who find my blog via searching. I’ve had a few spurts where I blogged quite frequently, but for the most part it was just here and there.

Presentation: Real Life Scaling

Sun, 11 Oct 2009 18:39:44 +0000

Here are my slides for the presentation that I gave at the Utah Open Source Conference on Friday. It was an awesome conference, and I am glad that I was able to be involved.

Real Life Scaling: A Tale of Two Websites http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=reallifescaling-ataleoftwowebsites-small-091010143045-phpapp02&stripped_title=real-life-scaling-a-tale-of-two-websites

View more presentations from Justin Carmony.

If you have any questions or comments, please feel free to leave them below.

So You Want To Be A Web Developer?

Mon, 14 Sep 2009 14:54:35 +0000

This last week I’ve been talking with my brother Brack, who to say the least, absolutely abhors his job. Well, maybe abhor is a little strong, but he is extremely frustrated and unsatisfied. He works at a air force base in the human resource department and he basically helps provide statistical data for departments on who, what, and where to hire new help. Since he is a “wiz” at solving problems, computers, and making this more efficient, he has streamlined his work process with several nifty Excel spreadsheets and Java programs. He can now take a weeks worth of work and do it in one morning.

Data Backups - There Are No Excuses

Sun, 19 Apr 2009 00:04:48 +0000

Today I just had the terrible experience of having a database lose data, need to restore, only to not have a recent backup. If you haven’t had this experience before, please, take this serious. My wife was home for lunch as it happened, and she watched as the blood drained from my face. It only took a few seconds for the loss to happen, and immediately I knew exactly what the repercussions where. The immediate second thought that passes through your brain is “Where are my backups?” That is when I realized I didn’t have my nightly backups set up on this server. I quickly checked the file date on the last known backup I had.

XAMPP for Mac - My Frustrations & Solutions

Sat, 14 Feb 2009 23:15:34 +0000

These last few days I’ve been trying to get XAMPP to work on my MacBook Pro these last couple of days and it has been frustrating! It seemed I kept running into more and more problems. For those who don’t know, XAMPP is tool for developers to run a LAMP stack (Linux, Apache, MySQL, PHP) on your machine locally. I’ve been running it on my Windows machines for years, and I have been running my production servers on Linux.

MySQL, 40 Million Rows, MyISAM to InnoDB, 45 Minutes

Tue, 13 Jan 2009 05:13:08 +0000

Warning: This blog post is NOT a walk-through or tutorial. If you don’t know what you’re doing, you could seriously screw up your database. This is just talking theory and ideas on how I solved my problem.

The other night (really, the other morning) I had the wonderful pleasure of trying to convert a table with 40 millions from MyISAM to InnoDB. The reason for the conversion was this table had a high amount of queries, and a high amount of writes. With this table growing and updating daily, the table locks were killing us. Because MyISAM needs to peform a lock on the entire table, we would have dozens of queries backlog as locks and writes were taking place. It was causing the site to screech to a halt, waiting for the locks the be released. It was a total nightmare.

The Dangers of Twitter!

Thu, 08 Jan 2009 07:00:34 +0000

Over Christmas my wife and I visited my father in San Diego, CA. Several members of my dad’s side of the family were there, and one day we had a discussion about being “wired.” What does that exactly mean? Well, how fast would a person respond after any form of electronic communication were to be sent.

We decided, that hands down, my father was the most wired. One evening my wife and I stayed in the car while my father ran into a grocery store to pick up something quickly. I made a tweet about our trip to Barnes & Nobel, and how I spend so much money there. By the time my father came back to the car three minutes later, he commented to me about my tweet. He had set up that all my tweets are sent to his phone. I laughed and told him he definitely was the most wired.

MySQL - Does Table Exist w/o Throwing Errors

Tue, 04 Nov 2008 00:34:33 +0000

There are times where you would like to know if a table exists before executing an query. Most solutions require having MySQL throw an error saying “table does not exist,” but I prefer a cleaner way. I found on this forum post a clean way to do it:

/* example with table name: table_name */
SHOW TABLES LIKE 'table_name';

This solution will return 1 row if it exists, and 0 rows if it doesn’t. Here is a PHP example using this:

PHP Design - Biggest Database Oversights

Sat, 25 Oct 2008 07:56:44 +0000

Over the last three years I’ve had the opportunity to work on several PHP projects, some of them having grown rapidly and required to scale quickly. Three in particular have been a fantastic learning experience for me. Now I don’t consider myself a total expert, but I thought I would share my experiences. Many PHP developers are self-taught, like I was, and are learning PHP to be able to create dynamic websites. Many haven’t had studied or learned best practices for Object-Oriented Programming (OOP). They are web designers first, and programming in PHP is just a way to do things simple HTML can’t do.

Web Development 10-Years Ago & Now

Wed, 16 Jul 2008 01:43:55 +0000

This week’s blogging challenge was our computer software now and then. Lorelle talked about the different versions of WordPress over time. I thought it was an interesting article.

10 Years Ago

I started web development back in 1997. My tools back then were:

Notepad - That right, the one and only. It was how most people did their websites back in the day. It hasn’t changed much over the years.
Adobe Photoshop v4 - The must have program for image editing:
Dreamweaver 1.0 - Thats right, I’ve been with dreamweaver since the beginning! (Sorry, I couldn’t find a screenshot)
ws_ftp95 - This old school FTP program was the bomb in my eyes since day 1. Boy have things come a long.

Technology wise I was serving up static HTML pages. When I had mastered Dreamweaver’s template system, I was able to run a website with 20+ pages not too bad. It was definitely interesting. Every time I made a change, it would go through and update every single page, and then I would upload every page to the server.

MySQL & PHP – SQL_CALC_FOUND_ROWS – An easy way to get the total number of rows regardless of LIMIT

Tue, 01 Jul 2008 20:30:48 +0000

Here is a little trick I found awhile back. I faced a challenge several months ago where I had to query a large, complex data result from MySQL. I “paged” through the results using LIMIT and OFFSET. However, I wanted to know the total number of rows w/o the LIMIT. The previous coder accomplished this by a query before appending the LIMIT statement at the end. This turned out to be very inefficient due to MySQL put together the 10000+ rows and sent it to PHP ready to be parsed, only to have the function mysql_num_rows() to be executed on it.

LT Pact '08 - Overview of Experience

Fri, 27 Jun 2008 22:52:45 +0000

I just left the last session of the LT Pact ‘08 conference and it has been fun. I’ve learned a lot and there were a lot of common themes among the conference. I’ll briefly touch on the themes and over the next week or so I’ll be writing more in-depth on each one. Besides, I need to enjoy Las Vegas while I’m here and not just blog.

Virtualization & Cloud Computing

Virtualization is the future of server hosting. It won’t be optimal for all scenarios of course, but there are so many advantages to virtualization that they can’t be passed up: