System Administration on Justin Carmony

Tracking Your Data with StatsD, Graphite, and Graphene

Mon, 04 Jun 2012 05:04:23 +0000

Update: So I spelt ~~Neil~~ Neal’s name wrong. Sorry Neal! I also added some other links to the bottom that he mentioned to me.

Its been a little over 48 hours, and StatsD, Graphite, and Graphene have already paid off major dividends. While speaking at Tek12, Neal Anders was giving a talk on Graphite which I wanted to attend, but I was speaking at the same time I was speaking on Redis so I couldn’t. Then, a week or so after the conference, someone posted a link about StatsD on twitter (I can’t remember who), so I started poking around, loved what I saw, and within an hour I had it running on a server for Dating DNA friday night (I still do some contract work for them to help keep their infrastructure running well). I quickly added about a dozen metrics to track, and went to bed.

Video: Look At Your Data - John Rauser - Velocity 2011

Sun, 13 May 2012 20:36:25 +0000

Last year I saw some videos from Velocity 2011 that were really enlightening, and today I wanted to re-watch one of them. Of course, I hadn’t bookmarked them, or remembered which conference they were from. So after some searching and remembering, I found the video. “Look at Your Data” by John Rauser. It is an excellent video and understanding your performance data and how to get more from it. I’ve embedded it below, but I would encourage everyone to checkout all the videos they posted from Velocity 2011, its a bunch of great stuff.

SMS Nagios Notifications with PHP & Twilio

Tue, 31 Jan 2012 01:12:14 +0000

I’ve had a few requests to share my Nagios SMS notifications using Twilio. I’m almost embarrassed to share them, since they are so dead simple. There was another plugin out there to do the same, but it was a lot more advanced and more work to setup, so I wrote my own in PHP.

In the past I would just use my iPhone’s email-to-txt email address. However, when I received the txt message, it wasn’t formated very pretty, and it would have a different “From Number.” So if we had a crazy day, I would have 20-30 message threads in my iPhone all about Nagios. I’ve been lazy in the past, and didn’t clear them out until I had about 200 of them, and it was a lot of “swipe, delete, swipe, delete, swipe, delete” to get rid of them.

Setting Up Nginx & PHP-FPM on Ubuntu 10.04

Tue, 25 Oct 2011 03:50:59 +0000

This is another wonderful setup that I’ve found myself using rather than the traditional Apache & mod_php setup.

What is Nginx?

Nginx (pronounced engine-x) is a fast, powerful, lightweight web server. I won’t go into the theory under-the-hood, but it’s focus is high concurrency with low memory usage. So while Apache is more robust in supporting many different features, nginx focuses on handling the important features very quickly. I still use Apache internally for our SVN & Trac web server. Heck, even at this time I’m using Apache to host this blog. However, Dating DNA, Clipish, CEVO, Alienware Arena, and some other high traffic sites/apis use nginx.

Setting Up Percona Server 5.5 on Ubuntu 10.04

Mon, 24 Oct 2011 18:43:25 +0000

I’m deploying some new servers today, and I realized I hadn’t documented my steps on my blog. So here are a few posts detailing my setup process.

What is Percona Server?

Percona Server is a fork of MySQL Server. So what does that mean? For the past few years, MySQL has had some “interesting” developments from the business stand point. Being bought by Sun, which was then sold to Oracle, and a lot of “drama” surrounding all of that. In turn, what seems to have happened is MySQL’s development stalled or slowed down as it took a back seat to the legal sides of things.

Working with Middle-Scale Websites

Tue, 19 Jul 2011 00:02:40 +0000

I’ve been thinking about this idea for awhile, and I thought I would put a name to the thought. I brought up this idea while I was giving my “Real Life Scaling” presentation at the Utah Open Source Conference in 2009. Here is the problem I think most individuals in the web development face:

Hopefully at some point, your website gets a lot of traffic. Yay, you’ve reached your goal of getting good traffic, but it is soon followed by issues with performance and load. I like to call these the growing pains of a website. So as a web developer, I suddenly have the epiphany of “Hey, I need to scale my website!” What follows next is the biggest mistake a web developer can make:

Restoring Large MySQL Dump - 900 Million Rows

Thu, 07 Apr 2011 03:05:12 +0000

This last weekend I had a fun opportunity of restoring roughly 912 Million Rows to a database. 902 Million belonged to one single table (902,966,645 rows, to be exact). To give an idea of growth, the last time I blogged about this database we had about 40 million rows. This giant table, the “Scores” table, is has a very small schema: two ints, a tiny int, and a DECIMAL(5,2).

Problem

Our current backup system uses mysqldump. It dumps a 25 GB sql dump file, which compresses to about 2.5GB using gzip. The last time we needed to restore a backup it was only about 9GB, and it took several hours.

Sending Email from non-email Ubuntu Server

Wed, 30 Mar 2011 19:22:11 +0000

I have a couple of scripts that use the linux “mail” command to send results of cron jobs, backups, and such. The problem is I don’t want to setup more overhead than I need, nor do I want to setup non-secure services. So after asking the guys on the #uphpu channel, they recommended postfix. So for a Ubuntu 10.04 server, just run the following commands:

aptitude install postfix #when prompted, select Internet Site aptitude install mailutils

Setting up Nagios for Servers

Tue, 04 May 2010 21:17:20 +0000

Ok, so I need to confess, I haven’t been a good blogger. I’ve gone two months without writing a post, which is broke a 2 year streak. But, I’m going to post date these posts, since I can.

How I’ve gone this long without setting up Nagios to monitor servers is beyond me. However, I was tired of my clients and so finding out a server was offline before me. So I threw up a 256MB virtual server on Slicehost, since I don’t have any servers with them at the moment, in case a data center went offline.

VirtualBox Networking Setup

Mon, 22 Mar 2010 06:32:43 +0000

Every time I go to set up VirtualBox and several VMs I always forget how I setup the networking for them. Once again, I went through the pain of trying to figure it all out, so I might as well document it for future use.

The Problem

When creating virtual machines, especially virtual linux servers, I want them to be able to do the following things:

Connect to the Internet
Connect over SSH from my Host Machine to the Guest Machine
Have the servers connect to each other on a private internal network

Out of the box, a VM is configured for #1, however it cannot do #2 and #3. There are some solutions that do some archaic port forwarding by issuing special commands from the Terminal. This is just painful, and if you just want one port to work, that’s manageable, but what if you want ssh, ftp, myql, http, https, etc? Any on multiple VMs? What a nightmare! What is painful is when googling for a solution, almost every “answer” is this crazy port forwarding method. There is a much cleaner, better way!

The Cloud to Bare Metal

Mon, 22 Mar 2010 06:02:22 +0000

A Little Background

It was the summer of 2008 when I attended the LT Pact conference that I first drank the “Cloud Kool-Aid.” As you can see from my notes, I was pretty excited about this whole concept. Excited enough, one client of mine signed up for Layered Tech’s grid layer (which is what they were demoing at the conference).

One interesting thing I heard at the LT Pact conference was one of their keynotes was as IT Researcher for Gartner. He outlined what phases the current industry would go through with “cloud computing”: