Justin Carmony » Linux

Siege with User Authentication

Justin Carmony — Thu, 03 Nov 2011 16:33:02 +0000

If you want to stress test your application, the quickest and best way to do so is with siege. From the siege website:

Siege is an http load testing and benchmarking utility. It was designed to let web developers measure their code under duress, to see how it will stand up to load on the internet. Siege supports basic authentication, cookies, HTTP and HTTPS protocols. It lets its user hit a web server with a configurable number of simulated web browsers. Those browsers place the server “under siege.”

On Linux, you can install it easily via yum or apt. On OS X, you can install it easily via homebrew or macports. I personally prefer homebrew. In your terminal just type:

brew install siege

Now you can stress test a given app by simply using the siege command:

siege -c 10 -n 10 http://www.example.com/

You can read the documentation and faq on their site to learn more. However, there is one question I get a lot when talking about siege: “How can I use siege to test users who are logged in?”

I would always recommend that they send a session header in the config, but today I learned there is an even easier way (although undocumented), thanks to this question in Server Fault. In your siege’s configuration you can add the login-url directive:

# Login URL. This is the first URL to be hit by every siege
# client. This feature was designed to allow you to login to 
# a server and establish a session. It will only be hit once
# so if you need to hit this URL more then once, make sure it
# also appears in your urls.txt file.
#
# ex: login-url = http://eos.haha.com/login.jsp POST name=jeff&pass=foo
#
# login-url =

Awesome! Now each connection will authenticate before continuing their siege. A very handy tool, especially if your load is generated by user account pages and such.

Setting Up Nginx & PHP-FPM on Ubuntu 10.04

Justin Carmony — Tue, 25 Oct 2011 03:50:59 +0000

This is another wonderful setup that I’ve found myself using rather than the traditional Apache & mod_php setup.

What is Nginx?

Nginx (pronounced engine-x) is a fast, powerful, lightweight web server. I won’t go into the theory under-the-hood, but it’s focus is high concurrency with low memory usage. So while Apache is more robust in supporting many different features, nginx focuses on handling the important features very quickly. I still use Apache internally for our SVN & Trac web server. Heck, even at this time I’m using Apache to host this blog. However, Dating DNA, Clipish, CEVO, Alienware Arena, and some other high traffic sites/apis use nginx.

Ngnix, unlike Apache, doesn’t actually load PHP. Instead, it hands it off as a proxy to a “php handler” which acts like an Application Server. So nginx by itself won’t serve PHP files, but just static files.

What is PHP-FPM?

In the past, when working with something like Nginx or lighttpd, you would use spawn-fcgi to host your PHP application. However, spawn-fcgi had some major drawbacks and problems. So a guy named Andrei Nigmatulin created PHP-FPM, which stands for “PHP FastCGI Process Manager.” Since then, several others have contributed and ultimately it was include into the PHP core in version 5.3.3.

So from a high level look, on every PHP request Apache will load the entire installed PHP environment each time. This is for every request, and while it has been optimized as much as it can, that is a lot of overhead! With PHP-FPM, it will spin up a configurable amount of children. Each load the PHP environment and then will serve as many requests as it can without having to reload the environment. This saves on a lot of overhead!

Why use Nginx & PHP-FPM?

I should note, it is possible to configure/compile Apache in such a way that it can have similar performance capabilities. However, it takes a ton of work. Meanwhile, Nginx & PHP-FPM are very fast from the start, so I prefer just using them. You do lose some features, like .htaccess files won’t work so you’ll have to do that configuration in your virtual hosts.

How to Setup Nginx & PHP-FPM

Nginx

First off, lets setup Nginx.

sudo aptitude update
sudo apt-get install nginx
/etc/init.d/nginx start

Thats it! If you go to your server’s IP Address or Domain Name you should see a “Welcome to Nginx!”

PHP-FPM

Because PHP-FPM is only included by default in PHP 5.3.3 and later, and Ubuntu 10.04 LTS only has PHP 5.2.3, we have two options. Either we can install by source, or we can add another repository to install PHP-FPM. The latter is much, much easier, and there is a good PHP-FPM Repo for Ubuntu 10.04. To add it, you just run the following commands:

sudo aptitude install python-software-properties
sudo add-apt-repository ppa:brianmercer/php
sudo aptitude update

Now that we have the new repository, we can install PHP5:

sudo aptitude install php5-cli php5-common php5-mysql php5-suhosin php5-gd php5-dev
sudo aptitude install php5-fpm php5-cgi php-pear php5-memcache php-apc
/etc/init.d/php5-fpm restart

Excellent! Now, if you need to change some of PHP-FPM’s configurations, they are found in /etc/php5/fpm/. The file php5-fpm.conf configures how FPM will opporate, and the php.ini is the settings file that PHP will use while running in FPM.

A few settings I like to change in /etc/php5/fpm/php5-fpm.conf:

pm.max_children = 20

The php5-fpm.conf that comes is pretty well documented on the different settings. Once you make a change, make sure to restart php5-fpm.conf: /etc/init.d/php5-fpm restart

Configuring Nginx

Now, we have a few settings for Nginx. The configuration files are found in /etc/nginx/. First we’ll edit nginx.conf. Here are a few settings we’ll want to change:

user www-data;
worker_processes  4; # 1 to 4, I normally put this to the number of cores

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
    multi_accept on; # uncomment this line
    use epoll; # Add This - We'll want Nginx to use epoll for event timing
}

http {
    include       /etc/nginx/mime.types;

    access_log  /var/log/nginx/access.log;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;
    tcp_nodelay        on;

    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

Now, we need to add a VirtualHost! Nginx uses the same layout in Ubuntu as Apache, so we’ll add configurations for each site we want under /etc/nginx/sites-available/. So using vi, nano, or whichever editor you prefer, create a /etc/nginx/sites-available/www.example.com file:

# rewrite from example.com to www.example.com
server { 
	listen 80;
	server_name example.com;
	rewrite ^(.*) http://www.example.com$1 permanent;
}

server {
    listen   80;
    server_name www.example.com;
    access_log /var/log/nginx/www.example.com.access.log;
    error_log /var/log/nginx/www.example.com.error.log;

	client_max_body_size 4M;
	client_body_buffer_size 128k;
	expires 24h;
 
    location / {
        root   /var/www/example.com/;
        index index.html index.php;
		
        # if file exists return it right away
        if (-f $request_filename) {
                break;
        }

        if (-e $request_filename)
        {
                break;
        }

        # Useful rewrite for most frameworks, wordpress
        if (!-e $request_filename) {
                rewrite ^(.+)$ /index.php last;
                break;
        }

    }

    location /nginx_status {
      # copied from http://blog.kovyrin.net/2006/04/29/monitoring-nginx-with-rrdtool/
      stub_status on;
      access_log   off;
      allow 127.0.0.1;
      deny all;
    }

    location ~ \.php$ {
        expires off;
        include /etc/nginx/fastcgi_params;
        fastcgi_pass  127.0.0.1:9000;
        fastcgi_index index.php;
        fastcgi_param  SCRIPT_FILENAME  /var/www/example.com/$fastcgi_script_name;
    }
}

Now, we need to create the symlink from sites-enabled to sites-available:

ln -s /etc/nginx/sites-available/www.example.com /etc/nginx/sites-enabled/www.example.com

Restart nginx with “/etc/init.d/nginx restart”. Go ahead and put a test.php file in your directory with a Hello World example, and see if it works. It should work, and you should be good to go.

Preparing a VMWare Ubuntu Guest OS

Justin Carmony — Tue, 13 Sep 2011 07:11:09 +0000

I forget these steps all the time, so I figured I should record them here.

A lot of times VMWare will auto-install the VMWare tools for you when you first setup your VM. However, many times after setting it up I’ll do updates, and updates to the kernel will be applied, knocking out the VMWare Tools changes. These tools are used for things like file sharing between the guest and host.

So here are the steps to take to fully update and re-install the VMWare Tools. I got most of these from an article on Ubuntu’s website.

First, sudo aptitude update to make sure all my repository information is up-to-date.

Second, sudo aptitude safe-upgrade to get any and all updates for my newly installed OS.

Third, sudo apt-get install build-essential linux-headers-`uname -r` psmisc to install the linux headers for my kernel.

Fourth, copy and install the VMWare Tools:

# make a mount point if needed : sudo mkdir /media/cdrom


# Mount the CD

sudo mount /dev/cdrom /media/cdrom
# Make a dir for the VMWare Tools files

mkdir ~/vmtools
# Copy and extract VMWareTools

sudo cp /media/cdrom/VMwareTools*.tar.gz ~/vmtools
# You can extract with archive manager, right click on the archive and extract ... or

cd ~/vmtools

tar xvf VMwareTools*.tar.gz

# Install the tools cd vmware-tools-distrib sudo ./vmware-install.pl

Just follow the prompts and hit for all the default values.

Working with Middle-Scale Websites

Justin Carmony — Tue, 19 Jul 2011 00:02:40 +0000

I’ve been thinking about this idea for awhile, and I thought I would put a name to the thought. I brought up this idea while I was giving my “Real Life Scaling” presentation at the Utah Open Source Conference in 2009. Here is the problem I think most individuals in the web development face:

Hopefully at some point, your website gets a lot of traffic. Yay, you’ve reached your goal of getting good traffic, but it is soon followed by issues with performance and load. I like to call these the growing pains of a website. So as a web developer, I suddenly have the epiphany of “Hey, I need to scale my website!” What follows next is the biggest mistake a web developer can make:

They start looking at articles on how Google scales, or maybe how Facebook manages all of their traffic.

This is a mistake! To be brutally honest, you are not Google. You are not Facebook. You are not Twitter. You are a website that receives less than 0.000001% of the traffic that some major websites receive.

Why is this dangerous for web developers to do? Google, Twitter, Facebook, and others like them are solving complicated at a very large scale. I remember a presentation by a Twitter engineer who developed a program for a unique ID generator that can generate millions of IDs per second. The probability of you needing this type of solution is about the same as being struck by lightening. Applying these same practices at a much smaller scale are not realistic. If a locally owned grocery store wanted to open a second store, they would not adopt the same practices that Wal-mart use to manage their 8970 stores.

A Little Reality Check

I’m sure that most of my readers know of StackExchange.com. They power the popular website StackOverflow and several others. They have about two million visitors per day. That is a lot of traffic. StackOverflow is ranked #123 on Alexa. So you would imagine that they have a very large infrastructure serving all of this traffic?

Earlier this year, Stack Exchange wrote an article about their production environment. I was surprised on what exactly they were using. In paticular, the number of Production Servers*:

12 Web Servers (Windows Server 2008 R2)

2 Database Servers (Windows Server 2008 R2 and SQL Server 2008 R2)

2 Load Balancers (Ubuntu Server and HAProxy)

2 Caching Servers (Redis on CentOS)

1 Router / Firewall (Ubuntu Server)

3 DNS Servers (Bind on CentOS)

That is 22 servers for 2 Million Visits per day, serving 800 HTTP requests per second. Now, StackExchange did clarify that they did have other servers for management and fail over, but 22 servers handle their production load. This is a website that is ranked the 123rd most visited website in the world.

Honestly, most websites could be run on half a dozen servers if designed and configured correctly, including redundancy. Some really busy websites could run off a dozen servers. Unless you’re in the top 5,000 websites on the web, you really shouldn’t be worried about large-scale techniques.

So when you’re website is starting to grow, and you leave small scale, you’ll enter the phase of “Middle-Scale.”

What is Middle-Scale?

Middle-Scale is like being an awkward teenager:

You know that you can’t be the only one suffering through this, but you’re unsure how to proceed. It feels like you’re missing missing out on things everyone else must already know, but aren’t talking about. Like everyone else are awesome vampires or something:

But the reality is this: they don’t have some awesome secret! They are just normal teenagers.

This same idea applies to Middle-Scale websites.

Middle-Scale is when the most important things are still the best practices. Only now when you deviate from them you can feel those consequences. When you only had 100 users, a couple of nested queries and missing indexes didn’t cause that much of a problem. Your database is powerful enough to hide the inefficiencies. However, when you get to 10,000 users, your database can no longer hide the inefficiencies.

Middle-Scale is when simply separating your web server and database server isn’t enough. You’ll probably need to add some sort of cache like memcached. You’ll need to start tweaking your MySQL, Apache, and PHP configurations.

Then, after you’ve ironed out your inefficiencies, you’ll start to use multiple servers. You’ll probably add a Load Balancer with multiple web servers. After that, you’ll probably have some sort of Master-Slave replication for your Database for backups and fail-overs.

You start to leave this “Middle-Scale” classification when you move to multiple data centers, and start to do some load balancing at the the DNS layer. This is when you’ll start to have a dedicated sys-admin team.

Okay, I’m Middle-Scale! So what should I do? Where do I look?

First off, you must adhere to best practices. If you are working with PHP, research PHP performance and best practices. Do they same for each of your technologies, like Apache and MySQL. You will need to stop treating your application as one big app, and start to understand all of it’s moving parts.

Second, you must understand your specific problems. Scaling ins’t a problem, nor is it a solution. It is a generic term for many different types of solutions. Without understanding why your website is running slow, or why it cannot handle the load, you will not be able to create an effective solution.

So you don’t have a scaling problem. You have a MySQL performance issue, or a Apache problem, or a PHP problem. Most likely, it is something extremely specific. You have a high volume of MySQL write operations (i.e. UPDATE, INSERT, DELETE, REPLACE), or perhaps you are missing some indexes and have too many full table scans.

Third, Googling for help will only get you so far. You are starting to enter a phase when it is harder and harder to find answers to your broad issues. Talking with other experienced people who have gone through the Middle-Scale pains before will help immensely. I cannot recommend highly enough going to User groups. Being able to communicate with someone, either face to face, on the Phone, over IRC, etc. is invaluable. While I’ve learned a lot at conference and usergroup presentations, I’ve learned even more by just talking with the people attending and at the social gatherings.

Profile & Performance will Naturally Lead to Scaling

When you want to scale, it can feel like a very daunting task. It seems like this big unknown complicated solution. What in the world am I going to do? I remember feeling these worries when I first started to investigate load balancing and sharding for some websites I was working on.

The thing is, if you start to profile your application, you will discover it’s inefficiencies. I remember when I spent a sold week, working 12-16 hours a day profiling and optimizing Dating DNA’s database. I found a lot of bad queries, and I was able to cut our load times from 2-5 seconds to under 0.1 seconds. The CPU on the database server went from 80-90% CPU utilization to under 10%. It was incredible, and then I promptly took the entire next week off. When we migrated to new servers, I was able to move to less powerful database server and still have the same great performance. So by profiling and optimizing our database, I didn’t need to worry about spinning up multiple master databases and sharding our data.

With Clipish, we faced almost opposite scaling problems. The database was rarely an issue, but our web server CPU’s were. We do a lot of ImageMagick manipulations of images, and at high volumes on virtual servers this can be a big issue. So over the last year we’ve introduced some load balancing and CDN tools to help serve all 10 TB of bandwidth for Clipish.

The thing is, when you start to profile your application, you start to understand it’s low areas better, so you have a much better idea on what do to. Even if you don’t know your solution, it is much easier to find a solution with a sound understanding of the problem. For example “scaling mysql” yields much less helpful results than “mysql full table scans” in Google.

So should I ignore what Facebook and Google do for scaling?

Of course not! First off, they do cool stuff. Just because I’ll watch NASA launch a space shuttle doesn’t mean I’ll try to make a rocket system for my broken lawn mower. But you have to put what they are doing into context. People from large websites have published several good “best practices” articles on techniques that help any website. Especially things on the client/browser side of things. Just use caution. I cringe when I hear someone say “we’re trying to use Cassandra to solve XYZ problem at work” when it is severe overkill.

Final Thoughts

Most of the time when I talk about performance and scaling with other people, it is when they are in “critical mode.” Their website is down, slow, unusable, etc, and they are looking to fix the problem. I will say, it is much more difficult to profile in “critical mode” than profiling before hand. The reason is you are much more desperately focused on getting it working again instead of understanding the problem.

I’ll be giving a presentation this Thursday at UPHPU on Profiling PHP Applications. I’ll post the slides, and most likely write some articles on the subject afterwards. As always, feel free to email me or leave a comment.

Simple Trick: History Command

Justin Carmony — Tue, 31 May 2011 21:05:08 +0000

So today I quickly installed Redis on a server, and I wanted to keep a copy of all the commands I had used to do so. I knew there was a better way than hitting the up arrow each time to copy and paste each command in reverse order. So after some Googling, I found out about the history command. So I was able to type the command “history” and it showed me the last two thousand or so commands I had used. I was able to copy and paste that into a wiki, and was good to go. You could also search your history by piping the history command to grep. Cool stuff I don’t want to forget.

Sharing Sessions Across Multiple Servers With Memcache

Justin Carmony — Wed, 15 Sep 2010 08:52:15 +0000

These last few days have been full of quickly implementing things I’ve only read about, but hadn’t done for myself.

One situation was looking at splitting web traffic across multiple servers. In our case, we had only 2% of our website that would spike and increase our load by %1000 percent (I’m not joking) for a short period of time. So instead of doing traditional load balancing, we opted for routing specific functions to a custom subdomain that would run on another server. That way if the thousands of crazy video game players (you know who you are), slam the servers trying to get a cool skin for their online avatar, it would only take down that one section.

Well, we were using PHP Sessions, and as you might know, they are file based. The problem is that when it’s file based, server A by default won’t have access to server B’s session files. This makes having a user stay logged in between servers a real pain.

After researching all sorts of possibilities, we decided to take the shortest route. You can write your own PHP Session Handlers, but dealing with race conditions and other pitfalls are left to the developer.

So, we decided to use the built in memcache session handler that has all the extra tricky stuff built in. All it takes is installing the php memcache module, and a few php.ini changes outlined here. I’ve copied the text here to keep a record of it.

The first thing is to install the memcached server on your Debian server :

apt-get install memcached

Then, since the memcache PECL extension now provides its own session handler, it’s easy to plug PHP and memcached servers. Just install the appropriate extension (from Dotdeb) :

apt-get install php5-memcache

and change some of your PHP settings :

session.save_handler = files
; session.save_path = “N;/path”

to :

session.save_handler = memcache
; change server:port to fit your needs…
session.save_path=”tcp://server:port?persistent=1&weight=1&timeout=1&retry_interval=15″

That’s all! After relaunching your Apache2 server, your PHP sessions will be stored on the memcached server.

Now, you’ll need to have memcached, the service, running somewhere that all your php web servers can access. In presentations I’ve given about memcached, I talk about how memcached as zero built in security (well, maybe 0.1% security, close enough to zero). So you need to secure it with iptables so that only the web servers can access your memcached service.

Tomorrow if I have time I might quickly document my experience of getting a Ubuntu 10.04 server running nginx + php. I was surprised at how easy it was, and even figuring out some more advanced configs were not bad.

Setting up SSH Key Authentication Between Servers

Justin Carmony — Wed, 14 Jul 2010 19:52:05 +0000

I always forget how to do this, so I have to look it up each time. This will make it easier the next time. It is so simple, just I always forget.

First, I need to generate my ssh pub and private keys on the host server: ssh-keygen

Second, I need to copy the pub key to the remote server: ssh-copy-id -i id_rsa.pub user@server.local

Then, that it! Now that I wrote it down, I won’t forget it… I hope.

Setting up Nagios for Servers

Justin Carmony — Tue, 04 May 2010 21:17:20 +0000

Ok, so I need to confess, I haven’t been a good blogger. I’ve gone two months without writing a post, which is broke a 2 year streak. But, I’m going to post date these posts, since I can.

How I’ve gone this long without setting up Nagios to monitor servers is beyond me. However, I was tired of my clients and so finding out a server was offline before me. So I threw up a 256MB virtual server on Slicehost, since I don’t have any servers with them at the moment, in case a data center went offline.

I put Ubuntu 10.04 LTS on the VM, and I followed the instructions found here on Linode:

http://library.linode.com/server-monitoring/nagios/ubuntu-10.04-lucid

It worked pretty well, but I think in the future I will just install the packages that come with the server distro. The reason being is there were several checking tools, like check_mysql and check_nrpe that were not installed because I didn’t have some dependencies installed.

Now after a few hours I have 9 servers being monitored with about two dozen services being monitored. I also found out that one server had a 96% full HDD, so it already is paying off.

On another note, for those who know me and are curious, I’m in full swing in finding sponsors for this years Utah Open Source Conference. I’ve talked to quite a few, just need to finalize and close their deals. If you know of a company interested in sponsoring this year’s conference, let me know (check the contact me link up above).

The Cloud to Bare Metal

Data Backups – There Are No Excuses

Justin Carmony — Sun, 19 Apr 2009 00:04:48 +0000

Today I just had the terrible experience of having a database lose data, need to restore, only to not have a recent backup. If you haven’t had this experience before, please, take this serious. My wife was home for lunch as it happened, and she watched as the blood drained from my face. It only took a few seconds for the loss to happen, and immediately I knew exactly what the repercussions where. The immediate second thought that passes through your brain is “Where are my backups?” That is when I realized I didn’t have my nightly backups set up on this server. I quickly checked the file date on the last known backup I had.

13 Days.

It could have been a lot worse, but it was still extremely bad. Those last thirteen days had been record setting days. Emails each day were going around about record new signups, records internal messages sent, etc. Those thirteen had been the best 13 days by far.

If some of you are wondering what had happened, and know me to be very diligent in my backups, I did the one wrong, terrible thing: I made an assumption. I’ve blogged before on how backups have saved me in the past, and how I am almost a fanatic about them. So what the heck happened?

This website was on some hardware that was starting to get overburdened. Then, out of the blue, our traffic exploded and our web server and database server started to grind to a halt. I spent long hours and sleepless nights migrating from these old servers from a terrible host to some new virtual machines. We then discovered our MySQL Database was so intense that the virtual server couldn’t handle the CPU and I/O requirements. Finally, in a last attempt of desperation I moved the Database to a spare box of another company who gave me permission to use it temporarily. That finally worked and allowed us to handle the load on our Database. By the time I finished this, it was about 8 AM in the morning and I went to bed.

I assumed we’d only be on this box for a day or two, so I didn’t setup the backup scripts. However, it gave us more breathing room than we expected, and other issues came up non-db related. The company lending the us the server said we could take our time, so the urgency on ordering our new hardware was pushed off more and more. I had completely forgotten about setting up backups scripts, and we ended up where we are now.

What I’m Changing Personally

I’ve decided to make two changes personally after this experience.

First, there are zero excuses for not having automated backups. Zero, zilch, nada! If a backup should have occurred, there is no excuse for it not to happen.

Secondly, I’m going to pick a day of the month where before I do anything else, I verify that all the backups are working. My father-in-law on the first business day of the month has the habit of doing his business’s billing and other accounting activities. He lets just about nothing stand in the way, and all ways checks his bank accounts and records to make sure everything is in order. I’m going to adopt this same idea, only with servers and data. The first business day of the month I’m going to go through all the servers under my care, verify the backups are working, check error logs, etc. I want to catch the problem before anyone else does.

How To Prevent Data Loss

Here are a few guidelines to make sure you don’t fall victim to data loss.

Select a Backup Schedule & Follow It 100% - I suggest for most websites, a daily backup will work out pretty well. If you have a lot of data that would really stink to lose that changes frequently through the day, you could backup several of the tables hourly.
Back Up To Several Locations – I like my servers to have two hard drives. One for the live data and another for backups. Then, after a backup has been created, I like to sync that backed up data to another server. It is important that if a meteor fell from the sky and hit your data center (or a flood, fire, earthquake), you would have a very recent backup somewhere else.
Verify Your Backups – I can’t stress this enough. After this terrible accident of not having a recent backup, I went and checked all my other website database backups. I found out that one critical database’s backups were broken and not running nightly. You never want to find out this information after you have to restore from a backup. Regularly verify that your backups are being created, and that you can restore from them.

Hopefully this will motivate at least one person in our profession to evaluate their backup strategy and make it better. You don’t ever want to tell a client that you just lost 13 days of their record setting work.

Recursively Add New Files to SVN

Justin Carmony — Fri, 20 Feb 2009 16:51:33 +0000

One great thing about running Windows is TortoiseSVN. I makes managing my SVN so easy. One of the problems I’ve ran into on other systems was I had been working for several hours. I would have dozens of new files and with TortoiseSVN it would show me a list of all the new files and I could hit “select all” and commit. However, on Mac I haven’t found an SVN client that I like, so I just use the terminal commands. There are also times, like when managing a WordPress installation, that WordPress and its plug-ins could auto update. I wanted to be able to add all the new files on the linux server back into the svn.

I never could figure out how to do it until I read a comment at a blog post:

svn st --ignore-externals | grep ^? | sed 's/\?/svn add/' | sh

Man, shell is just awesome. This is better than the first suggestion of using “svn add –force *” because it would add ignored files as well. Hope this helps someone and makes them happy like I am now!