Comments on: MySQL, 40 Million Rows, MyISAM to InnoDB, 45 Minutes

By: Restoring Large MySQL Dump – 900 Million Rows | Justin Carmony’s Blog

Thu, 07 Apr 2011 03:05:16 +0000

[...] Million belonged to one single table (902,966,645 rows, to be exact). To give an idea of growth, the last time I blogged about this database we had about 40 million rows. This giant table, the "Scores" table, is has a [...]

By: Lubor Nosek

Lubor Nosek — Wed, 10 Nov 2010 09:09:02 +0000

If you’re having trouble with tmp tables being created on disk, you could modify tmp_table_size for the thread doing the conversion.

Something like:

SET tmp_table_size=4*1024*1024*1024; — 4 GB

INSERT INTO … SELECT … FROM … ORDER BY …;

This way you end with simpler query and still use memory for the temporary table.

By: Large MySQL Conversion & InnoDB | Justin Carmony’s Blog

Large MySQL Conversion & InnoDB | Justin Carmony’s Blog — Sat, 16 Oct 2010 03:02:41 +0000

[...] off, converting from one large set of InnoDB tables to a new structure. I’ve done something similar in the past, and I think I’ve honed in a little bit more the technique. So here are some thoughts [...]

By: Justin Carmony

Justin Carmony — Thu, 07 Oct 2010 06:37:25 +0000

Looking at some of the latest comments, I thought I should post an update to my story. I had tried to have MySQL dump the data in an ordered fashion. However, it would have to sort through the 40 million rows and because of the memory limits on the DB server, it would have to do this with a temporary table on the disk. So I would issue the command, and it would sit there for an hour while I waited for it to perform it’s operations, and I had no idea how close or far I was for the query to finish, so I aborted it.

Having mysql dump the data as it was stored was much, much faster, and then inserting into a memory table, reorganizing the data, and re-exporting it was much faster than just summerizing it all into one query. I could look back and see why dumping it in an ordered fashion wasn’t working like I expected, but at the time, this was the shortest distance I could think of to get our website back up and running.

By: Derek

Derek — Mon, 27 Sep 2010 22:05:46 +0000

this doesn’t sound like an advanced topic.. more like someone who doesn’t know what they are doing…

By: Joris Machielse

Joris Machielse — Thu, 12 Aug 2010 21:05:04 +0000

Or actually: you don’t even have to do a dump. Just do:
Step 2, and then:
INSERT INTO new_table (id, etc) VALUES (SELECT id, etc FROM old_table ORDER BY id, etc);
Then step 4 and 5.

By: Joris Machielse

Joris Machielse — Thu, 12 Aug 2010 20:54:16 +0000

Wouldn’t using mysqldump with an order by argument, have been much simpler?
So:
1- mysqldump that outputs the rows in sorted order (without schema)
2- create new table (with new name) with existing schema but new engine.
3- import data into new table.
4- drop old table
5- rename new table to the original table’s name.
Cheers!

By: Augustina Dziedzic

Augustina Dziedzic — Thu, 15 Jul 2010 04:29:28 +0000

Terrific do the job! This could be the sort of information that ought to be shared around the web. Shame about the search engines for not positioning this post higher!

By: imran ahmed rahi

imran ahmed rahi — Mon, 08 Feb 2010 19:18:52 +0000

Hi,

I read your article after searching google for many hours and visiting several sites. I really liked the MEMORY Storage engine. I knew it theoratically, but never thought how useful it could in situation like yours.
I am facing similar problem but with inserts of 3 million records from csv file into mysql db.
I am uploading a csv file with 3 million email lists, parsing them and storing information in INNODB Table. The problem is it takes around 45 minutes to upload 1 lakh contacts. I am targetting to upload 3 million contacts and looking for optimal way to do that.
Can you give me suggestions towards my approach? I would be very happy.

Thanks,

By: Justin Carmony

Justin Carmony — Fri, 16 Oct 2009 06:35:24 +0000

@JDS

Interesting, I’ll have to try that out. My guess is that going between files like that skips any type of temporary tables that are created. I’ll have to test that out, because while putting everything into memory was cool, sometimes that much memory isn’t available. That table that has 40 Million Rows now has around 450 Million Rows, and I don’t think it could hold it in memory entirely.