InnoDB transportable tablespaces
By Calvin Sun on Apr 10, 2012
Note: this article was originally published on http://blogs.innodb.com on April 10, 2012 by Sunny Bains.
In the past, users were unable to take full advantage of the FLUSH TABLES WITH READ LOCK statement. InnoDB simply ignored the flush to disk part. If the table did not have any dirty pages in the InnoDB buffer that weren’t synced to disk (due to sheer luck) then it was safe to copy the .ibd file to another location. Also, the restore was not without its limitations and complications. The .ibd file could not be copied over to another server because InnoDB during import did not fix up metadata required for a trouble-free import. The main problems during import were:
- If the tablespace ID of the IMPORTing instance had changed, for example if the table had been dropped and re-created, then the import would fail.
- If the table and index IDs in the tablespace were different than on the importing server.
- InnoDB uses a global row ID for tables without an explicit primary key. Any such row IDs in an imported tablespace had to be lower than the current maximum on the new server.
- The maximum LSN of the tablespace had to be lower than the server’s current max LSN.
- The maximum transaction (trx) ID of the tablespace had to be lower than the server current max trx id.
- Purge and change buffer issues.
In short, if it worked you were lucky that the above constraints were satisfied, or your tables were probably read-only.
Do the flush (export) properly and also the import. During flushing we disable purge, merge all pending change buffer entries to the tablespace and then flush all the dirty pages to disk. Note: Purge will remain disabled until UNLOCK TABLES; is issued. Write out a metadata file that contains the table schema, hostname of the exporting server, page size of the exporting instance, the highest autoinc value in memory etc. So that when we do an import, we can check if the table schema matches (currently, it ignores foreign key relationships) and also restore the autoinc value. This metadata file is created in the same directory as the tablespace, with a suffix of .cfg. If the table is named customer then the meta-data file will be named customer.cfg and the tablespace will be named customer.ibd. Note: FLUSH TABLES and IMPORT only work for tables that are stored outside the system tablespace, in their own tablespace. The InnoDB config parameter innodb_file_per_table must be set when creating the table.
Because InnoDB stores data in big-endian format, this allows copying data between different architectures. Copying between different operating systems will also work.
If you want to backup a single table:
CREATE DATABASE ftwrl;
SET GLOBAL innodb_file_per_table=1;
CREATE TABLE ftwrl.t (C INT) ENGINE=InnoDB;
INSERT INTO ftwrl.t VALUES(1),(2),(3);
SELECT * FROM ftwrl.t;
FLUSH TABLES ftwrl.t WITH READ LOCK;
From the command line, copy the t.ibd and t.cfg to your backup directory. The files should be in the ftwrl/ sub-directory under datadir.
UNLOCK TABLES; — Note: This will remove the t.cfg file.
DROP TABLE ftwrl.t;
CREATE TABLE ftwrl.t (C INT) ENGINE=InnoDB;
ALTER TABLE ftwrl.t DISCARD TABLESPACE;
The DISCARD tablespace will rename the t.ibd file to t.ibt in the same directory; you have to remove this file manually for now. Once you’ve removed this file, copy the backed up tablespace t.ibd and the metadata file t.cfg to the ftwrl/ sub-directory. Then issue the folowing:
ALTER TABLE ftwrl.t IMPORT TABLESPACE;
CHECK TABLE ftwrl.t;
SELECT * FROM t;
You should be good to go. If you want to backup multiple tables together, the syntax for that is:
FLUSH TABLES t1, t2, …, tn WITH READ LOCK;
Copy as before but this time the multiple tablespaces and metadata files.
Then first discard the tablespace as above followed by an import as above.
Errors and warnings will be pushed to the client so that the user doesn’t need to poke around in the server error log files to check why an operation failed. The intention is to send enough information to the user so that the problem can be clearly identified and understood.
The syntax for “export” is not finalised yet and may change; for now the above examples using FLUSH TABLES WITH READ LOCK should work. To export tables that are in a foreign key relationship, you must export and import all the tables together, otherwise you may end up with inconsistencies. The export/import doesn’t work on partitioned tables, this limitation will be addressed later. A matching table definition must exist in the importing server instance, currently we don’t automagically create the table using the saved metadata during import. The .cfg format is not written in stone yet, we may add more data to this file, for example for partition support.
Your feedback is important to us, so that we can improve this feature further.