MySQL Cluster Start Phases
By LinuxJedi on Jan 31, 2010
When MySQL Cluster data nodes start they need to go through a process of determining roles in the cluster, copying the data back into RAM and synchronising everything up. This can take longer than expected and the process is not always very verbose. So in this post I will outline a simplified version of what MySQL Cluster is doing in each start phase so that you can see why it can take time.
You can see the last completed start phase by following the management logs or you can see the current start phase by doing 'x status' in ndb_mgm where 'x' is the node ID of the data node you want the information for.
Start Phase 0
This is part of the basic initialisation, and the configuration is received. If this is an initial start then the node's file system is created.
Start Phase 1
All the blocks that are not already running are initialised in this phase. Connections to other data nodes are established in this phase and the nodes determine which one is to be the president node (coordinating things during the start). If nodes stalls in this phase it usually means that the nodes can talk the the management node but not to each other.
Start Phase 2
The state of all other nodes is checked in this phase and at this point a new master may be elected, this is typically the president or the node with the newest complete Global Checkpoint. The master node co-ordinates things such as Global Checkpoints.
Start Phase 3
A lot of decisions are made in this phase as to what kind of start is required. Some block communications are also setup such as the DBLQH and DBTC.
Start Phase 4
The real workload start here! The schema and LCP (Local CheckPoint) data is loaded back into memory, this can take a little time if you have a lot of in-memory data. In an initial start the redo log files are created.
Start Phase 5
Until very recently this used to typically be the slowest phase, in some cases it could even take hours! The REDO log was processed in this phase and if there were a lot of long transactions in this log the backwards searching required took a long time. This is now heavily optimised in 6.3.28 and 7.0.9, in my own limited testing I have seen a 60x start speed improvement thanks to these changes but real mileage may vary.
Other things are also started in this phase such as LCPs as well as copying data from other nodes if our node has some (or all) missing.
Start Phase 6
This phase is mostly just to setup the node groups.
Start Phase 7
Primarily the president selects a node as an Arbitrator. Minor things also happen such as the normal DiskCheckPointSpeed is configured instead of DiskCheckPointSpeedInRestart.
Start Phase 8
Indexes are rebuilt in this stage. This is now multi-threaded in 6.3.30 and 7.0.10 with the new BuildIndexThreads option which can give much better performance on very large tables.
Start Phase 9
Some finial initialisation of variables.
Start Phase 101
The node is now responsible for the primary replica of its data.
This node can now accept transactions!
As you can see there have been some changes recently which can greatly improve the performance of starting a node (in some cases it is now down to mere seconds). This should also help you figure out roughly where the problem is happening if a node is taking excessively long to start or fails during start.