MySQL Cluster GCP Stop

One of the most common errors we come across whilst supporting MySQL Cluster is an error commonly referred to as 'GCP stop'.  These errors will occur most frequently in cluster setups which have high activity and more often than not use disk data.  So lets look into what these are, why they happen and how to prevent them.

What is a GCP Stop?

All data that needs to be written to MySQL cluster is first written to the REDO log, this is so that when a node starts the log can be played back from the position of the last good LCP (Local CheckPoint, a point at which all the cluster data memory is written to disk).  The REDO data needs to be consistent between all data nodes and that is where the GCP (or Global CheckPoint) comes in.  It synchronously flushes the REDO data across all data nodes to disk every 2 seconds (by default).  A GCP stop happens when a new GCP is trying to commit the REDO to disk and the previous one has not finished.  MySQL Cluster is a real-time database so this is a critical problem and the node in question is shut down to preserve data integrity.

Why does a GCP Stop happen?

GCP stop usually happens for one of two related reasons.  Firstly there is too much data to commit between GCPs for it to all be written to disk at once and secondly the disks are too slow.

You should now be able to get an idea of why this is more prominent on clusters using disk data, both the disk data and GCP are written to disk at the same time (as well as things like the LCP), lowering the disk bandwith available for the GCP.

This is also more common on multi-threaded data nodes (ndbmtd) in MySQL Cluster 7.0 because these can handle more data simultaneously and therefore can be in a situation where they need to write more to the REDO log.

How to prevent a GCP Stop

There are several effective ways to prevent a GCP stop:

1. Buy faster disks - may not be an option but if the data is written faster this can prevent a GCP Stop
2. Spread the different parts of the data node onto different disks - the REDO, LCP and disk data can all be separated onto different disks, giving a much better disk I/O bandwidth to each
3. Commit more often - if you have a really long transaction with lots of data this could create a commit which is too large for one GCP
4. Configuration - there are some configuration settings you can tweak to improve things, but these will only give small improvements over the above three points.  Settings like TimeBetweenGlobalCheckpoints which if decreased causes the data node to GCP more often which means there is less to write to disk per checkpoint (but checkpointing more often means less time to checkpoint, so not always a good option).  There are also settings affecting disk factors outside of GCP such as DiskPageBufferMemory, increasing this will buffer more disk data (much like innodb_buffer_pool_size for InnoDB) decreasing disk bandwidth disk data uses so that the GCP can use more disk bandwidth.

There are other settings that can be tweaked as a last resort depending on what kind of GCP Stop occurs (yes, there are a couple of different types) but the first three points should be a primary concern before thinking about doing this.

If you have any problems with GCP Stop I highly recommend asking on the MySQL Cluster forum or the MySQL Cluster mailing list.

Comments:

Hi

I remember when I was at a customer tuning an app with MySQL Cluster 7 running on a pair of 256 cpu Sparc servers. After we got it well tuned, we started seeing GCP errors followed by crashes.

When I finally found out what GCP errors mean, the customer realised that for all this time we had been running just agains 1 local sata disk! (per server of course). He reconfigured with a RAID of 6 disks and then everything was fine.

This was a really write intensive app, so my conclusion was that Cluster was actually quite easy on the disk there and you could press quite far before the disk couldn't keep up anymore.

Posted by Henrik Ingo on January 29, 2010 at 05:28 AM GMT #

Hi Henrik,

I have seen a few clusters recently with multiple data nodes per server running from one disk or all nodes running from a SAN (all in the same partition/disks array). In those kind of cases it doesn't even need to be heavy write-intensive unfortunately :(

But yes, as in your case it tends to be the write-intensive apps, especially with disk data and blobs that really stall the GCP.

Posted by LinuxJedi on January 29, 2010 at 06:20 AM GMT #

Yes, the tests I described are a year old now, so I'm sure with todays hardware this becomes more and more common. My point is not that there is a possibility to escape this problem, rather I was amazed that you can get even that far without worrying about disks. After all, we were using quite a lot of the capacity on that server before we finally hit GCP errors.

Posted by Henrik Ingo on January 29, 2010 at 12:36 PM GMT #

More specifically, GCP stop is detected when TimeBetweenEpochsTimeout is reached. So, increasing this parameter may ease the problem as well.

Posted by Mikiya Okuno on February 02, 2010 at 12:49 AM GMT #

Hi Mikiya,

For a gcp_commit stop that is correct, thanks for pointing it out. If it is a gcp_save stop then it is TimeBetweenGlobalCheckpoints. Also if you mess with TimeBetweenEpochsTimeout too much you can end up with gcp_save stops anyway :)

I was going to put about this in the posting but at the moment the way to tell the difference between the two kind of stops but since it means looking for specific numbers in the logs I thought I would save it for a later blog post and explain what the numbers mean.

Hmm...maybe I can write a quick patch so that it will automatically tell you the kind of GCP stop.

Posted by LinuxJedi on February 02, 2010 at 01:37 AM GMT #

Post a Comment:
  • HTML Syntax: NOT allowed
About

LinuxJedi is an ex-MySQL Senior Technical Support Engineer who previously worked at Oracle and specialised in MySQL Cluster as well C/C++ APIs.

Search

Categories
Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today