MySQL NDB Cluster offers high availability and scalability through its distributed architecture. However, managing and troubleshooting such a system requires robust monitoring and insightful diagnostics. At the heart of this lies effective logging. This post will guide you through the details of logging within MySQL NDB Cluster, explaining how it works across different kind of nodes, the valuable information contained within each log file and the underlying mechanisms that handle this process. Finally, we will present a new feature, added to both MySQL NDB Cluster 8.4 Community and Commercial editions, that allows NDB API users to supply their own logging functions to the logger. 

Logging Architecture in MySQL NDB Cluster

MySQL NDB Cluster employs a distributed logging system, where each node is responsible for recording events relevant to its specific role. Understanding these roles is key to understanding the logs effectively:

  • MySQL Server (SQL Node): As the interface for applications, the SQL node logs its interactions with the NDB storage engine, including connection attempts, query execution (and any errors encountered), and overall server status.
  • NDB Management Server (MGM Node): The central coordinator of the cluster, the MGM node logs critical events related to cluster configuration, node membership changes (startups, shutdowns, failures), arbitration processes, and overall cluster health.
  • NDB Data Nodes: These nodes are responsible for storing and serving data within the cluster. Their logs contain detailed information about transaction processing, data access patterns, inter-node communication, data checkpoints, etc.

Log Files and Their Content

  • MySQL Server (SQL Node) Logs:
    This log contains general MySQL Server messages, but also includes details  about the server’s interaction with the NDB storage engine, including connection status with MGM and data nodes, NDB-specific errors and warnings (such as failed transactions or timeouts), and information regarding the NDB API version and its usage.
  • Cluster Log and Management Node Log:
    • The Cluster Log
      This is the central log for the cluster’s management activities. Key information includes:
      •  Node connection and disconnection events.
      • Processing of configuration files (config.ini).
      • Results of management commands issued via ndb_mgm.
      • Detection and handling of node failures.
      • Arbitration processes to determine the active nodes.
      • Alerts about potential issues or inconsistencies within the cluster.
    • Management Node Log
      The management server also has a node log, but it contains mostly the same information as the cluster log.  It’s generally only useful for troubleshooting management server startup issues.

2025-08-06T09:35:07.568239Z [MgmtSrvr] INFO — Got initial configuration from ‘my.cnf’, will try to set it when all ndb_mgmd(s) started
2025-08-06T09:35:07.679778Z [MgmtSrvr] INFO — Node 1: Node 1 Connected
2025-08-06T09:35:07.689690Z [MgmtSrvr] INFO — Id: 1, Command port: *:14000
2025-08-06T09:35:07.689692Z [MgmtSrvr] INFO — Not requiring TLS
2025-08-06T09:35:07.690249Z [MgmtSrvr] INFO — MySQL Cluster Management Server mysql-9.5.0 ndb-9.5.0 started
2025-08-06T09:35:07.806423Z [MgmtSrvr] INFO — Node 1 connected
2025-08-06T09:35:07.806671Z [MgmtSrvr] INFO — Starting initial configuration change
2025-08-06T09:35:07.894264Z [MgmtSrvr] INFO — Configuration 1 commited
2025-08-06T09:35:07.894319Z [MgmtSrvr] INFO — Config change completed! New generation: 1
2025-08-06T09:35:09.366850Z [MgmtSrvr] INFO — Nodeid 2 allocated for NDB at 100.103.23.242
2025-08-06T09:35:09.691160Z [MgmtSrvr] INFO — Node 1: Node 2 Connected
2025-08-06T09:35:10.102865Z [MgmtSrvr] INFO — Alloc node id 3 rejected, no new president yet
2025-08-06T09:35:10.102968Z [MgmtSrvr] INFO — Nodeid 3 allocated for NDB at 100.103.23.153
2025-08-06T09:35:10.417539Z [MgmtSrvr] INFO — Node 1: Node 3 Connected
2025-08-06T09:35:10.835551Z [MgmtSrvr] INFO — Alloc node id 4 rejected, no new president yet
2025-08-06T09:35:10.835631Z [MgmtSrvr] INFO — Nodeid 4 allocated for NDB at 100.103.23.242
2025-08-06T09:35:11.148242Z [MgmtSrvr] INFO — Node 1: Node 4 Connected
2025-08-06T09:35:11.579317Z [MgmtSrvr] INFO — Alloc node id 5 rejected, no new president yet
2025-08-06T09:35:11.579396Z [MgmtSrvr] INFO — Nodeid 5 allocated for NDB at 100.103.23.153
2025-08-06T09:35:11.879165Z [MgmtSrvr] INFO — Node 1: Node 5 Connected
2025-08-06T09:35:12.015423Z [MgmtSrvr] INFO — Node 3: Start initiated (mysql-9.5.0 ndb-9.5.0)
2025-08-06T09:35:12.015648Z [MgmtSrvr] INFO — Node 2: Start initiated (mysql-9.5.0 ndb-9.5.0)
2025-08-06T09:35:12.015690Z [MgmtSrvr] INFO — Node 4: Start initiated (mysql-9.5.0 ndb-9.5.0)
2025-08-06T09:35:13.248112Z [MgmtSrvr] INFO — Node 3: Buffering maximum epochs 100
2025-08-06T09:35:13.264414Z [MgmtSrvr] INFO — Node 5: Buffering maximum epochs 100
2025-08-06T09:35:13.381284Z [MgmtSrvr] INFO — Node 2: Buffering maximum epochs 100
2025-08-06T09:35:13.392635Z [MgmtSrvr] INFO — Node 4: Buffering maximum epochs 100
2025-08-06T09:35:13.405442Z [MgmtSrvr] INFO — Node 3: Start phase 0 completed
2025-08-06T09:35:13.406587Z [MgmtSrvr] INFO — Node 3: Initial start, waiting for 2, 4 and 5 to connect, nodes [ all: 2, 3, 4 and 5 connected: 3 no-wait: ]

Cluster log excerpt during a start up

  • MySQL NDB Cluster Data Node Logs:
    These logs provide the most granular view of data-related operations within the cluster. It includes details on:
    • Data node startup and shutdown sequences.
    • The lifecycle of transactions (start, commit, rollback).
    • Details of read and write operations on data fragments.
    • Messages exchanged between data nodes for coordination and data transfer.
    • Information about node recovery processes after failures, including data synchronization.
    • Error conditions encountered during data handling or communication.

Log Levels and Filtering

MySQL NDB Cluster implements different log levels (e.g., ALERT, CRITICAL,ERROR, WARNING, INFO) to allow fine-grained control over the verbosity of the logs. By choosing the appropriate log levels, users can filter out less critical messages and focus on information relevant to their current needs.

MySQL NDB Cluster Log Handlers

MySQL NDB Cluster employs internal log handlers to manage the process of writing log messages. Currently MySQL NDB Cluster support 3 different log handler:

  1.  File Log Handler: This is the primary mechanism, where each MySQL NDB Cluster process (MGM node and Data Nodes) writes its log messages to to a local file. The location and name of each file is defined by the configuration.
  2. Console Log Handler, similar to the File Log Handler but in this case log messages and printed to the default stdout of the node process.
  3. Syslog Handler: MySQL NDB Cluster also provides the option to direct log messages to the system’s syslog facility. This can be beneficial for centralized log management and integration with other system monitoring tools.

Asynchronous Logging

MySQL NDB Cluster is designed for performance and reliability, and its logging mechanisms reflect this by employing Asynchronous Logging.

Dedicated Logging Thread: To minimize the impact of disk I/O on the main processing threads responsible for handling data and management operations, MySQL NDB Cluster uses dedicated threads for writing log messages. When an event needs to be logged, the information is passed to a logging thread, which then handles the actual writing to log. This asynchronous approach helps prevent blocking and ensures that critical operations are not delayed by logging.

Buffering: MySQL NDB Cluster may employ in-memory buffers to temporarily store log messages before they are flushed to disk. Buffering can improve performance by reducing the frequency of disk writes. However, it’s important to be aware that in the event of an immediate and ungraceful shutdown, some buffered log messages might be lost.

By default both Management nodes and NDB API nodes use Asynchronous logging independent on the type of LogHandler the node is using.

In NDB API programs Asynchronous Logger functionality can be activated/deactivated through the functions startAsync() /stopAsync() of the logger instance.

User-defined Logging Function for NDB API

NDB API was, since 8.4, enhanced to support user-defined logging functions. This new functionality enables developers to provide a custom log() function that the NDB API Logger will call with its internal log messages. The user-defined function receives the log level, category, and the message text, offering flexibility in handling NDB API logging.

This enhancement enables users to redirect NDB API logs from their default standard output destination, enabling them to route these messages to alternative destinations, apply custom filtering or processing, or wrap them within other logging frameworks. By providing a mechanism for user-defined log handling, the NDB API offers greater control and integration with application-specific logging requirements.

Example of user-defined log function:


  /**
   * Set log consumer
   *
   * Supply a consumer which will be passed NdbApi internal informational
   * log messages for output.
   *
   * This consumer will be used for all NdbApi logging in the process, even
   * with multiple ndb cluster connections in a process.
   *
   * The consumer must exist until all ndb_cluster_connections in the
   * process are deleted, or the log consumer is set to NULL.
   *
   * Setting to NULL reverts to the default behaviour.
   *
   * @param log_consumer Pointer to log consumer.  Setting NULL reverts to the
   *                     default behaviour.
   */
  static void set_log_consumer(NdbApiLogConsumer *log_consumer);

NDB API file ndb_cluster_connection.hpp

 #include "ndb_cluster_connection.hpp"

// Define the new log consumer and log function
class Ndb_log_consumer : public NdbApiLogConsumer {
  const char *filename;

 public:
  explicit Ndb_log_consumer(const char *out_filename) {
    filename = out_filename;
  }
  ~Ndb_log_consumer() override = default;

  void log(LogLevel level, const char *category, const char *message) override {
    char timestamp[64];
    std::ofstream ofs(filename);
    Logger::format_timestamp(time(nullptr), timestamp, sizeof(timestamp));
    ofs << timestamp << " [" << category << "]"
        << " [" << level << "] " << message << std::endl;
    ofs.close();
  }
};

void main() {
   ndb_init();
   const std::string file = "file.log";
   auto *log_consumer = new Ndb_log_consumer(file.c_str());
   Ndb_cluster_connection::set_log_consumer(log_consumer);
   for (int i = 0; i < 10; ++i) {
      g_eventLogger->info("[CONSUMER 1] message #%d", i);
   }
   // Default Handler re-activated 
   Ndb_cluster_connection::set_log_consumer(nullptr);
}

User program file

Configuring MySQL NDB Cluster Logging

MySQL NDB Cluster logging is primarily configured through the config.ini file for the MGM node and potentially through command-line options or node-specific configuration files for the data nodes. Key parameters to be aware of include:

  • DataDir (Data Nodes): While primarily for data files, the log files (ndbd.log) are created within this directory. The default is the data node process working directory.
  • DataDir (MGM nodes): Specifies the directory where output files, including log files, from the management server are placed. For log files, this location can be overridden by LogDestination parameter.
  • LogDestination: This parameter specifies where to send cluster logging information. There are three options: CONSOLE, SYSLOG, and FILE, with FILE being the default.
  • ndb-log-timestamps: Command-line option for both management and data nodes to control the format of timestamps prepended to log messages, with possible values of ‘LEGACY’, ‘UTC’ (default), or ‘SYSTEM’.

Depending on the chosen LogDestination option (CONSOLE, SYSLOG, or FILE), different sub-parameters become available for configuring the logging behavior, such as syslog facilities for SYSLOG and filename, maxsize, and maxfiles for FILE.

On a Windows environment, if the LogDestination is set to SYSLOG the MySQL NDB Cluster logging will automatically write events and error messages to the Windows Event Log.

Consult the MySQL NDB Cluster documentation for the specific configuration parameters available in your version.

Analyzing MySQL NDB Cluster Logs

Effectively analyzing MySQL NDB Cluster logs is crucial for troubleshooting and performance optimization. Here are some tips:

  • Correlate Events by Timestamp: Due to the distributed nature, ensure your system clocks are synchronized. This will help you piece together events that span multiple nodes.
  • Identify Node-Specific Issues: Focus on the logs of the node type that is exhibiting the problem.
  • Search for Error Keywords: Use tools like grep to search for keywords like ERROR, WARNING, Failed, Timeout, etc.
  • Understand MySQL NDB Cluster Error Codes: MySQL NDB Cluster often uses specific error codes. Referencing the documentation for these codes can provide valuable insights.
  • Look for Patterns: Recurring warnings or errors can indicate underlying issues.

Conclusion

MySQL NDB Cluster’s distributed nature requires robust logging for effective management. Logs from MySQL Servers, MGM nodes, and Data nodes provide essential information for troubleshooting, performance optimization, and maintaining high availability. MySQL NDB Cluster offers users control over logging with flexible configuration options and, since version 8.4, custom logging functions are available in both Community and Commercial versions. By leveraging these features and employing effective log analysis, users can proactively address issues and ensure reliable operation.

References

https://dev.mysql.com/doc/refman/8.4/en/mysql-cluster-logging-management-commands.html

https://dev.mysql.com/doc/refman/8.4/en/mysql-cluster-mgm-definition.html