Introducing TLS in MySQL NDB Cluster

MySQL NDB Cluster 8.3.0 introduces the use of TLS to secure the network protocols used inside MySQL NDB Cluster. NDB Cluster is a distributed database system that provides very high availability and real-time performance, and this important change makes it possible to deploy cluster nodes not just on trusted, private networks, but also on untrusted networks and public cloud services. NDB uses two protocols for inter-node communication: one we know as the management protocol, and the other as the transporter protocol. Securing these protocols requires understanding some design considerations that are particular to NDB, so the result may have some surprises in store, even for someone familiar with other applications of TLS.

TLS

Many people are familiar with TLS from the Web, where HTTPS servers present signed certificates to browsers, and the browsers verify these certificates. This provides both encryption on the network and protection against man-in-the-middle attacks. This type of security in itself is not quite sufficient for NDB. In the MySQL ecosystem, NDB sits at the storage level, below the layer of MySQL’s security subsystem; every NDB node joins the cluster as a peer and has the ability to manipulate the database. Merely encrypting NDB’s network protocols, without also authenticating the nodes that use them, would not provide meaningful security. Therefore, NDB TLS employs mutual TLS authentication. In a TLS connection between two NDB nodes, each node presents its own certificate and also verifies the other’s certificate. The certificates of the two nodes must both be valid, and both signed by the same Certification Authority. This means that all cluster nodes – including not just MySQL servers and NDB data nodes, but also internal client programs like ndb_mgm and ndb_restore – are required to have valid TLS certificates and private keys in order to use TLS.

Many applications support TLS version 1.2 and older versions of the protocol, but within an NDB cluster, TLS version 1.3 is strictly required. MySQL clients can still connect to the MySQL server using TLS 1.2, since nothing about NDB’s internal TLS changes the way TLS is used between the MySQL server and its clients.

Certificates and Expiration Dates

All TLS certificates have limited validity, and at the end of their lifetime, they expire. Once a node’s certificate expires, any subsequent TLS connection to that node will fail. For NDB, the possibility of an expiring certificate must be considered as a source of node failure. NDB is designed with availability in mind, so it is important to minimize node failures, contain them in scope, and recover from them. One helpful strategy is to use a set of certificates with different expiration dates. Ideally, no node will ever fail because of an expired certificate, but with randomly staggered dates, at least multiple simultaneous failures are avoided. To facilitate certificate management, we are introducing a new tool called ndb_sign_keys. This tool is designed to make it easy to create a CA on one host, and then create private keys and signed node certificates for each host in the cluster, without ever having to copy a private key across the network. Node certificates will have staggered, randomized expiration dates, according to this scheme:

• API node certificates begin to expire after 120 days
• Data node certificates begin to expire after 130 days
• Management server certificates expire on the 150th day.

We hope that ndb_sign_keys is useful, but it is not the only way to create valid certificates. You can  alternately use OpenSSL, cloud-vendor tools, or some other convenient toolchain.

The Certificate Search Path

At startup time, each NDB node will need to find its private key and the associated certificate in its local filesystem. This is configured using a search path: a list of directories delimited by a colon on Unix, or by a semicolon on Windows. The search path is supplied using the command-line option --ndb-tls-search-path, which is accepted by all NDB-related programs and is described in the online manual here. It can be supplied in the [mysql_cluster] section of a my.cnf file, as follows:

The NDB Management Protocol

The NDB management protocol uses “opportunistic” TLS: every connection is initially established using clear text, and then the client issues a command within the protocol to begin the TLS handshake. In MySQL 8.3, every cluster node and NDB tool has been modified to automatically make this START TLS request whenever it finds a usable key and certificate in its filesystem. This means that simply creating certificates and setting the TLS search path enables a significant step toward a system that is “secure by default.” This can be verified in the ndb_mgm client utility using a new “TLS INFO” command, which provides a set of statistics on total current management protocol connections, current management connections that are using TLS, and the total accumulated number of each of these since the management server was last restarted.

To enforce security, once TLS is configured and verified to be working, the management server can be configured to require TLS for all non-trivial commands. With this change, all  commands in the management protocol (except for the “START TLS” command itself, and a small number of other “bootstrap” commands) can only be executed on connections secured by TLS. This configuration change is made in the cluster configuration file, and can be deployed by restarting the management server.

When RequireTls is enabled on a management server, any client that attempts to run a non-bootstrapping command on a cleartext connection will receive an authorization error. The total count of authorization errors is reported in the TLS INFO output.

The NDB Transporter Protocol

Management protocol TLS is enabled by default; therefore any connection from an API node or data node to a management server will use TLS, as long as both nodes are TLS-capable. Connections that do not include a management server do not by default use TLS. When we consider a connection between two data nodes, or a connection from an API node to a data node, both of these connections use the NDB transporter protocol. Transporter TLS adds encryption and synchronization requirements to NDB’s core transaction messages, and this may impose a performance cost that could affect query performance. It might, for instance, require the thread configuration in data nodes to be tuned in order to meet the same performance requirements with transporter TLS enabled that were previously obtained without TLS. Because of this cost, a transporter connection from an API node to a DB node will use TLS only if the data node is configured to require TLS. For a connection between two data nodes, TLS is used if either of them requires TLS. The final step in fully securing networking in an NDB cluster is to require TLS for data node connections. This can be accomplished by editing the cluster configuration to set RequireTls for all data nodes, and then performing a rolling restart of the entire cluster.

Monitoring NDB TLS from the MySQL Server

To monitor the status of Transporter TLS in the MySQL server, the newly introduced ndbinfo.certificates table provides a view of all certificates as seen by the data nodes. The existing ndbinfo.transporters table now has an encrypted column indicating whether each transporter connection is secure. The following queries present an example from a cluster in which MGM connections use TLS, but data node connections do not. 

mysql> select * from ndbinfo.certificates;
+---------+------------------------------+-------------+-------------------------------+
| Node_id | Name                         | Expires     | Serial                        |
+---------+------------------------------+-------------+-------------------------------+
|       1 | NDB Data Node Mar 2023       | 19-Apr-2024 | B7:15:32:D5:AE:FF:C7:E1:E2:4A |
|       3 | NDB Management Node Mar 2023 | 19-Apr-2024 | 20:70:03:B8:BE:5F:C7:FB:A8:3E |
|       2 | NDB Data Node Mar 2023       | 19-Apr-2024 | B7:15:32:D5:AE:FF:C7:E1:E2:4A |
+---------+------------------------------+-------------+-------------------------------+
3 rows in set (0.08 sec)

mysql> select node_id, remote_node_id, encrypted from ndbinfo.transporters;
+---------+----------------+-----------+
| node_id | remote_node_id | encrypted |
+---------+----------------+-----------+
|       1 |              2 |         0 |
|       1 |              3 |         1 |
|       1 |             52 |         0 |
|       1 |             51 |         0 |
|       2 |              1 |         0 |
|       2 |             51 |         0 |
|       2 |              3 |         1 |
|       2 |             52 |         0 |
+---------+----------------+-----------+
8 rows in set (0.07 sec)

For More Information

In this post I have touched on the major design points of NDB TLS and the most fundamental things to know about it. There are many more details available in the TLS Link Encryption for NDB Cluster section in the online manual.