MySQL NDB Cluster Manager 8.4, or MCM 8.4, was recently released as part of the 8.4 LTS (Long Term Support) release series by the MySQL team.
If you are unfamiliar with MCM, it is a commercial addition, available with the commercial version of MySQL NDB Cluster. MCM simplifies the creation and management of your MySQL NDB Cluster database by automating common management tasks including on-line scaling, upgrades, backups, reconfiguration, and tuning. MCM works equally well both in OCI computes, and on-premise.
More information on MCM is available in the docs, at https://www.mysql.com/products/cluster/mcm/
TLS with MySQL NDB Cluster 8.4
Improved security was one of the most critical aspects when improving MySQL NDB Cluster 8.4.
The addition of TLS support to MySQL NDB Cluster makes it possible to have authenticated and encrypted connections between the cluster nodes – Management Nodes, Data Nodes and API/SQL Nodes. There is already a great blog covering the details of TLS in MySQL NDB Cluster – https://blogs.oracle.com/mysql/post/introducing-ndb-tls – and more details in the Documentation.
Do note that NDB requires TLSv1.3.
TLS with MCM 8.4
MCM 8.4 added support for TLS for client – mcmd agent connections, as well as support for enabling TLS in MySQL NDB Cluster itself. The following describes how to enable TLS with MCM 8.4.
Creating certificates
The first step is creating the certificates needed to enable TLS, using openssl.
Create a CA and its private key:
$ mkdir my_certs $ cd my_certs $ openssl genrsa 2048 > ca-key.pem $ openssl req -new -x509 -nodes -days 3600 -key ca-key.pem -out ca.pem
Create a server certificate and its private key, and sign it:
$ openssl req -newkey rsa:2048 -days 3600 -nodes -keyout server-key.pem -out server-req.pem $ openssl rsa -in server-key.pem -out server-key.pem $ openssl x509 -req -in server-req.pem -days 3600 -CA ca.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem
Lastly create a client certificate, remove its passphrase, and sign it:
$ openssl req -newkey rsa:2048 -days 3600 -nodes -keyout client-key.pem -out client-req.pem $ openssl rsa -in client-key.pem -out client-key.pem $ openssl x509 -req -in client-req.pem -days 3600 -CA ca.pem -CAkey ca-key.pem -set_serial 01 -out client-cert.pem
We now have 6 certificate files:
- ca.pem – the CA
- ca-key.pem – the CAs private-key
- server-cert.pem – the servers public key
- server-key.pem – the servers private key
- client-cert.pem – the client public key
- client-key.pem – the client private key
More information on creating keys for use with MySQL can be found in the documentation, in the chapter “Creating SSL Certificates and Keys Using openssl“.
Enabling TLS for mcmd client connections
Next we enable TLS for mcmd client connections.
For this example we will use the certificates files generated in the my_certs directory in the previous section.
Enable TLS by adding the TLS configuration options ssl-key and ssl-cert, and set ssl_mode to REQUIRED to only allow secure connections to mcmd:
$ ./mcm8.4.2/bin/mcmd --mcmd.ssl_key=my_certs/server-key.pem --mcmd.ssl_cert=my_certs/server-cert.pem --mcmd.ssl_mode=REQUIRED
The necessary configuration options may also be added to the mcmd.cnf file:
[mcmd] ssl_key=my_certs/server-key.pem ssl_cert=my_certs/server-cert.pem
The mcm client can then connect to mcmd using the client key and certificate files:
$ ./mcm8.4.2/bin/mcm --ssl-key=my_certs/client-key.pem --ssl-cert=my_certs/client-cert.pem ...
The client ‘status‘ command will show TLS details of the current connection in the ‘SSL‘ field:
mcm> status
--------------
./mcm8.4.2/bin/mcm Ver 8.4.2-commercial for Linux on x86_64 (MySQL Enterprise Server - Commercial)
Connection id: 0
Current database: <n/a>
Current user: mcmd
SSL: Cipher in use is TLS_AES_128_GCM_SHA256
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server version: 8.4.2 MySQL Cluster Manager
Protocol version: 10
Connection: 127.0.0.1 via TCP/IP
Server characterset: latin1
Db characterset: latin1
Client characterset: latin1
Conn. characterset: latin1
TCP port: 1862
Binary data as: Hexadecimal
Uptime: 4 min 21 sec
Agent no: 0 Connections: 1 Max msg: {c33f4fe7 117 0}
--------------
Enabling NDB TLS with MCM
If you have read the NDB TLS blog post, you may realize that the complexity increases rapidly both with the number of processes, different process types, as well as the number of hosts in your cluster. MCM 8.4 adds a ‘create certs‘ command to simplify the process, and improve the ease of use. Note that each NDB instance under MCM has its own separate set of certificate files, that are not the same as those used by MCM itself.
We will use an OCI compute with EL8 for the following. Once the compute is running and the binaries for mcmd and NDB are installed, we bootstrap a cluster for testing purposes using the mcmd –bootstrap option. Please refer to the MCM documentation for more details on –bootstrap, or for how to define your own MCM setup across multiple computes. As we want TLS to be required, we set the ssl_key,ssl_cert, and ssl_mode options to mcmd as discussed in the previous section.
$ ./mcm8.4.2/bin/mcmd --bootstrap --mcmd.ssl_key=my_certs/server-key.pem --mcmd.ssl_cert=my_certs/server-cert.pem --mcmd.ssl_mode=REQUIRED
MySQL Cluster Manager 8.4.2 (64bit) started
Connect with "/export/home/tmp/mcm8.4.2/bin/mcm" -h lagoon -P 1862
Configuring default cluster 'mycluster'...
Setting default_storage_engine to ndbcluster...
Starting default cluster 'mycluster' version '8.4.2-cluster'...
Cluster 'mycluster' started successfully
ndb_mgmd lagoon:1186
ndbmtd lagoon
ndbmtd lagoon
mysqld lagoon:3306
mysqld lagoon:3307
ndbapi *
Connect to the database by running "/export/home/tmp/cluster/bin/mysql" -h 127.0.0.1 -P 3306 -u root
Cluster booted and ready to use in about 30 seconds!
To enable TLS in a running MySQL NDB Cluster, there is a series of steps needed
- define ndb_tls_search_path for all processes
- generate a cluster CA and key, using ndb_sign_keys
- generate a set of cluster node certificates and keys, using ndb_sign_keys
- restart all cluster processes to load certificates
- enable the RequireTls with a rolling restart
- optionally: enable RequireCertificates and set ndb_mgm_tls to STRICT
MCM simplifies these tasks for you.
First we set and check the ndb_tls_search_path configuration attribute is set
mcm> set ndb_tls_search_path:ndb_mgmd=/export/home/tmp/mcm_data/clusters/mycluster/certs,ndb_tls_search_path:ndbmtd=/export/home/tmp/mcm_data/clusters/mycluster/certs,ndb_tls_search_path:mysqld=/export/home/tmp/mcm_data/clusters/mycluster/certs mycluster; +-----------------------------------+ | Command result | +-----------------------------------+ | Cluster reconfigured successfully | +-----------------------------------+ 1 row in set (1 min 29.21 sec) mcm> get -d ndb_tls*: mycluster; +---------------------+----------------------------------------------------+----------+---------+----------+---------+---------+--------------+ | Name | Value | Process1 | NodeId1 | Process2 | NodeId2 | Level | Comment | +---------------------+----------------------------------------------------+----------+---------+----------+---------+---------+--------------+ | ndb_tls_search_path | /export/home/tmp/mcm_data/clusters/mycluster/certs | ndb_mgmd | 145 | | | Process | Command Line | | ndb_tls_search_path | /export/home/tmp/mcm_data/clusters/mycluster/certs | ndbmtd | 1 | | | Process | Command Line | | ndb_tls_search_path | /export/home/tmp/mcm_data/clusters/mycluster/certs | ndbmtd | 2 | | | Process | Command Line | | ndb_tls_search_path | /export/home/tmp/mcm_data/clusters/mycluster/certs | mysqld | 146 | | | Process | | | ndb_tls_search_path | /export/home/tmp/mcm_data/clusters/mycluster/certs | mysqld | 147 | | | Process | | +---------------------+----------------------------------------------------+----------+---------+----------+---------+---------+--------------+ 5 rows in set (0.10 sec)
Next, create the CA and node certificates with the MCM ‘create certs‘ command to create the CA, certificates and keys using the ndb_sign_keys utility delivered with NDB:
mcm> create certs mycluster; +-----------------------------------+ | Command result | +-----------------------------------+ | Certificates created successfully | +-----------------------------------+ 1 row in set (8.41 sec)
Inspecting the contents of the ndb_tls_search_path folder, we find that MCM has created the CA, certificates and keys as required by NDB:
$ ls /export/home/tmp/mcm_data/clusters/mycluster/certs/ NDB-Cluster-cert NDB-Cluster-private-key ndb-api-cert ndb-api-private-key ndb-data-node-cert ndb-data-node-private-key ndb-mgm-server-cert ndb-mgm-server-private-key
The cluster must now be restarted for cluster nodes to load the certificates:
mcm> restart cluster mycluster; +--------------------------------+ | Command result | +--------------------------------+ | Cluster restarted successfully | +--------------------------------+ 1 row in set (1 min 52.65 sec)
Examining the logfile of datanode 1, we see that the certificate file has been loaded, from the location set earlier, during restart:
2024-08-23 23:02:19 [ndbd] INFO -- NDB TLS 1.3 available using certificate file '/export/home/tmp/mcm_data/clusters/mycluster/certs/ndb-data-node-cert'
Configure cluster to use TLS by enabling RequireTls for the cluster proceses:
mcm> set RequireTls:ndb_mgmd=true,RequireTls:ndbmtd=true mycluster; +-----------------------------------+ | Command result | +-----------------------------------+ | Cluster reconfigured successfully | +-----------------------------------+ 1 row in set (1 min 54.75 sec)
Cluster is now running with TLS enabled and required.
One may also enable RequireCertificate, and/or set either ndb-mgm-tls mode to STRICT to secure NDB further:
mcm> set RequireCertificate:ndb_mgmd=true,RequireCertificate:ndbmtd=true,ndb_mgm_tls:ndb_mgmd=strict,ndb_mgm_tls:ndbmtd=strict mycluster; +-----------------------------------+ | Command result | +-----------------------------------+ | Cluster reconfigured successfully | +-----------------------------------+ 1 row in set (1 min 56.05 sec)
Verifying TLS is active
Using the example from the NDB TLS blog, we can verify that TLS is active by looking at the encrypted column of the ndbinfo.transporters table:
mysql> select * from ndbinfo.certificates; +---------+---------------------------------+-------------+-------------------------------+ | Node_id | Name | Expires | Serial | +---------+---------------------------------+-------------+-------------------------------+ | 1 | NDB Data Node August 2024 | 01-Jan-2025 | 5A:25:47:8A:23:EA:A8:CE:4C:CC | | 2 | NDB Data Node August 2024 | 01-Jan-2025 | 5A:25:47:8A:23:EA:A8:CE:4C:CC | | 145 | NDB Management Node August 2024 | 20-Jan-2025 | 47:AE:E9:78:B2:FA:70:F5:A9:B4 | | 146 | NDB Node August 2024 | 30-Dec-2024 | 3E:0A:E3:DA:69:DE:F3:B8:BB:4F | | 147 | NDB Node August 2024 | 30-Dec-2024 | 3E:0A:E3:DA:69:DE:F3:B8:BB:4F | +---------+---------------------------------+-------------+-------------------------------+ 5 rows in set (0.02 sec) mysql> select node_id, remote_node_id, encrypted from ndbinfo.transporters; +---------+----------------+-----------+ | node_id | remote_node_id | encrypted | +---------+----------------+-----------+ | 1 | 2 | 1 | | 1 | 145 | 1 | | 1 | 146 | 1 | | 1 | 147 | 1 | | 1 | 148 | 0 | | 2 | 1 | 1 | | 2 | 145 | 1 | | 2 | 146 | 1 | | 2 | 147 | 1 | | 2 | 148 | 0 | +---------+----------------+-----------+ 10 rows in set (0.04 sec)
Another option is verifying that TLS is active by using the TLS INFO command in the ndb_mgm client:
$ ./cluster/bin/ndb_mgm --ndb-tls-search-path=mcm_data/clusters/mycluster/certs -- NDB Cluster -- Management Client -- ndb_mgm> TLS INFO Connected to management server at localhost port 1186 (using TLS) Main interactive connection is using TLS Event listener connection is using TLS Server reports 10 TLS connections. Session ID: 23 Peer address: :: Certificate name: NDB Node August 2024 Certificate serial: 3E:0A:E3:DA:69:DE:F3:B8:BB:4F Certificate expires: 30-Dec-2024 Session ID: 22 Peer address: 127.0.0.1 Certificate name: NDB Node August 2024 Certificate serial: 3E:0A:E3:DA:69:DE:F3:B8:BB:4F Certificate expires: 30-Dec-2024 Session ID: 21 Peer address: 127.0.0.1 Certificate name: NDB Node August 2024 Certificate serial: 3E:0A:E3:DA:69:DE:F3:B8:BB:4F Certificate expires: 30-Dec-2024 Session ID: 20 Peer address: 100.103.30.7 Certificate name: NDB Node August 2024 Certificate serial: 3E:0A:E3:DA:69:DE:F3:B8:BB:4F Certificate expires: 30-Dec-2024 Session ID: 19 Peer address: 100.103.30.7 Certificate name: NDB Node August 2024 Certificate serial: 3E:0A:E3:DA:69:DE:F3:B8:BB:4F Certificate expires: 30-Dec-2024 Session ID: 18 Peer address: 100.103.30.7 Certificate name: NDB Node August 2024 Certificate serial: 3E:0A:E3:DA:69:DE:F3:B8:BB:4F Certificate expires: 30-Dec-2024 Session ID: 17 Peer address: 100.103.30.7 Certificate name: NDB Node August 2024 Certificate serial: 3E:0A:E3:DA:69:DE:F3:B8:BB:4F Certificate expires: 30-Dec-2024 Session ID: 16 Peer address: 100.103.30.7 Certificate name: NDB Data Node August 2024 Certificate serial: 5A:25:47:8A:23:EA:A8:CE:4C:CC Certificate expires: 01-Jan-2025 Session ID: 8 Peer address: 100.103.30.7 Certificate name: NDB Node August 2024 Certificate serial: 3E:0A:E3:DA:69:DE:F3:B8:BB:4F Certificate expires: 30-Dec-2024 Session ID: 1 Peer address: 127.0.0.1 Certificate name: NDB Management Node August 2024 Certificate serial: 47:AE:E9:78:B2:FA:70:F5:A9:B4 Certificate expires: 20-Jan-2025 Server statistics since restart Total accepted connections: 23 Total connections upgraded to TLS: 23 Current connections: 9 Current connections using TLS: 9 Authorization failures: 0
The output tells us that both the ndb_mgm client connection to the management server, as well as the internal connections between the cluster nodes, all use TLS.
It’s easy to enable TLS in MySQL NDB Cluster 8.4 with MCM 8.4, and this improves the security of NDB. Enjoy!