MySQL Operator for Kubernetes?
The MySQL Operator for Kubernetes supports the lifecycle of a MySQL InnoDB Cluster inside a Kubernetes Cluster. This goes from simplifying the deployment of MySQL Server and MySQL Router instances, including management of TLS certificates and replication setup, over ongoing management of those as well as support for backups, be it one-of backups or following a schedule.
The MySQL Operator for Kubernetes is a controller, in the terms of Kubernetes, that manages MySQL InnoDBClusters (IC) on Kubernetes. The clusters are created and managed by creating custom resources which adhere to Custom Resource Definitions (CRDs) defined by the operator during installation. Amongst the tasks that the operator does is automatization of the deployment, operation, and management of IC instances, handling tasks like:
- Provisioning and scaling
- High Availability
- Backup and restore
- Log collection / Log shipment
- TLS life cycle handling
- TDE and keyring support
- Configuration management
- Failover and recovery
In this blogpost, we will see how to install the MySQL Operator for Kubernetes and how to create MySQL InnoDB Clusters in K8s with the help of the operator. In a later, we will explore two advanced topics, “High Availability” (HA) and “Failover and recovery” (FnR). As of MySQL Operator version 9.3.0-2.2.4 it is possible to create ClusterSets, which bring the HA and FnR to a new level. But as mentioned, we will leave that for the next post.
An IC is created by creating a custom resource (CR) of kind InnoDBCluster. This could be done in two ways:
- By creating a manifest file, typically written in YAML, but could also be a JSON, of this kind and then applying it using kubectl, the K8s client CLI.
- By using Helm, which is a package manager for Kubernetes. The MySQL Operator has a Helm Chart, a resource, that when imported into the local Helm repository, is used for installing, modifying and upgrading IC. The Helm chart takes care of creating the InnoDBCluster CR by using values passed by the administrator as arguments to the Helm chart (either on the command line with using –set arguments or by providing a yaml file, typically named values.yaml, which contains are parameters needed)
Installing the Helm charts
Before creating our first IC, we need to add the MySQL Operator Charts (there are two of them) to the local repository. A very simple command will achieve this:
$ helm repo add mysql-operator-repo https://mysql.github.io/mysql-operator/
$ helm repo update
To see the just installed charts the following command is very helpful:
$ helm search repo mysql
NAME CHART VERSION APP VERSION DESCRIPTION
mysql-operator-repo/mysql-innodbcluster 2.2.4 9.3.0 MySQL InnoDB Cluster Helm Chart for deploying M...
mysql-operator-repo/mysql-operator 2.2.4 9.3.0-2.2.4 MySQL Operator Helm Chart for deploying MySQL I..
Installing the Operator
The next step to install the MySQL Operator will be using the mysql-operator-repo/mysql-operator chart, where the first part is the name of the repository which we used in the helm repo command, and the second part is the name of the chart.
$ helm install mysql-operator mysql-operator-repo/mysql-operator --namespace mysql-operator --create-namespace
mysql-operator is the name of our release, which is the Helm term for a deployed instance of a Helm chart in a K8s cluster. Every release has to have an unique name and inherently has a version number, starting from 1, which is incremented every time the release is modified. Versions are important, as they provide auditability (diffs) but also a way to revert changes (a rollback) by going back to a previous release.
We instructed Helm to install the operator in the mysql-operator namespace and because this namespace doesn’t exist we instructed Helm to create it.
We can quickly check if the operator is up and running:
$ kubectl -n mysql-operator get pods
NAME READY STATUS RESTARTS AGE
mysql-operator-78dff6cd5c-28qv5 1/1 Running 0 1m15s
Alright. Let’s quickly inspect the operator container log, as usually it contains valuable information. We can do that by inspecting the operator pod log or by using the less known but very nice feature of Kubernetes to inspect the mysql-operator deployment log, as the operator runs as a K8s Deployment). It is very handy, because we don’t need to write the pod name suffix, which is unique to a pod, and changes from installation to installation, from pod incarnation to pod incarnation.
$ kubectl -n mysql-operator logs deploy/mysql-operator | head -n 15
2025-05-30 09:48:59 [1:1]: Info: mysqlsh Ver 9.3.0 for Linux on x86_64 - for MySQL 9.3.0 (MySQL Community Server (GPL)) - build 18939594 - commit_id 1e2c0b770d0dd8b0d4cdac135ccef1b6dcbc134e - product_id el9-x86-64bit rpm
2025-05-30 09:48:59 [1:1]: Info: Using credential store helper: /usr/bin/mysql-secret-store-login-path
2025-05-30 09:48:59 [1:1]: Info: Setting Python home to '/usr/lib/mysqlsh'
2025-05-30 09:48:59 [1:1]: Info: Loading startup files...
2025-05-30 09:48:59 [1:1]: Info: Loading plugins...
[2025-05-30 09:49:00,956] root [INFO ] Auto-detected cluster domain: cluster.local
[2025-05-30 09:49:00,961] kopf.activities.star [INFO ] MySQL Operator/operator.py=2.2.4 timestamp=2025-05-08T12:07:42 kopf=1.37.4 uid=2
[2025-05-30 09:49:00,969] kopf.activities.star [INFO ] KUBERNETES_VERSION =1.28
[2025-05-30 09:49:00,969] kopf.activities.star [INFO ] OPERATOR_VERSION =2.2.4
[2025-05-30 09:49:00,970] kopf.activities.star [INFO ] OPERATOR_EDITION =community
[2025-05-30 09:49:00,970] kopf.activities.star [INFO ] OPERATOR_EDITIONS =['community', 'enterprise']
[2025-05-30 09:49:00,970] kopf.activities.star [INFO ] SHELL_VERSION =9.3.0
[2025-05-30 09:49:00,970] kopf.activities.star [INFO ] DEFAULT_VERSION_TAG=9.3.0
[2025-05-30 09:49:00,970] kopf.activities.star [INFO ] SIDECAR_VERSION_TAG=9.3.0-2.2.4
[2025-05-30 09:49:00,970] kopf.activities.star [INFO ] DEFAULT_IMAGE_REPOSITORY =container-registry.oracle.com/mysql
There are quite a few details in the first log lines. Let’s take a look at them and see what they mean and some further details regarding them
- The operator running is based on MySQL Shell version 9.3.0
- The Kubernetes cluster domain got autodetected to be cluster.local. Sometimes, for various reasons, the operator can’t detect the domain name and the actual domain name needs to be provided by the administrator. To achieve this –set envs.k8sClusterDomain=<domain.tld> has to be passed to the Helm chart, or be set in the values file.
- The version of the Kubernetes Control Plane is 1.28 . Why the control plane version? Because the data plane, the worker nodes, might have different versions, like unupgraded worker nodes after control plane upgrade. In most cases, however, all nodes, control and data plane are on the same version.
- The community edition operator is running. We see that the possible versions are community and enterprise.
- The default version for the MySQL Servers and Routers is 9.3.0, while the sidecar containers in the IC StatefulSet (STS) will use the 9.3.0-2.2.4 version (or tag).
The container images will be pulled from the official container registry/repository at container-registry.oracle.com/mysql
Using own container registry
In some cases the administrator might want to use a private container registry:
- Because image caching for faster startup times is important
- Because the Kubernetes cluster is on an air-gapped network
In these cases, the image repository from where the operator image will be pulled has to be changed by passing two arguments that combined generate the full URI to the images. If, for example, our images are at mycr.example.com:5000/mysql then use:
- –set image.registry=mycr.example.com:5000 to set the CR host
- –set image.repository=mysql to set the path of the CR host
In the advanced case of an authenticated registry, the administrator has to provide additional arguments:
- –set image.pullSecrets.enabled=true
- –set image.pullSecrets.secretName=<secret_name> , where <secret_name> should already exist and should be of type docker-registry
An quick example showing creating such a secret in the operator namespace (the –docker-email field is not important) is:
$ kubectl -n mysql-operator create secret docker-registry priv-reg-secret \
--docker-server=https://mycr.example.com:5000/v2/ \
--docker-username=user \
--docker-password=pass \
--docker-email=user@example.com
In case of a private registry, the administrator probably will also need to use:
- –set envs.imagesDefaultRegistry and –set envs.imagesDefaultRepository, which will be used for the server, router and the sidecar images
- If the ImagePullPolicy is to be changed from the default IfNotPresent, to something else, then use –set envs.imagesPullPolicy
In addition, when using the mysql-operator-repo/mysql-innodbcluster chart, which we will use below, the name of the private registry secret, if any, needs to be provided. The Helm chart will create a Kubernetes ServiceAccount and list the secret as an imagePullSecret set in that ServiceAccount. Then this ServiceAccount will be added to the IC manifest and be used for pulling container images. To achieve this, use :
- –set image.pullSecrets.enabled=true
- –set image.pullSecrets.secretName=name_of_secret
This secret should exist in the cluster namespace. See above how to create it and don’t forget to use the correct namespace.
Creating our first IC
Once the operator is up and running we can create this values.yaml file for our first IC:
credentials:
root:
user: root
password: sakila
host: "%"
tls:
useSelfSigned: true
serverInstances: 3
router:
instances: 1
baseServerId: 1000
We set values for the root user name, password and host mask. For the sake of simplicity, in this blogpost we don’t explore the creation of TLS certificates and just ask the IC servers to use self-signed certificates by itself. We want to have an IC with three MySQL Server instances, the minimal number of servers for a local HA setup, but only one MySQL Router instance. The base server ID will be 1000. As we will have 3 IC servers, their server IDs will be 1000, 1001 and 1002. Using base server ID in this case is not obligatory but in a ClusterSet environment it will be so.
Then we can create our IC by issuing the following command, which uses the second Helm chart, mysql-operator-repo/mysql-innodbcluster, that we added to our local Helm repository:
$ helm install mycluster mysql-operator-repo/mysql-innodbcluster --values values.yaml --namespace myclusterns --create-namespace
NAME: mycluster
LAST DEPLOYED: Fri May 30 10:22:48 2025
NAMESPACE: myclusterns
STATUS: deployed
REVISION: 1
TEST SUITE: None
Where mycluster is the name of our IC release.
For the sake of clarity, we could have decided not to use a values.yaml file and pass all arguments on the command line. In this case the command would have looked like this:
$ helm install mycluster mysql-operator-repo/mysql-innodbcluster \
--namespace myclusterns --create-namespace \
--set credentials.root.user=root \
--set credentials.root.password=sakila \
--set credentials.root.password=% \
--set tls.useSelfSigned=true \
--set serverInstances=3 \
--set router.instances=1 \
--set baseServerId=1000
We can also mix –values and –set in one command. The usage of –set for passing values that contain lists is complicated and in this case the best solution is to use a values.yaml file. For more information on this topic, please refer to the Helm documentation.
As already mentioned previously, in case of a private registry, when using the mysql-operator-repo/mysql-innodbcluster chart, which we will use below, the name of the private registry secret, if any, needs to be provided.
Here is an example output (shortened a bit) of the first command:
$ kubectl -n myclusterns get pods -w
NAME READY STATUS RESTARTS AGE
mycluster-0 0/2 Init:2/3 0 28s
mycluster-1 0/2 Init:2/3 0 28s
mycluster-2 0/2 Init:2/3 0 28s
mycluster-1 0/2 PodInitializing 0 62s
mycluster-2 0/2 PodInitializing 0 62s
mycluster-0 0/2 PodInitializing 0 62s
mycluster-1 1/2 Running 0 63s
mycluster-2 1/2 Running 0 63s
mycluster-0 1/2 Running 0 63s
mycluster-0 2/2 Running 0 74s
mycluster-1 2/2 Running 0 74s
mycluster-2 2/2 Running 0 74s
mycluster-router-dfc4f9d6f-jng4d 0/1 Pending 0 0s
mycluster-router-dfc4f9d6f-jng4d 0/1 ContainerCreating 0 0s
mycluster-router-dfc4f9d6f-jng4d 0/1 ContainerCreating 0 1s
Let’s elaborate on the output we see:
- We specified mycluster as the name of our Helm release and this name is reused for the IC StatefulSet (STS). Hence, the names of the server pods being mycluster-0, mycluster-1 and mycluster-2. The numbering is performed by the K8s STS controller. mycluster is also used as a prefix for the router Deployment. Deployments don’t get predictable ordinal numbers as suffixes, like STS pods do, but unique following the schema <deployment-name>-<replicaset-id>-<pod-id>, where:
- <deployment-name> is the name of the Deployment
- <replicaset-id> is an unique identifier for the ReplicaSet managing the current generation (version) of the Deployment (extended ReplicaSet) and this is typically a hash or random string. This remains the same for all Pods in the same ReplicaSet (i.e., for the same generation/version of the Deployment).
- <pod-id> is an unique identifier for the specific pod instance, and also a random string. If a pod is terminated, the Kubernetes Deployment controller (via the ReplicaSet) creates a new pod to maintain the desired replica count. The new Pod will have the same <replicaset-id>, since it’s part of the same Deployment version, and a different <pod-id>, since it’s a new pod instance.
- The server pods go through the Init, PodInitializing phases and end up in the Running phase. In the Init phase, in the Status column we see 2/3. This means, two of the init containers have already started and have finished (execution of init containers is sequential), and the third init container will be executed or is currently executing. The init containers prepare the pod for the MySQL Server run. Depending on the environment the IC pods have two or three init containers, that set file system rights in the Kubernetes PV and PVCs, create the proper configuration (think my.cnf) for MySQL, and then start the MySQL Server to initialize the datadir in the PV/PVC.
- After that the pods move to the PodInitializing and the Running phases. The column Status gets to 1/2, which shows that one of the two Readiness Gates (readiness flags) of the STS has been set. The names of the gates are mysql.oracle.com/configured and mysql.oracle.com/ready . The former is set by the sidecar container of a particular pod after the MySQL Server in the pod is up and running, and additional configuration and possible restore from a clone or a backup, has been performed. A configured pod is still not part of the IC. The operator watches for events like setting a readiness gate and reacts accordingly. If there is still no IC created it will create one, as of Operator 9.3.0-2.2.4 it will create a ClusterSet and then a primary IC in this CS. Pre 9.3.0-2.2.4, only a primary IC will be created and there is no ClusterSet support. Hence, IC created with Operator pre-9.3.0-2.2.4 cannot become a part of a ClusterSet, and so cannot get replica clusters attached to them. For the second, third, etc. pod, the operator will join them to the IC. In case of 9.3.0-2.2.4 or later they will be added to their IC, which could be a primary or replica. More information on ClusterSet will be provided in another blog post.
- Once the servers are up and running, both readiness gates are set, the CS is formed, the IC has been created and all pods/servers have joined the IC (being it primary or replica), eventually the router pods will be created. In fact, the Router Deployment is created together with the STS but the replica count is set to 0, so no router pods are created. Once the IC is formed the replica count is updated to the number specified by the administrator. This update forces the Kubernetes Deployment controller to create and initialize all router pods. Hence, the creation of the router pods is an external sign that the IC is in a running state. If no router pods ever start there is probably an error and the operator log needs to be inspected for more information.
We can now inspect the IC using kubectl:
$ kubectl -n myclusterns describe ic mycluster
Name: mycluster
Namespace: myclusterns
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: mycluster
meta.helm.sh/release-namespace: myclusterns
mysql.oracle.com/cluster-info:
{"initialDataSource": "blank", "createOptions": {"gtidSetIsComplete": true, "manualStartOnBoot": true, "memberSslMode": "REQUIRED", "exitS...
mysql.oracle.com/fqdn-template: {service}.{namespace}.svc.{domain}
mysql.oracle.com/mysql-operator-version: 9.3.0-2.2.4
API Version: mysql.oracle.com/v2
Kind: InnoDBCluster
Metadata:
Creation Timestamp: 2025-05-30T10:29:34Z
Finalizers:
mysql.oracle.com/cluster
kopf.zalando.org/KopfFinalizerMarker
Generation: 1
Resource Version: 4303
UID: e6db5575-b92b-422f-b65f-03d10cf4ed3f
Spec:
Base Server Id: 1000
Image Pull Policy: IfNotPresent
Instances: 3
Router:
Instances: 1
Secret Name: mycluster-cluster-secret
Service Account Name: mycluster-sa
Tls Use Self Signed: true
Version: 9.3.0
Status:
Cluster:
Last Probe Time: 2025-05-30T10:45:59Z
Online Instances: 3
Status: ONLINE
Type: UNKNOWN
Create Time: 2025-05-30T10:31:08Z
Kopf:
Progress:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Join 1m operator Joining mycluster-2 to cluster
Normal Join 1m operator Joining mycluster-1 to cluster
Normal ResourcesCreated 2m operator Dependency resources created, switching status to PENDING
Normal StatusChange 1m operator Cluster status changed to INITIALIZING. 0 member(s) ONLINE
Normal StatusChange 1m operator Cluster status changed to ONLINE. 1 member(s) ONLINE
We can use the MySQL Shell that is in the server and the sidecar containers to quickly check the cluster status. The warning that you see warns you that directly passing the password to the CLI is insecure. The password will be stored in the shell history in the container and will stay there until the container gets destroyed.
$ kubectl -n myclusterns exec -it mycluster-0 -c sidecar -- bash
bash-5.1$ mysqlsh -uroot -psakila --js -e "print(dba.getClusterSet().status())"
{
"clusters": {
"mycluster": {
"clusterRole": "PRIMARY",
"globalStatus": "OK",
"primary": "mycluster-0.mycluster-instances.myclusterns.svc.cluster.local:3306"
}
},
"domainName": "mycluster",
"globalPrimaryInstance": "mycluster-0.mycluster-instances.myclusterns.svc.cluster.local:3306",
"primaryCluster": "mycluster",
"status": "HEALTHY",
"statusText": "All Clusters available."
}
bash-5.1$ mysqlsh -uroot -psakila --js -e "print(dba.getCluster().status({extended:1}))"
{
"clusterName": "mycluster",
"clusterRole": "PRIMARY",
"defaultReplicaSet": {
"GRProtocolVersion": "8.0.27",
"communicationStack": "MYSQL",
"groupName": "31ce73c7-3d41-11f0-8320-9e52e6d32fce",
"groupViewChangeUuid": "AUTOMATIC",
"groupViewId": "17486010669292270:3",
"name": "default",
"paxosSingleLeader": "OFF",
"primary": "mycluster-0.mycluster-instances.myclusterns.svc.cluster.local:3306",
"ssl": "REQUIRED",
"status": "OK",
"statusText": "Cluster is ONLINE and can tolerate up to ONE failure.",
"topology": {
"mycluster-0.mycluster-instances.myclusterns.svc.cluster.local:3306": {
"address": "mycluster-0.mycluster-instances.myclusterns.svc.cluster.local:3306",
"applierWorkerThreads": 4,
"fenceSysVars": [],
"memberId": "1a537c10-3d41-11f0-b2a5-9e52e6d32fce",
"memberRole": "PRIMARY",
"memberState": "ONLINE",
"mode": "R/W",
"readReplicas": {},
"role": "HA",
"status": "ONLINE",
"version": "9.3.0"
},
"mycluster-1.mycluster-instances.myclusterns.svc.cluster.local:3306": {
"address": "mycluster-1.mycluster-instances.myclusterns.svc.cluster.local:3306",
"applierWorkerThreads": 4,
"fenceSysVars": [
"read_only",
"super_read_only"
],
"memberId": "1a517d69-3d41-11f0-b3bb-f60700172020",
"memberRole": "SECONDARY",
"memberState": "ONLINE",
"mode": "R/O",
"readReplicas": {},
"replicationLagFromImmediateSource": "",
"replicationLagFromOriginalSource": "",
"role": "HA",
"status": "ONLINE",
"version": "9.3.0"
},
"mycluster-2.mycluster-instances.myclusterns.svc.cluster.local:3306": {
"address": "mycluster-2.mycluster-instances.myclusterns.svc.cluster.local:3306",
"applierWorkerThreads": 4,
"fenceSysVars": [
"read_only",
"super_read_only"
],
"memberId": "1a53a0fa-3d41-11f0-b342-6aaf20c9aa03",
"memberRole": "SECONDARY",
"memberState": "ONLINE",
"mode": "R/O",
"readReplicas": {},
"replicationLagFromImmediateSource": "",
"replicationLagFromOriginalSource": "",
"role": "HA",
"status": "ONLINE",
"version": "9.3.0"
}
},
"topologyMode": "Single-Primary"
},
"domainName": "mycluster",
"groupInformationSourceMember": "mycluster-0.mycluster-instances.myclusterns.svc.cluster.local:3306",
"metadataVersion": "2.3.0"
}
As we are using Operator 9.3.0-2.2.4 a ClusterSet was created and there is just one IC in the ClusterSet. If {extended:1} is passed to the status() call, there will be more detailed info provided. With the second command we inspect the IC, of which the pod, mycluster-0, we are running MySQL Shell is part of.
At this point there is a ClusterSet with one IC in it, which also happens to be the primary IC in the ClusterSet.