Introduction

Mosquitto is an open-source message broker that supports MQTT 3.1 and 3.1.1. This article will explain how to setup and integrate a Mosquitto cluster in OKE on OCI, over a Load Balancer.

OKE Setup

  • Navigate to your Oracle Cloud Infrastructure console.

  • Go to Solutions and Platform – Developer Services – Kubernetes Clusters.

  • Click on Create Cluster and follow the wizard to set up your OKE instance (documentation available here).

  • Once your cluster has been created, you can either use the Cloud Shell or a local configuration to connect to it. To set it up, click on your cluster and then click on Access Cluster. This guide will assume we are using a local connection.

  • For local access, you must have downloaded and installed OCI CLI version 2.6.4 (or later) and configured it for use. If your version of the OCI CLI is earlier than version 2.6.4, download and install a newer version from here.

  • Create a directory to contain the kubeconfig file.

$ mkdir -p $HOME/.kube

  • To access the kubeconfig for your cluster, run the following command:

$ oci ce cluster create-kubeconfig --cluster-id YOUR_CLUSTER_ID --file $HOME/.kube/config --region YOUR_REGION --token-version 2.0.0 --profile YOUR_PROFILE

  • If you chose key authentication for your user when setting up the OCI CLI, please edit your ~/.kube/config file and make sure the correct profile is selected:

[...]
users:
  - name: user_name
  user:
    exec:
    apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - ce
      - cluster
      - generate-token
      - --cluster-id
      - CLUSTER_ID
      - --region
      - REGION
      - --profile
      - YOUR_PROFILE
[...]

For more information on managing kubeconfig files, please refer to the official Kubernetes documentation. More information on the available commands for OCI’s Container Engine for Kubernetes CLI can be found here.

Mosquitto Setup (with Helm)

$ git clone https://github.com/halkeye-helm-charts/mosquitto.git

  • Modify values.yaml to include the OCI-specific storageClass:

storageClass:  oci-bv

User and Password setup

  • To configure a user and a password, you need to use a tool called mosquitto_passwd (part of the Mosquitto installation). For example, to add a new user, use the following command:

$ mosquitto_passwd -c /etc/mosquitto/passwd mqttuser

  • Modify the values.yaml file to add the newly created user and password:

passwd: |-
mqttuser:$6$CugH57SaJ4bXB5Ui$jHWGbmIw1XmBCe9aIAbZHD7O45YeLbKOCAE4dTWz6LxT1yiek6CVcpMiJd3YUbVvfEo4ScpqT9JT60wLFN3vQA=

This defines a user called mqttuser with the password test.

  • In the mosquitto/templates folder, create a new ConfigMap file called configmap-passwd.yaml with the following contents:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mqtt-passwd
  labels:
    app: {{ template "fullname" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
data:
  passwd: |-
{{- if .Values.passwd }}
{{ .Values.passwd | indent 4 }}
{{- end -}}

  • To use this ConfigMap, you need to modify deployment.yaml. Add the following lines under volumeMounts:

- name: passwd
  mountPath: /etc/mosquitto/passwd
  subPath: passwd
  readOnly: true

  • Under volumes in deployment.yaml, add the following:

- name: passwd
  configMap:
    name: mqtt-passwd

  • Modify configmap.yaml to use the passwd file and to turn off anonymous authentication:

config: |-
data:
  mosquitto.conf:  |
    persistence true
    persistence_location /mosquitto/data/
    log_dest stdout
    listener 1883
    listener 8080
    protocol websockets
    allow_anonymous false
    password_file /mosquitto/etc/passwd

  • Modify service.yaml to use those same listener ports as defined above, 1883 for MQTT and 8080 for WebSocket:

ports:
  -  port:  1883
     targetPort:  default
     protocol:  TCP
     name:  default
  -  port:  8080
     targetPort:  websocket
     protocol:  TCP
     name:  websocket

  • Define the service of type NodePort, by adding the following configuration to service.yaml:

service:
type:  NodePort
port:  1883
port:  8080

  • From your CLI, deploy Mosquitto using this configuration to OKE with Helm:

$ helm install mosquitto --generate-name

Where mosquitto is the cloned folder containing the config files.

  • You can check the deployment of the service by using kubectl:

$ kubectl get services
NAME TYPE  CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1  <none>  443/TCP 24d
mosquitto-1600171724 NodePort  10.96.117.26 <none>  1883:31521/TCP,8080:31070/TCP 22d

Load Balancer Setup with Frontend SSL

Prerequisites

  • A domain name from a Registrar, such as GoDaddy.com

  • Server certificates from a CA, like for instance ZeroSSL.com

Configuration

On OCI, navigate to Core Infrastructure – Networking – Load Balancers in the Oracle Cloud console and set up a new public Load Balancer.

DNS

Once you have your domain name, configure the DNS on the Registrar to use the Oracle DNS:

  • Navigate to Networking – DNS Zone Management in the OCI console.

  • Create a new DNS zone, with the name being your domain name

  • In the DNS settings of your Registrar, replace the nameservers with the ones shown in your DNS Zone in OCI, so that this can be managed in Oracle Cloud.

  • Go to Records in your DNS Zone in OCI and add a new record for the public Load Balancer:

Domain             TTL  Type  RDATA
lb.yourdomain.com  30   A     [LOAD_BALANCER_PUBLIC_IP]

Where lb can be any subdomain name of your choice.

Certificates

  • In your CA of choice, generate the Certificate Bundle and follow their instructions to validate them.

  • For example, in ZeroSSL, click on New Certificate, provide the Load Balancer domain, ex: lb.yourdomain.com (where lb can be any subdomain name of your choice).

  • Choose DNS (CNAME) as a verification method and add a new record in your DNS Zone in OCI with the information provided by ZeroSSL.

  • Verify your domain to complete the setup and you can now download your certificate from ZeroSSL.

  • After downloading your certificate, you should have a ZIP containing the following certificate files:

    certificate.crt
ca_bundle.crt
private.key

  • In the Load Balancer on OCI, set up a listener on TCP, port 8883 (MQTT SSL default port, can be whatever you want).

  • Check the Use SSL box and upload the three files that ZeroSSL provided for the certificate.

  • Create a backend set for your Mosquitto nodes.

  • Add your cluster nodes to each of the backend sets with the respective port for the connection you want to use.

  • The ports are the NodePorts from the Kubernetes service:

$ kubectl get services
NAME TYPE  CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1  <none>  443/TCP 24d
mosquitto-1600171724 NodePort  10.96.117.26 <none>  1883:31521/TCP,8080:31070/TCP 22d

So in this example we would use 31521 as a listener port for MQTT, and 31070 for Websockets.

  • Configure the health check on the same port.

For more information about how to setup a Load Balancer in OCI, please refer to the official documentation available here.

Examples

You will now be able to connect to the Mosquitto cluster from an external client using a secure connection to the Load Balancer.

  • Recommended GUI clients: MQTT Explorer MQTTfx

  • For testing the connection to queues and pub/sub, here’s a sample Python client using Paho:

import paho.mqtt.client  as mqtt #import the client
import time
import ssl
​
############
​
def  on_message(client, userdata, message):
  print("message received ", str(message.payload.decode("utf-8")))
  print("message topic=", message.topic)
  print("message qos=", message.qos)
  print("message retain flag=", message.retain)
def  on_log(client, userdata, level, buf):
  print("log: ", buf)
  
############
​
broker_address =  "lb.yourdomain.com"
broker_port =  8883
print("creating new instance")
client = mqtt.Client("MQTT_Client",  transport='tcp')  # create new instance
client.on_message  = on_message # attach function to callback
client.on_log  = on_log
​
client.username_pw_set("mqttuser",  password="test")
client.tls_set(cert_reqs=ssl.CERT_NONE,tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None)
​
############
​
client.connect(broker_address, broker_port)  # connect to broker
client.loop_start()  # start the loop
print("Subscribing to topic",  "test/topic")
client.subscribe("test/topic")
print("Publishing message to topic",  "test/topic")
client.publish("test/topic",  "Test message")
time.sleep(5)  # wait
client.loop_stop()  # stop the loop