Multiclustering between OCI and Azure with Verrazzano

April 19, 2022 | 10 minute read
Rafael Benevides
Principal Product Manager
Text Size 100%:

​One of the most desired features of companies that are looking to improve their technical or business requirements is the usage of multiple clusters. A multi-cluster strategy can improve availability, isolation, and scalability. 

 

OCI itself can provide everything that you need to achieve availability, isolation, and scalability. Verrazzano, on the other hand, allows you to achieve a multi-cluster environment even if you are not using OCI. 

 

It's important to mention that a multi-cluster environment is not simply running multiple Kubernetes clusters. Instead, it should consist of proper application deployment strategies to distribute and maintain the application across multiple clusters with appropriate tools and practices baked into the DevOps process.

 

Having understood what to expect from the DevOps perspective on a multi-cluster environment, we can finally understand how Verrazzano allows you to centralize the management and deployment of applications.

 

Verrazzano may be installed in a multi-cluster environment, consisting of an admin cluster and optionally, one or more managed clusters.

  • The admin cluster is a central point from which Verrazzano applications in managed clusters can be deployed and monitored.
  • Managed clusters are registered with an admin cluster.
  • Verrazzano multi-cluster resources are used to target applications to any cluster in a multicluster Verrazzano environment.

 

Verrazzano Multicluster

 

Note that Verrazzano federates the metrics from multiple clusters into a single Prometheus instance, and the Kiali component, which is part of Verrazzano, can build graphical flows for all the applications deployed to all the clusters in your Verrazzano environment. This is a really awesome feature that turns multi-cluster management into an easy task.

 

In this blog post, we will see how to create a multi-cluster environment that uses OCI to host the admin cluster, and Azure (which interconnects with OCI, accelerating the multi-cloud adoption) will host the managed cluster for our multi-cluster demo.

 

Installing the Multi Cluster environment

This section will be divided in two parts: Installation of admin cluster, and installation of the managed cluster.

 

During the setup process, we will need to alternate between the clusters. To make it easy to switch between them, you can use the kubectx tool.

 

Let's start by installing the admin cluster on OCI.

 

Admin Cluster on OCI

The admin cluster doesn't require anything special. Just install a regular Verrazzano, following the instructions in the Verrazzano page, using the prod or dev profile.

 

For more information about the different Verrazzano profiles, check this page: Verrazzano Installation Profiles.

 

Managed Cluster on Azure 

The creation of a Kubernetes Cluster using AKS is not in the scope of this blog post. 

 

Considering that you have created your Kubernetes Cluster and configured your kubectl command to access it, let's install Verrazzano. The only difference in this case, is that you should use the managed-cluster profile using the command:

 

kubectl apply -f - <<EOF
apiVersion: install.verrazzano.io/v1alpha1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: ${VZ_PROFILE:-managed-cluster}
EOF

 

Register the managed cluster with the admin cluster

In this blog post example, we used a self-signed certificate for both clusters (Admin and Managed).  We need to register the managed cluster certificate in the admin cluster, so the admin cluster can scrape metrics from the managed cluster’s Prometheus endpoint via HTTPS.

 

The following instructions will guide you on the steps needed to extract the managed cluster certificate and add it to the admin cluster. Note that If you have customized Verrazzano certificates to use a Well-Known CA, or LetsEncrypt, I suggest you follow the instructions on the Verrazzano Website.

 

Given that you are still "connected" to the managed cluster, run the following commands to create a file called "managed1.yaml" containing a secret with the Self-Signed certificate from the managed cluster

 

MGD_CA_CERT=$(kubectl \
     get secret system-tls \
     -n verrazzano-system \
     -o jsonpath="{.data.ca\.crt}" | base64 --decode)

echo $MGD_CA_CERT


kubectl \
  create secret generic "ca-secret-managed1" \
  -n verrazzano-mc \
  --from-literal=cacrt="$MGD_CA_CERT" \
  --dry-run=client \
  -o yaml > managed1.yaml


 

The next step is to create this secret in the admin cluster. Let's use the kubectx tool to change the Kubernetes context to the admin cluster in OCI. Type "kubectx", to show all contexts available in your Kubernetes configuration file. Type "kubectx <name of admin cluster context in OCI>" to switch the context. Example:

 

Now you can create the secret in the admin cluster using the command:

 

kubectl apply -f managed1.yaml

 

The next step is to get the externally accessible API server address of the admin cluster from its kubeconfig file. Execute the command:

 

kubectl config view --minify


# Sample output
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://129.159.51.7:6443
  name: cluster-cf4kwdczkzq

 

Create an environment variable called ADMIN_K8S_SERVER_ADDRESS with the content of the URL (protocol + ip + port):

 

export ADMIN_K8S_SERVER_ADDRESS=https://129.159.51.7:6443

 

Now create a ConfigMap with the address of the admin cluster:

kubectl apply -f <<EOF -
apiVersion: v1
kind: ConfigMap
metadata:
  name: verrazzano-admin-cluster
  namespace: verrazzano-mc
data:
  server: "${ADMIN_K8S_SERVER_ADDRESS}"
EOF

 

To begin the registration process for a managed cluster named managed1, apply the VerrazzanoManagedCluster object on the admin cluster:

 

kubectl apply -f <<EOF -
apiVersion: clusters.verrazzano.io/v1alpha1
kind: VerrazzanoManagedCluster
metadata:
  name: managed1
  namespace: verrazzano-mc
spec:
  description: "Test VerrazzanoManagedCluster object"
  caSecret: ca-secret-managed1
EOF

 

Wait for the VerrazzanoManagedCluster resource to reach the Ready status:

 

kubectl wait --for=condition=Ready \
    vmc managed1 -n verrazzano-mc

 

Yet, in the admin cluster, export the YAML file created to be used in the managed cluster:

 

kubectl get secret verrazzano-cluster-managed1-manifest \
    -n verrazzano-mc \
    -o jsonpath={.data.yaml} | base64 --decode > register.yaml

 

Now, let's change to the managed cluster using the kubectx tool to change the Kubernetes context to the managed cluster: 

 

kubectx <managed-cluster-name>
# Switched to context "Verrazzano-Managed".


kubectl apply -f register.yaml

 

After this step, the managed cluster will begin connecting to the admin cluster periodically. When the managed cluster connects to the admin cluster, it will update the Status field of the VerrazzanoManagedCluster resource for this managed cluster.

 

We can check the VerrazzanoManagedCluster resource on the admin cluster, and check whether the lastAgentConnectTime, prometheusUrl, and apiUrl fields are populated. This may take up to two minutes after completing the registration steps.

 

kubectx <admin-cluster-name>

kubectl get vmc managed1 -n verrazzano-mc -o yaml

# Sample output showing the status field
spec:
  ....
  ....
status:
  apiUrl: https://verrazzano.default.172.18.0.211.nip.io
  conditions:
  - lastTransitionTime: "2021-07-07T15:49:43Z"
    message: Ready
    status: "True"
    type: Ready
  lastAgentConnectTime: "2021-07-16T14:47:25Z"
  prometheusHost: prometheus.vmi.system.default.172.18.0.211.nip.io

 

You can also verify that the managed cluster is successfully registered with Rancher. Let's open the Rancher UI. First, let's get Verrazzano URLs:

kubectl get vz -o jsonpath="{.items[].status.instance}" | jq .

 

And get the credentials to access Rancher for the user admin:

kubectl get secret \
    --namespace cattle-system rancher-admin-secret \
    -o jsonpath={.data.password} | base64 \
    --decode; echo

 

 


 

Trying a Multicluster Hello World Helidon demo.

To try the Multicluster that we have created, let's use a demo application that comes with Verrazzano. The Hello World Helidon demo is a Helidon-based service that returns a “Hello World” response when invoked. Although it's very simple, this demo shows how to deploy an application in a multi-cluster environment, and allows changing the placement of the application to a different cluster.

 

The first thing that we need to do is to make sure that our kubectl configuration points to the admin cluster to create the application namespace. Note that although we use the kubectl command in the admin cluster, the VerrazzanoProject YAML has a placement for the managed1 cluster:

 

kind: VerrazzanoProject
metadata:
  name: hello-helidon
  namespace: verrazzano-mc
spec:
  template:
    namespaces:
      - metadata:
          name: hello-helidon
  placement:
    clusters:
      - name: managed1

 

Let's deploy this file, and all other YAML files needed to deploy the demo (hello-helidon-comp.yaml and mc-hello-helidon-app.yaml).

 

kubectx <admin-cluster-name>

kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.2.0/examples/multicluster/hello-helidon/verrazzano-project.yaml
kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.2.0/examples/multicluster/hello-helidon/hello-helidon-comp.yaml 
kubectl apply -f https://raw.githubusercontent.com/verrazzano/verrazzano/v1.2.0/examples/multicluster/hello-helidon/mc-hello-helidon-app.yaml

 

Access the application in the Managed Cluster 

To access the application in the managed cluster, point the kubectl to the managed cluster and run the following commands:

 

kubectx <managed-cluster-name>


HOST=$(kubectl get gateways.networking.istio.io hello-helidon-hello-helidon-appconf-gw \
     -n hello-helidon \
     -o jsonpath='{.spec.servers[0].hosts[0]}')

echo $HOST


ADDRESS=$(kubectl get service \
     -n istio-system istio-ingressgateway \
     -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

echo $ADDRESS


curl -sk \
   -X GET \
   https://${HOST}/greet \
   --resolve ${HOST}:443:${ADDRESS}

 

You should see the output: {"message":"Hello World!"}

 

Change the cluster placement.

To change the cluster placement, you just need to modify the MultiClusterApplicationConfiguration and change the placement info. We can do this easily by applying the following patch in the admin cluster:

kubectx <managed-cluster-name>

kubectl patch --type=merge \
   -n hello-helidon \
   mcappconf hello-helidon-appconf \
   --patch '{"spec":{"placement":{"clusters":[{"name":"local"}]}}}'

 

You can check the placement of the application by running the following command:

 

kubectl get mcappconf hello-helidon-appconf \
    -n hello-helidon \
    -o jsonpath='{.spec.placement}';echo

 

The placement should be local instead of managed1. At this moment, you can try the application running the same commands that we executed before. However, this time we should use the admin cluster.

 

kubectx <admin-cluster-name>


HOST=$(kubectl get gateways.networking.istio.io hello-helidon-hello-helidon-appconf-gw \
     -n hello-helidon \
     -o jsonpath='{.spec.servers[0].hosts[0]}')

echo $HOST


ADDRESS=$(kubectl get service \
     -n istio-system istio-ingressgateway \
     -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

echo $ADDRESS


curl -sk \
   -X GET \
   https://${HOST}/greet \
   --resolve ${HOST}:443:${ADDRESS}

 

You should see the output: {"message":"Hello World!"}

 

Conclusion

Multi-clustering in Verrazzano is a first class citizen. You can easily use your admin cluster to deploy and manage an application in different clusters, no matter if they are on the same cloud. A muti-cluster enabled environment allows proper application deployment strategies to distribute and maintain the application across multiple clusters, with appropriate tools and practices baked into the DevOps process.

 

The observability is also centralized in the admin cluster, allowing you to monitor how your application is behaving on these different clusters. For example, you can open the Prometheus console and watch the metric vendor_requests_count_total. You should see the metrics from both clusters that we have created in this blog post.

 

 

Check out more details about Verrazzano on its webpage: https://verrazzano.io  and https://www.oracle.com/java/verrazzano/. At the Verrazzano's Youtube channel, you will see a Verrazzano tour video and much more. News about this project is shared on Verrazzano's Twitter profile. Verrazzano is a fast paced evolving open source project. You can follow, and also contribute to this awesome project on the Github project page

 

 

Rafael Benevides

Principal Product Manager

Rafael Benevides is a Principal Product Manager at Oracle. With many years of experience in several fields of the IT industry, he helps developers and companies all over the world to be more effective in software development. Rafael considers himself a problem solver who has a big love for sharing. He is a member of Apache DeltaSpike PMC - a Duke’s Choice Award winner project, and a speaker in conferences like JavaOne, Devoxx, TDC, DevNexus and many others.
Twitter | LinkedIn | rafabene.com


Previous Post

Strengthen Your Developer Experience and Deployment Velocity with OKE and Shipa Cloud

Robert Ronan | 11 min read

Next Post


Partnership Announcement: Oracle Cloud and WunderGraph

Robert Ronan | 5 min read