On June 28, the Verrazzano team released Oracle Verrazzano Enterprise Container Platform 1.6. In this minor release, the team focused on two major themes:
1.    Make Kubernetes cluster management easier
2.    Make observability more flexible and efficient

Let’s dig into the details.

OCNE Cluster Lifecycle Management

Let’s start with cluster management – there are several changes in Verrazzano 1.6 that make it easier to manage Kubernetes clusters. First, we further integrated Verrazzano with Oracle Cloud Native Environment (OCNE), which is an upstream Kubernetes distribution with CNI, CSI, an API, and lifecycle tooling. Verrazzano 1.6 and OCNE 1.7 demonstrate the progress of aligning the two products as layers of a complete solution by combining the components of the two products into a single component model: when you create an OCNE cluster, you can layer Verrazzano on top of the cluster, leveraging the components in OCNE as part of the Verrazzano configuration. 

With Verrazzano 1.6, you can use the user interface to create OCNE clusters on Oracle Cloud Infrastructure (OCI). The process is pretty easy: provide cloud credential information, cluster node shape information, and cluster size. Verrazzano does the rest. Also, by default, the system installs the Verrazzano platform components in the cluster, and registers the cluster with Verrazzano as a fully managed cluster: you can manage the cluster lifecycle through Verrazzano, as well as deploy and manage workloads in the cluster. More is coming in this area, including creating Oracle Container Engine for Kubernetes (OKE) clusters, and installing in other environments. See https://verrazzano.io/latest/docs/setup/provision-cluster/ocne-oci/ for more details.

Create cluster controls in the Verrazzano UI

Cluster API in Verrazzano

Under the covers, Verrazzano uses Cluster API (CAPI) to drive cluster creation and management. CAPI is an API for creating and managing Kubernetes clusters. It has a pluggable architecture with various plugin types, including infrastructure provider, control plane provider, and bootstrap provider plugin types. Verrazzano 1.6 ships with the following CAPI providers:

  • CAPI core provider – the CAPI implementation in Verrazzano.
  • OCI Infrastructure provider – enables creation of OCI objects, including compute, storage, load balancers, and so forth.
  • OCNE Bootstrap provider – enables creation of OCNE Kubernetes clusters. It creates the OCNE cloudinit file for the control plane nodes and worker nodes.
  • OCNE Controlplane provider – drives creation of the Kubernetes cluster control plane and its required artifacts. 

Cluster API Providers in Verrazzano

You can extend the CAPI infrastructure in Verrazzano with other CAPI infrastructure, bootstrap and control plane providers. For more information, see the Verrazzano documentation and the cluster-api-provider-ocne source code.

Managed Cluster Minimal Footprint

Also in the cluster management theme in Verrazzano 1.6 is the ability to create Verrazzano managed clusters with a minimal footprint. Recently, we’ve heard a number of requests for managing clusters and workloads in edge environments where there is limited compute capacity. 

Verrazzano edge application and cluster management

With Verrazzano 1.6, you can install Verrazzano with profile set to “none,” which results in installation of only the Verrazzano Platform Operator and the Verrazzano ClusterAgent, which provides a very small footprint. This minimal installation may not be fully useful, though, since there is no infrastructure for monitoring or communicating with the cluster. A more typical installation would include certmanager, authProxy, NGINX, Prometheus, and Thanos, which would enable full observability and communication with the cluster. For more details, see the Verrazzano documentation. The combination of this minimal footprint Verrazzano installation with Verrazzano’s OCNE cluster management, and OCNE’s single node cluster configuration enables you to manage clusters and workloads running at the edge where resources may be restricted. There is more to come with this flexible installation of components.

Better Observability with Thanos

Verrazzano 1.6 includes Thanos as an optional observability component. There are some very good reasons to use Thanos with Verrazzano:

  • Reduces cost: Uses inexpensive object storage, which is typically less expensive on cloud providers, and also has lower egress charges.
  • Eases compliance: Provides long-term storage when needed for compliance or other reasons.
  • Improves availability: Provides several high-availability features:
    • Deduplication of metrics at the storage layer
    • No loss from node failure/block volume loss
    • Tolerance for connection issues between Admin and Managed clusters
  • Reduces resource usage and network traffic:
    • Queries across managed clusters for specific metrics, rather than all metrics
    • Caches metrics at the admin cluster
    • Enables global queries across clusters

Thanos in a Verrazzano Multicluster Environment

When you enable Thanos in Verrazzano, the Verrazzano Platform Operator sets up the default Thanos configuration. You can see the default configuration in the values.yaml file for the Thanos helm chart. 

Verrazzano uses the “sidecar” pattern for Thanos where Thanos takes metrics from Prometheus and writes them to object storage. The Thanos Query and the Thanos Gateway are then used to access those metrics stored in object storage. 

In a multicluster configuration, when Thanos is enabled on all of the clusters, Thanos global queries are used to consolidate metrics on the Admin cluster from the managed clusters, in place of Prometheus federation. If only some clusters have Thanos enabled, Verrazzano uses a mix of Thanos queries and Prometheus federation to capture all of the necessary metrics. 

Read more in the Verrazzano documentation.

Efficient Log Collection with Fluent Bit

Verrazzano 1.6 includes Fluent Operator and Fluent Bit as optional components. Fluentd is still the default log collection technology, and you can run both Fluent Bit and Fluentd in the same clusters. 

Fluent Bit is very similar to Fluentd with its flexible source and destination plugins that enable you to collect logs from many sources, and then process and forward those logs to many destinations. Fluent bit is different than Fluentd in that Fluent Bit uses far less resources than Fluentd, and it processes logs much faster than Fluentd! 

Fluent Operator and Fluent Bit in Verrazzano

In Verrazzano, you use the Fluent Operator to manage Fluent Bit. By default, the Fluent Operator deploys Fluent Bit as a daemonset on each node in the cluster with the following inputs and outputs:

  • Default Inputs
    • ClusterInput of type tail and systemd
    • ClusterFilters to enrich logs with k8s metadata
  • Default Outputs
    • opensearch-system-clusteroutput
    • opensearch-application-clusteroutput

You can change, add, and remove inputs and outputs, and you can custom filtering and parsing. See the Verrazzano documentation for more information.

Conclusion

Verrazzano 1.6 is the latest release of our enterprise container platform. As with all Verrazzano releases, new features are driven by customer requirements. With this release, we focused on cluster management, including OCNE integration, and observability efficiency. The result is more value with more relief for challenges that customers face with multicluster workload management. Upgrades to Verrazzano 1.6 from 1.4 and 1.5 remain simple and straightforward: update the Verrazzano Platform Operator, and tell it to upgrade the cluster. 

Why not give Verrazzano a try? Take a look at https://verrazzano.io/latest/docs/setup/quickstart/. If you are looking for more of a guided tour, check out our latest lab: Deploy a Springboot application to OKE using Verrazzano.