We are proud to announce a reference architecture for Elasticsearch on Oracle Cloud Infrastructure. Starting today, you can deploy Elasticsearch, an open source, distributed, RESTful search and analytics engine on Oracle's high-performance cloud by using Terraform templates.
With this announcement, Oracle Cloud Infrastructure enhances the Big Data ISV ecosystem of partners. To get started with Elasticsearch on Oracle Cloud Infrastructure, you can use the Terraform automated deployment template. We have two templates: "simple" and "cluster". The "simple" template deploys Elasticsearch, Kibana, and Logstash on a single VM and the "cluster" template deploys a highly available cluster of Elasticsearch and Kibana. This blogs covers the cluster deployment. The cluster template performs the steps to deploy and configure Elasticsearch and Kibana: provisions instances and storage, deploys and configures the software, sets up networking and a load balancer, and starts it all.
The following diagram shows the Elasticsearch and Kibana deployment using the template.
The following components are part of the deployment:
Bastion host: A bastion host is used as a NAT instance for Elasticsearch master and data nodes to update and install software from the internet.
Load balancer: Oracle Cloud Infrastructure Load Balancing is used to load balance index operations onto the data nodes and Kibana access to master nodes. It uses two listeners, one for Kibana and one for index data access, backed by backend sets with master node backends and data node backends. The load balancer is launched into a public subnet with a public IP address, but you can modify this by changing lbaas.tf to make it a private load balancer.
Elasticsearch, master nodes: Master nodes perform cluster management tasks like creating new indexes and rebalancing shards. They don’t store data. Three master nodes (recommended for bigger clusters) are deployed across three availability domains to ensure high availability.
Elasticsearch, data nodes: Four data nodes are deployed across two availability domains (two nodes in each availability domain) for high availability. Memory-optimized compute instances are recommended because Elasticsearch is dependent on the amount of memory available. A complete list of compute instance shapes to select is available here. Each data node is configured with a 200-GiB block storage. In addition to VMs, Oracle Cloud Infrastructure offers powerful bare metal instances that are connected in clusters to a nonoversubscribed 25-gigabit network infrastructure. This configuration guarantees low latency and high throughput, which is a key requirement for high-performance distributed streaming workloads. Oracle Cloud Infrastructure is the only cloud with a network throughput performance SLA.
Kibana: Like the master nodes, Kibana has relatively light resource requirements. Most computations are pushed to Elasticsearch. In this deployment, Kibana runs on the master nodes.
To customize your Terraform deployment, you can perform the following actions:
For details about the Terraform templates, see the Readme.md file.
We hope you are as excited as we are about Elasticsearch on Oracle Cloud Infrastructure. Let us know what you think!