Deploying Elasticsearch on Oracle Cloud Infrastructure Using a Terraform Template

Pinkesh Valdria
Principal Solutions Architect

We are proud to announce a reference architecture for Elasticsearch on Oracle Cloud Infrastructure. Starting today, you can deploy Elasticsearch, an open source, distributed, RESTful search and analytics engine on Oracle's high-performance cloud by using Terraform templates.

With this announcement, Oracle Cloud Infrastructure enhances the Big Data ISV ecosystem of partners. To get started with Elasticsearch on Oracle Cloud Infrastructure, you can use the Terraform automated deployment template. We have two templates: "simple" and "cluster".  The "simple" template deploys Elasticsearch, Kibana, and Logstash on a single VM and the "cluster" template deploys a highly available cluster of Elasticsearch and Kibana. This blogs covers the cluster deployment. The cluster template performs the steps to deploy and configure Elasticsearch and Kibana: provisions instances and storage, deploys and configures the software, sets up networking and a load balancer, and starts it all.

Deployment Architecture

The following diagram shows the Elasticsearch and Kibana deployment using the template.

The following components are part of the deployment:

Bastion host: A bastion host is used as a NAT instance for Elasticsearch master and data nodes to update and install software from the internet.

Load balancer: Oracle Cloud Infrastructure Load Balancing is used to load balance index operations onto the data nodes and Kibana access to master nodes. It uses two listeners, one for Kibana and one for index data access, backed by backend sets with master node backends and data node backends. The load balancer is launched into a public subnet with a public IP address, but you can modify this by changing lbaas.tf to make it a private load balancer.

Elasticsearch, master nodes: Master nodes perform cluster management tasks like creating new indexes and rebalancing shards. They don’t store data. Three master nodes (recommended for bigger clusters) are deployed across three availability domains to ensure high availability.

Elasticsearch, data nodes: Four data nodes are deployed across two availability domains (two nodes in each availability domain) for high availability. Memory-optimized compute instances are recommended because Elasticsearch is dependent on the amount of memory available. A complete list of compute instance shapes to select is available here. Each data node is configured with a 200-GiB block storage. In addition to VMs, Oracle Cloud Infrastructure offers powerful bare metal instances that are connected in clusters to a nonoversubscribed 25-gigabit network infrastructure. This configuration guarantees low latency and high throughput, which is a key requirement for high-performance distributed streaming workloads. Oracle Cloud Infrastructure is the only cloud with a network throughput performance SLA.

Kibana: Like the master nodes, Kibana has relatively light resource requirements. Most computations are pushed to Elasticsearch. In this deployment, Kibana runs on the master nodes.   

To customize your Terraform deployment, you can perform the following actions:

  • Choose the shape for master node and data node instances.
  • Specify storage capacity for data node instances
  • Change CIDR block sizes for the virtual cloud network and subnets, and other configuration settings.

For details about the Terraform templates, see the Readme.md file.

What’s Next?

  1. If you don’t have an Oracle Cloud Infrastructure account yet, you can sign up for a 30-day free trial account.
  2. Follow the instructions on the GitHub oci-elasticsearch page to install an Elasticsearch cluster on Oracle Cloud Infrastructure.

We hope you are as excited as we are about Elasticsearch on Oracle Cloud Infrastructure. Let us know what you think!


Join the discussion

Comments ( 1 )
  • Lokesh Madan Friday, January 11, 2019
    Hi Pinkesh,

    Nice article, but clicking on Terraform automated deployment template. gives 404.

Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.