X

News, tips, partners, and perspectives for the Oracle Linux operating system and upstream Linux kernel work

  • Linux
    February 25, 2020

Easy Provisioning Of Cloud Instances On Oracle Cloud Infrastructure With The OCI CLI

As a developer, I often provision ephemeral instances in OCI for small projects or for testing purposes.
Between the Browser User Interface which is not very convenient for repetitive tasks and Terraform which would be over-engineered for my simple needs the OCI Command Line Interface (CLI) offers a simple but powerful interface to the Oracle Cloud Infrastructure.

In this article I will share my experience with this tool and provide as example the script I am using to provision cloud instances.

The OCI CLI

The OCI CLI requires python version 3.5 or later, running on Mac, Windows, or Linux.
Installation instructions are provided on the OCI CLI Quickstart page.

The examples from this article have been been tested on Linux, macOS and Windows.
Windows users can use either Windows Subsystem for Linux or Git BASH.

These examples assume that the OCI CLI is already installed and configured; and that the compartment is saved in the ~/.oci/oci_cli_rc file:

[DEFAULT]
compartment-id = ocid1.compartment.oc1..xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Handling the OCI CLI output: JMESPath

The main challenge of using the OCI CLI in scripts is handling its responses.

By default, all responses to a command are returned in JSON format. E.g.

$ oci os ns get
{
  "data": "mynamespace"
}

Alternatively, a table format is also available:

$ oci os ns get --output table
+-------------+
| Column1     |
+-------------+
| mynamespace |
+-------------+

But none of these formats are directly usable in a shell script. One could use the well known jq JSON processor, but the OCI CLI is built with the JMESPath library which allows JSON manipulation without the need of an third party tool.
With the same simple request we can select the data field:

$ oci os ns get --query 'data'
"mynamespace"

Finally we can get rid of the quotes using the raw output format:

$ oci os ns get --query 'data' --raw-output
mynamespace

And to capture the output in a shell variable:

$ ns=$(oci os ns get --query 'data' --raw-output)
$ echo $ns
mynamespace

As a less trivial example, the following returns the image OCID of the latest Oracle Linux 7.7 image compatible with the VM.Standard2.1 shape:

$ ocid=$(oci compute image list \
    --operating-system "Oracle Linux" \
    --operating-system-version "7.7" \
    --shape "VM.Standard2.1" \
    --sort-by TIMECREATED \
    --query 'data[0].id' \
    --raw-output)
$ echo $ocid
ocid1.image.oc1.eu-frankfurt-1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

The --raw-output option is only effective when the output of the query returns a single string value. When multiple values are expected we will concatenate them in the query.

Depending on the format of the fields, I typically use two different constructions to retrieve the data: concatenate with space or new line separators.

The space construct is the simplest, but it obviously won’t work if your fields are free text.

$ response=$(oci compute image list \
    --operating-system "Oracle Linux" \
    --operating-system-version "7.7" \
    --shape "VM.Standard2.1" \
    --sort-by TIMECREATED \
    --query '[data[0].id, data[0]."display-name"] | join('"'"' '"'"',@)' \
    --raw-output)
$ read ocid display_name <<< "${response}"
$ echo $ocid
ocid1.image.oc1.eu-frankfurt-1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$ echo $display_name
Oracle-Linux-7.7-2020.01.28-0    

Note: never use pipes to read and store data in shell variables as pipes are run in sub-shells!

The new line construct is slightly more complex, but can be used with fields containing spaces:

$ response=$(oci compute image list \
    --operating-system "Oracle Linux" \
    --operating-system-version "7.7" \
    --shape "VM.Standard2.1" \
    --sort-by TIMECREATED \
    --query '[data[0].id, data[0]."display-name"] | join(`\n`,@)' \
    --raw-output)
$ { read ocid; read display_name; } <<< "${response}"
$ echo $ocid
ocid1.image.oc1.eu-frankfurt-1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
$ echo $display_name
Oracle-Linux-7.7-2020.01.28-0    

Notes:

  • You can use inverted quotes instead of quotes for strings in JMESPath queries, it makes the overall quoting more readable.
  • If you use a bash shell under Windows (Git BASH), make sure it properly handles DOS type end-of-line by setting the IFS environment variable: IFS=$' \t\r\n'

Provisioning script

Using the above constructions you can easily write a script to facilitate image provisioning.

The provisioning script from which the code snippets are extracted is part of the ol-sample-scripts project on GitHub.

My goal is to be able to swiftly provision instances in the same environment, so the script assumes there are a Virtual Cloud Network (VCN) and a Subnet already defined in the tenancy. A Public IP is always assigned.

The following sections describe the high level steps needed to provision an image.

Platform images

This is the easiest case: the image OCID can be retrieved with a simple query:

image_list=$(oci compute image list \
  --operating-system "${operating_system}" \
  --operating-system-version "${operating_system_version}" \
  --shape ${shape} \
  --sort-by TIMECREATED \
  --query '[data[0].id, data[0]."display-name"] | join(`\n`,@)' \
  --raw-output)

it is important to include the target shape in the query to only retrieve compatible images.

The Availability Domain is retrieved using pattern matching:

availability_domain=$(oci iam availability-domain list \
  --all \
  --query 'data[?contains(name, `'"${availability_domain}"'`)] | [0].name' \
  --raw-output)

We also need the VCN and Subnet OCIDs:

ocid_vcn=$(oci network vcn list \
  --query "data [?\"display-name\"=='${vcn_name}'] | [0].id" \
  --raw-output)
ocid_subnet=$(oci network subnet list \
  --vcn-id ${ocid_vcn} \
  --query "data [?\"display-name\"=='${subnet_name}'] | [0].id" \
  --raw-output)

We now have all the data needed to launch the instance:

ocid_instance=$(oci compute instance launch \
  --display-name ${instance_name} \
  --availability-domain "${availability_domain}" \
  --subnet-id "${ocid_subnet}" \
  --image-id "${ocid_image}" \
  --shape "${shape}" \
  --ssh-authorized-keys-file "${public_key}" \
  --assign-public-ip true \
  --wait-for-state RUNNING \
  --query 'data.id' \
  --raw-output)

We use the --wait-for-state option to wait until the image is up and running. This allows us to retrieve and print the IP address, so we can immediately connect to our new instance:

public_ip=$(oci compute instance list-vnics \
  --instance-id "${ocid_instance}" \
  --query 'data[0]."public-ip"' \
  --raw-output)

Marketplace images

Unfortunately, the oci compute image list command only returns Platform and Custom images. What if we want to provision Oracle images from the Marketplace (Cloud Developer, Autonomous Linux, …)?

This is a bit more complex as these images require you to accept the Oracle Standard Terms and Restrictions before using them.

The Marketplace is also known as the Product Image Catalog (PIC) and the corresponding API calls are done with the oci pic commands.

To instantiate an image from the Marketplace we need to:

Get the image listing OCID – the query must be specific enough to return a single row.

pic_listing=$(oci compute pic listing list \
  --all \
  --query 'data[?contains("display-name", `'"${image_name}"'`)].join('"'"' '"'"', ["listing-id", "display-name"]) | join(`\n`, @)' \
  --raw-output)

Using that listing OCID, find the latest image OCID in that listing:

version_list=$(oci compute pic version list --listing-id "${ocid_listing}" \
  --query 'sort_by(data,&"time-published")[*].join('"'"' '"'"',["listing-resource-version", "listing-resource-id"]) | join(`\n`, reverse(@))' \
  --raw-output)

The above query does not allow to specify a shape like we do for the Platform images. We have to browse the list until we find a compatible image:

available=$(oci compute pic version get --listing-id "${ocid_listing}" \
  --resource-version "${image_version}" \
  --query 'data."compatible-shapes"|contains(@, `'${shape}'`)' \
  --raw-output)

Now that we have a compatible image OCID, we need to retrieve the agreement for the listing OCID:

agreement=$(oci compute pic agreements get --listing-id "${ocid_listing}" \
  --resource-version  "${image_version}" \
  --query '[data."oracle-terms-of-use-link", data.signature, data."time-retrieved"] | join(`\n`,@)' \
  --raw-output)

And eventually subscribe to the agreement:

subscription=$(oci compute pic subscription create --listing-id "${ocid_listing}" \
  --resource-version  "${image_version}" \
  --signature "${signature}" \
  --oracle-tou-link "${oracle_tou_link}" \
  --time-retrieved "${time_retrieved}" \
  --query 'data."listing-id"' \
  --raw-output)

Once subscribed, we can proceed as we did for the Platform images.

Cloud-init

Beyond the simple provisioning, I like to have a ready to use instance with my favorite tools installed and configured (shell, editor preferences, …).
This can be done with a cloud-init file.

Cloud-init files can be very complex (see the cloud-init documentation), but in its simplest form it can just be a shell script.
The file is passed as paramter to the oci compute instance launch command.
As illustration, the project repository contains a simple oci-cloud-init.sh file.

Sample session

$ ./oci-provision.sh --help
Usage: oci-provision.sh OPTIONS

  Provision an OCI compute instance.

Options:
  --help, -h                show this text and exit
  --os                      operating system (default: Oracle Linux)
  --os-version              operating system version
  --image IMAGE             image search pattern in the Marketplace
                            os/os-version are ignored when image is specified
  --name NAME               compute VM instance name
  --shape SHAPE             VM shape (default: VM.Standard2.1)
  --ad AD                   Availability Domain (default: AD-1)
  --key KEY                 public key to access the instance
  --vcn VCN                 name of the VCN to attach to the instance
  --subnet SUBNET           name of the subnet to attach to the instance
  --cloud-init CLOUD-INIT   optional clout-init file to provision the instance

Default values for parameters can be stored in ./oci-provision.env
$ ./oci-provision.sh --image "Cloud Dev" \
  --name Development \
  --ad AD-3 \
  --key ~/.ssh/id_rsa.pub \
  --vcn "VCN-Dev" \
  --subnet "Public Subnet" \
  --cloud-init oci-cloud-init.sh
+++ oci-provision.sh: Getting image listing
    oci-provision.sh: Selected image:
    oci-provision.sh: Image      : Oracle Cloud Developer Image
    oci-provision.sh: Summary    : Oracle Cloud Developer Image
    oci-provision.sh: Description: An Oracle Linux 7-based image with the latest development tools, languages, Oracle Cloud Infrastructure Software Development Kits and Database connectors at your fingertips
+++ oci-provision.sh: Getting latest image version
    oci-provision.sh: Version Oracle_Cloud_Developer_Image_19.11 selected
+++ oci-provision.sh: Getting agreement and subscribing...
    oci-provision.sh: Term of use: https://objectstorage.us-ashburn-1.oraclecloud.com/n/partnerimagecatalog/b/eulas/o/oracle-apps-terms-of-use.txt
    oci-provision.sh: Subscribed
+++ oci-provision.sh: Retrieving AD name
+++ oci-provision.sh: Retrieving VCN
+++ oci-provision.sh: Retrieving subnet
+++ oci-provision.sh: Provisioning Development with VM.Standard2.1 (oci-cloud-init.sh)
Action completed. Waiting until the resource has entered state: ('RUNNING',)
+++ oci-provision.sh: Getting public IP address
    oci-provision.sh: Public IP is: xxx.xxx.xxx.xxx

Demo

Join the discussion

Comments ( 4 )
  • Torsten Boettjer Tuesday, March 17, 2020
    First listening under Platform images, line 6:

    --query '[data[0].id, data[0]."display-name"] | join(`n`,@)'

    doesn't that have to be:

    --query '[data[0].id]
  • Philippe Vanhaesendonck Tuesday, March 17, 2020
    Torsten,
    You are perfectly right, you can limit the query to "data[0].id" if you only want to retrieve the image OCID.

    The code snippets from this post are actually taken from the https://github.com/oracle/ol-sample-scripts/blob/master/oci-provision/oci-provision.sh script, where I also use the "display-name" which explains why we have both fields here.
  • Rajeev Goel Wednesday, March 18, 2020
    Can you please tell me how to handle the Service Error using JMES:
    oci os object put -ns verisurenext -bn sbnnogl-incomin --file test.csv --query 'ServiceError/message' --raw-output


    Uploading object [####################################] 100%ServiceError:
    {
    "code": "BucketNotFound",
    "message": "Either the bucket named 'sbnnogl-incomin' does not exist in the namespace 'verisurenext' or you are not authorized to access it",
    "opc-request-id": "fra-1:3gWF2SSWJ1l7ZS_UFS0UD46IWatXIz1RA66QFlVwhHzFUcZ3bVsNNSm_7LJex1lb",
    "status": 404
    }

    i need to pick the message from the above
  • Philippe Vanhaesendonck Wednesday, March 18, 2020
    Rajeev,

    The "Service Error" message is an error message and as such doesn't go through JMSPath as the query is typically set to match a valid JSON response rather than an error.

    As the oci command will return a non-zero status code, you can still handle error conditions,but you will have to use JQ to process these.

    I have posted a short gist to illustrate what can be done: https://gist.github.com/AmedeeBulle/84c1fe6f964e6989ea2cf0d23c59ff80
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.