X

Customize Block Volume Backups with the Oracle Cloud Infrastructure CLI

Changbin Gong
Principal Product Manager and Cloud Advisor

It is a common IT Operations practice to manage the data protection of compute instances through command line and scripts.  This post provides detailed instructions on how to customize your application compute instance's block volume backup by using the Oracle Cloud Infrastructure Command Line Interface (CLI). With the CLI, you can perform a block volume backup based on your schedule and remove old backups based on your retention period.

Environment

You can run this customized block volume backup task from a centralized system or inside the application compute instance itself. In the example in this post, the task is run inside a compute instance, and is created as a bash shell script.

Because this task runs inside a compute instance, we recommend using the instance principal feature to avoid storing user credentials locally. 

Volume Group

For this customized volume backup script, we recommend using the volume group feature to a create block volume backup. This feature enables you to group multiple block volumes and create a collection of volumes from which you can create consistent volume backups and clones. You can restore an entire group of volumes from a volume group backup. 

Customized Volume Backup Script

Before you can run this customized script, you need to install the CLI on your compute instance. Detailed instructions for installing the CLI are located in the documentation.

Step 1

The first step of this script gets required information about where this compute instance is located, such as availability domain, compartment OCID, and instance OCID. You can get this information through the metadata of the compute instance. 

# Get availability domain

AD=$(curl -s http://169.254.169.254/opc/v1/instance/ |grep availabilityDomain | awk '{print $3;}' | awk -F\" '{print $2;}')

echo "AD=$AD"

# Get Compartment-id

COMPARTMENT_ID=$(curl -s http://169.254.169.254/opc/v1/instance/ |grep compartmentId | awk '{print $3;}' | awk -F\" '{print $2;}')

echo "COMPARTMENT_ID=$COMPARTMENT_ID"

# Get Instance-id

INSTANCE_ID=$(curl -s http://169.254.169.254/opc/v1/instance/ |grep ocid1.instance | awk '{print $3;}' | awk -F\" '{print $2;}')

echo "INSTANCE_ID=$INSTANCE_ID"

 

Step 2

The second step of the script gets the tagging information from the boot volume of the compute instance. Then you can use the same tagging information to create the volume group and its backups. With the same tags, you can easily to sort or filter your volumes and their backups.     

# Get tags of boot volume of this instance

# We will use these tags for volume group created for this instance's boot volume and other attached volumes

 

# Get boot Volume tag

BOOTVOLUME_DEFINED_TAGS=$(oci compute boot-volume-attachment list --compartment-id=$COMPARTMENT_ID --availability-domain=$AD --instance-id=$INSTANCE_ID --auth instance_principal | jq '.data[] | ."defined-tags"')

 

BOOTVOLUME_FREEFORM_TAGS=$(oci compute boot-volume-attachment list --compartment-id=$COMPARTMENT_ID --availability-domain=$AD --instance-id=$INSTANCE_ID --auth instance_principal | jq '.data[] | ."freeform-tags"')

 

Note: The jq command is very useful for parsing the JSON output from the CLI.  

Step 3

The third step of the script gets the boot volume OCID and a list of attached block volumes' OCIDs for the compute instance. These OCIDs will be used to construct JSON data for the volume group creation command. 

# Get boot volume-id

BOOTVOLUME_ID=$(oci compute boot-volume-attachment list --compartment-id=$COMPARTMENT_ID --availability-domain=$AD --instance-id=$INSTANCE_ID --auth instance_principal | grep boot-volume-id | awk '{print $2;}'|awk -F\" '{print $2;}')

echo $BOOTVOLUME_ID

 

# Get a list of attached block volumes

BLOCKVOLUME_LIST=($(oci compute volume-attachment list --compartment-id=$COMPARTMENT_ID --availability-domain=$AD --instance-id=$INSTANCE_ID --auth instance_principal | grep volume-id | awk '{print $2;}'|awk -F\" '{print $2;}'))

 

# Construct JSON for volume group creat command

LIST="[\"$BOOTVOLUME_ID\""

for volume in ${BLOCKVOLUME_LIST[*]}

do

   LIST="${LIST}, \"${volume}\""

done

LIST="${LIST}]"

SOURCE_DETAILS_JSON="{\"type\": \"volumeIds\", \"volumeIds\": $LIST}"

 

Step 4

The fourth step of the script checks whether existing volume groups have been created by the script before.

  • If there is no existing volume group, the script creates the volume group based on the information from the previous steps, such as list OCIDs of the boot volume and all the attached block volumes.
  • If there is an existing volume group, the script checks whether there are any changes to the member volumes inside the volume group; for example, new block volumes are attached to the compute instance. If there are changes, the script updates the volume group with the latest volumes. 

# Check whether there is an existing available volume group created by the script.

VOLUME_GROUP_NAME="volume-group-$INSTANCE_ID"

VOLUME_GROUP_ID=$(oci bv volume-group list --compartment-id $COMPARTMENT_ID --availability-domain $AD --display-name $VOLUME_GROUP_NAME --auth instance_principal | jq '.data[] | select(."lifecycle-state" == "AVAILABLE") | .id' |awk -F\" '{print $2;}')

 

echo "VOLUME_GROUP_ID=$VOLUME_GROUP_ID"

 

# If volume group does not exist, then create a new volume group

if [ -z "$VOLUME_GROUP_ID" ]; then

 

# Create volume group

VOLUME_GROUP_ID=$(oci bv volume-group create --compartment-id $COMPARTMENT_ID --availability-domain $AD --source-details "$SOURCE_DETAILS_JSON" --defined-tags="$BOOTVOLUME_DEFINED_TAGS" --freeform-tags="$BOOTVOLUME_FREEFORM_TAGS" --display-name=$VOLUME_GROUP_NAME --wait-for-state AVAILABLE --max-wait-seconds 24000 --auth instance_principal | grep ocid1.volumegroup | awk '{print $2;}' |awk -F\" '{print $2;}')

 

echo "VOLUME_GROUP_ID=$VOLUME_GROUP_ID"

 

else

# volume group exists and then check whehter there are any changes for the attached block volumes

VOLUME_LIST_IN_VOLUME_GROUP=$(oci bv volume-group get --volume-group-id $VOLUME_GROUP_ID --auth instance_principal| jq '.data | ."volume-ids"' | grep ocid1.volume | awk -F\" '{print $2;}')

# compare with attached block volume list

LIST3=$(echo $BLOCKVOLUME_LIST $VOLUME_LIST_IN_VOLUME_GROUP | tr ' ' '\n' | sort | uniq -u)

if [ -z "$LIST3" ]; then

    echo "no change for volume group"

else

    # update volume group with updated volume ids list

    VOLUME_GROUP_ID=$(oci bv volume-group update --volume-group-id $VOLUME_GROUP_ID --volume-ids "$LIST" --defined-tags="$BOOTVOLUME_DEFINED_TAGS" --freeform-tags="$BOOTVOLUME_FREEFORM_TAGS" --display-name=$VOLUME_GROUP_NAME --wait-for-state AVAILABLE --max-wait-seconds 24000 --auth instance_principal | grep ocid1.volumegroup | awk '{print $2;}' |awk -F\" '{print $2;}')

fi

fi

 

Step 5

The last step of the script creates the backup for this volume group. The script uses the same tags, defined-tags and freeform-tags, from the boot volume of the compute instance. However, you can define your own customized tags as needed. 

# Create Backup

VOLUME_GROUP_BACKUP_NAME="Volume-group-backup-$VOLUME_GROUP_ID"

 

VOLUME_GROUP_BACKUP_ID=$(oci bv volume-group-backup create --volume-group-id $VOLUME_GROUP_ID --defined-tags="$BOOTVOLUME_DEFINED_TAGS" --freeform-tags="$BOOTVOLUME_FREEFORM_TAGS" --display-name=$VOLUME_GROUP_BACKUP_NAME --wait-for-state AVAILABLE --max-wait-seconds 24000 --auth instance_principal | grep ocid1.volumegroupbackup | awk '{print $2;}' |awk -F\" '{print $2;}')

 

echo "VOLUME_GROUP_BACKUP_ID=$VOLUME_GROUP_BACKUP_ID"

echo "VOLUME_GROUP_BACKUP_NAME=$VOLUME_GROUP_BACKUP_NAME"

 

You can configure the cron job to run this customized volume backup script according to your backup schedule. 

Volume Backup Retention Script

Based on your requirements, you might need to define a customized and flexible retention period for your volume backups. For example, say you want the retention period of the volume backups to be 14 days.  Following example script checks the creation times for your volume backups and then deletes the old backups beyond the retention period. You can configure and run this script in your cron job based on how often you want to conduct a backup retention check. 

# get all the volume group backup

RETENTION_DAYS=14

VOLUME_GROUP_BACKUP_LIST=$(oci bv volume-group-backup list --compartment-id $COMPARTMENT_ID --volume-group-id $VOLUME_GROUP_ID --display-name=$VOLUME_GROUP_BACKUP_NAME --auth instance_principal | jq -r '.data[] | select (."time-created" | sub("\\.[0-9]+[+][0-9]+[:][0-9]+$"; "Z") | def daysAgo(days):  (now | floor) - (days * 86400); fromdateiso8601 < daysAgo(14)) | .id')

 

echo $VOLUME_GROUP_BACKUP_LIST

for backup in ${VOLUME_GROUP_BACKUP_LIST[*]}

do

   DELETED_VOLUME_GROUP_BACKUP_ID=$(oci bv volume-group-backup delete --volume-group-backup-id ${backup} --force --wait-for-state TERMINATED --max-wait-seconds 24000 --auth instance_principal | grep ocid1.volumegroupbackup | awk '{print $2;}' |awk -F\" '{print $2;}')

   echo $DELETED_VOLUME_GROUP_BACKUP_ID

done

 

Join the discussion

Comments ( 1 )
  • Vijay Monday, March 4, 2019
    Very nice article 👌
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha
Oracle

Integrated Cloud Applications & Platform Services