X

News, tips, partners, and perspectives for the Oracle Linux operating system and upstream Linux kernel work

Using rclone to copy data in and out of Oracle Cloud Object Storage

Introduction

In this blog post I’ll show how to use rclone on Oracle Linux with free object storage services included in Oracle Cloud Free Tier. Free tier includes 20GiB of object storage.

Rclone is a command line program to sync files and directories to and from various cloud-based storage services. Oracle Cloud Object Storage is Amazon S3 compatible, so I’ll use Rclone’s S3 capabilities to move data between my local Oracle Linux system and object storage.

One way to configure Rclone is to run rclone config and step through a series of questions, adding Oracle Cloud Object Storage as an S3 compatible provider. Instead, I’m going to use Oracle Cloud Infrastructure’s Cloud Shell to gather the relevant data and construct what’s ultimately a small configuration file.

The high level steps are:

  • On Oracle Cloud Infrastructure:
    • Create an object storage bucket
    • Create an Access Key/Secret Key pair
    • Gather relevant values for Rclone configuration
  • On your local Linux system
    • Install Rclone
    • Create an Rclone config file

Create Object Storage Bucket

One of the benefits of Cloud Shell is that it includes pre-configured OCI Client tools so you can begin using the command line interface without any configuration steps.

Accessing OCI Cloud Shell
 

Starting in Cloud Shell, set up environment variables to make running subsequent commands easier. The following stores your region and tenancy OCID, and storage namespace in environment variables.

export R=`curl -s http://169.254.169.254/opc/v1/instance/ | jq -r '.region'`
export C=`oci iam compartment list | jq -r '.data[] ."compartment-id"'`
export N=`oci os ns get | jq -r '.data'`
Cloud Shell in action
 

To create a storage bucket:

oci os bucket create --name mybucket --compartment-id $C

Create an Access Key/Secret Key pair

The Amazon S3 Compatibility API relies on a signing key called a Customer Secret Key.

export U=`oci os bucket list --compartment-id=$C | jq -r '.data[] ."created-by"'`
oci iam customer-secret-key create --display-name storagekey --user-id $U
export K=`oci iam customer-secret-key list --user-id $U | jq -r '.data[] | select (."display-name"=="storagekey") | ."id"'`

In the response, id corresponds to the access key and key represents the secret key. Make a note of the key immediately because it will not be shown to you again!

Finally, gather up the relevant values for the Rclone configuration. Remember to copy the secret key and save it somewhere.

Run the following to collect and display the information you need.

echo "ACCESS KEY: $K"; echo "SECRET KEY: check your notes"; echo "NAMESPACE: $N"; echo "REGION: $R"

Set Up Linux System with Rclone

Over to the local system on which Rclone will be used to move files to- and from object storage.

Install Rclone

To install Rclone:

$ sudo yum install -y oracle-epel-release-el7 && sudo yum install -y rclone

Create the Rclone Configuration File

In your home directory, create a file, .rclone.conf using the contents below, replacing the values you gathered earlier:

[myobjectstorage]
type = s3
provider = Other
env_auth = false
access_key_id = <ACCESS KEY>
secret_access_key = <SECRET KEY>
endpoint = <NAMESPACE>.compat.objectstorage.<REGION>.oraclecloud.com

Running Rclone

You are now ready to start copying files to object storage. The following copies a file, myfile.txt to object storage. You can show the contents of object storage using rclone ls.

$ echo `date` > myfile.txt
$ rclone copy myfile.txt myobjectstorage:/mybucket
$ rclone ls myobjectstorage:/mybucket

Conclusion

Rclone is a useful command line utility to interact with, among other types, S3 compatible cloud-based object storage. Oracle Cloud Object Storage has an S3 compatible API. In this blog post, I showed how to install Rclone from Oracle Linux yum server and configure it using free Oracle Cloud Object Storage.

References

Join the discussion

Comments ( 2 )
  • Greg Tuesday, May 12, 2020
    Great blog thanks Sergio - got me using the Cloud Shell, more of the OCI CLI tools and worked out how to get rclone working - so a great lesson in one blog!
  • Matthieu Heimer Thursday, July 16, 2020
    A note if you are trying to access a bucket outside of your home region. I followed these steps and was getting an error:

    Failed to ls: SignatureDoesNotMatch: The required information to complete authentication was not provided.
    status code: 403, request id: ...

    The important note on https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm is actually important.

    You can add the region to the .rclone.conf file with a line like:

    region = us-phoenix-1
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.