Announcing native OCI Object Storage provider backend support in rclone

We’re pleased to announce the general availability of native Oracle Cloud Infrastructure (OCI) Object Storage API support in the popular open source Rclone data migration and sync tool. Until now, OCI Object Storage users who wanted to use Rclone had to set up, manage, and distribute access keys and secret keys for OCI Identity and Access Management (IAM) domain users just to use the S3 compatible API and S3 backend. By providing native Object Storage access support in Rclone that works with user, instance, and resource principals in OCI, the need for managing s3 compatible secrets is eliminated!

OCI has now added native access backend to the open source Rclone tool. It’s available for download in v1.60.0 - 2022-10-21 from GitHub.

What’s available in the new version of Rclone

With this version of Rclone, you can access the OCI Object Storage with user, instance, resource, or no-auth principals when using Rclone to synchronize data to the cloud. Earlier, OCI Object Storage users had to manage S3 access-key and secret-key pairs and store them in KMS Vault stores for multiple users wanting to use Rclone.

Large enterprise customers with thousands of users required managing secrets for all of them. By using OCI native API instead, you can create an OCI Object Storage configuration for user, instance, and resource principals by running the Rclone config interactive setup command and setting up a new remote.

Authentication with an OCI user and API key

You have a few options for setting up authorization for your user principals. You can set up user principals using an OCI user and an API key for authentication. The following code block shows a sample Rclone config file for the authentication provider user principal:

[oos]
type = oracleobjectstorage 
namespace = id(redacted)34 
compartment = ocid1.compartment.oc1..aa(redacted)ba 
region = us-ashburn-1 
provider = user_principal_auth 
config_file = ~/.oci/config 
config_profile = Default

An OCI user config file, ~/.oci/config, would include the following as an example:

[DEFAULT] 
user=ocid1.user.oc1..aa(redacted)ja 
fingerprint=60:03:63:4b:(redacted):bf:5b:e7:22 
tenancy=ocid1.tenancy.oc1..aa(redacted)6q 
region=us-ashburn-1 
key_file=(path to your private keyfile)

You use this method from any server within OCI or on-premises or from other cloud provider, but review the following considerations:

• The actions allowed to run depends on the user’s privileges and policies.

• Other users on the machine can run the commands in the name of that user.

• This configuration isn’t possible when using federated users.

• Overhead of managing users and keys

• If the user is deleted, the config file no longer works and can cause automation regressions that use that user’s credentials.

Using Rclone with Compute instance

You can authorize an OCI Compute instance to use Rclone by using its identity and certificates as instance principal. With this approach, you don’t need to store or manage credentials.

The following code block show a sample Rclone configuration file for authentication provider instance principal:

[opc ~]$ cat ~/.config/Rclone/Rclone.conf
[oos]
type = oracleobjectstorage
namespace = id(redacted)fn
compartment = ocid1.compartment.oc1..aa(redacted)k7a
region = us-ashburn-1
provider = instance_principal_auth

This method has the following advantages and considerations:

• With instance principals, you don’t need to configure user credentials and save them to disk in your Compute instances or rotate the credentials.

• You don’t need to deal with users and keys.

• You don’t have to manage access keys, user private keys, store them in vault, or use KMS, greatly helping with automation.

• You need to configure a dynamic group with this instance as member and add policies to read Object Storage.

• Everyone who has access to this machine can run the CLI commands.

• In OCI Auditing, the instance runs a command, so you can’t map the actions to a real user.

• This method is applicable for OCI Compute Instances only. You can’t use it on external instance or resources.

The following code block gives a sample Rclone configuration file for authentication provider resource principal:

[oos]
type = oracleobjectstorage
namespace = id(redacted)34
compartment = ocid1.compartment.oc1..aa(redacted)ba
region = us-ashburn-1
provider = resource_principal_auth

Using Rclone with no authorization

You can also create an Rclone remote that refers to a public bucket by instructing Rclone not to use any authentication method. Authentication isn’t needed to access public buckets. The following code block gives a sample Rclone configuration file for no authentication provider:

[oos] 
type = oracleobjectstorage 
namespace = id(redacted)34 
compartment = ocid1.compartment.oc1..aa(redacted)ba 
region = us-ashburn- 1
provider = no_auth

Conclusion

Get started with Rclone by downloading v1.60.0 - 2022-10-21 from GitHub. For more information on the concepts discussed in this post, see the following resources:

• Rclone.org

• Download version 1.60.0 with native OCI support

• Rclone documentation for Oracle Cloud Infrastructure Object Storage

• Rclone changelog and documentation

• OCI Object Storage PM: Ashutosh Mate

• OCI native Rclone dev lead: Manoj Ghosh