We’re pleased to announce the general availability of native Oracle Cloud Infrastructure (OCI) Object Storage API support in the popular open source Rclone data migration and sync tool. Until now, OCI Object Storage users who wanted to use Rclone had to set up, manage, and distribute access keys and secret keys for OCI Identity and Access Management (IAM) domain users just to use the S3 compatible API and S3 backend. By providing native Object Storage access support in Rclone that works with user, instance, and resource principals in OCI, the need for managing s3 compatible secrets is eliminated!
OCI has now added native access backend to the open source Rclone tool. It’s available for download in v1.60.0 - 2022-10-21 from GitHub.
With this version of Rclone, you can access the OCI Object Storage with user, instance, resource, or no-auth principals when using Rclone to synchronize data to the cloud. Earlier, OCI Object Storage users had to manage S3 access-key and secret-key pairs and store them in KMS Vault stores for multiple users wanting to use Rclone.
Large enterprise customers with thousands of users required managing secrets for all of them. By using OCI native API instead, you can create an OCI Object Storage configuration for user, instance, and resource principals by running the Rclone config interactive setup command and setting up a new remote.
You have a few options for setting up authorization for your user principals. You can set up user principals using an OCI user and an API key for authentication. The following code block shows a sample Rclone config file for the authentication provider user principal:
[oos]
type = oracleobjectstorage
namespace = id(redacted)34
compartment = ocid1.compartment.oc1..aa(redacted)ba
region = us-ashburn-1
provider = user_principal_auth
config_file = ~/.oci/config
config_profile = Default
An OCI user config file, ~/.oci/config, would include the following as an example:
[DEFAULT]
user=ocid1.user.oc1..aa(redacted)ja
fingerprint=60:03:63:4b:(redacted):bf:5b:e7:22
tenancy=ocid1.tenancy.oc1..aa(redacted)6q
region=us-ashburn-1
key_file=(path to your private keyfile)
You use this method from any server within OCI or on-premises or from other cloud provider, but review the following considerations:
The actions allowed to run depends on the user’s privileges and policies.
Other users on the machine can run the commands in the name of that user.
This configuration isn’t possible when using federated users.
Overhead of managing users and keys
If the user is deleted, the config file no longer works and can cause automation regressions that use that user’s credentials.
You can authorize an OCI Compute instance to use Rclone by using its identity and certificates as instance principal. With this approach, you don’t need to store or manage credentials.
The following code block show a sample Rclone configuration file for authentication provider instance principal:
[opc ~]$ cat ~/.config/Rclone/Rclone.conf [oos] type = oracleobjectstorage namespace = id(redacted)fn compartment = ocid1.compartment.oc1..aa(redacted)k7a region = us-ashburn-1 provider = instance_principal_auth
This method has the following advantages and considerations:
With instance principals, you don’t need to configure user credentials and save them to disk in your Compute instances or rotate the credentials.
You don’t need to deal with users and keys.
You don’t have to manage access keys, user private keys, store them in vault, or use KMS, greatly helping with automation.
You need to configure a dynamic group with this instance as member and add policies to read Object Storage.
Everyone who has access to this machine can run the CLI commands.
In OCI Auditing, the instance runs a command, so you can’t map the actions to a real user.
This method is applicable for OCI Compute Instances only. You can’t use it on external instance or resources.
The following code block gives a sample Rclone configuration file for authentication provider resource principal:
[oos] type = oracleobjectstorage namespace = id(redacted)34 compartment = ocid1.compartment.oc1..aa(redacted)ba region = us-ashburn-1 provider = resource_principal_auth
You can also create an Rclone remote that refers to a public bucket by instructing Rclone not to use any authentication method. Authentication isn’t needed to access public buckets. The following code block gives a sample Rclone configuration file for no authentication provider:
[oos]
type = oracleobjectstorage
namespace = id(redacted)34
compartment = ocid1.compartment.oc1..aa(redacted)ba
region = us-ashburn- 1
provider = no_auth
Get started with Rclone by downloading v1.60.0 - 2022-10-21 from GitHub. For more information on the concepts discussed in this post, see the following resources:
Rclone documentation for Oracle Cloud Infrastructure Object Storage
OCI Object Storage PM: Ashutosh Mate
OCI native Rclone dev lead: Manoj Ghosh
Previous Post
Next Post