bbcp and rclone for Oracle Linux

Last week we packaged up a few more RPMs for Oracle Linux 7 that will help make life easier for Cloud users.

bbcp 15.02.03.01.1-3  in ol7_developer:

# yum install bbcp

bbcp is what I would call ssh on steroids. If you want to copy files from a local node to a remote node (say in Oracle Cloud) then this is a great tool. It might require some tuning but the idea is that you can open up parallel TCP streams. When you do large file transfers this should be able to give you a bit of a performance boost. I would also recommend using UEK5 and enable BBR as the congestion control algo. (see an old blog entry). The combination of enabling BBR (only has to be done on one of the 2 nodes (src or dest)) and using bbcp to copy large files using parallel streams should provide you the best throughput. By making this into an RPM for OL, it makes it easily available for everyone to use.

rclone 1.42 in ol7_developer

# yum install rclone

rclone is a very cool command line tool to move files around from/to local storage and cloud object storage. This works very well with Oracle Cloud Infrastructure's Object Storage. Now that it's packaged as an RPM with OL you can just install it directly from the command line instead of having to go download a file from a website. rclone works like scp.

Example could be  # rclone copy localdir ocistorage:remotedir

In order to configure rclone for Oracle Cloud Infrastructure's Object Storage, you have to create an "Amazon S3 Compatible API Key". This generates a secret key that you have to use during rclone config along with the access key (looks like an OCID in Object Storage   ocid1.credential.oc1.<string>) .

Configuration example:

# sudo yum install -y rclone

-> In the OCI console you go to Identity -> Users -> User Details -> Amazon S3 Compatible API Key and generate a new Secret Key.

-> copy the secret key because you need that to configure rclone, and you will also need the  Access Key (which is an OCID)

-> configure rclone on your OL7 client.

Example :

# rclone config

-> type n (new remote) and give it a name

name> ocistorage

Type of storage to configure.

-> type 3  (Amazon S3 Compliant Storage Providers (AWS, Ceph, Dreamhost, IBM COS, Minio))

Choose your S3 provider.

type 8 (Any other s3 compatible provider)

-> Next type 1 (1 / Enter AWS credentials in the next step) 

For access key provide the ocid

-> access_key_id> ocid1.credential.....

For the secret access key use your secret key that was just generated.

secret_access_key> tyjXhM7eUuB2v........

Region to connect to.

-> hit enter

For endpoint (example, phoenix) enter a https url

example :  https://orclwim.compat.objectstorage.us-phoenix-1.oraclecloud.com

my tenant name is orclwim  so replace it with your tenant name.

The end point URLs are

https://<tenantname>.compat.objectstorage.us-phoenix-1.oraclecloud.com

https://<tenantname>.compat.objectstorage.us-ashburn-1.oraclecloud.com

https://<tenantname>.compat.objectstorage.eu-frankfurt-1.oraclecloud.com

https://<tenantname>.compat.objectstorage.uk-london-1.oraclecloud.com

Location Constraint hit enter

and ACL hit enter

type y OK to store the settings

you should get something like

Current remotes:

Name                 Type
====                 ====
ocistorage           s3

That's it - we have some code changes pending that will include oracle and the endpoints in rclone but those are being reviewed still.