Get the Most Out of Storage Gateway with Cloud Sync

Douglas Copas
Solutions Architect

Oracle Cloud Infrastructure Storage Gateway's Cloud Sync feature helps keep files synchronized across two NFS servers.

In my previous blog post, I introduced Storage Gateway as an Object Storage backed NFS server designed to run on Oracle Cloud Infrastructure. The solution is ideal for data transfer, backup, and archive use cases, and allows software that's not cloud native to use Object Storage without needing to know how to read and write objects from an Object Storage bucket.

But what if your application makes frequent writes to a large set of files? In this case, writing directly to Storage Gateway is not practical. Reading and writing files from an external cloud resource takes time, which slows down an application that needs performant storage. However, you still want to leverage the practically limitless storage offered by Object Storage. A common solution in this case involves running two NFS servers: one backed by block volumes for performance, and one backed by Object Storage for archiving. And it would be great if the files on the local NFS could be kept synchronized with the files on Storage Gateway. You could do this manually, if you're excited by the possibility of human error. Or, you could use Cloud Sync.

What is Cloud Sync?

The Cloud Sync feature of Storage Gateway uses the Linux standard rsync command to synchronize the files from a locally backed NFS server to an Object Storage backed Storage Gateway NFS server, and vice versa. Additionally, the tool also performs error handling and progress monitoring. It makes meta-automation (for example, a nightly cron job) that much easier because you don’t need to add this to your scripting. Connectivity issues are handled seamlessly through auto retry, and both the web console and command line interface (CLI) invocations provide status feedback, with the CLI's monitoring output being ideal for consumption by higher-level automation.

But wait, that’s not all! As the CLI parameters reveal, Cloud Sync performs a task that a human would almost always fail to perform—synchronize files in parallel:

sudo ocisg cloudsync create [--auto-delete] [--parallel=<number>] [--files-from=<file>] <job_name> <source_path> <target_path>

You simply create a job, specify the number of parallel processes (up to a maximum of 10), and run it:

sudo ocisg cloudsync run <job_name> 

This feature is particularly helpful when you have a large amount of data to transfer from a locally backed NFS server to Storage Gateway. Perhaps you have multiple terabytes of data that you want to move to Object Storage. Using the Cloud Sync feature, it's possible to ingest multiple tens of terabytes of data per day. With the auto-retry and resume features, no manual intervention is required.

And although I always recommend using a CLI, the web console provides a fast, intuitive, and convenient method for creating, running, querying, and deleting Cloud Sync jobs.

I hope you’ve enjoyed reading about this exciting new addition to Oracle Cloud Infrastructure Storage Gateway. Until next time, keep it cloudy!

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha