Data migration to and between OCI storage services using Resilio Connect

March 1, 2024 | 5 minute read
Ashutosh Mate
Sr. Principal Solutions Architect
Vinoth Krishnamurthy
Solutions Architect / Product Manager - OCI Storage Services
Text Size 100%:

Oracle Cloud Infrastructure (OCI) customers are looking to migrate increasingly larger amounts of data from on-premises or other cloud vendors into OCI File Storage service and OCI Object Storage service. They also want to be able to move data between these services in the same or different regions and tenancies. They want to be able to start off data transfer jobs and monitor progress periodically. To help our customers with the data migration challenge, OCI has partnered with Resilio, who offer Resilio Connect for high-speed data transfer at wire speed.

The OCI Storage Outbound PM team recently conducted validation of using Resilio Connect for various data migration scenarios for OCI customers and recommended it to be listed in Oracle Cloud Marketplace for download. This blog post provides a high-level overview of the tests, results, and recommendation.

Resilio Connect tests

For the validation tests, we deployed the Resilio server and Management Console in the Ashburn (IAD) region, while Resilio Connect agents were deployed on-premises and on Compute instances in OCI Phoenix (PHX) and Frankfurt (FRA) regions. These agents were connected to the Object Storage and File Systems in the regions for data transfer. The Resilio Connect agent was also deployed in Amazon Web Services (AWS)’s Ohio region for transferring data from AWS Elastic File System (EFS) to OCI File Storage. The following diagram provides the deployment overview.

OCI deployment architecture diagram

We evaluated Resilio Connect solution from the partner to test various data migration scenarios and measure time taken. We also compared this measurement against the time taken by open source tools, such as rclone, rsync, and fpsync, and OCI internal features like file and object replication. The Resilio Management Console makes it very easy to deploy and configure agents, configure OCI Object Storage endpoint as S3-compatible cloud object storage and mount File Storage shares on the agent. You can create jobs for data transfer and sync. You can run these jobs on a schedule and monitor them through the console as shown in the following screenshots.

Resilio agents deployed on Compute virtual machine (VM) instances in different OCI regions and AWS regions (agents view)

Adding OCI Object Storage as S3-compatible cloud storage to Resilio

Resilio Connect data transfer jobs created

Resilio Management Console: Job Runs view

Data flow migration from AWS EFS to OCI File Storage diagram

Test results

Tests were conducted with data sets from single 10-GB file to file system with thousands of files totaling 462 GB. Tests also included data sets with hundreds of files totaling 100GB and a bucket with hundreds of objects totaling 107 GB.

The data transfer scenarios tested with Resilio Connect included the following examples:

  • From on-premises to OCI Object Storage and File Storage in different regions

  • Between File Storage and File Storage in different regions (one time and periodic sync)

  • Between Object Storage and Object Storage in different regions

  • Between File Storage and Object Storage across different regions and tenancies

  • From AWS EFS to OCI Object Storage

Transfer scenario Transfer speed
On-premises to OCI Object Storage or File Storage 20–373 Mbps
AWS EFS to OCI File Storage 1.6 Gbps
OCI File Storage to OCI File Storage in another region 980–1.5 Gbps
OCI File Storage to OCI Object Storage in another region 932 Mbps
OCI Object Storage to OCI Object Storage in another region 1.3 Gbps

Our finding was that Resilio Connect data migration times are comparable to tools like rclone. OCI internal features like crossregion copy and replication are faster than Resilio Connect because they use OCI backbone. The network speed and bandwidth have the biggest impact on overall data transfer performance.

Our recommendation

Based on the evaluation conducted, we have provided Resilio Connect in Oracle Cloud Marketplace as a recommended, tested data migration solution for the following reasons:

  • Resilio Connect provides intuitive GUI management console to deploy agents, create profiles, optimize performance settings, run and monitor jobs, and create support tickets if needed.

  • Resilio is responsible for supporting large data migration projects for key OCI storage customers.

  • Provides easy-to-use GUI, CLI, and APIs for data migration with Resilio instead of open source rclone CLI

The latest version of Resilio Connect is available in Oracle Cloud Marketplace. You can download the app and get the required one-time or annual subscription licenses from Resilio for Bring Your Own License (BYOL) deployment in Oracle Cloud Infrastructure.

Ashutosh Mate

Sr. Principal Solutions Architect

Vinoth Krishnamurthy

Solutions Architect / Product Manager - OCI Storage Services

Previous Post

Powering the AI revolution: Oracle at NVIDIA GTC

Richard Wang | 6 min read

Next Post

Announcing new features and implementation playbook for OCI Secure Desktops