We’re thrilled to announce a major enhancement for Oracle Cloud Infrastructure (OCI) customers: Bi-Directional Object Copy is now available for OCI File Storage with Lustre File System (LFS). With this new capability, you can now copy data seamlessly and efficiently between high-performance Lustre file systems and OCI Object Storage—in both directions.

Unlocking the Next Level in Data Pipeline Efficiency

OCI File Storage with Lustre powers critical AI/ML and HPC workloads with a fully managed, scalable, and high-throughput parallel file system. Customers depend on OCI Lustre for use cases such as large language model (LLM) training, genetic research, robotics, and engineering simulation—where data volumes routinely scale to petabytes and require ultra-low latency.

Before the launch of object copy feature, you needed to rely on manual processes such as rclone to export data back and forth. These processes are error-prone and incur operational overhead. Starting today you can:

  • Automate 100% of Your Data Copy Workflows: Eliminate routine manual intervention. With full integration into the OCI Console, CLI, SDKs, and APIs, data copying becomes a background operation, minimizing human oversight and freeing teams to focus on innovation.
  • Improve reliability and reduce risk: Manual scripts (E.g., rclone) are not only operationally intense but also error-prone—even a single copy failure can lead to hours of GPU or AI training downtime, potentially costing thousands of dollars in wasted compute cycles. With Bi-Directional Object Copy, reliability is significantly improved through native, monitored copy operations. Built-in checks and integrated metrics help ensure that your critical datasets arrive safely and punctually, minimizing the risk of costly resource idle time or failed training runs.
  • Lower TCO: Using Object Storage for persistent storage and reserving Lustre for high-performance processing windows means compute and storage costs are tightly optimized. Early adopters are seeing up to 30% cost savings on long-running pipelines.
  • Scalability: Support for linking up to 10 object storage buckets or prefixes per Lustre file system enables unprecedented flexibility without operational bottlenecks.

Bringing the Best of Both Worlds to Your AI Pipeline

AI/ML data pipelines often combine object storage and file systems. Object Storage enables unlimited scalability and cost-effective durability, making it ideal for storing large datasets, raw inputs, and long-term archives. Lustre, on the other hand, delivers the performance needed for iterative training, simulation, and analytics, with POSIX compliance for applications that expect a file system interface.

  • Import raw datasets from Object Storage directly into Lustre clusters for preprocessing and training by copying.
  • Export simulation outputs, AI checkpoints, or model artifacts back to Object Storage for long-term durability and sharing by copying.
  • Synchronize results between iterations with automated rules for skipping or overwriting files, making data lifecycle management simple and repeatable.

How It Works—From Hours to Minutes

A research team developing a new LLM previously spent days managing up to 5 PB of data between storage tiers. Now, with Object Copy:

  1. Bulk copy training data to Lustre in minutes.
  2. Run training jobs directly on the Lustre file system attached to GPU or HPC nodes.
  3. Checkpoint and archive models back to Object Storage with a single copy operation, ready for hand-off or cluster teardown.

Teams report reducing end-to-end data copying time by up to 90%, and operational overhead by nearly 100% in many batch workflows.

Getting Started is Simple

  1. In the OCI Console, navigate to File Storage with Lustre.
Screen to create object storrage links
Figure 1.  Create Object Storage Links

 

  1. Enter the Lustre directory and the Object Storage bucket and/or prefix you want to copy (see Figure 2). You can configure up to 10 such links per file system.
Screen to enter Lustre Directory and Object STorage Bucket details
Figure 2. Enter Lustre Directory and Object Storage Bucket details

 

  1. For each Object Storage link, choose whether to Import (Object Storage → Lustre) or Export (Lustre → Object Storage) to start the data transfer. You can also set rules to skip or overwrite files that already exist in the file system or bucket (see Figure 3).
Screen where you can set Import and Export rules for file handling
Figure 3. Set Import and Export rule for file handling

 

  1. Monitor progress directly in the Console (see Figure 4), or automate copy operations using the OCI CLI, SDKs, and REST APIs.
Screen where you can monitor progress
Figure 4. Object Copy Metrics

 

Available Now—Start Building Faster Pipelines

Bi-Directional Object Copy for OCI Lustre File System is available in all regions where Lustre is supported. This feature brings transformational gains in speed, automation, and operational efficiency.

Ready to accelerate your workloads? Get started with OCI File Storage with Lustre and unlock the full potential of automated, high-performance data copying today.