Building storage systems for the future: The OCI roadmap

September 10, 2024 | 6 minute read
Sabrinath Rao
Sr. Director, Product Management, OCI Storage
Text Size 100%:

Fast, cost-effective, and reliable are the three orthogonal forces that drive storage innovation, with the common refrain “pick two.” These driving forces are more pronounced now than they have ever been, as data-hungry applications such as AI and machine learning (ML), Hadoop, PostgreSQL and others blow through conventional boundaries of scale and performance. Oracle CloudWorld (OCW) is a good time to reflect on how much progress we have made this past year in all these areas and to look forward to the future. It has been quite the year for the Oracle Cloud Infrastructure (OCI) Storage team. In the past year, we enabled the following capabilities:

  • A large AI customer with a 20-PB Lustre file system, one of the largest Lustre file systems in the cloud
  • 240 Gb/s aggregate throughput with a 5-PB file system for a large language model (LLM) customer with OCI File Storage
  • An industry-leading 750-node OpenSearch cluster and reduced costs  
  • Migration of one of the largest Hadoop deployments from on-premises to OCI Object Storage
  • Zero recovery point objective (RPO) with Database Optimized Storage, for OCI Database with PostgreSQL powered by OCI Block Storage

Join Cameron Bahar, SVP and general manager of OCI Storage and Data Management, on Wednesday, September 11 at 8:30 a.m. PT, as he shares more details on these innovations and more at his OCW session, The Future of Cloud Storage—OCI Storage Strategy and Roadmap [LRN1758].

Oracle CloudWorld 2024 OCI Storage Sessions

 

OCI Storage at a glance

Our team has the privilege and responsibility of running a Day 0 foundational cloud service. Our storage services must always be available and can never lose data. Thousands of customers trust us with their business-critical systems. In addition, OCI services and Oracle software-as-a-service (SaaS) applications depend on the reliability and availability of OCI Storage. Our customers and our services push us toward industry-leading performance and ease of use with features, such as OCI Block Volume’s dynamic performance scaling.

OCI Storage Services

 

What’s top of mind for customers

The following themes emerge when we talk to customers, and they anchor our roadmap: 

  • 10X scale: Customers with hundreds of Petabytes on day 1 is the norm today. Customers expect us to scale seamlessly without compromising on reliability and availability.
  • Price-performance: Customers expect us to deliver on hundreds to millions of IOPS and 10s of Gb/s per TB on the same data sets and vary the price based on the consumed performance.
  • Security and data protection: Interactive applications, such as big data systems, require access controls at a granular level as multiple applications access and use the same data.
  • Enabling partner ecosystems: Customers who lift and shift from on-premises to the cloud want minimal disruption. They expect us to enable partners, such as Commvault, for backup. 

 

What we have shipped this year

Price-performance

Industry-leading 1.3 Million IOPS with OCI Block Volumes: We shattered the one million I/O operations barrier! On May 30, 2024, we shared that customers can now achieve up to 1.3 Mil IOPS and 12 GB/s throughput per OCI Compute instance and attach 32 Ultra High Performance (UHP) volumes per single instance. This capacity is particularly useful for high performance I/O workloads, such as AI and machine learning (ML), online transaction processing, data warehousing and analytics, and databases with large data sets that are I/O sensitive.

Security and data protection

File Storage Clone Detach: OCI File Storage customers can now easily make fully independent file systems clones to quickly create thin, read-write copies of a file system for disaster recovery testing, application testing and development, and other use cases. You can go from testing to production in minutes by creating a dedicated file system completely detached from the parent file system.

Object Level Identity & Access Management: Earlier this year we shared that customers now have granular access controls at the object level. With Object Identity and Access Management (IAM), you can now set access control policies at an object level and enable workloads, such as Hadoop, data lakes, and internet of things (IoT), that share a single bucket with millions of objects with multiple applications.    

 

New features we are announcing today 

Price-performance

File Storage High Performance Mount Targets (HPMT): OCI File Storage already delivers an industry-leading 10-PB file system in production with a single file system. Starting today, customers can get up to 80 Gb/s per mount target with File Storage. You can add multiple mount targets and scale the aggregate throughput on a Gb/s per provisioned TB of storage basis. We have several AI and ML customers running their training workloads with high performance mount targets. Learn more at the OCW session,  OCI Storage—Performance Breakthroughs Enabling AI and HPC Workloads (LRN2934) on Wednesday, September 11 at 4:45 p.m.

Object Storage HDFS Connector: We offer a Hadoop Distributed File System (HDFS) connector that enables customers to connect HDFS natively to Object Storage. We worked with a customer to double the performance that they get on-premises. We also delivered observability and Kerberos authentication to enable a seamless migration to OCI. The HDFS Connector is open source, maintained by us, and available for free. Learn more at the OCW session, Uber’s Data and Streaming Stack on Oracle Cloud Infrastructure [LRN2108] on Wednesday, September 11 at 9:45 a.m.

Security and data protection

Object Storage support for Private Endpoints: Private endpoints help enable secure, private connectivity using a private IP address to access OCI Object Storage from your virtual cloud network (VCN) or on-premises network. The traffic doesn’t go over the public IP address (a public endpoint) or the internet.

 

Looking forward to the next few months 

Scale 10X

Scaling 10X requires us to fundamentally rethink our storage stack. This is a multi-quarter investment that is well underway. Learn more at the OCW session, The Future of Cloud Storage—OCI Storage Strategy and Roadmap [LRN1758] on Wednesday, September 11 at 3:30 p.m. 

Price-performance

File Storage with Lustre: We are enabling a large customer, training a foundational model with an industry-leading 20-PB Lustre file system. Later this year, we plan to on-board customers on to our upcoming fully managed File Storage with Lustre service. We’re enrolling interested customers to help us test this in our limited availability program. If you’re interested, contact your account representative.

File Storage User Quotas: OCI File Storage administrators can soon start managing quotas at the user and group level. This feature enables customers to set capacity limits and manage their consumption and costs.

Security and data protection

Object Storage support for multiple Checksums:  Regulators and compliance auditors have varying standards for which Checksum algorithms are acceptable. In a few weeks, Object Storage will support CRC32C, SHA256, and SHA384. We’re also building the ability to add other Checksum algorithms faster as the industry evolves.

File Storage 30 minute Recovery Point Objective (RPO): We enable File Storage customers with more granular recovery points, with as low as 15-minute intervals. File Storage is the storage backend for many enterprise applications. With flexible recovery point objective (RPO) up to 15 min intervals, we enable customers with options to optimize their costs with their disaster recovery and protection needs. 

Block Volumes support for customer managed keys for cross-region replication: Customers can soon start using their own encryption keys with Block Storage crossregion replication and policy-based crossregion backup copies. This feature enables enterprise customers that use their own keys, such as banks, financial institutions, and customers in highly regulated industries, to build a robust disaster recovery and data protection strategy for their enterprise applications.

Enabling partner ecosystem

Direct APIs: Change block tracking enables fast and cost-effective backups by capturing the changes since the last backup. We are partnering with ISVs such as Commvault to enable this capability for Block Storage volumes. The API for this capability will be available to all storage partners in the next few months. If you’re interested in enabling this API for efficient backups, contact your account manager.

 

Exciting transformation that is well underway

Today, Oracle Cloud Infrastructure Storage looks very different from two years ago. Our footprint has grown by orders of magnitude, and customers trust us to run business-critical workloads, such as Hadoop, AI and ML, banking, healthcare, and blockchain applications. We look forward to hearing your stories and learning how we can continue to enable your success.

Sabrinath Rao

Sr. Director, Product Management, OCI Storage

Sabrinath partners with customers and various OCI teams to build service capabilities that help customers succeed in their cloud journey. 


Previous Post

First Principles: Robust data breach protection with Zero Trust Packet Routing

Pradeep Vincent | 13 min read

Next Post


Zero Data Loss Autonomous Recovery Service is now available for Oracle Database@Azure

Kelly Smith | 4 min read