Exadata Exascale (Exascale) is Exadata’s new revolutionary software architecture. Whether deployed on-premises or in the cloud, Exascale combines the best of Exadata—extreme performance, high availability, scalability, security, and efficiency—with cloud characteristics—multitenancy, resource pooling, and elasticity. Exascale reimagines how compute and storage resources are managed on Exadata platforms by decoupling and simplifying storage management and adding many innovative new capabilities. It ensures industry-leading database performance, availability, and security standards that organizations expect from Exadata.

Exascale is available as a multitenant cloud service, Exadata Database Service on Exascale Infrastructure, enabling organizations of all sizes to gain the benefits of Exadata in the cloud, starting with just a few cores and a few hundred GB/s storage capacity, scaling elastically, and paying only for what you use! For on-premises customers, Exadata Cloud@Customer, and very soon, Exadata Database on Dedicated Infrastructure customers in the cloud, Exascale can be easily deployed into existing environments. Exascale’s multiple deployment offerings enable all our Exadata customers to take advantage of the many amazing features Exascale offers, including the subject of this post—leveraging the full potential of Exascale Volumes, Advanced Cluster File System (ACFS), and Exascale’s advanced snapshot and clone capabilities.

Exascale Files and Volumes

The smallest unit of storage in Exascale that a customer will generally see is a file. A file is a segment of storage in an Exascale vault that contains data with a defined set of redundancy and other characteristics. An Exascale vault is a defined policy that segregates storage of an Exascale cluster, is securely independent of other vaults, and serves as a logical container for groups of files.

Files in Exascale can be accessed in two ways:

  • Similar to ASM files, by the database and other Exascale vault-aware applications.
  • Similar to ASM Dynamic Volume Manager (ADVM), through a block volume device known as Exascale Direct Volumes.

Exascale Direct Volume (EDV) is a high-speed method for presenting Exascale storage as a block volume device in the Linux OS. Specifically, an EDV is stored as a file in Exascale and attached as a block device to a server using an RDMA-enabled protocol. This allows applications that are not natively Exascale-aware to utilize the features and performance of Exascale storage. Since EDVs are built on top of Exascale, they retain all the best-in-class characteristics, including the security, speed, and features of Exascale Files.

One of Exascale’s ground-breaking features is its reimagined approach to snapshots and clones on Exadata. It enables space-efficient thin clones from any EDV, significantly boosting developer productivity. Exascale seamlessly integrates with development, test, and deployment pipeline requirements while providing native Exadata performance.

In addition to EDV, Oracle provides ACFS, a file system developed to utilize Exascale’s advanced features. ACFS is a best-in-class component of the Exascale stack, providing the same high availability, performance, and scalability as Exascale itself. With ACFS and EDV, every application can utilize Exascale’s new features, including snapshots and clones.

Diagram showing the relationship between Exascale Storage Pools, Vaults, Volumes, and ACFS

Administrators can create Exascale volume snapshots or clones from an existing volume or another snapshot or clone of a volume as a source. This capability opens the door to many use cases for these powerful capabilities. All Exascale clones and snapshots are thinly provisioned, meaning they do not consume additional space except when the clone has been changed from the original data.

In addition, I/O is not impacted during the snapshot or clone operation, with performance maintained for all ongoing usage of the Exascale Volume.

Let’s define what snapshots and clones are in Exascale:

  • Clone – A clone is a thinly-provisioned point-in-time copy of a file that is readable and writeable. Clones use redirect-on-write to store new data in the clone; there is no duplication of the original (unchanged) data.
  • Snapshot – A snapshot is a thinly-provisioned read-only copy of a file.

While volume snapshots work at the volume layer and are managed by Exascale tools, ACFS enables additional types of snapshots at the file system layer that are managed by ACFS tools. ACFS snapshots and clones use redirect-on-write technology (known as copy-on-write in ACFS) and may be read-only or read-write.

  • ACFS
    • File snapshot (fshare) – creates a new reference to a single file, similar to the Linux reflink concept.
    • File system snapshot – creates a new reference to an entire file system.
  • Exascale Volumes:
    • Volume snapshot/clone – creates a new reference to an entire Exascale volume.

Visibility

ACFS snapshots and clones are only visible in the file system in which they are created. Administrators create ACFS snapshots using the ‘acfsutil’ command. If the snapshot is created in file system /u01, the snapshot will only be visible in /u01. You can see the snapshot from anywhere that the volume containing /u01 is mounted. The snapshot is an inherent characteristic of the file system. It cannot be separated from the file system, be used in a separate file system, or be used on another ACFS cluster without moving the entire volume.

Exascale volume snapshots and clones are visible from anywhere that the Exascale vault in which they reside is visible. For example, if a snapshot is taken of the volume containing the /u01 file system, a clone of the snapshot can be independently attached to another cluster or node as an EDV device.

Creation

ACFS snapshots are managed with the ‘acfsutil’ command. If the Exascale volume is mounted, ‘acfsutil’ is also used to manage the volume snapshots.

  • ‘acfsutil snap’ – File system snapshots
  • ‘acfsutil fshare’ – File snapshots (fshares)
  • ‘acfsutil volsnap’ – Volume snapshots

The Exascale command-line interface utility (escli) manages volume snapshots and clones when the volume is not mounted.

  • ‘escli mkvolumesnapshot’ – Create a volume snapshot
    • Note—this command will fail if the volume has a mounted ACFS file system. In this case, ‘acfsutil’ must be used.
  • ‘escli rmvolumesnapshot’ – Remove a volume snapshot

In Oracle Cloud, ‘escli’ may not be available as a command line utility; however, its functions are available via the OCI Cloud Console and related REST APIs.

The following table summarizes the differences between the ACFS and Exascale volume snapshots and clones.

 

Can be used by a different VM cluster

Results in a new Exascale volume

Read-Only

Read-Write

Thin Provisioned

Exascale Read-Only Volume Snapshot / acfsutil volsnap create -r

Y (can be attached as a new device)

Y

Y

N

Y

Exascale Writable Volume Snapshot / acfsutil volsnap create -w

Y (can be attached as a new device)

Y

N

Y

Y

ACFS fshare

N (single file in file system)

N

Y

Y

Y

ACFS Snapshot

N (contained in the file system)

N

Y

Y

Y

Exascale integration with ACFS

Exascale volume snapshots and clones are created with the same attributes as the source volume. Therefore, when a snapshot or clone is created of a volume, the access to that snapshot or clone will default to being specific to that host.

To maintain the consistency of the ACFS file system when creating an volume snapshot, the ‘-f’ option of the acfsutil volsnap create command “freezes” the file system, which temporarily suspends I/O on the volume and prevents write-tearing, where a portion of the modification to a file is not fully committed to the volume when the snapshot takes place. Use the ‘acfsutil volsnap create -f’ command when creating snapshots of a mounted volume.

Use Cases

There are many use cases for snapshots and clones. The scope of the snapshot and any further operations that may be necessary help define the correct technology to use.

  • ACFS Snapshots
    • Oracle pluggable database (PDB) thin clones
      • ACFS snapshots are integrated into the Oracle Database PDB clone functionality and can be used to easily create and maintain PDBs. When a PDB is created via snapshot copy on an ACFS file system, the Oracle Database automatically utilizes ACFS snapshots.
    • Test and development databases
      • ACFS snapshots can easily create and maintain test and dev clones of entire databases, either container databases (CDB) or non-CDB. gdbClone is an Oracle-provided tool that automates cloning of entire databases on ACFS. ACFS’s ease of use means these can be quickly and efficiently created for developers.
    • Creating a consistent file system for backup
      • ACFS snapshots are consistent, meaning that in-flight writes are written to disk before the snapshot is taken. ACFS snapshots can be taken at a regular interval (snapshot carousel) to create a first-level backup with quick restore capabilities. They can also be used as a source for traditional backup tools.
    • Gold image creation
      • ACFS snapshots can be used to create a master copy of software that has been installed and configured on an ACFS file system.
  • ACFS fshares
    • Oracle Database thin clone
      • When stored on ACFS, Oracle Database automatically leverages ACFS fshares when creating thin clones of PDBs via snapshot carousel or snapshot copy. In Oracle Database 21c and later, ACFS fshares are the default for PDB thin clones.
    • VM image files
      • ACFS fshares are used to create a master copy of VM image files, which can be used to create a backup of the VM or as a source to create clone VMs. This separates the VM’s configuration and instantiation while preserving storage efficiency.
    • Efficient space usage
      • fshares, like Linux reflinks, can be used to quickly create space-efficient file copies within the same file system. This can be a tremendous advantage when working with large files like AI data sets. The ‘cp’ command on newer versions of Oracle Linux uses fshares by default for copying files on ACFS file systems.
  • Exascale Volume Snapshots
    • Gold Images for Other VM Clusters
      • Exascale volume snapshots can be attached to other clusters to configure storage on one cluster and use it on many other clusters. Simply create a clone of your master copy and adjust configuration values for the intended target cluster.
    • Efficient Data Access
      • Exascale volume snapshots enable rapid and efficient access to data on different clusters and nodes. The snapshot operation completes quickly in the background, because data does not have to be copied to a new volume, which can then be cloned and used on a new cluster.
    • VM Files
      • Using an Exascale Volume clone allows the VM to take advantage of Exascale Volumes’ native ability to move between cluster nodes while maintaining the VM’s space efficiency and providing for read-write configuration of an intended VM instantiation.
    • Test and Development
      • With the ability to modify the configuration of running software, quickly take and delete clones, and modify stored information, Exascale Volume clones can quickly and efficiently work with various configurations, software versions, updated configs, and other system-level operations.
    • Backup
      • Since Exascale Snapshots create read-only copies of the data, and this data is hosted in the Exascale Vault, they represent an efficient way to create a backup of data.  A simple snapshot attachment provides access if the data is needed again (such as restoring it).
    • Master Images
      • Since Exascale Snapshots are read-only, they are a great way to freeze changes to software or VM images when the configuration is complete.  To use these frozen images, administrators would create a clone to allow access and changes for a specific software instantiation.

As you can see, there are many ways in which ACFS and Exascale Volume snapshots and clones can benefit. In upcoming posts, we will dive deeper into the mechanics of ACFS and Exascale and many of the use cases above. Stay tuned! In the meantime, many valuable resources are already available that provide more detail on ACFS and Exascale. Check them out below.

Additional resources