Exascale is the next generation of storage management that combines multitenancy, resource pooling, and elastic database infrastructure with the performance of data-intelligent Exadata storage and Remote Direct Memory Access (RDMA). This article series aims to get you started quickly by introducing some Exascale fundamentals. This article shows through demonstration how Exadata storage is managed by Exascale—starting from the physical disks, all the way to the database files.
Starting at the lowest layer is the storage media, which describes the physical drives in the storage servers. The physical drives are partitioned and formatted as grid disks or pool disks. Storage pools are a collection of pool disks that are the same storage server type—i.e., high capacity or extreme flash. Vaults are a metadata construct and are a collection of rules and quotas for how the storage pools, flash cache, RDMA memory, and flash log will be used by the database or virtual machine. Templates determine the redundancy and storage media type of each type of database file, as well as the performance profile used to write the blocks to disk.
Storage Media
Exascale storage is provided by Exadata storage servers, and each storage device is categorized according to its media type. The supported media types are:
- High Capacity (HC) – Identifies high capacity storage media, which uses hard disk drives (HDD)
- Extreme Flash (EF) – Identifies extreme flash storage media, which uses capacity-optimized flash devices
Creating a storage server cell disk reserves a physical disk for use by Exadata System Software. A cell disk has a one-to-one relationship with a physical disk. Creating a grid disk or pool disk reserves part or all of a cell disk to be used by ASM or Exascale. Grid disks can be added to an ASM disk group. Pool disks can be added to an Exascale storage pool.
Pool Disks
Exascale uses pool disks and storage pools to organize the physical storage provided by Exadata storage servers. A pool disk is a storage server disk allocation that is reserved for use by Exascale. The lsgriddisk command in escli displays all Exadata grid disks used by ASM, as well as pool disks used by Exascale, that have been created and are accessible to the Exascale cluster. In this example, the pool disks are labeled with a prefix of pool. There are also grid disks labeled with a prefix of DATA and RECO, which indicates that this storage server is also being utilized in ASM disk groups.
@>lsgriddisk size diskType cellDisk status name 3298534883328 HardDisk CD_00_cell01 active DATAC1_CD_00_cell01 3298534883328 HardDisk CD_01_cell01 active DATAC1_CD_01_cell01 ... 2199023255552 HardDisk CD_00_cell01 active RECOC1_CD_00_cell01 2199023255552 HardDisk CD_01_cell01 active RECOC1_CD_01_cell01 ... 14300596928512 HardDisk CD_00_cell01 active pool1_CD_00_cell01 14300596928512 HardDisk CD_01_cell01 active pool1_CD_01_cell01 ...
If you want to only see the pool disks belonging to Exascale storage pools, you can add a filter to the query. ASM grid disks do not have a value in the storagePool attribute.
@>lsgriddisk --filter storagePool>"" size diskType cellDisk status name 14300596928512 HardDisk CD_00_cell01 active pool1_CD_00_cell01 14300596928512 HardDisk CD_01_cell01 active pool1_CD_01_cell01 ...
Viewing the details of a pool disk with the –detail parameter, shows additional information common to both pool and grid disks.
@>lsgriddisk --detail --filter name="pool1_CD_00_cell01" id pool1_CD_00_cell01@cell01 name pool1_CD_00_cell01 cachedBy FD_02_cell01 cachingPolicy default cell cell01 cellDisk CD_00_cell01 creationTime 1970-01-01T00:00:00-00:00 diskType HardDisk poolDiskStatus online size 14300596928512 spaceUsed 42147774464 status active storagePool pool1 uuid e09863fc-279a-48c3-924f-5a1a7eeeaf00
The lspooldisk command in escli shows similar information to the lsgriddisk command, but only shows the storage partitions allocated as pool disks.
@>lspooldisk --detail --count 1 id 148 name cell01_.28_5042_8/pool1_CD_00_cell01 cellId 73 gdName pool1_CD_00_cell01 healthy true incarnation 13 size 13.006317138671875T state Uninitialized status ONLINE storagePool pool1
Storage pools
A storage pool is a collection of pool disks that provides persistent physical storage for Exascale vaults and files. Each storage pool is a collection of pool disks using only one type of storage media—i.e., one of High Capacity (HC) or Extreme Flash (EF). You cannot define a storage pool with pool disks having a mixture of storage media types.
An Exascale cluster has one or more storage pools. A storage pool can contain pool disks that reside on different generations of Exadata storage server hardware, which allows for easy expansion to new generations of Exadata hardware.
The lsstoragepool command in escli shows a significant amount of detail about the storage pools. In this example, there is only one storage pool in the Exascale cluster called pool1. The cellsCount and disksCount attributes show that there are five different storage servers with a total among them of 60 grid disks allocated to this storage pool. The mediaType attribute shows that this storage pool is comprised of High Capacity (HC) storage disks.
@>lsstoragepool --detail id pool1 name pool1 cellOfflineTimerInMins 1440 cellsCount 5 diskOfflineTimerInMins 120 disksCount 60 flashCacheProvisionableSize 86.94542255174383171834051609039306640625T flashCacheProvisionedSize 0 guid 4081be6e-e54f134d-dd230bf-d6722bd5 highDataStatus available highRecoStatus available inReconfig false iopsProvisionable 3520560 iopsProvisioned 0 iopsRaw 3520560 mediaType HC normalDataStatus available normalRecoStatus available spaceProvisionable 780.3790283203125T spaceProvisioned 50T spaceRaw 780.3790283203125T spaceUsed 2.572264492511749267578125T sysDataStorePresent true sysDataStoreShardsCount 20 xrmemCacheProvisionableSize 4.4261037441538064740598201751708984375T xrmemCacheProvisionedSize 0
Files
Exascale is optimized to directly store and manage files associated with Oracle Database and Oracle Grid Infrastructure. Exascale automatically understands the type of files that it is managing. By using a template that is associated with the file type, Exascale automatically applies the appropriate file storage attributes. This process indirectly determines the storage pool that houses the file. Unlike ASM, Exascale does not need separate storage entities like disk groups for files needing different redundancy, performance, or disk placement characteristics. For example, a datafile may be assigned the characteristics of high redundancy, placement on high capacity disk storage, and extents located on separate physical drives from online log files—whereas an online log file may be assigned the characteristics of high redundancy, placement on capacity-optimized flash storage, and extents located on separate physical drives from datafiles. Both of these files can be created in the same logical storage container (vault), and Exascale will handle where and how the extents are written to and read from storage.
Vaults
An Exascale vault is a logical container that groups and limits the storage resources provided by the storage servers. To an administrator and Oracle Database, a vault appears like a top-level directory that contains files. A fully qualified Exascale file path always begins with the ampersand (@) character and the vault name, followed by the rest of the file path (e.g., @MYVAULT/myexample/myfilename).
A vault (and all the databases/VMs with access to it) will be able to use all the storage resources in all of the storage pools in the cluster, by default. For example, the cluster in the following example has both an HC storage pool and an EF storage pool. When I create the MYVAULT_ALL vault without any attributes, it has access to all the capacity and performance resources of both storage pools.
@>mkvault MYVAULT_ALL Vault @MYVAULT_ALL created. @>lsvault --detail MYVAULT_ALL id MYVAULT_ALL flashCacheProv unlimited flashLogProv true iopsProvEF unlimited iopsProvHC unlimited spaceProvEF unlimited spaceProvHC unlimited xrmemCacheProvEF unlimited xrmemCacheProvHC unlimited
The escli command mkvault MYVAULT_HC_UNLIMITED –attributes spaceProvHC=unlimited restricts the type and amount of storage resources accessible to the vault by adding the spaceProvHC attribute. Vault attributes are explicit for each storage pool, meaning that once you add an attribute to the mkvault command, the vault is restricted to only those storage pool resources. The MYVAULT_HC_UNLIMITED vault has unlimited access to the capacity and performance resources of only the HC storage pool by adding the spaceProvHC=unlimited attribute. Alternatively, the command mkvault MYVAULT_HC_UNLIMITED –attributes spaceProvEF=0,iopsProvEF=0 would achieve the same result.
@>mkvault MYVAULT_HC_UNLIMITED --attributes spaceProvHC=unlimited Vault @MYVAULT_HC_UNLIMITED created. @>lsvault --detail MYVAULT_HC_UNLIMITED id MYVAULT_HC_UNLIMITED flashCacheProv unlimited flashLogProv true iopsProvEF 0 iopsProvHC unlimited spaceProvEF 0 spaceProvHC unlimited xrmemCacheProvEF 0 xrmemCacheProvHC unlimited
Adding additional attributes will further restrict the storage resources available to the vault.
@>mkvault MYVAULT_HC_LIMITED --attributes spaceProvHC=1T,iopsProvHC=1000000,flashCacheProv=500G,xrmemCacheProvHC=500G Vault @MYVAULT_HC_LIMITED created. @>lsvault --detail MYVAULT_HC_LIMITED id MYVAULT_HC_LIMITED flashCacheProv 500G flashLogProv true iopsProvEF 0 iopsProvHC 1000000 spaceProvEF 0 spaceProvHC 1T xrmemCacheProvEF 0 xrmemCacheProvHC 500G
Templates
An Exascale template is a named collection of file storage attribute settings. When a file is created in Exascale, the template associated with that file type determines how the file is stored and managed.
Templates are defined at two levels:
- Cluster Templates are defined at the cluster level. A cluster template is used if no vault-level template exists to override it.
- Vault Templates are defined at the vault level. A vault template defines vault-specific attributes that override the corresponding cluster template.
The templates define redundancy, media type, and content type. Here is a list of some of the file type templates that should look familiar.
@>lstemplate --cluster redundancy mediaType contentType name high HC RECO ARCHIVELOG high HC DATA DATAFILE high HC DATA ONLINELOG high HC DATA PARAMETERFILE high HC DATA PASSWORD ...
When a vault is created, it will use the cluster templates, but each template can be overridden at the vault level by creating a vault template. For example, I can override the DATAFILE template for the MYVAULT_HC_AND_EF_UNLIMITED vault so that the datafiles are stored on EF storage.
@>mktemplate --vault MYVAULT_HC_AND_EF_UNLIMITED --file-type DATAFILE --media-type EF --redundancy high --content-type DATA Template created: MYVAULT_HC_AND_EF_UNLIMITED/DATAFILE
I can see that there is one vault-level DATAFILE template overriding the cluster level template.
@>lstemplate --vault-level-only --vault MYVAULT_HC_AND_EF_UNLIMITED redundancy mediaType contentType name high EF DATA DATAFILE
When a tablespace is created in the MYVAULT_HC_UNLIMITED vault, its datafiles are stored according to the cluster-level DATAFILE template, which dictates that the datafiles are stored in the HC storage pool. When a tablespace is created in the MYVAULT_HC_AND_EF_UNLIMITED vault, its datafiles are stored according to the vault-level DATAFILE template, which dictates that the datafiles are stored in the EF storage pool.
@>ls @MYVAULT_HC_UNLIMITED/ --attributes id,contentType,fileType,mediaType id contentType fileType mediaType @MYVAULT_HC_UNLIMITED/.../DATAFILE/MYTBSHC.OMF.776DDAB1 DATA DATAFILE HC @MYVAULT_HC_UNLIMITED/.../DATAFILE/MYTBSHC.OMF.074E62A8 DATA DATAFILE HC @MYVAULT_HC_UNLIMITED/.../DATAFILE/MYTBSHC.OMF.0331B5F7 DATA DATAFILE HC @>ls @MYVAULT_HC_AND_EF_UNLIMITED/ --attributes id,contentType,fileType,mediaType id contentType fileType mediaType @MYVAULT_HC_AND_EF_UNLIMITED/.../DATAFILE/MYTBSEF.OMF.6B04ABFE DATA DATAFILE EF @MYVAULT_HC_AND_EF_UNLIMITED/.../DATAFILE/MYTBSEF.OMF.3A3B1E63 DATA DATAFILE EF @MYVAULT_HC_AND_EF_UNLIMITED/.../DATAFILE/MYTBSEF.OMF.0D42E7F8 DATA DATAFILE EF
Next Steps
Exadata Exascale introduces a powerful new way to approach database performance and scalability, much like the game-changing innovations that came before it—such as container databases (CDBs), Smart Scan, Real Application Clusters (RAC), and Exadata itself. Exascale brings a wealth of new functionality and opens up exciting possibilities for optimization and efficiency. This series of Exascale articles is designed to make your journey smoother, helping you quickly harness the full potential of Exascale technology.
Exadata Exascale Series
- Exascale User Interfaces and Wallets
-
Exascale Storage Fundamentals