Relational database systems form the core of a company’s Information Technology (IT) business infrastructure. Application downtime and data unavailability translates into lost productivity and revenue, dissatisfied customers, and damage to corporate reputation. A basic building block to building a High Availability (HA) infrastructure is deploying redundant hardware and software resources. Nowhere is this as important than in the realm of storage. Ensuring data reliability and availability is paramount to an enterprise’s RDBMS infrastructure and application environment. To this end, Oracle Maximum Availability Architecture (Oracle MAA) prescribes HA best practices for a complete range of availability and data protection requirements, spanning all company sizes and lines of business. For greater discussion of Oracle MAA please review other blogs here or visit MAA material on Oracle Technology Network.
Oracle MAA prescribes Oracle Automatic Storage Management (ASM) as an critical component providing a clustered volume manager and file system for the Oracle database environment. Through its low cost, ease of administration and high performance, ASM is the storage technology of choice for Oracle databases. An important capability of ASM is its data protection features inherent in ASM’s ability for mirroring and rebalancing data distributed on individual disks making up an ASM disk group. This blog describes the details of how ASM mirrors data and important architectural details associated with mirroring.
ASM mirrors data as a means for protecting against data loss as a result of disk or other storage component failure. Unlike functionality found in a storage array, ASM only mirrors blocks of data in existing files. A storage array implementing RAID1 functionality mirrors all physical blocks, regardless of whether the blocks are allocated in a file or simply unallocated. This reduces mirroring overhead and improves efficiency. ASM mirroring has a few different ways it mirrors referred to as redundancy type. The redundancy type determines how many disk or component failures can be tolerated without forcibly dismounting a disk group or otherwise losing access to data. The redundancy types are:
When a customer chooses external redundancy for a disk group, this tells ASM not to protect data through mirroring and it is expected that underlying storage hardware provides all necessary redundancy and resiliency against component failures. In a disk group using external redundancy, if a disk becomes unavailable then the disk group is taken offline until the failing ASM disk is made available and brought online.
Choosing normal redundancy for a disk group tells ASM to provide two-way mirroring for data in the disk group. Normal redundancy disk groups require at least two failure groups, however three failure groups are preferable.
Choosing high redundancy for a disk group tells ASM is to provide three-way mirroring for data in the specified disk group. High redundancy disk groups require at least three failure groups, however four failure groups are preferable.
Two new disk group redundancy types introduced in Oracle Database 12c Release 2 and 18c are flex redundancy and extended redundancy disk groups. From a mirroring perspective, these disk group types are a hybrid of the above disk group types in that they enable a data redundancy to be dynamically changed at the file level as required. Please review the ASM Administration guide for other features of these redundancy types.
High-end storage arrays generally provide hardware RAID protection. Minimally, Oracle MAA recommends using ASM mirroring when not using robust hardware RAID. You can also use ASM mirroring in configurations when mirroring between sites is desired (RAC on Extended Distance Clusters). Even with hardware RAID, ASM mirroring is useful in situations where you want to mirror data across separate physical storage arrays. In such cases, data in a disk group remain available even when one storage array is offline.
One significant advantage of ASM mirroring as compared to data protection through storage array RAID, is that ASM understands the internal data structure in an Oracle database. ASM is able to detect data corruption that a storage array often can not detect. When there is data corruption, and if ASM is mirrored the data, then ASM will automatically correct the corrupted data and ensure the integrity of the database.
For every extent in a file that is mirrored, there is a primary copy and one or more secondary mirror copies of the extent. The combination of a primary and secondary extents is called an extent set. ASM chooses a disk for storing the secondary extent copy that is in a different failure group from the primary copy. Failure group definitions provide the means for specifying placement of mirrored extent copies so that each extent copy does not share a common component that could fail.
Failure groups are determined by the customer and define the hardware that they wish to protect against failure. For example, a disk group can be created where each individual disk is defined as an individual failure group, one failure group for each ASM disk. This way ASM mirroring protects against the loss of an individual disk. In Exadata, storage cells containing multiple disks, are typically defined as failure groups. This means file extents are mirrored in different storage cells and data is protected against the loss of an entire storage cell.
The default failure group definition is at the individual disk shown in the picture below.
Normal redundancy disk group with three failure groups
In the above illustration, the shaded boxes represent three different file extents mirrored on different disks. Each disk is its own failure group. Data availability is maintained with the loss of a disk as shown below.
Failure Group can fail with no data loss
Space allocation problems can arise with a disk group having failure groups of uneven capacity. The problem presents itself when new file allocation is unable to succeed even though the space on all the disks is not fully utilized. The situation is that at least one or more failure groups are at capacity and the new allocation can find free space. For this reason Oracle Database Release 12c require disks in a disk group to be of the same size and individual failure groups to have the same number of disks.
The primary extents for a file are spread evenly across all the disks in the disk group without considering failure groups. Once a primary extent is allocated on a disk, secondary extents are allocated on disks belonging in different failure groups. Every copy of a secondary extent is in a different failure group than the primary extent copy. Additionally, multiple secondary extent copies of an extent set can not reside in the same failure group.
When a data block is written to a file, the write operation goes to all extents in that block's extent set. The write operation is not acknowledged until all the individual writes complete. When a block is read from a file, the primary extent is typically read unless it is unavailable. This ensures reads are evenly distributed across all available disks no matter how they are placed into failure groups. However, preferred reads and special considerations associated with engineered systems such as Exadata can alter this logic to the benefit of the customer.
Choosing the best failure group definition depends on the kinds of failures needing to be tolerated. For small numbers of disks (less than 20) it is usually best to use the default failure group creation that puts every disk in its own failure group. This is also true for large numbers of disks when the main concern is individual disk failure. If there is a need to protect against the simultaneous loss of multiple disk drives because of a single component failure, then failure group specification at a higher component level should be considered. For example, a disk group may be constructed of disks from several small modular disk arrays. If the disk group needs to be continually available when an entire modular array fails, then a failure group should consist of all the disks in one array module. If one array module fails, data will continue to be available and after a period of time all the data on that array module will be relocated to other array modules in order to restore redundancy. As a general guideline, disks should be placed in the same failure group if they depend on a common component whose failure needs to be tolerated with no loss of availability.
Once a disk has been assigned to a failure group it cannot be reassigned to another failure group. If a disk needs to be moved to another failure group then the disk can be dropped from the disk group and then added back, but in a different failure group. A failure group is always a subset of the disks within a single disk group. However there could be disks in different disk groups that share the same hardware. It would be reasonable to use the same failure group name for these disks even though they are in different disk groups. This would give the impression of being in the same failure group even though that is not strictly the case.
In summary, Oracle ASM, as an important element of Oracle MAA, provides the means to protect against the loss of data and ensure data availability through mirroring data in different failure groups. The customer chooses the failure groups by determining what components they want to protect against a failure. In situations where data protection is provided by highly reliable and robust storage array hardware, then external redundancy disk group may be use, but ASM does not protect the storage infrastructure from component failures.