Short Q&A for Oracle Exalogic Elastic Cloud X2-2
By uwes on Mar 16, 2012
Why should a customer consider buying an Exalogic system rather than assemble one themselves using commodity off-the-shelf components (COTS)?
- Exalogic is pre-integrated and tested to provide fastest time to market and superior overall experience - we build the whole system, test it and ensure that it is functioning correctly before our installation engineer leaves the site
- Because Oracle develops the Exalogic system and all of it's major components, we are able to test the Exalogic system to discover component defects or limitations which impair overall system performance, manageability or stability and then execute programs to enhance individual components in ways that make the Exalogic system better - when you consider the case of an Oracle Application workload, there is clearly no other vendor that can make a long term commitment to ensuring that the entire application-to-disk deployment will be increasingly efficient, reliable, easier to manage and lower cost to own over time.
- Only Oracle can support Oracle's applications in a fully virtualized, mission-critical production deployment, and the only platform for such a deployment is Exalogic.
- Complete monitoring and management stack for the system infrastructure (IB Gateway, compute nodes, storage, OS, OVM), which includes system-level command line diagnostic and configuration utilities
- Oracle Linux, OVM, all 7320 Storage Device features (including replication and snapshots), HW/VM/IB Fabric management (deployment automation), Exabus APIs (and the underlying InfiniBand stack including EoIB support) software license and support is included in the Exalogic Elastic Cloud Software license - a comparable suite of software and support from any combination of component vendors is both very costly and very complex to deploy and support
- The Exalogic system is the same system used by our support organization, greatly improving our ability to diagnose and correct issues - our TTR is much shorter on Exalogic than for the same Oracle software products on COTS platforms, largely because the platform is both widely used (Oracle alone has more than 75 Exalogic configurations in production across the company) and is essentially identical in all deployments - every new deployment improves the product quality by exposing it to additional workloads and environments, which feeds back into our support capability and engineering program
- The number and severity of defects in Exalogic systems is greatly reduced (versus a COTS multi-vendor system) by the preemptive distribution of fully tested full-stack patches (which include device firmware, device drivers, OS patches, and software updates for OVM, OTD, OpsCenter, Exalogic Deployer) and pre-tested non disruptive patching procedures
What is included in an Exalogic system?
- Exalogic is an engineered system comprised of Hardware and Software.
- Exalogic hardware and software are separate items on the Exalogic Price List, but they are closely coupled and are of little value separately. As with the Exadata Storage Server, the Exalogic Elastic Cloud Software is installed on the Exalogic Elastic Cloud Hardware at the factory.
- The Oracle Exalogic Elastic Cloud hardware is available in three basic configurations: Eighth, Quarter, Half and Full rack. Each configuration contains x86 servers with flash SSDs, InfiniBand switches and gateways that comprise I/O fabric and an integrated disk storage subsystem.
- Each Exalogic hardware configuration is pre-installed with major components of the Exalogic Elastic Cloud Software called the 'Base Image' at the factory. The Exalogic Elastic Cloud Software is the result of thousands of hours of testing, tuning, optimization and hardening, which are the source for Exalogic's greatly improved overall performance, stability and manageability. It is extremely unlikely that any customer (or any competitor) could replicate the Exalogic Elastic Cloud Software, even if they were willing and able to invest the thousands of hours in testing and development that Oracle has since mid 2009.
- The Exalogic Elastic Cloud Software is the only OS platform on which the Exalogic optimizations in the latest releases of HotSpot, JRockit, WebLogic Server and Oracle Coherence are supported. All of the Exalogic performance optimizations made to those upper-stack products rely on Ethernet-over-InfiniBand (EoIB) and Sockets Direct protocol are enabled through the Exalogic Networking Stack. In practice, this means that they are logically extensions of the Exalogic platform as well, although obviously those features cannot be used unless the customer has licensed the appropriate products that contain them (i.e. WebLogic Suite, Coherence, etc.).
How does the Exalogic hardware provide 99.9999% (six-nines) high-availability?
Exalogic is designed to be extremely reliable and tolerant of hardware component failures and applies a no-single-point-of-failure redundancy strategy. Any given Exalogic rack configuration will remain continuously available throughout it's life in any given production deployment, uninterrupted by either individual component failures or regular servicing of the system.
Every Exalogic X2-2 configuration is fully redundant and provides automated fault detection and fail-over using techniques which are completely independent of any external software in the following ways:
- Exalogic Power Distribution Units (PDU) in each Exalogic rack configuration are redundant. For complete redundancy, each PDU should be wired to a different AC source.
- Exalogic has 2 (two) independent power supplies in each component (InfiniBand switch/gateway, compute node and storage head) actively balance power - if one fails, or is connected to a PDU that fails, the other takes over (2N redundancy, continuous availability)
- Exalogic has excess fan capacity in each component (InfiniBand switch/gateway, compute node and storage head) and if a fan dies the temperature sensors will up the RPM on remaining fans to maintain safe operating temperature (N+1 redundancy, continuous availability)
- Exalogic compute nodes use on-board enterprise-grade SSDs in a RAID1 configuration (continuous availability). These SSDs are hot-swappable.
- The InfiniBand ports in each HCA are bonded and by default have a link failure detection in the single-digit milliseconds or less. (fast fail-over)
- Each InfiniBand port on each HCA is connected to a different physical InfiniBand switch, and each port is capable of handling more I/O traffic than a single compute node can generate or receive, making it possible for an Exalogic system to operate at full capacity with up to ½ (half) of the InfiniBand switches disabled.
- Each InfiniBand gateway provides redundant up to 8 (eight) physical 10GbE connections to the data center service network, allowing each gateway to connect to multiple (redundant) external modular switches. Each Exalogic compute node is configured (by default) with a bonded EoIB interface that is associated with minimum two 10G ports on separate InfiniBand Gateways.
- Exalogic compute nodes talk to the storage heads using NFS and the fail-over delay is governed by the NFS client configuration and can take 30 seconds to a minute. Following fail-over, the read performance of the storage device will be impaired while the read cache is rebuilt. (fail-over)
- The disks in the Exalogic storage array are each separately cabled and are connected to the storage heads in a ZFS cluster. It is possible to configure the storage subsystem for multiple levels of redundancy, including striping and mirroring. The hard disks in the storage array are hot-swappable.
- The Exalogic storage heads support block-level storage replication, which is the foundation of Exalogic's Disaster Recovery capability. It is possible to use up to 2 (two) GbE ports on each storage head for redundant direct connection to the datacenter network that will be used to connect the primary Exalogic site to the backup Exalogic site. It is also possible to bond these ports for the purposes of automated fault detection and fail-over.
- The GbE management network is not redundant because the failure of that network will not cause the system to cease serving end-user traffic. The only exception would be in a case where the customer has configured the system to access external NDIS, DNS or similar resources exclusively over the management network. We recommend having those external resources available on the Data Center service network or deployed on one of the compute nodes directly attached to the InfiniBand fabric.
The likelihood of the failure of any single component is so low that it is extremely unlikely that it would not be possible to repair/replace/restart a failed component before it's backup/replacement also failed. This means that the reliability of the Exalogic X2-2 configuration as a whole is extremely high, and system downtime is unlikely to ever exceed the time required for applications to fail-over to the secondary storage head in the system. Average cumulative downtime of an Exalogic configuration, during any one-year window, is very likely to be less than 5 minutes, even in the case of multiple sequential component failures.
It is important to note that all software executing on a given compute node will be unavailable if that compute node fails or is taken out of service. Exalogic does not provide any mechanism for continuous availability of applications that do not perform their own state replication or clustering. All sessions on a given application server instance, for example, will be lost if the compute node hosting that instance fails or is taken out of service unless the application server is deployed in a fault-tolerant configuration.
What is the Exalogic Elastic Cloud Software?
The Oracle Exalogic Elastic Cloud Software is the unique set of software components, tools and documentation required to make the Exalogic Elastic Cloud Hardware functional and usable as a platform for Oracle's Fusion Middleware and business applications. The Exalogic Elastic Cloud Software consists of a number of components, many of which are pre-integrated with the specific Oracle Solaris and Oracle Linux operating system images and device firmware installed on the Exalogic Elastic Cloud Hardware at the time of manufacture. There is no practical means of using the Exalogic Elastic Cloud Hardware that does not require use of the Exalogic Elastic Cloud Software, nor are there any supported approaches to "hard partitioning" that would allow customers to avoid licensing all physical processors for a given Exalogic Elastic Cloud X2-2 Hardware compute node (server) that is powered on and in use.
The principal components of the Exalogic Elastic Cloud Software are as follows:
- Exabus: an assembly of special InfiniBand gateway hardware, device drivers, device firmware, software libraries and configuration files that allow other software ("applications" to make use of the Exalogic Elastic Cloud hardware and ensure the optimal performance and reliability of the system. The Exabus firmware and software extends and integrates Oracle Linux, Oracle Solaris, the OpenFabrics Enterprise Distribution (OFED) with the unique hardware design of the InfiniBand gateways and switches in the so-called "I/O backplane" of the Exalogic system. This software is installed on the Exalogic Elastic Cloud Hardware at the time of manufacture.
- Exalogic Configuration Utility: A desktop tool used to configure the Exalogic system management and data center service network interfaces and internal subnets.
- Exalogic Distributed Command Line Interface: A command-line tool that allows commands to be executed on some or all of the Exalogic nodes simultaneously, at the discretion of the operator. This software is installed on the Exalogic Elastic Cloud Hardware at the time of manufacture.
- Exalogic Topology Verifier: This command-line tool automatically verifies the InfiniBand topology of the Exalogic system, ensuring that the correct topology is applied for each given system configuration: Quarter Rack, Half Rack or Full Rack. This software is installed on the Exalogic Elastic Cloud Hardware at the time of manufacture.
- Exalogic InfiniCheck: This tool verifies the correct operation of every InfiniBand device and port on the fabric, ensuring that all ports and connectors are functioning correctly. This software is installed on the Exalogic Elastic Cloud Hardware at the time of manufacture.
- Exalogic Hardware & Firmware Profiler: This tool automatically verifies that the all of the hardware devices and firmware versions connected to the Exalogic system fabric are verified and supported, with the correct and compatible device firmware versions. This software is installed on the Exalogic Elastic Cloud Hardware at the time of manufacture.
- Exalogic Software Profiler: This tool verifies that all of the Linux or Solaris software packages installed on any of the system's compute nodes are of the correct version and do not jeopardize the Exalogic system's performance, security or stability. This software is installed on the Exalogic Elastic Cloud Hardware at the time of manufacture.
- Exalogic Boot Manager: This tool allows system operators to easily re-image individual Exalogic compute nodes, via external PXE servers or network-mounted disk images. This software is installed on the Exalogic Elastic Cloud Hardware at the time of manufacture.
- Exalogic Elastic Cloud Software options for WebLogic Suite, coherence and Tuxedo: This is a set of features implemented within the other Fusion Middleware products that are technically dependent on the underlying Exalogic Elastic Cloud Hardware and Software.