Inside the Sun Oracle Database machine : The F20 PCie cards
By mrbenchmark on Dec 09, 2009
We are living a drastic change, something close to a revolution with the new Sun Oracle Database machine. Why ? With the critical use of enterprise flash memory, this architecture is not anymore reserved to data warehouses but very well suited to Online Transaction Processing. We are preparing benchmark results on this platform and actively shipping systems to customers. In the meantime, and in a suite of short entries, I will describe the key innovations at the heart of this environment.
Let's start with the Sun Flash Accelerator F20 PCIe Cards.
Each Exadata cell (or Sun Oracle Exadata Storage Server) include 384 GB total of flash storage producing up to 75,000 IOPS [8k block]. This capacity is obtained via four 96GB F20 PCIe cards detailed below. A full rack configuration comes equipped with 5.3TB of Flash storage and can produce an amazing 1 million IOPS [8k block]. This huge cache is not only used smartly and automatically by the Database machine but can also be user-managed via the ALTER TABLE STORAGE command and the CELL_FLASH_CACHE argument.
Here is the detailed architecture of the F20 PCIe card :
As you can see, we obtain the total 96GB capacity via four Disk on Module (DOM). Each DOM contains four 4GB SLC NAND component on the front side and four on the back side. It gives us a 32GB capacity of which 24GB is addressable. To even accelerate flash performance, 64MB of DDR-400 DRAM per DOM provide a local buffer cache.
Finally, the DOM need to manage all of its components, track faulty blocks, handle load balancing and communicate with the outside world using standard SATA protocols. This is achieved with a Marvell SATA2 Flash Memory Controller.
Outside of the four DOMS, a Supercapacitor module provides enough backup power to flush data from DRAM to non-volatile flash devices, therefore maintaining data integrity during a power outage. With a 5 years lifespan in a well-cooled chassis, this modules are superior to classic batteries.Finally, a LSI eight-port SAS controller connect the four DOMS to a 12-port SAS expander for external connectivity.
We measured in Sun labs a 16.5W power consumption per F20 card. We were able to produce 1GB/s in sequential read (1MB I/O) and 100,110 IOPS (4k random read) for each card. In addition, we can replace about two hundred 15k HDDs latest generation for a power consumption of 0.165 milliwatts per 4K I/O and an estimated MTBF of 227 years. Amazing !