OCI Intel Ice Lake compute provides the highest throughput and lowest cost for HPC file systems

Maximum IO throughput delivered by a file server depends on the available maximum network bandwidth (Gbps) on a file server and the IO throughput of the attached storage.

With Oracle Cloud Infrastructure (OCI), we offer optimized Compute shapes in both bare metal and virtual machine (VM) shapes: BM.Optimized3.36 and VM.Optimized3.Flex. Optimized shapes are designed for computing workloads that require high frequency processor cores or high-performance computing (HPC) workloads that require high performance with low latency or 100 Gbps RDMA (RoCEv2) cluster networking.

In this blog, I demonstrate how optimized Compute shapes are most ideal for file system workloads, including network and parallel file systems, such as Lustre, BeeGFS, IBM Spectrum Scale, and Quobyte, from a performance and cost perspective. This blog is for ethernet- and TCP-based file servers; we cover RDMA, RoCE, and InfiniBand file servers in another blog.

Specifications

The following tables show the specs for optimized Compute shapes. BM.Optimized3 uses and Intel processor with a base frequency of 3.0 GHz and a max turbo frequency 3.6 GHz. VM.Optimized3 uses an Intel processor with a base frequency of 3.0 GHz and a max turbo frequency of 3.6 GHz. The VM.Optimized3.Flex Compute series allows you to select the number of physical cores and memory size in GB independently to best suit your workload needs.

These shapes offer sufficient physical cores and memory to run any file server. Each Oracle CPU (OCPU) is equal to two vCPUs.

Performance

Based on these specifications, a BM.Optimized3.36 has two 50-Gbps network interface cards (NICs), so one 50 Gbps NIC can handle read/write traffic from file system client nodes to the file server and the other 50 Gbps NIC can manage read/write traffic to OCI’s network-attached block volume storage. 50 Gbps is equivalent to 6,250 MB/s.

To ensure that we fully utilize the network bandwidth for maximum performance, we need OCI Block Volumes, which can deliver 6,250 MB/s or more. To achieve maximum performance from your file server, plan for aggregate storage performance to be always more than or equal to max NIC bandwidth.

OCI Block Volume service is designed for consistency, durability, and offers flexible and elastic performance. Compute instances can attach up to 32 block volumes, and each volume can be 50 GB–32 TB. So, I can attach up to 1 PB of storage to a single file server, which is ideal for building large PB file systems without deploying hundreds of file servers to provision the required storage capacity. Block Volume service offers 13 performance tiers to choose from: Ultra High (which has 10 different performance levels to choose from), High, Balanced, and Lower Cost performance tiers. In this blog, we use the Balanced performance tier, which provides a good balance between performance and cost savings for most workloads, especially large storage capacity file servers.

To deliver approximately 6,250 MB/s of IO throughput, we have the following options:

• Attach 14 Balanced tier block volumes of 1–32 TB capacity to get aggregate throughput of 6,720 MB/s (14 x 480)

• Attach 14 High Performance tier block volumes of 800-GB capacity to get aggregate throughput of 6,720 MB/s (14 x 480)

• Attach 9 High Performance tier block volumes of 1.2–32 TB to get aggregate throughput of 6,120 MB/s (Close enough to the NIC bandwidth of 6,250 MB/s)

You can always attach more volumes. These examples give the minimum volume count based on theoretical specs. In the benchmark section, you can see real performance achieved.

Earlier, we saw how Oracle Cloud core platform services (Compute, storage, and network) can be used to build high-performance file servers. We compared all OCI bare metal shapes for maximum network bandwidth in Gbps per OCPU and concluded that the BM.Optimized3.36 shape offers the most network bandwidth per OCPU (2 x 50 Gbps / 36 cores = 2.78), making it the most ideal for file servers.

Similarly, among all virtual machines shapes, the VM.Optimized3.Flex shape with 10 OCPU offers the most network bandwidth per OCPU (40 Gbps / 10 cores = 4), making it the most ideal for VM-based file servers.

Benchmarking

To showcase the performance of BM.Optimized3.36, we deployed BeeGFS parallel file system and ran the well-known IOR benchmark for IO performance measurement.

Cluster-1 uses two file servers and the following specifications:

• Two Object Storage servers using BM.Optimized3.36 with 15 1-TB Balanced block volumes

• One metadata Server using VM.Optimized3.Flex (4 cores/60 GB) with a 400-GB block volume

• One management server using VM.Standard2.2 with a 50-GB block volume

• Eight clients using BM.Optimized3.36

Cluster-2 uses four file servers and the following specifications:

• Four Object Storage servers using BM.Optimized3.36 with 15 1-TB Balanced block volumes

• One metadata server using VM.Optimized3.Flex (4 cores/60GB) with a 400-GB block volume

• One management server using VM.Standard2.2 with a 50-GB block volume

• 20 clients using BM.Optimized3.36

Benchmark results

As shown in the following chart, cluster-2 delivered 22.6 GB/s for write and 20.9 GB/s for read. Similarly, cluster-1 delivered 11.2 GB/s for write and 10.5 GB/s for read. On average, a single BM.Optimized3.36 file server can deliver 5.2–5.6 GB/s for read/write throughput.

Cost

The following table shows the file server cost per network bandwidth in Gbps per hour for bare metal Compute shapes. The BM.Optimized3.36 Compute shape has the lowest cost per Gbps/hour of $0.027.

Conclusion

BM.Optimized3.36 Compute shape is the ideal Compute shape to run high-performance file servers on OCI with the ability to deliver 5.2–5.6 GB/s for read/write throughput per node at the lowest price point of $0.027/Gbps/hour.

Are you planning to run a file server on OCI? Do you want to know more about high-performance storage and file system on Oracle Cloud Infrastructure? You can ask your Oracle account manager to contact the OCI HPC Storage team or you can reach me directly.