The authors want to thank Dr. Sanjay Basu, Senior Director of OCI Engineering, and Rob Dolin, Senior Manager of OCI Engineering, for their assistance in publishing these results.

Oracle Cloud Infrastructure (OCI) has achieved strong results across multiple benchmarks in the MLCommons Inference Datacenter v3.1 suite, the industry standard for measuring AI infrastructure performance. OCI was tested across several shapes powered by NVIDIA GPUs, including the NVIDIA H100 Tensor Core GPU, the NVIDIA A100 Tensor Core GPU, and the NVIDIA A10 Tensor Core GPU, and has the following key highlights:

• OCI’s BM.GPU.H100.8 shape with eight NVIDIA H100 GPUs delivered top results, outperforming or matching competitors on RESNET, RetinaNet, BERT, DLRMv2, and GPT-J benchmarks.
• BM.GPU.A100-v2.8 with eight NVIDIA A100 GPUs also showed strong performance across the board.
• BM.GPU.A10.4 with four NVIDIA A10 GPUs demonstrated cost-effective performance on select benchmarks like RetinaNet and RNNT.

OCI’s focus on high performance

From its inception, OCI has been focused on providing high-performance infrastructure for any workload. We were one of the first cloud providers to offer support for bare metal instances natively and high-performance RDMA network for internode communications. We can support all phases of the AI workflow, including training and inference. Many organizations are driving innovation on OCI with AI and NVIDIA GPUs, including MosaicML, Twist Bioscience, and Emory University. 

This announcement is for our inaugural publication of OCI’s MLPerf results through the MLCommons Inference Datacenter v3.1 benchmarks. MLCommons is a collaborative engineering organization focused on developing the AI ecosystem through benchmarks, public datasets, and research. OCI’s portfolio of GPU shapes include NVIDIA H100, NVIDIA A100, and NVIDIA A10 GPUs, among others. We have obtained inference benchmark results for all three of these industry-leading NVIDIA GPU shapes.

OCI Compute shapes benchmarked for MLCommons Inference v3.1

Bare metal shape powered by NVIDIA H100 GPUs: BM.GPU.H100.8

The BM.GPU.H100.8 shape includes eight NVIDIA H100 GPUs, with 80 GB of GPU memory per GPU. The CPU processors are Intel Xeon Platinum 8480+ with 2-TB system memory and 112 cores. The shape also includes 16 local NVMe drives with a capacity of 3.84 TB each.

The following table shows the benchmark results of BM.GPU.H100.8. The results are comparable or superior to those from alternatives. For the complete results, see the ML Commons Inference Data Center page.

Bare metal shape powered by NVIDIA A100 GPUs: BM.GPU.A100-v2.8

This shape includes eight NVIDIA A100 GPUs, with 80 GB of GPU memory per GPU. The CPUs are AMD EPYC 7J13 64-Core Processors with 128 cores and 2 TB of system memory. The benchmark results indicated that the results were on par with, if not superior to, the other hyperscalers.

Bare metal shape powered by NVIDIA A10 GPUs: BM.GPU.A10.4

The BM.GPU.A10.4 shape includes four NVIDIA A10 GPUs, with 24 GB of GPU memory and 1-TB system memory. The shape also includes two 3.5-TB NVMe local storage.

The benchmark identified that OCI Compute instances based on NVIDIA A10 GPUs are a suitable option for inferencing specific models at optimum price-performance. GPTJ and DLRMv2 benchmarks were not run on A10.4 for this iteration.

The complete results are available at the ML Commons Inference Data Center page.

Takeaway

Oracle Cloud Infrastructure provides a comprehensive portfolio of GPU options optimized for AI workloads including training and inference. The MLPerf Inference results showcase OCI’s competitive strength in AI infrastructure and ability to handle demanding workloads like large language models. For further information on our products, see our GPU and AI infrastructure pages.