X

Reproducible Performance Benchmarks for Oracle Cloud Infrastructure Compute Instances

Sanjay Pillai
Principal Product Manager

Enterprise workloads are often performance sensitive. At Oracle, we designed our cloud to deliver consistently high performance. As we improve our cloud over time, we periodically measure performance and share that data with our customers, so that you can understand what to expect. In this post, we show test results from two common benchmarks and present the methodology to re-create our results.

We measured four of our commonly used Compute instances by using two well-known and reliable benchmarking suites: UnixBench on Linux, and LINPACK on Windows. For each OS, there were two levels of workload complexity: with UnixBench, varying concurrency in the tests; with LINPACK, the number of equations solved.

Compute Instance Performance on Linux OS

We used UnixBench to test the performance of Linux instances. UnixBench is a set of tests that measure the performance of various aspects of a UNIX or Linux based system. The result is an aggregate score that measures the overall system performance as opposed to any one individual component.

The following table shows the aggregate UnixBench performance for each of the tested Compute instance shapes with single-threaded and concurrent (multi-threaded) test configurations. Higher mean/median results indicate better performance, and standard deviation is provided to show consistency (smaller is better).

Table 1: Aggregate UnixBench Performance for Tested Compute Shapes

Shape

Test Concurrency

Mean

Median

Standard Deviation

VM.Standard2.1

1

584

587

10

VM.Standard2.1

2

802

804

10

VM.Standard2.2

1

637

640

12

VM.Standard2.2

4

1493

1500

32

VM.Standard2.4

1

671

673

12

VM.Standard2.4

8

2452

2455

34

VM.Standard2.8

1

689

692

11

VM.Standard2.8

16

4017

4027

57

 

As expected, we saw relatively consistent single-threaded performance across these instances, and a roughly proportional increase in performance as the tests are run in a concurrent configuration.

The following graph plots these results:

Graph that shows the results of the performance test on Linux instances with UnixBench.

We performed these tests in different regions and found them to be highly consistent across all tested regions. Here are the results by region:

Graph that shows the test results by region.

Compute Instance Performance on Windows OS

We used LINPACK to test the performance of Windows instances. The LINPACK benchmark measures how quickly a system can solve several linear equations. The results show the average number of floating-point operations per second that an instance can perform, measured in gigaFLOPS (GFLOPS). We ran it with a small workload of 2,000 equations, and with a larger workload of 40,000 equations.

The following table shows the aggregate LINPACK performance for each of the tested Compute instance shapes. A higher number indicates better performance, and a lower standard deviation indicates reduced variation between each test.

Table 2: Aggregate LINPACK Performance for Tested Compute Shapes

Shape

Test Size

Mean

Median

Standard Deviation

VM.Standard2.1

2,000

27

27

3

VM.Standard2.1

40,000

49

51

6

VM.Standard2.2

2,000

69

73

11

VM.Standard2.2

40,000

97

102

13

VM.Standard2.4

2,000

119

126

16

VM.Standard2.4

40,000

197

203

21

VM.Standard2.8

2,000

200

216

31

VM.Standard2.8

40,000

402

410

32

 

The following chart shows a summary of the average scores for small and large test sizes by instance type. The results show that the performance increases as the size of the instance increases, with a steeper increase shown for tests with a larger number of equations. You can expect the floating-point performance on Intel Xeon based virtual machine Compute instances to scale linearly with a shape’s core count.

Graph that shows the results of the performance test on Windows instances with LINPACK.

Similar to the Linux data, the Windows results are fairly consistent across the tested regions. The following graph shows minimal variation between regions:

Graph that shows the test results by region.

Testing Methodology

We intended our testing to be easily reproducible. We used standard open source benchmarks and a straightforward testing methodology.

Our performance tests used the following parameters:

To reproduce these results, you need the following items:

  • Access to an Oracle Cloud Infrastructure account with the following permissions:
    • Create and delete dynamic groups, and the associated access policy
    • Create and delete Object Storage buckets
    • Create and delete virtual cloud networks (VCNs) and associated objects
    • Create and delete Compute instances
    • (Optional) Create and delete streams
  • Access to a system that can run Python 3. The benchmark code has been tested on OS X and Oracle Linux 7.7. Although you can run the code on your personal workstation, we recommend that you use a Compute VM instance to run the tests.
  • A working installation of the Oracle Cloud Infrastructure Python SDK.
  • The code to run the benchmark tests. Download the code, extract the files, and then customize the files for your environment.

Note: The code to run the benchmark is provided only as an example. Although the code creates a working setup, Oracle Cloud Infrastructure doesn’t support this code.

Customizing the Benchmark Tests for Your Environment

The config.py file shows you the relevant customizations. Although you can edit the file directly, we recommend that you create an override file that contains your specific changes and then activate the file by setting the OVERRIDES environment variable, before you run any scripts.

At a minimum, modify the following parameters in the config.py file:

  • homeregion
  • compartment
  • cname
  • tenancy
  • launches: Don't set this value too large, because the tests could cause your tenancy to reach its Compute service limits for the shape that you're testing.
  • limit: Don't set this value too large, because the tests could cause your tenancy to reach its Compute service limits for the shape that you're testing.
  • public_key: Although a readable file name is required, the file contents aren't validated. If you want to prevent cloud-init from configuring Linux instances for SSH access, provide a dummy file.
  • regions: The test code launches instances in all commercial availability domains, which is excessive for most users. Include only the regions where you want to launch instances.

Launching Instances Using the Example Benchmark Tests

If you decide to launch instances with these scripts, you typically run the scripts in this order:

  1. setupenv.py: Creates the objects that are needed by the global runtime environment
  2. launchinstance.py: Creates the regional objects, launches the instances, and waits for the instances to terminate
  3. download_data.py: Stores the run artifacts in a directory that you choose
  4. teardown.py: Terminates (or reterminates, if necessary) any instances that were launched and removes the regional objects
  5. teardownenv.py: Removes the global objects and associated artifacts that setupenv.py created

We’re excited to help you understand the performance that you can expect from Oracle Cloud implementations, and this post describes a transparent methodology for measuring the performance that our Compute instances deliver.

We want you to experience the features and enterprise-grade capabilities that Oracle Cloud Infrastructure offers. It’s easy to try them out with our 30-day trial and our Always Free tier of services. For more information, see the Oracle Cloud Infrastructure Getting Started guide, Compute service overview, and Compute FAQ.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.