Introduction

This blog is about comparing the performance of popular language runtimes using two simple micro benchmarks.

I am writing a series of blogs on using different languages to access Oracle databases [eg Python, Node.js,  Rust and Julia].  Eventully, I want to compare the performance of various languages accessing Oracle. But first I wanted to get a performance baseline for some popular language runtimes.  Last week I covered Making Java faster for the same micro benchmarks.

 

The languages covered in this blog are:

• Python 3.12
• JavaScript with Node.js 18.12.1
• TypeScript 4.9.3 witth Node.js 18.12.1
• Java 19.0.1 with GraalVM Enterprise Editon 22.3
• Scala 3.2.1 with GraalVM Enterprise Editon 22.3
• Ruby 3.1.2
• R 4.2.2
• Julia 1.8.3

 

This blog covers the following topics:

• The two micro benchmarks that I created
• The results
• My source code for all of those languages
• How I did the builds and tests
• How I calculated the results
• Summary

 

This blog is not a tutorial on these computer languages.  This is also not a blog on how to download and configure various language runtimes.

In my next blog, I plan to cover the same micro benchmarks for compiled languages [eg C, C++, C#, Go and Rust].

 

 

 

My micro benchmarks

I am not trying to state that one computer language is better than another.  There are many factors that influence which language that you choose to use and performance is only one of them.

I needed some trivial workloads, so I chose to use the same micro benchmarks that I used for my blog on Making Java faster:

• Calculate the Fibonacci sequence with an input of 1475, repeated one million times
• Some trivial string processing with strings. ie creating, concatenating and using substrings for strings under 2000 characters with a huge number of iterations

 

 

 

How valid are these results

Micro benchmarks are, by definition, only relevant to the specific workload that they cover.  These workloads do not try to cover everything, they only cover what I care about. The only workload that matters to you is your workload.  So compare your own workloads with your favourite languages.  I have found that string processing and simple maths are important to enable fast SQL database drivers, so that is what I tested.

 

 

 

Results

I chose to split the eight languages into two groups, ie strong vs weakly typed languages:

This is not the best definition for types, but it is useful as type strength tends to have a dramatic effect on performance.  Strongly typed lanugages tend to be faster than weakly typed languages as the extra type metadata can enable more compiler optimizations.

 

 

Micro benchmarks with weak typing

This chart shows the total execution time of micro benchmarks for simple math and string processing:

• Julia is a general purpose language from 2012 that is gaining popularity in machine learning and scientic computing
  • Julia has optimized floating point libraries for numerical analysis
  • I choose to use my trivial code in the Julia interpreter rather than to use optimized floating point libraries
  • Using compilers to create executables from scripting languages is a blog for another week
• Python is a general purpose language from 1991 which is popluar in machine learning and as a scripting language
• Ruby is a general purpose language from 1995 which enabled the popular Ruby on Rails framework
• R is a language from 1993 designed for statistical computing and is popular in machine learning

 

The results look fairly definitive, but the reality is more complicated than that.

• The string processing dominated the total execution time
• Julia was OK at processing the Fibonacci function, but was slow for string processing
• R, Ruby and Python have opportunities for optimization for both simple math and string processing

 

 

 

 

Micro benchmarks with strong typing

• Java 19.0.1 using GraalVM Enterprise Edition 22.3 gave the best performance for these micro benchmarks
  • Other Java runtimes were considerably slower
• TypeScript 4.9.3 run with Node.js 18.12.1 was almost as fast as Java
  • The V8 JavaScript engine continues to be optimized for performance and memory usage
  • TypeScript gets compiled into JavaScript code via tsc
• JavaScript run on Node.js 18.12.1 gave exactly the same performance as TypeScript
  • My initial JavaScript code was not as fast as my TypeScript code. Oops
  • I tweaked my JavaScript code to be more like the JavaScript generated from the TypeScript compiler until they gave the same performance
• Scala 3.1.2 run with the GraalVM Enteprise Edition was the slowest of the four strongly typed runtimes
  • Scala code compiles to Java byte code and runs on a Java Virtual Machine
  • For these micro benchmarks, both Java and Scala used the same JVM [GraalVM Enterprise Edition 22.3]]
  • If both Java and Scala used the same JVM, why was Java so much faster than Scala?
  • I assume that the javac compiler created more optimized byte code than the scalac compiler for this workload

 

 

This table gives a break down of the execution time for the fibonacci and string workloads for strongly typed languages:

• The strings workload was less dominant for these strongly typed languages
• The total time [column both] can be less than the sum of the Fibonacci and Strings columns as the process startup and shutdown time [+ runtime initialization] is only counted once
• Java is still considerably slower than languages that compile to standalone executables [eg C and Rust]
  • I will cover standalone executable languages in my next blog

 

 

Strong vs Weak Typing

• The slowest strongly typed language [Scala] was still significantly faster than the fastest weakly typed language [Julia]
• Ultimately, whether the language runtime was designed for the workload and how effective the runtime optimization is will tend to determine the performance for a given workload

 

 

My trivial source code

 

The Scala Main Function

• The fibonacci function has an input of 1475 and was called one million times
  • Why 1475, to avoid numeric overflow in some of the other languages which I tested this workload against
  • I am using the type double [or equivalent] for all languages to avoid numeric overflow for the large numbers from the Fibonacci sequence
• Both the strings and long_strings methods are called with an input of 1475

 

 

The Scala Fibonacci Function

Why am I using a double for the variables?

• The values of the Fibonacci sequence rapidly get larger
• I also implemented these micro benchmarks in many other languages
• Some of these languages had issues with integer overflow for large values in the Fibonacci sequence
• So I used the type double to be fair and consistent across all of the languages

I am not using recursion as it is against my religion.

 

 

The Scala Strings Function

• This function does some trivial operations on strings
  • The operations include constructors, append, length, substring and copy
• There are three nested loops, so the operations in the inner-most loop are executed about 26 million times
  • 1475 * 12 * 1475 = 26,107,500
  • n = 1475
  • The string length is 12 characters

 

 

The Scala long_strings Function – Part 1

The logic for function long_strings was the same as for method strings, but there were significantly more string concatenation operations.

• The fully appended string is 1965 bytes long
• Why did I not create the strings outside the loops?
  • I wanted to make the comparisons with other languages fair and consistent
  • I am not trying to optimize the code for this workload, I am trying to see how common / ‘bad’ code performs
• The number of iterations of the string and operations is significantly larger
• This also creates the opportunity of garbage collection

 

 

The Scala long_strings Function – Part 2

• The ‘j’ for loop iterates based on the length of the string, ie 1965 times
• The ‘k’ for loop iterates n times, ie 1475
• The outer ‘i’ for loop also iterates n times, ie 1475
• 1475 * 1965 * 1475 = 4,275,103,125 iterations
• So there are 4.2 billion iterations of the ‘k’ loop which creates strings from substrings

 

 

 

The TypeScript Main Function

• The fibonacci function has an input of 1475 and was called one million times
  • Why 1475, to avoid numeric overflow in some of the other languages which I tested this workload against
  • I am using the type double [or equivalent] for all languages to avoid numeric overflow for the large numbers from the Fibonacci sequence
• Both the strings and long_strings methods are called with an input of 1475

 

 

The TypeScript Fibonacci Function

Why am I using a number rather than bigint for the variables?

• The values of the Fibonacci sequence rapidly get larger
• I also implemented these micro benchmarks in many other languages
• Some of these languages had issues with integer overflow for large values in the Fibonacci sequence
• So I used the type double to be fair and consistent across all of the languages

I am not using recursion as it is against my religion.

 

 

The TypeScript Strings Function

• This function does some trivial operations on strings
  • The operations include constructors, append, length, substring and copy
• There are three nested loops, so the operations in the inner-most loop are executed about 26 million times
  • 1475 * 12 * 1475 = 26,107,500
  • n = 1475
  • The string length is 12 characters

 

 

 

The TypeScript long_strings Function – Part 1

The logic for function long_strings was the same as for function strings, but there are significantly more string concatenation operations.

• The fully appended string is 1965 bytes long
• Why did I not create the strings outside the loops?
  • I wanted to make the comparisons with other languages fair and consistent
  • I am not trying to optimize the code for this workload, I am trying to see how common / ‘bad’ code performs
• The number of iterations of the string and operations is significantly larger
• This also creates the opportunity of garbage collection

 

 

The TypeScript long_strings Function – Part 2

• The ‘j’ for loop iterates based on the length of the string, ie 1965 times
• The ‘k’ for loop iterates n times, ie 1475
• The outer ‘i’ for loop also iterates n times, ie 1475
• 1475 * 1965 * 1475 = 4,275,103,125 iterations
• So there are 4.2 billion iterations of the ‘k’ loop which creates strings from substrings

 

 

 

The JavaScript Main Function

• The fibonacci function has an input of 1475 and was called one million times
  • Why 1475, to avoid numeric overflow in some of the other languages which I tested this workload against
  • I am using the type double [or equivalent] for all languages to avoid numeric overflow for the large numbers from the Fibonacci sequence
• Both the strings and long_strings methods are called with an input of 1475

 

 

The JavaScript Fibonacci Function

I am not using recursion for this function as it is against my religion.

 

 

The JavaScript Strings Function

• This function does some trivial operations on strings
  • The operations include constructors, append, length, substring and copy
• There are three nested loops, so the operations in the inner-most loop are executed about 26 million times
  • 1475 * 12 * 1475 = 26,107,500
  • n = 1475
  • The string length is 12 characters

 

 

 

The JavaScript long_strings Function – Part 1

The logic for function long_strings was the same as for function strings, but there were significantly more string concatenation operations.

• The fully appended string is 1965 bytes long
• Why did I not create the strings outside the loops?
  • I wanted to make the comparisons with other languages fair and consistent
  • I am not trying to optimize the code for this workload, I am trying to see how common / ‘bad’ code performs
• The number of iterations of the string and operations is significantly larger
• This also creates the opportunity of garbage collection

 

 

 

The JavaScript long_strings Function – Part 2

• The ‘j’ for loop iterates based on the length of the string, ie 1965 times
• The ‘k’ for loop iterates n times, ie 1475
• The outer ‘i’ for loop also iterates n times, ie 1475
• 1475 * 1965 * 1475 = 4,275,103,125 iterations
• So there are 4.2 billion iterations of the ‘k’ loop which creates strings from substrings

 

 

 

The Java code for these micro benchmarks was covered in my blog How to make Java faster.

 

 

 

 

 The R Main Function

• The fibonacci function has an input of 1475 and was called one million times
  • Why 1475, to avoid numeric overflow in some of the other languages which I tested this workload against
  • I am using the type double [or equivalent] for all languages to avoid numeric overflow for the large numbers from the Fibonacci sequence
• Both the strings and long_strings methods are called with an input of 1475

 

 

The R Fibonacci Function

I am not using recursion for this function as it is against my religion.

 

 

The R Strings Function

• This function does some trivial operations on strings
  • The operations include constructors, append, length, substring and copy
• There are three nested loops, so the operations in the inner-most loop are executed about 26 million times
  • 1475 * 12 * 1475 = 26,107,500
  • n = 1475
  • The string length is 12 characters

 

 

 

The R long_strings Function – Part 1

The logic for function long_strings was the same as for function strings, but there were significantly more string concatenation operations.

• The fully appended string is 1965 bytes long
• Why did I not create the strings outside the loops?
  • I wanted to make the comparisons with other languages fair and consistent
  • I am not trying to optimize the code for this workload, I am trying to see how common / ‘bad’ code performs
• The number of iterations of the string and operations is significantly larger
• This also creates the opportunity of garbage collection

 

 

The R long_strings Function – Part 2

• The ‘j’ for loop iterates based on the length of the string, ie 1965 times
• The ‘k’ for loop iterates n times, ie 1475
• The outer ‘i’ for loop also iterates n times, ie 1475
• 1475 * 1965 * 1475 = 4,275,103,125 iterations
• So there are 4.2 billion iterations of the ‘k’ loop which creates strings from substrings

 

 

 

The Ruby Main Function

• The fibonacci function has an input of 1475 and was called one million times
  • Why 1475, to avoid numeric overflow in some of the other languages which I tested this workload against
  • I am using the type double [or equivalent] for all languages to avoid numeric overflow for the large numbers from the Fibonacci sequence
• Both the strings and long_strings methods are called with an input of 1475

 

 

The Ruby Fibonacci Function

I am not using recursion for this function as it is against my religion.

 

 

The Ruby Strings Function

• This function does some trivial operations on strings
  • The operations include constructors, append, length, substring and copy
• There are three nested loops, so the operations in the inner-most loop are executed about 26 million times
  • 1475 * 12 * 1475 = 26,107,500
  • n = 1475
  • The string length is 12 characters

 

 

 

The Ruby long_strings Function – Part 1

The logic for function long_strings was the same as for function strings, but there were significantly more string concatenation operations.

• The fully appended string is 1965 bytes long
• Why did I not create the strings outside the loops?
  • I wanted to make the comparisons with other languages fair and consistent
  • I am not trying to optimize the code for this workload, I am trying to see how common / ‘bad’ code performs
• The number of iterations of the string and operations is significantly larger
• This also creates the opportunity of garbage collection

 

 

The Ruby long_strings Function – Part 2

• The ‘j’ for loop iterates based on the length of the string, ie 1965 times
• The ‘k’ for loop iterates n times, ie 1475
• The outer ‘i’ for loop also iterates n times, ie 1475
• 1475 * 1965 * 1475 = 4,275,103,125 iterations
• So there are 4.2 billion iterations of the ‘k’ loop which creates strings from substrings

 

 

 

 The Python Main Function

• The fibonacci function has an input of 1475 and was called one million times
  • Why 1475, to avoid numeric overflow in some of the other languages which I tested this workload against
  • I am using the type double [or equivalent] for all languages to avoid numeric overflow for the large numbers from the Fibonacci sequence
• Both the strings and long_strings methods are called with an input of 1475

 

 

 

The Python Fibonacci Function

I am not using recursion for this function as it is against my religion.

 

 

 

The Python Strings Function

• This function does some trivial operations on strings
  • The operations include constructors, append, length, substring and copy
• There are three nested loops, so the operations in the inner-most loop are executed about 26 million times
  • 1475 * 12 * 1475 = 26,107,500
  • n = 1475
  • The string length is 12 characters

 

 

 

The Python long_strings Function – Part 1

The logic for function long_strings was the same as for function strings, but there were significantly more string concatenation operations.

• The fully appended string is 1965 bytes long
• Why did I not create the strings outside the loops?
  • I wanted to make the comparisons with other languages fair and consistent
  • I am not trying to optimize the code for this workload, I am trying to see how common / ‘bad’ code performs
• The number of iterations of the string and operations is significantly larger
• This also creates the opportunity of garbage collection

 

 

 

The Python long_strings Function – Part 2

• The ‘j’ for loop iterates based on the length of the string, ie 1965 times
• The ‘k’ for loop iterates n times, ie 1475
• The outer ‘i’ for loop also iterates n times, ie 1475
• 1475 * 1965 * 1475 = 4,275,103,125 iterations
• So there are 4.2 billion iterations of the ‘k’ loop which creates strings from substrings

 

 

 

 

 

 The Julia Main Function

• The fibonacci function has an input of 1475 and was called one million times
  • Why 1475, to avoid numeric overflow in some of the other languages which I tested this workload against
  • I am using the type double [or equivalent] for all languages to avoid numeric overflow for the large numbers from the Fibonacci sequence
• Both the strings and long_strings methods are called with an input of 1475

 

 

 

 

The Julia Fibonacci Function

I am not using recursion for this function as it is against my religion.

 

 

 

The Julia Strings Function

• This function does some trivial operations on strings
  • The operations include constructors, append, length, substring and copy
• There are three nested loops, so the operations in the inner-most loop are executed about 26 million times
  • 1475 * 12 * 1475 = 26,107,500
  • n = 1475
  • The string length is 12 characters

 

 

 

The Julia long_strings Function – Part 1

The logic for function long_strings was the same as for function strings, but there were significantly more string concatenation operations.

• The fully appended string is 1965 bytes long
• Why did I not create the strings outside the loops?
  • I wanted to make the comparisons with other languages fair and consistent
  • I am not trying to optimize the code for this workload, I am trying to see how common / ‘bad’ code performs
• The number of iterations of the string and operations is significantly larger
• This also creates the opportunity of garbage collection

 

 

 

The Julia long_strings Function – Part 2

• The ‘j’ for loop iterates based on the length of the string, ie 1965 times
• The ‘k’ for loop iterates n times, ie 1475
• The outer ‘i’ for loop also iterates n times, ie 1475
• 1475 * 1965 * 1475 = 4,275,103,125 iterations
• So there are 4.2 billion iterations of the ‘k’ loop which creates strings from substrings

 

 

 

 

My environment

I repeated these tested on two different machines:

• Oracle Linux 8.6 on Oracle Cloud. 4 OCPU with 128 GB RAM
• Ubuntu 22.04 on Oracle Cloud. 4 OCPU with 128 GB RAM
• As these were VMs, to avoid the risk of a noisey neighbor, I repeated the tests many times over three days
• My micro benchmarks were not doing any disk nor network IO. Instead they were CPU bound for a single threaded workload.
• As measured by ‘top‘, the VIRT and RSS memory was stable for the duration of the tests and there was 128 GB of RAM
  • VIRT was about 34 GB for Java and Scala and the VMs
  • VIRT for the other languages was less than 1 GB

 

 

 

How I built and ran each test

 

For Scala with graalvm-ee-java19-22.3.0

• scala -v    # to verify the version and JVM runtime
• scalac Main.scala
• time scala Main.scala

 

For Java jdk 19.0.1 from graalvm-ee-java19-22.3.0

• java -version  # to verify the version and JVM runtime
• javac fibStr.java
• time java fibStr

 

For TypeScript

• tsc –version
• tsc fibStrTS.ts
• time node fibStrTS.js

 

For JavaScript

• time node fibStr.js

 

For Python

• time python fibStr.py

 

For R

• time Rscript fibStr.R

 

For Ruby

• time ruby fibStr.rb

 

For Julia 1.8.3

• julia -v  # to verify the version
• time julia fibStr.jl

 

 

How I calculated the results

On three different days, I did the following:

• Run the tests for each runtime 10 times using the Linux time command until I got stable results
• I eliminated the highest and lowest results
• I took the average of the remaining eight results
  • The Linux time command gives a resolution of 1 millisecond
  • The fastest results for the three functions took several seconds
  • Most of the results took many minutes
  • So measurement error did not seem to be a factor
• There was always some variation between the runs, however the relative performance was always the same

 

 

 

Summary

• Based on my micro benchmarks, Java gave the best performance with TypeScript and JavaScript a close second
• The weakly typed languages [R, Ruby, Python and Julai] were significantly slower than the strongly typed languages [Java, TypeScript and Scala]
• In my next blog, I will repeat these micro benchmarks with C, C++, C#, Go and Rust

 

 

Disclaimer: These are my personal thoughts and do not represent Oracle’s official viewpoint in any way, shape, or form.