TimesTen Scaleout

TimesTen Scaleout is the distributed version of the Oracle TimesTen In-Memory Database.  TimesTen Scaleout has a sharded, shared nothing architecture which enables massive scalability while still supporting SQL joins and data transparency.  TimesTen Scaleout supports ACID transactions and active-active writes using an optimzed two phase commit [2PC] protocol.

TimesTen supports SQL and PLSQL via drivers for Oracle APIs such as JDBC, OCI, ODBC, Pro*C and ODP.Net as well as the open source Oracle SQL APIs for Node.js, Python, Go, Ruby, Rust, PHP and Tcl. 

TimesTen Scaleout can either be used as a system of record, or as a read only cache.  The source for the read only cache can either be an Oracle database, or Oracle GoldenGate non-Oracle databases [eg MySQL, PostgreSQL, MariaDB, SQL Server, DB2, Sybase, Informix, Teradata etc].

K-Safety

TimesTen Scalout 22.1 enables up to five copies of each database shard:

• K=1 means that there is only one copy of each shard.  In this configuration, each shard is a single point of failure
• K=2 means that there are two copies of each shard
• K=3 means that there are three copies of each shard
• K=4 means that there are four copies of each shard
• K=5 means that there are five copies of each shard

In the following picture, the TimesTen Scaleout database has six shards [horizontal axis] and four copies of each shard [vertical axis] for a total of 24 elements.

When using K-Safety, each copy of a shard should be on a separate physical machine so as not to become a single point of failure. Ideally each copy of a shard should use separate failure domains [eg different racks with independent power supplies] within a data centre or different availability zones in a public cloud. The elements [copies of a shard] can be in Linux containers, VMs or bare metal machines.

In TimesTen Scaleout, all of the copies of a shard are called a replicaset. In the above picture, a column of elements is a replicaset.  In the picture above, a row of elements is called a dataspace.

The NxM naming convention can be used to explain different configurations.  With an NxM grid, the ‘N’ is the number of shards and the ‘M’ is the number of copies of each shard. The above picture is an example of a 6×4 grid.

Scaleout 4×1

The following picture shows a TimesTen Scaleout database with four shards and only one copy of each shard [4×1]:

For this 4×1 configuration, there is one dataspace, four replicasets and four elements.

In the above example, having four shards enables each machine to store about one quarter of the rows in the database. Although this is a very simple configuration, it suffers from the fact that each one of the shards is a single point of failure. ie if any of the shards becomes un-available, then all of the data on that shard also becomes un-available. This is not a recommend configuration unless you have a use-case for read only caching where it does not matter if some of the cache data is not available.

Scaleout 4×2

The following picture shows a TimesTen database with four shards and two copies of each shard [4×2].

For this 4×2 configuration, there are two dataspaces, four replicasets and eight elements.

There is no single point of failure as there are two elements for each shard.

Scaleout 4×5

The following picture shows a TimesTen database with four shards and five copies of each shard [4×5].

For this 4×5 configuration, there are five dataspaces, four replicasets and twenty elements.

There is no single point of failure as there are five elements for each shard.

Scalability

Having more shards will enable greater throughput for read/write workloads. Having more elements (ie copies of each shard) enables greater throughput for read only workloads.

The following example uses a 32×2 grid, ie 32 shards with two copies for each shard for a total of 64 elements.

Recently we benchmarked a 64×2 grid in Oracle Cloud and achieved 3.1 Billion SQL selects per second.

This 64×2 configuration has almost linear scalability:

• One shard with two elements enabled 51 million SQL selects per second
• Two shards with four elements enabled 113 million SQL selects per second
• Four shards with eight elements enabled 205 million SQL selects per second
• Eight shards with sixteen elements enabled 452 million SQL selects per second
• Sixteen shards with 32 elements enabled 935 million SQL selects per second
• 32 shards with 64 elements enabled 1.6 billion SQL selects per second
• 64 shards with 128 elements enabled 3.1 billion SQL selects per second

This benchmark used Kubernetes 1.19 on Oracle Cloud.  Each element used a single container running Oracle Linux 7.9 on a bare metal Kubernetes node with AMD Rome CPUs and 2 TB RAM.

Database Capacity

TimesTen Scaleout is based on the Oracle TimesTen In-Memory Database. For an in-memory database like TimesTen, all of the data needs to be in memory all of the time.

To enable TimesTen to store large amounts of data, you either need a single machine with a large amount of RAM, or use many machines with TimesTen Scaleout. With TimesTen Scaleout, the sum of the RAM for number of the shards determines the database capacity.

The elements for a TimesTen Scaleout database can be Linux containers, VMs or bare metal machines.

In the above picture, the 6×4 has six shards. The sum of the RAM for those six shards determines the database capacity.  The other 18 elements are copies of the shards and do not increase the capacity of the database.

I recently tested TimesTen Scaleout 22.1 with the following configurations with Kubernetes on Oracle Cloud:

• 14×2 where each element had 1.95 TB RAM giving a database capacity of 27.3 TB
• 64×2 where each element had 1.95 TB RAM giving a database capacity of 124.8 TB
• 128×1 where each element had 1.95 TB RAM giving a database capacity of 249.6 TB

Each element ran in a single container on a Kubernetes node with 2 TB RAM.  I allocated a shared memory segment of 1950 GB. This left 50 GB of RAM for the Linux kernel, the Kubelet and any local applications.

TimesTen 22.1.1.1.0 supports 64 elements. TimesTen Scaleout has been benchmarked with 128 elements, but that configuration is not yet officially supported.

Summary

• TimesTen Scaleout uses K-Safety for high availability within a database
• The K-safety factor determines how many copies there are of each shard [between 1 and 5]
• The sum of the RAM for the number of shards determines the database capacity
• The total number of elements determines the read only scalability
• The number of shards determines the read/write scalability
• The customers use case, data volume and throughput requirements will determine how many shards & copies of each shard are needed
• The elements can run on Linux containers, VMs or bare metal hosts