X
  • December 19, 2017

Supporting 4 Million transactions per second and 7 TB of data per node with Scylla on Oracle Cloud Infrastructure

Gilson Melo
Principal Product Manager

Why Scylla?

Scylla on Oracle Cloud Infrastructure provides excellent performance results and customers can expect to serve a very high rate of write and read transactions with efficient usage of infrastructure, while minimizing the need to configure auxiliary systems and software. Scylla is an innovative data store that delivers the functionality of Apache Cassandra with the speed of a fast key/value store.
 Scylla’s order of magnitude improvement in performance opens a wide range of possibilities. Instead of designing a complex data model to achieve adequate performance, use a straightforward data model to eliminate complexity and finish your NoSQL project in less time with fewer bugs.

Performance improvements enable not just reduction in resources, but also innovation in architecture and devops systems. Scylla combines the simple and familiar Cassandra architecture with new power to build data models that fit the application. Used in conjunction with Oracle Cloud Infrastructure bare metal shapes, you are able to support 4 million transactions per second and 7 TB of data per instance.

Using Oracle Cloud Infrastructure


Getting Started

Environment:

  • Oracle Cloud Infrastructure Bare Metal Instances - BM.DenseIO1.36
, BM.Standard1.36 and Virtual Machine Shapes. More details below.
  • CentOS 7.4 with kernel 3.10 Instances.
  • The latest 2.0 release of Scylla is used in the testing.
  • 
Installing Scylla is conducted by using scylla_setup script.
  • The script will configure all the needed drives, CPU, and networking settings and Scylla will auto tune the I/O and CPU configuration. 

The following was obtained after the installation on each Scylla server:

Cpuset.conf file: CPUSET="--cpuset 1-35,37-71 "

Io.conf file:
SEASTAR_IO="--num-io-queues 70 --max-io-requests=771"

perftune.yaml:
cpu_mask: 0x000000ff,0xffffffff,0xffffffff mode: sq_split
nic: ens4f0
tune:
- net

Scylla invoking script arguments:
SCYLLA_ARGS="--background-writer-scheduling-quota 0.5 --auto-adjust-flush-quota 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --collectd-address=1 27.0.0.1:25826 --collectd=1 --collectd-poll-period 3000 --network-stack posix"


Scylla uses 70 out of the 72 available threads, with two threads reserved for network interrupt handling. The setting and configurations are automatically applied during Scylla installation, and are mentioned above for reference. 

Networking Considerations

On Oracle Cloud Infrastructure, make sure the proper ports are enabled in your security list. Define a security port list to enable the needed communication between the servers.


Scylla requires the following ports to sustain communication between Scylla servers and Scylla clients: 

Protocol Port Type
CQL (native_transport_port)  9042  TCP 
Inter-node communication (RPC)  7000   TCP 
SSL inter-node communication (RPC)  7001 TCP 
JMX management   7199 TCP 
Scylla REST API   10000 TCP 
Scylla Prometheus API   9180 TCP 
node_exporter (Optional)   9100 TCP 

 


Installation process

In the following sections Scylla uses a six node cluster of BM.DenseIO1.36. Check Scylla public Deploying Scylla on Oracle Bare Metal Cloud Services documentation for more details about how to install and configure Scylla on Oracle Cloud Infrastructure. Benchmarking the cluster is using a 5x BM.StandardIO1.36 and 4x VM.Standard1.16 nodes. A total of nine servers are used to stress the Scylla cluster, and in every instance we are deploying three Cassandra stress processes. A total of 27 stress processes are deployed in each test.

Populating data into the cluster

Using Cassandra Stress tool the cluster is populated to ~42TB of data with replication factor of three, letting each node store ~7TB of data, or ~2.3TB of data pre replication. The reason for the high data volume is to have a data set is that is larger (3x) than the server memory.

With the data write workload rate at 900K writes per second, initially the workload pushes more data points and over time resources are used to accommodate compaction workloads 

Results obtained from the Cassandra stress tool shows latencies are kept below 15ms for a high-rate, strong consistent write workload. The limiting factor in the data loading case is the CPU, we can see that reactor load is near its top.

Latency mean [ms]

4.9

Latency median [ms]

3.9

Latency 95th percentile [ms]

8.8

Latency 99th percentile [ms]

13.0

 

With slightly more than 8 millisecond difference between average and 99%, which is less than 3x difference, services using Scylla on Oracle Cloud Infrastructure can be sure to provide a fast and stable response time to their clients. Users requiring lower latency can rate limit the ingestion rate and maintain CPU reactor load at less than 90%. 

Read data tests

Retrieving data from a cluster has several aspects, as data may reside in memory or it may reside in disk. The first case, of data in memory, delivers the fastest data read option. With the provided setup, Scylla on Oracle Cloud Infrastructure is able to pull 4 Millions data points from the servers.

During this phase the latency results are:

Latency mean [ms]

2.7

Latency median [ms]

2.5

Latency 95th percentile [ms]

3.9

Latency 99th percentile [ms]

4.7

 

With less than 5ms 99th percentile latency, Oracle Cloud Infrastructure and Scylla provide a more than adequate latency budget for user facing applications. Load and latency are consistent throughout the test. Data is read mostly from memory, as the test conducted is querying information ranges that can fit into the server's memory. 

When data is required to be read from disk the results obtained are in the range of 1.3M operations per second
, which means data is read from the servers at the rate of 600MB/sec per server or a total of 3.6GB form the cluster.


The latency results for reads coming from disk are:

Latency mean [ms]

6.9

Latency median [ms]

3.7

Latency 95th percentile [ms]

9.3

Latency 99th percentile [ms]

13.6

 

For clarification, the latency results are obtained under the consistency level of Quorum, which means that at least two server replicas must provide the same coherent data. 

Workload balance during node drop

One of the primary operational tasks for a Database Administrators is monitoring capacity and availability of their system. Node drops are a “granted feature” of every distributed system. Scylla Heat weighted load balancing feature, introduced in Scylla 2.0, helps elevate the impact of read performance once a node departs from the cluster. Combined with Oracle Cloud Infrastructure 25Gbps Network, operators no longer need to configure additional tuning to maintain a constant throughput from a system in the event of a node failure. 

The following graphs show the impact of node drop from the cluster.

Workload balance during node drop 

As seen above, the cluster is pulling ~2M read operations per second when a node is dropped out of the ring. The operations per seconds serviced by the cluster is not affected by the missing node as each of the other servers takes the additional load. The coordinator node is informed there is a missing node, and is not trying to send a query request to that missing node, eliminating dropped request scenarios.

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha
Oracle

Integrated Cloud Applications & Platform Services