X

The Oracle NoSQL Database Blog covers all things Oracle NoSQL Database. On-Prem, Cloud and more.

  • November 29, 2016

Expanding an Oracle NoSQL Cluster

A Chandak
Product Manager

Introduction 

This is extension of my last blog, where I described how to setup
a single node NoSQL database. You'd realize that setting up a single
node NoSQL db or kvlite should take less than 5 min - that's how simple
it is! In this blog, I want to talk about how to expand the kvlite
instance to a 3x3 cluster -  3 shards each having 3 copies of data

Motivation for NoSQL Cluster Expansion 

Before we see the actual steps of expansion, lets look at what could be motivations to expand the NoSQL cluster :

Increase Data Capacity - A Company’s Oracle NoSQL Database
application is now obtaining it’s data from several unplanned new
sources.  The utilization of the existing configuration as more than
adequate  to meet requirements, with one exception, they anticipate
running out of disk space later this year. The company would  like to
add the needed disks to the existing servers in existing slots,
establish mount points, ask NoSQL Database to fully utilize the new
disks along with the disks already in place while the system is up and
running Oracle NoSQL Database. 

Increase Throughput -  As a result of an unplanned
corporate merger, the live Oracle NoSQL Database will see a substantial
increase in write operations. The read write mix of transactions will go
from 50/50 to 85/15.  The need workload will exceeds the I/O capacity
available of the available storage nodes. The company would like to add
new hardware and have it be utilized by the existing Oracle NoSQL
Database (kvstore) currently in place.  Oh, and of course the
Application needs to continue to be available while this upgrade is
occurring. 

Increase Replication Factor-  A new requirement has been
placed on an existing Oracle NoSQL Database to increase the overall
availability of the Oracle NoSQL Database by increasing the replication
factor by utilizing new storage nodes added in a second geographic
location.  This is accomplished by adding at least 1 replication node
for every existing shard.  The current configuration has a replication
factor of 3. And, yes the expansion happens, while KVStore continues to
service the existing workload.

Of the above, Increase Data Capacity, is obviously understood.
Improvement in the storage capacity is quite obvious. When you add
replication nodes, you have additional space to store your data, 

Expansion

We want to transform our existing 1x1 NoSQL store to 3x3  (3 shards each having 3 copies of data) cluster.

Prerequisite 

  • Make sure that you have Java SE 6 (JDK 1.6.0 u25) or later installed on
    all of the hosts that you are going to use for the Oracle NoSQL Database
    installation
  • Make sure that some
    sort of reliable clock synchronization is running on each of
    the machines. Generally, a synchronization delta of less than
    half a second is required. ntp is
    sufficient for this purpose.

Installation

Follow the installation steps highlighted on the doc guide

Configuration Steps

Step 1 : Create the initial "boot config" configuration file
using the makebootconfig
utility.
You should do this on each Oracle NoSQL Database
node, here we are doing for storage node 2 and node 3 (storage node 1 is
already bootstrapped). Notice that we are defining three different port (-admin, -harangue, -port )ranges for various inter-intra node communication. Number of replication-nodes (RN) this storage node can hosts defined by the capacity as ‘-capacity’. You can define storage directories where data is going to be written using -storagedir argument.Follow the link to understand the meaning of different configuration parameters.

Please note, in this expansion exercise, we have added storage nodes (SN2 and SN3) with the greater capacities (i.e capacity =3). we defined capacity=3, which means we would like three RNs to be hosted on each of the three storage node with each RN using its own disk .This is important aspect as independent spindles improves performance and the availability of the system.

########### Bootstrap Storage Node 2 ###########
java -jar $KVHOME/lib/kvstore.jar makebootconfig \
       -root $KVROOT \

        -store-security none \
       -capacity 3 \
        -harange 6010,6030 \
        -admin 6001 \
        -port 6000 \
        -memory_mb 200\
        -host kvhost02 \
        -storagedir <PATH TO STORAGEDIR 1 ON SN2> \
        -storagedir <PATH TO STORAGEDIR 2 ON SN3> \
        -storagedir <PATH TO STORAGEDIR 3 ON SN3> \
################################################

############ Bootstrap Storage Node 3 ##########
java -jar $KVHOME/lib/kvstore.jar makebootconfig \
        -root $KVROOT\
        -store-security none \
        -capacity 3 \
        -harange 7010,7030 \
        -admin 7001 \
        -port 7000 \
        -memory_mb 200\
        -host kvhost03 \
        -storagedir <PATH TO STORAGEDIR 1 ON SN3> \
        -storagedir <PATH TO STORAGEDIR 2 ON SN3> \
        -storagedir <PATH TO STORAGEDIR 3 ON SN3> \

################################################

Step 2 : Start the Oracle NoSQL Database Storage Node Agent (SNA) on each of the
Oracle NoSQL Database nodes. The SNA manages the
Oracle NoSQL Database processes on each node.

java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT &
java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT &

Step 3 : Deploy 3x3 topology

In this step we deploy the
storage nodes and configure the topology, since Storage Node (SN1) is
already deployed, we are not deploying that again, but instead just
increasing the capacity to 3 (from 1) by using the plan change parameter
command. We also need to add the  path to the storage directory that will contain the environment
associated with a 2 additional Replication Node that would be running on SN1. Next we, deploy the new storage nodes SN2 and SN3. To change from one configuration to another, you
either create a new initial topology, or youclone an existing topology and modify it into
your target topology and then deploy this target topology- we take the clone of the current topology. Remember, we are changing the Replication Factor (RF) from 1 to 3, increase in RF creates
more copies of the data that  improves read throughput
and availability. RF is increased by using the topology change-repfactor command. For additional information on how to identify your
replication factor and its implications, seeReplication Factor.We then call the redistribute command, which will create new shards, replication nodes and move partitions, RN to re-balance the cluster. Finally, we preview and deploy the topology

java -Xmx256m -Xms256m -jar ${KVHOME}/lib/kvstore.jar runadmin -port 5000
       -host ${HOSTNAME} << EOF

plan change-parameters -service sn1 -wait -params capacity=3
plan change-storagedir -sn sn1 -storagedir<PATH TO STORAGEDIR 2 ON SN1> \ -add -wait
plan change-storagedir -sn sn1 -storagedir <PATH TO STORAGEDIR 2 ON SN1> \ -add -wait

plan deploy-sn -znname "Boston" -port 6000 -wait -host kvhost02
plan deploy-admin -sn sn2 -port 6001 -wait

plan deploy-sn -znname "Boston" -port 7000 -wait -host kvhost03
plan deploy-admin -sn sn3 -port 7001 -wait

topology clone -current -name 3x3
topology change-repfactor -name 3x3 -pool BostonPool-znname "Boston" -rf 3
topology redistribute -name 3x3 -pool BostonPool -partitions
topology preview -name 3x3
plan deploy-topology -name 3x3 -wait
show  plan
EOF

Once done. You can verify by logging into the admin console and using the show topology command

kv-> show topology;
store=kvstore  numPartitions=120 sequence=217
  zn: id=zn1 name=Boston repFactor=3 type=PRIMARY allowArbiters=false

sn=[sn1] zn:[id=zn1 name=Boston] kvhost01:5000 capacity=3 RUNNING
    [rg1-rn1] RUNNING
             single-op avg latency=1.2094289 ms   multi-op avg latency=5.4368057 ms
    [rg2-rn1] RUNNING
             single-op avg latency=0.0 ms   multi-op avg latency=7.688985 ms
    [rg3-rn1] RUNNING
             single-op avg latency=0.0 ms   multi-op avg latency=1.6008012 ms
  sn=[sn2] zn:[id=zn1 name=Boston] kvhost02:6000 capacity=3 RUNNING
    [rg1-rn2] RUNNING
             single-op avg latency=0.0 ms   multi-op avg latency=3.631618 ms
    [rg2-rn2] RUNNING
             single-op avg latency=0.0 ms   multi-op avg latency=3.1090064 ms
    [rg3-rn2] RUNNING
             single-op avg latency=1.5529954 ms   multi-op avg latency=5.964887 ms
  sn=[sn3] zn:[id=zn1 name=Boston] kvhost03:7000 capacity=3 RUNNING
    [rg1-rn3] RUNNING
             single-op avg latency=0.0 ms   multi-op avg latency=3.7453163 ms
    [rg2-rn3] RUNNING
             single-op avg latency=1.3022312 ms   multi-op avg latency=3.657524 ms
    [rg3-rn3] RUNNING
             single-op avg latency=0.0 ms   multi-op avg latency=5.796298 ms

  shard=[rg1] num partitions=40
    [rg1-rn1] sn=sn1
    [rg1-rn2] sn=sn2
    [rg1-rn3] sn=sn3
  shard=[rg2] num partitions=40
    [rg2-rn1] sn=sn1
    [rg2-rn2] sn=sn2
    [rg2-rn3] sn=sn3
  shard=[rg3] num partitions=40
    [rg3-rn1] sn=sn1
    [rg3-rn2] sn=sn2
    [rg3-rn3] sn=sn3

Be the first to comment

Comments ( 0 )
Please enter your name.Please provide a valid email address.Please enter a comment.CAPTCHA challenge response provided was incorrect. Please try again.Captcha
Oracle

Integrated Cloud Applications & Platform Services