Revisiting Oracle NoSQL and Docker Swarm

Some years ago I wrote an article on how to run an Oracle NoSQL cluster on Docker Swarm.

We know that Oracle NoSQL is previous to modern clustering technologies such as Docker Swarm or Kubernetes, specially because it use RMI and requires some static naming conventions on node names and registration.

But with a little trick, using bash, We can easily have NoSQL cluster up and running in a few minutes.

First we start building a custom Oracle NoSQL image extending the official provides by Oracle, here a Dockerfile:

To build our custom image simple run from command line:

$ docker build -t "oracle-nosql/net" .
$ docker image ls|grep oracle-nosql/net
oracle-nosql/net latest 8c9784377824 3 weeks ago 571MB

or use buildDockerImage.sh script.

Note that apart from defining /data and /kvroot as persistent directories We added start-nosql.sh as custom startup script. Here the magic to use NoSQL on Swarm automatically, here the logic of the script, first create a set of directories either for KVROOT and DATA (configs files and NoSQL store files), note that We embedded host-name information in directory path.

Directory preparation

Next We have start and stop functions to operate with NoSQL

start/stop NoSQL

Below, functions to deploy admin and nodes, see below which one is used either if is a master or slave node

deploy node as admin or slave

create_bootconfig function creates Oracle NoSQL config.xml it selects which functionality to use using the environment var NODE_TYPE

config.xml creation depend on NODE_TYPE value

traps functions to properly shutdown nodes when docker stop a container and startup functionality, here again depending on NODE_TYPE value We call to deploy_admin or deploy_node functions, also We check that if config.xml file exists we assuming the stack was stop/remove and We will preserve configuration and data stored in previous start.

start functionality

Testing our cluster

Here docker-compose.yml stack for deploying our simple NoSQL cluster, it includes a master node and two slave nodes.

Only master-1 service expose to outside 5000/5001 ports to connect to our cluster. To simplify the deployment We define a service for each slave node instead of using replica=2, this is due to a RMI limitation on node naming, if We use replica 2 the DNS name for locating an specific docker node will be like nosql_slave.{TASK_ID}.jtfmewlpyq0d5rdli9j4wmnwz.nosql_mycorp, using a service for each slave node allow us to use a short host-name version (slave-1/slave-2) which is valid for RMI, adding an external name server could be other options but it will be a more complicated deployment.

Lets deploy the stack

$ docker stack deploy -c docker-compose.yml nosql
Creating network nosql_mycorp
Creating service nosql_master-1
Creating service nosql_slave-1
Creating service nosql_slave-2

To see NoSQL master log use:

$ docker service logs -f nosql_master-1 
nosql_master-1.1.zqahw3pt1ml3@pocho | Picked up _JAVA_OPTIONS: -Djava.security.egd=file:/dev/./urandom
nosql_master-1.1.zqahw3pt1ml3@pocho | WARNING: the -admin argument is obsolete and was benignly ignored.
nosql_master-1.1.zqahw3pt1ml3@pocho | Picked up _JAVA_OPTIONS: -Djava.security.egd=file:/dev/./urandom
nosql_master-1.1.zqahw3pt1ml3@pocho | Picked up _JAVA_OPTIONS: -Djava.security.egd=file:/dev/./urandom
nosql_master-1.1.zqahw3pt1ml3@pocho | Joining as sn1 to master-1 using hostname:master-1
nosql_master-1.1.zqahw3pt1ml3@pocho | Picked up _JAVA_OPTIONS: -Djava.security.egd=file:/dev/./urandom
nosql_master-1.1.zqahw3pt1ml3@pocho | Connected to Admin in read-only mode
nosql_master-1.1.zqahw3pt1ml3@pocho | Jun 30, 2020 9:44:32 PM org.jline.utils.Log logr
nosql_master-1.1.zqahw3pt1ml3@pocho | WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
nosql_master-1.1.zqahw3pt1ml3@pocho | kv-> Connected to Admin in read-only mode
nosql_master-1.1.zqahw3pt1ml3@pocho | Store configured: mystore
nosql_master-1.1.zqahw3pt1ml3@pocho | kv-> Executed plan 2, waiting for completion...
nosql_master-1.1.zqahw3pt1ml3@pocho | Plan 2 ended successfully
nosql_master-1.1.zqahw3pt1ml3@pocho | kv-> Executed plan 8, waiting for completion...
nosql_master-1.1.zqahw3pt1ml3@pocho | Plan 8 ended successfully
nosql_master-1.1.zqahw3pt1ml3@pocho | kv-> WARNING: the -port argument is obsolete and was benignly ignored.
nosql_master-1.1.zqahw3pt1ml3@pocho | Executed plan 10, waiting for completion...
nosql_master-1.1.zqahw3pt1ml3@pocho | Plan 10 ended successfully
nosql_master-1.1.zqahw3pt1ml3@pocho | kv-> Added pool MyPool
nosql_master-1.1.zqahw3pt1ml3@pocho | kv-> Added Storage Node(s) [sn1] to pool MyPool
nosql_master-1.1.zqahw3pt1ml3@pocho | kv-> 2020-06-30 21:44:22.509 UTC INFO [snaService] Service status changed from STARTING to WAITING_FOR_DEPLOY
nosql_master-1.1.zqahw3pt1ml3@pocho | 2020-06-30 21:44:23.015 UTC INFO [snaService] BootstrapAdmin.5000: ProcessMonitor: Picked up _JAVA_OPTIONS: -Djava.security.egd=file:/dev/./urandom
nosql_master-1.1.zqahw3pt1ml3@pocho | 2020-06-30 21:44:23.701 UTC INFO [snaService] BootstrapAdmin.5000: ProcessMonitor: ManagedServiceStarted: BootstrapAdmin.5000
nosql_master-1.1.zqahw3pt1ml3@pocho | 2020-06-30 21:44:33.605 UTC INFO [snaService] Register: root: /kvroot/master-1, store: mystore, hostingAdmin: true
nosql_master-1.1.zqahw3pt1ml3@pocho | 2020-06-30 21:44:33.606 UTC INFO [snaService] System architecture is amd64
nosql_master-1.1.zqahw3pt1ml3@pocho | 2020-06-30 21:44:33.635 UTC INFO [snaService] Starting, configuration file: /kvroot/master-1/config.xml
nosql_master-1.1.zqahw3pt1ml3@pocho | 2020-06-30 21:44:33.646 UTC INFO [snaService] Setting java.net.InetAddress cache ttl to: networkaddress.cache.ttl=10, networkaddress.cache.negative.ttl=10
nosql_master-1.1.zqahw3pt1ml3@pocho | 2020-06-30 21:44:33.648 UTC INFO [snaService] RMI registry serial filter is configured as: sun.rmi.registry.registryFilter=oracle.kv.**;java.lang.Enum
nosql_master-1.1.zqahw3pt1ml3@pocho | 2020-06-30 21:44:33.648 UTC INFO [snaService] Registered startup, config file: /kvroot/master-1/mystore/sn1/config.xml
nosql_master-1.1.zqahw3pt1ml3@pocho | 2020-06-30 21:44:33.662 UTC INFO [snaService] Changing log files to directory: /kvroot/master-1/mystore/log

If is a master node it will wait 10 seconds and then deploy_admin functionality is used, for slaves nodes We wait 10 seconds plus a random wait to avoid collisions.

To check the deployment status We can ping the cluster, let see the status

$ docker_pid=$(docker ps --filter label=com.docker.swarm.service.name=nosql_master-1 -q)
$ docker exec -t $docker_pid java -jar lib/kvstore.jar runadmin -host master-1 -port 5000 ping
Pinging components of store mystore based upon topology sequence #4
0 partitions and 3 storage nodes
Time: 2020-06-30 21:48:54 UTC Version: 19.5.19
Shard Status: healthy:0 writable-degraded:0 read-only:0 offline:0 total:0
Admin Status: healthy
Zone [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false] RN Status: online:0 read-only:0 offline:0
Storage Node [sn1] on master-1:5000 Zone: [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false] Status: RUNNING Ver: 19.5.19 2020-01-27 07:15:29 UTC Build id: 6783109c3c07 Edition: Community
Admin [admin1] Status: RUNNING,MASTER
Storage Node [sn2] on slave-2:5000 Zone: [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false] Status: RUNNING Ver: 19.5.19 2020-01-27 07:15:29 UTC Build id: 6783109c3c07 Edition: Community
Storage Node [sn3] on slave-1:5000 Zone: [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false] Status: RUNNING Ver: 19.5.19 2020-01-27 07:15:29 UTC Build id: 6783109c3c07 Edition: Community

Now We have a NoSQL cluster up and running lets deploy a topology on it using deploy-topo.sh script, it parses line by line script-topo.txt and send each NoSQL command to our cluster, here the output:

$ ./deploy-topo.sh
Created: MyStoreLayout
Topology transformation from current deployed topology to MyStoreLayout:
Create 3 shards
Create 9 RNs
Create 10 partitions
shard rg1
3 new RNs : rg1-rn1 rg1-rn2 rg1-rn3
4 new partitions
shard rg2
3 new RNs : rg2-rn1 rg2-rn2 rg2-rn3
3 new partitions
shard rg3
3 new RNs : rg3-rn1 rg3-rn2 rg3-rn3
3 new partitions

Executed plan 15, waiting for completion...
Plan 15 ended successfully
2 Deploy Zone SUCCEEDED
8 Deploy Storage Node SUCCEEDED
10 Deploy Admin Service SUCCEEDED
11 Deploy Storage Node SUCCEEDED
12 Deploy Storage Node SUCCEEDED
15 Deploy Topo SUCCEEDED

store=mystore numPartitions=10 sequence=26
zn: id=zn1 name=MyZone repFactor=3 type=PRIMARY allowArbiters=false masterAffinity=false
sn=[sn1] zn:[id=zn1 name=MyZone] master-1:5000 capacity=3 RUNNING
[rg1-rn1] RUNNING
No performance info available
[rg2-rn1] RUNNING
No performance info available
[rg3-rn1] RUNNING
No performance info available
sn=[sn2] zn:[id=zn1 name=MyZone] slave-2:5000 capacity=3 RUNNING
[rg1-rn2] RUNNING
No performance info available
[rg2-rn2] RUNNING
No performance info available
[rg3-rn2] RUNNING
No performance info available
sn=[sn3] zn:[id=zn1 name=MyZone] slave-1:5000 capacity=3 RUNNING
[rg1-rn3] RUNNING
No performance info available
[rg2-rn3] RUNNING
No performance info available
[rg3-rn3] RUNNING
No performance info available
numShards=3
shard=[rg1] num partitions=4
[rg1-rn1] sn=sn1
[rg1-rn2] sn=sn2
[rg1-rn3] sn=sn3
shard=[rg2] num partitions=3
[rg2-rn1] sn=sn1
[rg2-rn2] sn=sn2
[rg2-rn3] sn=sn3
shard=[rg3] num partitions=3
[rg3-rn1] sn=sn1
[rg3-rn2] sn=sn2
[rg3-rn3] sn=sn3

Verify: starting verification of store mystore based upon topology sequence #26
10 partitions and 3 storage nodes
Time: 2020-06-30 21:52:23 UTC Version: 19.5.19
See master-1:/kvroot/master-1/mystore/log/mystore_{0..N}.log for progress messages
Verify: Shard Status: healthy:3 writable-degraded:0 read-only:0 offline:0 total:3
Verify: Admin Status: healthy
Verify: Zone [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false] RN Status: online:9 read-only:0 offline:0 maxDelayMillis:1 maxCatchupTimeSecs:0
Verify: == checking storage node sn1 ==
Verify: Storage Node [sn1] on master-1:5000 Zone: [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false] Status: RUNNING Ver: 19.5.19 2020-01-27 07:15:29 UTC Build id: 6783109c3c07 Edition: Community
Verify: Admin [admin1] Status: RUNNING,MASTER
Verify: rg1-rn1: Storage directory on rg1-rn1 is running low [/data/master-1/1 size:1 GB used:17 KB]
Verify: Rep Node [rg1-rn1] Status: RUNNING,REPLICA sequenceNumber:42 haPort:5011 available storage size:1023 MB delayMillis:0 catchupTimeSecs:0
Verify: rg2-rn1: Storage directory on rg2-rn1 is running low [/data/master-1/2 size:1 GB used:15 KB]
Verify: Rep Node [rg2-rn1] Status: RUNNING,REPLICA sequenceNumber:40 haPort:5012 available storage size:1023 MB delayMillis:1 catchupTimeSecs:0
Verify: rg3-rn1: Storage directory on rg3-rn1 is running low [/data/master-1/3 size:1 GB used:15 KB]
Verify: Rep Node [rg3-rn1] Status: RUNNING,MASTER sequenceNumber:39 haPort:5013 available storage size:1023 MB
Verify: == checking storage node sn2 ==
Verify: Storage Node [sn2] on slave-2:5000 Zone: [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false] Status: RUNNING Ver: 19.5.19 2020-01-27 07:15:29 UTC Build id: 6783109c3c07 Edition: Community
Verify: rg1-rn2: Storage directory on rg1-rn2 is running low [/data/slave-2/1 size:1 GB used:14 KB]
Verify: Rep Node [rg1-rn2] Status: RUNNING,MASTER sequenceNumber:42 haPort:5010 available storage size:1023 MB
Verify: rg2-rn2: Storage directory on rg2-rn2 is running low [/data/slave-2/2 size:1 GB used:14 KB]
Verify: Rep Node [rg2-rn2] Status: RUNNING,REPLICA sequenceNumber:40 haPort:5011 available storage size:1023 MB delayMillis:1 catchupTimeSecs:0
Verify: rg3-rn2: Storage directory on rg3-rn2 is running low [/data/slave-2/3 size:1 GB used:15 KB]
Verify: Rep Node [rg3-rn2] Status: RUNNING,REPLICA sequenceNumber:39 haPort:5012 available storage size:1023 MB delayMillis:1 catchupTimeSecs:0
Verify: == checking storage node sn3 ==
Verify: Storage Node [sn3] on slave-1:5000 Zone: [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false] Status: RUNNING Ver: 19.5.19 2020-01-27 07:15:29 UTC Build id: 6783109c3c07 Edition: Community
Verify: rg1-rn3: Storage directory on rg1-rn3 is running low [/data/slave-1/1 size:1 GB used:1 KB]
Verify: Rep Node [rg1-rn3] Status: RUNNING,REPLICA sequenceNumber:42 haPort:5010 available storage size:1023 MB delayMillis:0 catchupTimeSecs:0
Verify: rg2-rn3: Storage directory on rg2-rn3 is running low [/data/slave-1/2 size:1 GB used:1 KB]
Verify: Rep Node [rg2-rn3] Status: RUNNING,MASTER sequenceNumber:40 haPort:5011 available storage size:1023 MB
Verify: rg3-rn3: Storage directory on rg3-rn3 is running low [/data/slave-1/3 size:1 GB used:1 KB]
Verify: Rep Node [rg3-rn3] Status: RUNNING,REPLICA sequenceNumber:39 haPort:5012 available storage size:1023 MB delayMillis:1 catchupTimeSecs:0
Verification complete, 1 violation, 9 notes found.
Verification violation: [zn1] zn1 needs 2 Admins to meet the required repFactor of 3
Verification note: [rg1-rn1] Storage directory on rg1-rn1 is running low [/data/master-1/1 size:1 GB used:17 KB]
Verification note: [rg1-rn2] Storage directory on rg1-rn2 is running low [/data/slave-2/1 size:1 GB used:14 KB]
Verification note: [rg1-rn3] Storage directory on rg1-rn3 is running low [/data/slave-1/1 size:1 GB used:1 KB]
Verification note: [rg2-rn1] Storage directory on rg2-rn1 is running low [/data/master-1/2 size:1 GB used:15 KB]
Verification note: [rg2-rn2] Storage directory on rg2-rn2 is running low [/data/slave-2/2 size:1 GB used:14 KB]
Verification note: [rg2-rn3] Storage directory on rg2-rn3 is running low [/data/slave-1/2 size:1 GB used:1 KB]
Verification note: [rg3-rn1] Storage directory on rg3-rn1 is running low [/data/master-1/3 size:1 GB used:15 KB]
Verification note: [rg3-rn2] Storage directory on rg3-rn2 is running low [/data/slave-2/3 size:1 GB used:15 KB]
Verification note: [rg3-rn3] Storage directory on rg3-rn3 is running low [/data/slave-1/3 size:1 GB used:1 KB]

We defined our persistent store, self managed by Docker, or could be defined to an specific directory as

local directory

or a NFS backed storage.

Put some data in our cluster

A simple way to check our store is using runadmin utility

$ docker exec -ti $docker_pid java -jar lib/kvstore.jar runadmin -host master-1 -port 5000
Picked up _JAVA_OPTIONS: -Djava.security.egd=file:/dev/./urandom
kv-> connect store -host master-1 -port 5000 -name mystore
Connected to mystore at master-1:5000.
kv-> put kv -key /test -value val
Operation successful, record inserted.
kv-> get kv -key /test
val
kv-> exit

If We want to verify that our data survives a docker stack removal, lets check

$ docker stack rm nosql 
Removing service nosql_master-1
Removing service nosql_slave-1
Removing service nosql_slave-2
Removing network nosql_mycorp

Re-deploy the stack without removing persistent volumes

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ docker stack deploy -c docker-compose.yml nosql
Creating network nosql_mycorp
Creating service nosql_master-1
Creating service nosql_slave-1
Creating service nosql_slave-2
$ docker_pid=$(docker ps --filter label=com.docker.swarm.service.name=nosql_master-1 -q)
$ docker exec -ti $docker_pid java -jar lib/kvstore.jar runadmin -host master-1 -port 5000
kv-> connect store -host master-1 -port 5000 -name mystore
Connected to mystore at master-1:5000.
kv-> get kv -key /test
val

It works!!!! Have fun using Oracle NoSQL on Docker Swarm.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store