Open in app

Sign in

Write

Sign in

Revisiting Oracle NoSQL and Docker Swarm

Marcelo Ochoa
10 min readJun 30, 2020

Some years ago I wrote an article on how to run an Oracle NoSQL cluster on Docker Swarm.

We know that Oracle NoSQL is previous to modern clustering technologies such as Docker Swarm or Kubernetes, specially because it use RMI and requires some static naming conventions on node names and registration.

But with a little trick, using bash, We can easily have NoSQL cluster up and running in a few minutes.

First we start building a custom Oracle NoSQL image extending the official provides by Oracle, here a Dockerfile:

To build our custom image simple run from command line:

$ docker build -t "oracle-nosql/net" .
$ docker image ls|grep oracle-nosql/net
oracle-nosql/net      latest     8c9784377824   3 weeks ago    571MB

or use buildDockerImage.sh script.

Note that apart from defining /data and /kvroot as persistent directories We added start-nosql.sh as custom startup script. Here the magic to use NoSQL on Swarm automatically, here the logic of the script, first create a set of directories either for KVROOT and DATA (configs files and NoSQL store files), note that We embedded host-name information in directory path.

Directory preparation

Next We have start and stop functions to operate with NoSQL

start/stop NoSQL

Below, functions to deploy admin and nodes, see below which one is used either if is a master or slave node

deploy node as admin or slave

create_bootconfig function creates Oracle NoSQL config.xml it selects which functionality to use using the environment var NODE_TYPE

config.xml creation depend on NODE_TYPE value

traps functions to properly shutdown nodes when docker stop a container and startup functionality, here again depending on NODE_TYPE value We call to deploy_admin or deploy_node functions, also We check that if config.xml file exists we assuming the stack was stop/remove and We will preserve configuration and data stored in previous start.

start functionality

Testing our cluster

Here docker-compose.yml stack for deploying our simple NoSQL cluster, it includes a master node and two slave nodes.

Only master-1 service expose to outside 5000/5001 ports to connect to our cluster. To simplify the deployment We define a service for each slave node instead of using replica=2, this is due to a RMI limitation on node naming, if We use replica 2 the DNS name for locating an specific docker node will be like nosql_slave.{TASK_ID}.jtfmewlpyq0d5rdli9j4wmnwz.nosql_mycorp, using a service for each slave node allow us to use a short host-name version (slave-1/slave-2) which is valid for RMI, adding an external name server could be other options but it will be a more complicated deployment.

Lets deploy the stack

$ docker stack deploy -c docker-compose.yml nosql
Creating network nosql_mycorp
Creating service nosql_master-1
Creating service nosql_slave-1
Creating service nosql_slave-2

To see NoSQL master log use:

$ docker service logs -f nosql_master-1 
nosql_master-1.1.zqahw3pt1ml3@pocho    | Picked up _JAVA_OPTIONS: -Djava.security.egd=file:/dev/./urandom
nosql_master-1.1.zqahw3pt1ml3@pocho    | WARNING: the -admin argument is obsolete and was benignly ignored.
nosql_master-1.1.zqahw3pt1ml3@pocho    | Picked up _JAVA_OPTIONS: -Djava.security.egd=file:/dev/./urandom
nosql_master-1.1.zqahw3pt1ml3@pocho    | Picked up _JAVA_OPTIONS: -Djava.security.egd=file:/dev/./urandom
nosql_master-1.1.zqahw3pt1ml3@pocho    | Joining as sn1 to master-1 using hostname:master-1
nosql_master-1.1.zqahw3pt1ml3@pocho    | Picked up _JAVA_OPTIONS: -Djava.security.egd=file:/dev/./urandom
nosql_master-1.1.zqahw3pt1ml3@pocho    | Connected to Admin in read-only mode
nosql_master-1.1.zqahw3pt1ml3@pocho    | Jun 30, 2020 9:44:32 PM org.jline.utils.Log logr
nosql_master-1.1.zqahw3pt1ml3@pocho    | WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
nosql_master-1.1.zqahw3pt1ml3@pocho    | kv-> Connected to Admin in read-only mode
nosql_master-1.1.zqahw3pt1ml3@pocho    | Store configured: mystore
nosql_master-1.1.zqahw3pt1ml3@pocho    | kv-> Executed plan 2, waiting for completion...
nosql_master-1.1.zqahw3pt1ml3@pocho    | Plan 2 ended successfully
nosql_master-1.1.zqahw3pt1ml3@pocho    | kv-> Executed plan 8, waiting for completion...
nosql_master-1.1.zqahw3pt1ml3@pocho    | Plan 8 ended successfully
nosql_master-1.1.zqahw3pt1ml3@pocho    | kv-> WARNING: the -port argument is obsolete and was benignly ignored.
nosql_master-1.1.zqahw3pt1ml3@pocho    | Executed plan 10, waiting for completion...
nosql_master-1.1.zqahw3pt1ml3@pocho    | Plan 10 ended successfully
nosql_master-1.1.zqahw3pt1ml3@pocho    | kv-> Added pool MyPool
nosql_master-1.1.zqahw3pt1ml3@pocho    | kv-> Added Storage Node(s) [sn1] to pool MyPool
nosql_master-1.1.zqahw3pt1ml3@pocho    | kv-> 2020-06-30 21:44:22.509 UTC INFO [snaService] Service status changed from STARTING to WAITING_FOR_DEPLOY
nosql_master-1.1.zqahw3pt1ml3@pocho    | 2020-06-30 21:44:23.015 UTC INFO [snaService] BootstrapAdmin.5000: ProcessMonitor: Picked up _JAVA_OPTIONS: -Djava.security.egd=file:/dev/./urandom
nosql_master-1.1.zqahw3pt1ml3@pocho    | 2020-06-30 21:44:23.701 UTC INFO [snaService] BootstrapAdmin.5000: ProcessMonitor: ManagedServiceStarted: BootstrapAdmin.5000
nosql_master-1.1.zqahw3pt1ml3@pocho    | 2020-06-30 21:44:33.605 UTC INFO [snaService] Register: root: /kvroot/master-1, store: mystore, hostingAdmin: true
nosql_master-1.1.zqahw3pt1ml3@pocho    | 2020-06-30 21:44:33.606 UTC INFO [snaService] System architecture is amd64
nosql_master-1.1.zqahw3pt1ml3@pocho    | 2020-06-30 21:44:33.635 UTC INFO [snaService] Starting, configuration file: /kvroot/master-1/config.xml
nosql_master-1.1.zqahw3pt1ml3@pocho    | 2020-06-30 21:44:33.646 UTC INFO [snaService] Setting java.net.InetAddress cache ttl to: networkaddress.cache.ttl=10, networkaddress.cache.negative.ttl=10
nosql_master-1.1.zqahw3pt1ml3@pocho    | 2020-06-30 21:44:33.648 UTC INFO [snaService] RMI registry serial filter is configured as: sun.rmi.registry.registryFilter=oracle.kv.**;java.lang.Enum
nosql_master-1.1.zqahw3pt1ml3@pocho    | 2020-06-30 21:44:33.648 UTC INFO [snaService] Registered startup, config file: /kvroot/master-1/mystore/sn1/config.xml
nosql_master-1.1.zqahw3pt1ml3@pocho    | 2020-06-30 21:44:33.662 UTC INFO [snaService] Changing log files to directory: /kvroot/master-1/mystore/log

If is a master node it will wait 10 seconds and then deploy_admin functionality is used, for slaves nodes We wait 10 seconds plus a random wait to avoid collisions.

To check the deployment status We can ping the cluster, let see the status

$ docker_pid=$(docker ps --filter label=com.docker.swarm.service.name=nosql_master-1 -q)
$ docker exec -t $docker_pid java -jar lib/kvstore.jar runadmin -host master-1 -port 5000 ping
Pinging components of store mystore based upon topology sequence #4
0 partitions and 3 storage nodes
Time: 2020-06-30 21:48:54 UTC   Version: 19.5.19
Shard Status: healthy:0 writable-degraded:0 read-only:0 offline:0 total:0
Admin Status: healthy
Zone [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false]   RN Status: online:0 read-only:0 offline:0
Storage Node [sn1] on master-1:5000    Zone: [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false]    Status: RUNNING   Ver: 19.5.19 2020-01-27 07:15:29 UTC  Build id: 6783109c3c07 Edition: Community
 Admin [admin1]  Status: RUNNING,MASTER
Storage Node [sn2] on slave-2:5000    Zone: [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false]    Status: RUNNING   Ver: 19.5.19 2020-01-27 07:15:29 UTC  Build id: 6783109c3c07 Edition: Community
Storage Node [sn3] on slave-1:5000    Zone: [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false]    Status: RUNNING   Ver: 19.5.19 2020-01-27 07:15:29 UTC  Build id: 6783109c3c07 Edition: Community

Now We have a NoSQL cluster up and running lets deploy a topology on it using deploy-topo.sh script, it parses line by line script-topo.txt and send each NoSQL command to our cluster, here the output:

$ ./deploy-topo.sh
Created: MyStoreLayout
Topology transformation from current deployed topology to MyStoreLayout:
Create 3 shards 
Create 9 RNs 
Create 10 partitionsshard rg1
  3 new RNs : rg1-rn1 rg1-rn2 rg1-rn3 
  4 new partitions 
shard rg2
  3 new RNs : rg2-rn1 rg2-rn2 rg2-rn3 
  3 new partitions 
shard rg3
  3 new RNs : rg3-rn1 rg3-rn2 rg3-rn3 
  3 new partitions
Executed plan 15, waiting for completion...
Plan 15 ended successfully
     2 Deploy Zone              SUCCEEDED
     8 Deploy Storage Node      SUCCEEDED
    10 Deploy Admin Service     SUCCEEDED
    11 Deploy Storage Node      SUCCEEDED
    12 Deploy Storage Node      SUCCEEDED
    15 Deploy Topo              SUCCEEDED
store=mystore  numPartitions=10 sequence=26
  zn: id=zn1 name=MyZone repFactor=3 type=PRIMARY allowArbiters=false masterAffinity=falsesn=[sn1] zn:[id=zn1 name=MyZone] master-1:5000 capacity=3 RUNNING
    [rg1-rn1] RUNNING
          No performance info available
    [rg2-rn1] RUNNING
          No performance info available
    [rg3-rn1] RUNNING
          No performance info available
  sn=[sn2] zn:[id=zn1 name=MyZone] slave-2:5000 capacity=3 RUNNING
    [rg1-rn2] RUNNING
          No performance info available
    [rg2-rn2] RUNNING
          No performance info available
    [rg3-rn2] RUNNING
          No performance info available
  sn=[sn3] zn:[id=zn1 name=MyZone] slave-1:5000 capacity=3 RUNNING
    [rg1-rn3] RUNNING
          No performance info available
    [rg2-rn3] RUNNING
          No performance info available
    [rg3-rn3] RUNNING
          No performance info availablenumShards=3
  shard=[rg1] num partitions=4
    [rg1-rn1] sn=sn1
    [rg1-rn2] sn=sn2
    [rg1-rn3] sn=sn3
  shard=[rg2] num partitions=3
    [rg2-rn1] sn=sn1
    [rg2-rn2] sn=sn2
    [rg2-rn3] sn=sn3
  shard=[rg3] num partitions=3
    [rg3-rn1] sn=sn1
    [rg3-rn2] sn=sn2
    [rg3-rn3] sn=sn3
Verify: starting verification of store mystore based upon topology sequence #26
10 partitions and 3 storage nodes
Time: 2020-06-30 21:52:23 UTC   Version: 19.5.19
See master-1:/kvroot/master-1/mystore/log/mystore_{0..N}.log for progress messages
Verify: Shard Status: healthy:3 writable-degraded:0 read-only:0 offline:0 total:3
Verify: Admin Status: healthy
Verify: Zone [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false]   RN Status: online:9 read-only:0 offline:0 maxDelayMillis:1 maxCatchupTimeSecs:0
Verify: == checking storage node sn1 ==
Verify: Storage Node [sn1] on master-1:5000    Zone: [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false]    Status: RUNNING   Ver: 19.5.19 2020-01-27 07:15:29 UTC  Build id: 6783109c3c07 Edition: Community
Verify:  Admin [admin1]  Status: RUNNING,MASTER
Verify:         rg1-rn1: Storage directory on rg1-rn1 is running low [/data/master-1/1 size:1 GB used:17 KB]
Verify:  Rep Node [rg1-rn1] Status: RUNNING,REPLICA sequenceNumber:42 haPort:5011 available storage size:1023 MB delayMillis:0 catchupTimeSecs:0
Verify:         rg2-rn1: Storage directory on rg2-rn1 is running low [/data/master-1/2 size:1 GB used:15 KB]
Verify:  Rep Node [rg2-rn1] Status: RUNNING,REPLICA sequenceNumber:40 haPort:5012 available storage size:1023 MB delayMillis:1 catchupTimeSecs:0
Verify:         rg3-rn1: Storage directory on rg3-rn1 is running low [/data/master-1/3 size:1 GB used:15 KB]
Verify:  Rep Node [rg3-rn1] Status: RUNNING,MASTER sequenceNumber:39 haPort:5013 available storage size:1023 MB
Verify: == checking storage node sn2 ==
Verify: Storage Node [sn2] on slave-2:5000    Zone: [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false]    Status: RUNNING   Ver: 19.5.19 2020-01-27 07:15:29 UTC  Build id: 6783109c3c07 Edition: Community
Verify:         rg1-rn2: Storage directory on rg1-rn2 is running low [/data/slave-2/1 size:1 GB used:14 KB]
Verify:  Rep Node [rg1-rn2] Status: RUNNING,MASTER sequenceNumber:42 haPort:5010 available storage size:1023 MB
Verify:         rg2-rn2: Storage directory on rg2-rn2 is running low [/data/slave-2/2 size:1 GB used:14 KB]
Verify:  Rep Node [rg2-rn2] Status: RUNNING,REPLICA sequenceNumber:40 haPort:5011 available storage size:1023 MB delayMillis:1 catchupTimeSecs:0
Verify:         rg3-rn2: Storage directory on rg3-rn2 is running low [/data/slave-2/3 size:1 GB used:15 KB]
Verify:  Rep Node [rg3-rn2] Status: RUNNING,REPLICA sequenceNumber:39 haPort:5012 available storage size:1023 MB delayMillis:1 catchupTimeSecs:0
Verify: == checking storage node sn3 ==
Verify: Storage Node [sn3] on slave-1:5000    Zone: [name=MyZone id=zn1 type=PRIMARY allowArbiters=false masterAffinity=false]    Status: RUNNING   Ver: 19.5.19 2020-01-27 07:15:29 UTC  Build id: 6783109c3c07 Edition: Community
Verify:         rg1-rn3: Storage directory on rg1-rn3 is running low [/data/slave-1/1 size:1 GB used:1 KB]
Verify:  Rep Node [rg1-rn3] Status: RUNNING,REPLICA sequenceNumber:42 haPort:5010 available storage size:1023 MB delayMillis:0 catchupTimeSecs:0
Verify:         rg2-rn3: Storage directory on rg2-rn3 is running low [/data/slave-1/2 size:1 GB used:1 KB]
Verify:  Rep Node [rg2-rn3] Status: RUNNING,MASTER sequenceNumber:40 haPort:5011 available storage size:1023 MB
Verify:         rg3-rn3: Storage directory on rg3-rn3 is running low [/data/slave-1/3 size:1 GB used:1 KB]
Verify:  Rep Node [rg3-rn3] Status: RUNNING,REPLICA sequenceNumber:39 haPort:5012 available storage size:1023 MB delayMillis:1 catchupTimeSecs:0Verification complete, 1 violation, 9 notes found.
Verification violation: [zn1] zn1 needs 2 Admins to meet the required repFactor of 3
Verification note: [rg1-rn1] Storage directory on rg1-rn1 is running low [/data/master-1/1 size:1 GB used:17 KB]
Verification note: [rg1-rn2] Storage directory on rg1-rn2 is running low [/data/slave-2/1 size:1 GB used:14 KB]
Verification note: [rg1-rn3] Storage directory on rg1-rn3 is running low [/data/slave-1/1 size:1 GB used:1 KB]
Verification note: [rg2-rn1] Storage directory on rg2-rn1 is running low [/data/master-1/2 size:1 GB used:15 KB]
Verification note: [rg2-rn2] Storage directory on rg2-rn2 is running low [/data/slave-2/2 size:1 GB used:14 KB]
Verification note: [rg2-rn3] Storage directory on rg2-rn3 is running low [/data/slave-1/2 size:1 GB used:1 KB]
Verification note: [rg3-rn1] Storage directory on rg3-rn1 is running low [/data/master-1/3 size:1 GB used:15 KB]
Verification note: [rg3-rn2] Storage directory on rg3-rn2 is running low [/data/slave-2/3 size:1 GB used:15 KB]
Verification note: [rg3-rn3] Storage directory on rg3-rn3 is running low [/data/slave-1/3 size:1 GB used:1 KB]

We defined our persistent store, self managed by Docker, or could be defined to an specific directory as

local directory

or a NFS backed storage.

Put some data in our cluster

A simple way to check our store is using runadmin utility

$ docker exec -ti $docker_pid java -jar lib/kvstore.jar runadmin -host master-1 -port 5000
Picked up _JAVA_OPTIONS: -Djava.security.egd=file:/dev/./urandom
kv-> connect store -host master-1 -port 5000 -name mystore
Connected to mystore at master-1:5000.
kv-> put kv -key /test -value val
Operation successful, record inserted.
kv-> get kv -key /test
val
kv-> exit

If We want to verify that our data survives a docker stack removal, lets check

$ docker stack rm nosql 
Removing service nosql_master-1
Removing service nosql_slave-1
Removing service nosql_slave-2
Removing network nosql_mycorp

Re-deploy the stack without removing persistent volumes

$ docker ps
CONTAINER ID   IMAGE    COMMAND   CREATED    STATUS    PORTS   NAMES
$ docker stack deploy -c docker-compose.yml nosql
Creating network nosql_mycorp
Creating service nosql_master-1
Creating service nosql_slave-1
Creating service nosql_slave-2
$ docker_pid=$(docker ps --filter label=com.docker.swarm.service.name=nosql_master-1 -q)
$ docker exec -ti $docker_pid java -jar lib/kvstore.jar runadmin -host master-1 -port 5000
kv-> connect store -host master-1 -port 5000 -name mystore
Connected to mystore at master-1:5000.
kv-> get kv -key /test
val

It works!!!! Have fun using Oracle NoSQL on Docker Swarm.

Oracle
NoSQL
Docker Swarm

--

--

Marcelo Ochoa

Written by Marcelo Ochoa

148 Followers

https://apexapps.oracle.com/pls/apex/r/ace_program/oracle-aces/ace?ace_id=240&clear=2

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams