pcdunio MongoDB Cluster

This years X-mas project was to build a little ARM based cluster of pcdunio (Raspberry Pie 2+ like) boards running a MongoDB 3.0 NoSQL database for under $200.

Hardware

The base board was a pcDunio3 nano lite that was a mere $15 on Amazon and is more powerful and has better ethernet connectivity than a RP2.

A stack of 4 of these boards needs to be build, and I opted to use a laser cut acrylic plate (had some left) and some M2 and M3 nylon spacers for the job. Below is the progression from circuitboard scan, to final cut plate. SVG files for laser cutters are here and here.

The boards were equiped with a 16GB UHCI SD card for the OS and a 8GB USB2 flash drive for swap space. The stack was wired with a 5-port switch to the LAN and a 4-port charge station supplied power.

The final stack looked like this:

Update (1/6/2015)

Added also a smallish (aka cheap) SSD for good measures; mostly to see if the performance is any better than the SD card or USB (of course it is).

Software

The OS installed was armbian (latest Debian version is fine, since I did not care about the missing accelerated graphics support), the database was a self-compiled MongoDB3.0 (vs. the stock 2.4), and rockmongo was used as web based UI for the database.

BOM

… under $200 🙂

+ $26 for the SSD (cables I had laying around)

Installation Steps

– local network is DHCP based and 192.168.1.x
– download and flash Armbian_4.5_Pcduino3nano_Debian_jessie_4.2.3 to SD card using win32diskimager-v0.7-binary

– boot and wait

– log into initial user using ssh (or putty)
root/1234

– edit hostname and set it to cluster[#]

# install preferred editor, here: joe
apt-get -y install joe

joe /etc/hostname
cluster[#]

joe /etc/hosts
127.0.0.1   localhost cluster[#]
::1         localhost cluster[#] ip6-localhost ip6-loopback
fe00::0     ip6-localnet
ff00::0     ip6-mcastprefix
ff02::1     ip6-allnodes
ff02::2     ip6-allrouters
192.168.1.11 cluster1
192.168.1.12 cluster2
192.168.1.13 cluster3
192.168.1.14 cluster4

joe /etc/network/interfaces
iface eth0 inet static
address 192.168.1.1[#]
netmask 255.255.255.0
gateway 192.168.1.1

– Add user

adduser aschiffler
usermod -aG sudo aschiffler

– Initial kernel upgrade:
apt-get -y update
apt-get -y upgrade

– Prep swap

# Swap on SD card (file)
#dd if=/dev/zero of=/swapfile4G bs=4096 count=1024000
#mkswap /swapfile4G
#chmod 0600 /swapfile4G
#swapon /swapfile4G

# Swap on USB drive (primary partition, type 82)
fdisk /dev/sda
# optional slow wipe: dd if=/dev/zero of=/dev/sda1
mkswap /dev/sda1
swapon /dev/sda1 # add to startup /etc/rc.local

– Activate changes

reboot

– Install default MongoDB (2.4)

apt-get -y install mongodb mongodb-server

– switch to custom version of MongoDB (3.0, see below on how to compile the new version)

#/root/mongodb/bin/mongo
#/root/mongodb/bin/mongod
#/root/mongodb/bin/mongoperf
#/root/mongodb/bin/mongos
# from
#/usr/bin/mongo
#/usr/bin/mongod
#/usr/bin/mongoperf
#/usr/bin/mongos

/etc/init.d/mongodb stop

cp /usr/bin/mongo /usr/bin/mongo.orig
cp /usr/bin/mongod /usr/bin/mongod.orig
cp /usr/bin/mongoperf /usr/bin/mongoperf.orig
cp /usr/bin/mongos /usr/bin/mongos.orig

mv /var/lib/mongodb /var/lib/mongodb.orig

cp -f /root/mongodb/bin/mongo /usr/bin/mongo
cp -f /root/mongodb/bin/mongod /usr/bin/mongod
cp -f /root/mongodb/bin/mongoperf /usr/bin/mongoperf
cp -f /root/mongodb/bin/mongos /usr/bin/mongos

mkdir /var/lib/mongodb
chown mongodb.mongodb /var/lib/mongodb

/etc/init.d/mongodb start

– Add a user to MonoDB

joe /etc/mongodb.conf
auth = false

/etc/init.d/mongodb restart

mongo

# Mongo 2.4 (old, see  also https://docs.mongodb.org/v2.4/tutorial/add-user-to-database/)
#use admin
#db.system.users.remove(
#   { user:”aschiffler” }
#)
#db.addUser( { user:”aschiffler”, pwd:”***EDITME***”, roles: [ ‘clusterAdmin’, ‘userAdminAnyDatabase’, ‘readAnyDatabase’ ] } )
#db.system.users.find()
#exit

# Mongo 3.0
use admin
db.system.version.remove({ “_id” : “authSchema” })
db.system.version.insert({ “_id” : “authSchema”, “currentVersion” : 3 })
db.system.users.remove({ user:”aschiffler” })
db.createUser( { user:”aschiffler”, pwd:”***EDITME***”, roles: [ ‘clusterAdmin’, ‘userAdminAnyDatabase’, ‘readAnyDatabase’ ] } )
db.system.users.find()
exit

joe /etc/mongodb.conf
auth = true

/etc/init.d/mongodb restart

– Install rockmongo (cluster1)

apt-get install apache2 php5 php-pear

apt-get install php5-mongo
cd /var/www/html
tar xvf ~aschiffler/rockmongo-1.1.7.tar
ln -s rockmongo-1.1.7 rockmongo
/etc/init.d/apache2 restart

joe /var/www/html/rockmongo/config.php
$MONGO[“servers”][$i][“mongo_name”] = “cluster1”;//mongo server name
$MONGO[“servers”][$i][“mongo_host”] = “cluster1”;//mongo host
$MONGO[“servers”][$i][“mongo_port”] = “27017”;//mongo port
$MONGO[“servers”][$i][“mongo_timeout”] = 0;//mongo connection timeout
$MONGO[“servers”][$i][“mongo_auth”] = true;

– configure firewall and bind

joe /etc/rc.local
iptables -A INPUT -s 192.168.1.0/24 -p tcp –destination-port 27017 -m state –state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -d 192.168.1.0/24 -p tcp –source-port 27017 -m state –state ESTABLISHED -j ACCEPT

. /etc/rc.local

joe /etc/mongodb.conf
bind_ip = 0.0.0.0

/etc/init.d/mongodb restart

– prepare replication

cp /etc/mongodb.conf /etc/mongodb.conf.orig
joe /etc/mongodb.conf

systemLog:
destination: file
path: “/var/log/mongodb/mongodb.log”
logAppend: true
processManagement:
fork: false
net:
bindIp: 0.0.0.0
port: 27017
storage:
dbPath: “/var/lib/mongodb”
journal:
enabled: false
engine: “mmapv1”
mmapv1:
smallFiles: true
replication:
replSetName: “rs1”
setParameter:
enableLocalhostAuthBypass: false

/etc/init.d/mongodb restart

– enable replication

mongo cluster1/admin -u aschiffler –p ***EDITME*** –authenticationMechanism MONGODB-CR

rs.initiate()
rs.conf()

rs.add(“cluster2”)
rs.add(“cluster3”)
rs.add(“cluster4”)
rs.status()

MongoDB 3.0 on ARM

These steps are based on the steps provided on:
http://andyfelong.com/2015/12/mongodb-3-0-7-on-raspberry-pi-2/

– login as user

sudo bash

# — ensure we have 4G+ swap

apt-get -y install build-essential libssl-dev git scons libboost-filesystem-dev libboost-program-options-dev libboost-system-dev libboost-thread-dev

cd
mkdir build_mongodb
cd build_mongodb/
git clone git://github.com/mongodb/mongo.git
cd mongo
git checkout r3.0.8

– patch data_view.h

cd ~/build_mongodb/mongo
joe src/mongo/base/data_view.h

and add
#define MONGO_HAVE_STD_IS_TRIVIALLY_COPYABLE 0
or
#undef MONGO_HAVE_STD_IS_TRIVIALLY_COPYABLE

– patch SConscript

cd ~/build_mongodb/mongo/src/third_party/v8-3.25/
# wget http://andyfelong.com/wp-content/uploads/2015/12/SConscript
wget https://gist.githubusercontent.com/kitsook/f0f53bc7acc468b6e94c/raw/93ebc8dc0adf7afb0a38c1b6bf702f8a8c6b70c2/SConscript
mv -f SConscript SConscript.old && mv -f SConscript.1 SConscript

– patch boost

cd ~/build_mongodb/mongo/src/third_party/boost/boost/smart_ptr/detail/
wget “https://svn.boost.org/trac/boost/changeset/77315?format=diff&new=77315″ -O patch.diff
patch <patch.diff

– update linux_syscall_support

cd ~/build_mongodb/mongo/src/third_party/gperftools-2.2/src
wget https://raw.githubusercontent.com/plexinc/breakpad-lss/master/linux_syscall_support.h
mv -f linux_syscall_support.h linux_syscall_support.h.old && mv -f linux_syscall_support.h.1 linux_syscall_support.h

– build (takes hours)

cd ~/build_mongodb/mongo
scons -j 1 –ssl –wiredtiger=off –js-engine=v8-3.25 –c++11=on –disable-warnings-as-errors CXXFLAGS=”-std=gnu++11″ –allocator=system core

– install (takes hours)

cd ~/build_mongodb/mongo
scons -j 1 –ssl –wiredtiger=off –js-engine=v8-3.25 –c++11=on –disable-warnings-as-errors CXXFLAGS=”-std=gnu++11″ –allocator=system –prefix=/root/mongodb install

– check binaries

cd ~/mongodb/bin
ls -l

– clean up

rm -rf ~/build_mongodb

 

Benchmark Results

CPU

sysbench –test=cpu –cpu-max-prime=20000 run

Maximum prime number checked in CPU test: 20000
Test execution summary:
total time:                          764.2113s
total number of events:              10000
total time taken by event execution: 764.1989
per-request statistics:
min:                                 76.34ms
avg:                                 76.42ms
max:                                141.03ms
approx.  95 percentile:              76.64ms
Threads fairness:
events (avg/stddev):           10000.0000/0.00
execution time (avg/stddev):   764.1989/0.00

 

SD

# Random read write — 514.5MB in 300sec = 1.7 MB/sec

sysbench –test=fileio –file-total-size=4G prepare
sysbench –test=fileio –file-total-size=4G –file-test-mode=rndrw –init-rng=on –max-time=300 –max-requests=0 run
sysbench –test=fileio –file-total-size=4G cleanup

Operations performed:  19757 Read, 13171 Write, 42112 Other = 75040 Total
Read 308.7Mb  Written 205.8Mb  Total transferred 514.5Mb  (1.715Mb/sec) 109.76 Requests/sec executed
Test execution summary:
total time:                          300.0024s
total number of events:              32928
total time taken by event execution: 27.9480
per-request statistics:
min:                                  0.05ms
avg:                                  0.85ms
max:                                423.34ms
approx.  95 percentile:               1.57ms
Threads fairness:
events (avg/stddev):           32928.0000/0.00
execution time (avg/stddev):   27.9480/0.00

# 512MB block write — 512MB in 0:28min = 18.3 MB/sec
cd /
time sh -c “dd if=/dev/zero of=ddfile bs=8k count=65536 && sync”; rm ddfile

# 64MB block read — 66 MB in  3.09 seconds =  21.35 MB/sec
hdparm -t /dev/mmcblk0p1

 

USB

# 512MB block write — 512MB in 1:54min = 4.5 MB/sec
mke2fs /dev/sda1
mkdir /x
mount /dev/sda1 /x
cd /x
time sh -c “dd if=/dev/zero of=ddfile bs=8k count=65536 && sync”; rm ddfile
cd /
umount /x

# 64MB block read — 64 MB in  3.09 seconds =  20.73 MB/sec
hdparm -t /dev/sda1

 

SSD

# 512MB block write — 512MB in 14.38s = 35.6 MB/s
mke2fs -t ext4 /dev/sdb1
mkdir /x
mount /dev/sdb1 /x
cd /x
time sh -c “dd if=/dev/zero of=ddfile bs=8k count=65536 && sync”; rm ddfile
cd /
umount /x

# 440MB block read — 440 MB in  3.00 seconds = 146.55 MB/sec
hdparm -t /dev/sdb1