Elvenware

MongoDb

Table of Contents

MongoDb

MongoDb is no-sql database.

Install

Before getting started, there are two facts you should know:

At this time, the latest LTS version of Ubuntu that is targeted by MongoDb is 14.04. The 16.04 of Ubuntu has shipped, but the MongoDb release is not yet ready at the time of this writing.

This means the official Ubuntu release of MongoDb is 2.6, which is from April 2014. Why is the Ubuntu version of MongoDb so old? Because Ubuntu releases LTS versions every two years, and at the time of this writing it has been about two years since the last LTS release for which there is a MongoDb release. (More specifically, the last LTS version was 14.04. The new 16.04 release is out, but the MongoDb apt install script has still not been updated.)

To get a new version of MongoDb on Ubuntu, there needs to be:

Right now those three things have not come into alignment. Hopefully they will soon. But for now, we have to live with our current situation.

Despite the annoyance, if all you are trying to do is learn MongoDb, it really doesn't matter that you have an older release. The commands are identical or very similar to those on the new release. Furthermore, you don't really need the latest unless you are trying to serve data to a huge number of users on a busy site. That is unlikely to be the case if you are just learning MongoDb. As a result, I don't advise wasting time trying to install the latest. Just live with the older version until the APT script is updated. In the meantime you can simply install MongoDb with this command:

sudo apt-get install mongodb

Some information:

Or just run the following bash script from /JsObjects/Utilities called InstallMongoDb.sh:

On Ubuntu 16.04 use these techniques based on a Digital Ocean article:

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
sudo echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
sudo apt-get update
sudo apt-get install -y --allow-unauthenticated mongodb-org

Then create a file called /etc/systemd/system/mongodb.service. This file will likely need to be replaced when official 16.04 MongoDb binaries are released. One way to create the file is with your favorite text editor, such as geany or nano. To create the file, you will need to be in sudo mode. You should therefore issue a command like this:

sudo nano /etc/systemd/system/mongodb.service

And here is the text that should be pasted into the file:

[Unit]
Description=High-performance, schema-free document-oriented database
After=network.target

[Service]
User=mongodb
ExecStart=/usr/bin/mongod --quiet --config /etc/mongod.conf

[Install]
WantedBy=multi-user.target

Now start the service and ensure that is restarted each time you boot up:

sudo systemctl start mongodb
sudo systemctl enable mongodb

Trouble Shoot

If you are working in Pristine Lubuntu, you may get an error starting MongoDb due to a lack of space. One possible solution would be to use the cloud 9 commands with a variation on the --bind_ip in the third line:

cd
mkdir data
echo 'mongod --bind_ip=127.0.0.1 --dbpath=data --nojournal --rest "$@"' > ~/mongod
chmod a+x ~/mongod
./mongod

On Cloud 9, we set the bind_ip to $IP, but here we set it to local host.

Uninstall

The following commands should remove mongodb from your system:

sudo apt-get purge mongodb-org  <= TRY THIS
sudo apt-get purge mongodb      <= OR TRY THIS
sudo apt-get autoremove

You should also remove mongodb.list if it exists:

sudo rm /etc/apt/sources.list.d/mongodb.list

Older Versions of Ubuntu

Older install script for previous versions of ubuntu such as 15.04:

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list
sudo apt-get update
sudo apt-get install mongodb-org

The second line above should be typed as one line, so just run JsObjects script or block copy the above and paste it into the command line. Note, however, that the second line has two parts:

PART01: 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen'
PART02: sudo tee /etc/apt/sources.list.d/mongodb.list

When you paste in the whole line, you are, in effect, running two commands separated by the pipe symbol. In pseudo code, it looks like this:

echo PART01 | PART02  // Don't actually run this

But don't try to paste that in, paste in the line from the script. The above pseudo command is just an attempt to explain what the second line in the script does. It is not a valid method of running the second line.

There is an alternative method, but it is not recommended, as it probably won't install the latest MongoDb. Nevertheless, it could work. So if the above fails, you could try:

sudo apt-get install mongodb  // Only do this if the above fails

Install on Cloud 9

MongoDb is preinstalled on Cloud 9. To set it up do this:

cd
mkdir data
echo 'mongod --bind_ip=$IP --dbpath=data --nojournal --rest "$@"' > ~/mongod
chmod a+x ~/mongod
mongod

I got the above here:

Then to view the database:

mongo

Here is a typical URL for accessing a database:

"mongodb://0.0.0.0:27017/test",

And here is what a MongoTalk.json configuration file might look like:

{
    "urls": [
        "mongodb://0.0.0.0:27017/test",
        "mongodb://127.0.0.1:27017/test",
        "mongodb://USERNAME:PASSWORD@dsXXX.mongolab.som:XXXX/DATABASE"
    ],
    "selectedUrl": 0
}

Install the NPM Package

If the package.json file for your project does not already include mongodb, use npm to install the Node mongodb driver into your local project:

sudo npm install mongodb

Start and Stop MongoDb

I believe these are the commands for the recent versions of Ubuntu:

    sudo systemctl start mongodb
    sudo systemctl stop mongodb
    sudo systemctl enable mongodb

Use these commands to stop, to start and restart MongoDb on older versions of Ubuntu:

    sudo service mongodb start
    sudo service mongodb stop
    sudo service mongodb restart

To see if MongoDb is running, or to learn a bit about your instance, go to this address:

The key is that it is a port 1000 above the port you use to query the database. The page you find will reveal at least a few minimal facts about your running instance of MongoDb.

On recent versions of MongoDb http://localhost:28017/ won't work unless you add this to /etc/mongod.conf and then restart mongodb:

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1
  http.enabled: true
  http.RESTInterfaceEnabled: true

Sample Session

You can view the content on your MongoDb server without writing any code. First you need to start the mongo client:

$ mongo
MongoDB shell version: 2.6.1
connecting to: test

To quit the shell, you type exit and press return. But let's not do that quite yet. Instead you might want to see the databases on your instance of MongoDb:

> show dbs
admin  (empty)
bar    (empty)
local  0.078GB
test   0.078GB

Select or create one of the databases:

> use test
switched to db test

NOTE: The command use foo will switch the user to the foo database. If foo does not exist, it will create it and then switch to it.

Instead of tables, MongoDb has collections. Show the collections in the database.

> show collections
finalWalk
system.indexes
testData
test_insert

Insert data into a collection called test01. The collection will be created if it does not exist:

db.test01.insert({firstName: "Susan"})
WriteResult({ "nInserted" : 1 })

A somewhat more complex example is shown here:

db.test02.insert({  
   "firstName": "Abe10001",
   "lastName": "Lincoln10001",
   "address": "10001 Green Street",
   "city": "Bellevue",
   "state": "WA",
   "zip": 98002
})

Note that we are inserting into a collection called test02. When you paste in the above, the output should look like this:

    >  db.test02.insert({  
    ...   "firstName": "Abe10001",
    ...   "lastName": "Lincoln10001",
    ...   "address": "10001 Green Street",
    ...   "city": "Bellevue",
    ...   "state": "WA",
    ...   "zip": 98002
    ... })

Use find to see the contents of test02:

    > db.test02.find()    
    {
        "_id": ObjectId("537a6ba7ceccfa98291d56dd"),
        "firstName": "Abe10001",
        "lastName": "Lincoln10001",
        "address": "10001 Green Street",
        "city": "Bellevue",
        "state": "WA",
        "zip": 98002
    }
    etc...
    >

By default, you can can insert nearly any valid JSON into your tables. This means we could insert the data we inserted into test01 into test02 or vice versa. Though the JSON has a different structure, it will not cause problems. For instance:

db.test02.insert({longshot: "RubyFind"})

> db.test02.find()
{ "\_id" : ObjectId("5754550586f9d37ae11eb9cd"), "firstName" : "Abe10001", "lastName" : "Lincoln10001", "address" : "10001 Green Street", "city" : "Bellevue", "state" : "WA", "zip" : 98002 }
{ "\_id" : ObjectId("5754568286f9d37ae11eb9ce"), "longshot" : "RubyFind" }

As you can see, the first and second objects have very different structures, yet they both co-exist in the collection. This behavior is, of course, very different from what we see in a SQL database.

You can ask to project only certain fields. The following projection asks not to see the _id field, which is displayed by default, and to see the firstName and lastName fields. (What you see here assumes that additional rows have been inserted):

> db.test02.find({}, {"\_id": 0, firstName: 1, "lastName": 1});
{ "firstName" : "Abe10000", "lastName" : "Lincoln10000" }
{ "firstName" : "Abe10001", "lastName" : "Lincoln10001" }
{ "firstName" : "Abe10002", "lastName" : "Lincoln10002" }
{ "firstName" : "Abe10003", "lastName" : "Lincoln10003" }
{ "firstName" : "Abe10004", "lastName" : "Lincoln10004" }

This query asks to only the rows where firstName is equal to Abe0001.

> db.test02.find({firstName: "Abe10001"}, {"\_id": 0, firstName: 1, "lastName": 1});
{ "firstName" : "Abe10001", "lastName" : "Lincoln10001" }
>

Now let's delete all the records:

> db.test02.remove({})
WriteResult({ "nRemoved" : 5 })
>

You can use the similar rules to those you used for find if you want to remove only specific records.

> db.test_insert.remove({firstName: "Abe10003"})
WriteResult({ "nRemoved" : 1 })
>

Mongo: Getting Help

Mongo is the command line utility you can use to query your MongoDB collections and to maintain your database. Start mongo by typing mongo at the command line.

When in mongo, type help to get a list of things you can do:

db.help()                    help on db methods
db.mycoll.help()             help on collection methods
sh.help()                    sharding helpers
rs.help()                    replica set helpers
help admin                   administrative help
help connect                 connecting to a db help
help keys                    key shortcuts
help misc                    misc things to know
help mr                      mapreduce

show dbs                     show database names
show collections             show collections in current database
show users                   show users in current database
show profile                 show most recent system.profile entries with time >= 1ms
show logs                    show the accessible logger names
show log [name]              prints out the last segment of log in memory, 'global' is default
use <db_name>                set current database
db.foo.find()                list objects in collection foo
db.foo.find( { a : 1 } )     list objects in foo where a == 1
it                           result of the last line evaluated; use to further iterate
DBQuery.shellBatchSize = x   set default number of items to display on shell
exit                         quit the mongo shell

Try, for instance, db.help():

DB methods:
db.addUser(userDocument)
db.adminCommand(nameOrDocument) - switches to 'admin' db, and runs command [ just calls db.runCommand(...) ]
db.auth(username, password)
db.cloneDatabase(fromhost)
db.commandHelp(name) returns the help for the command
db.copyDatabase(fromdb, todb, fromhost)
db.createCollection(name, { size : ..., capped : ..., max : ... } )
db.currentOp() displays currently executing operations in the db
db.dropDatabase()
db.eval(func, args) run code server-side
db.fsyncLock() flush data to disk and lock server for backups
db.fsyncUnlock() unlocks server following a db.fsyncLock()
db.getCollection(cname) same as db['cname'] or db.cname
db.getCollectionNames()
db.getLastError() - just returns the err msg string
db.getLastErrorObj() - return full status object
db.getMongo() get the server connection object
db.getMongo().setSlaveOk() allow queries on a replication slave server
db.getName()
db.getPrevError()
db.getProfilingLevel() - deprecated
db.getProfilingStatus() - returns if profiling is on and slow threshold
db.getReplicationInfo()
db.getSiblingDB(name) get the db at the same server as this one
db.hostInfo() get details about the server's host
db.isMaster() check replica primary status
db.killOp(opid) kills the current operation in the db
db.listCommands() lists all the db commands
db.loadServerScripts() loads all the scripts in db.system.js
db.logout()
db.printCollectionStats()
db.printReplicationInfo()
db.printShardingStatus()
db.printSlaveReplicationInfo()
db.removeUser(username)
db.repairDatabase()
db.resetError()
db.runCommand(cmdObj) run a database command.  if cmdObj is a string, turns it into { cmdObj : 1 }
db.serverStatus()
db.setProfilingLevel(level,<slowms>) 0=off 1=slow 2=all
db.setVerboseShell(flag) display extra information in shell output
db.shutdownServer()
db.stats()
db.version() current version of the server

Set up a MongoLab URL

When you want to connect to MongoLab from your program, you need to compose the URL that let's you in.

Near the top of the page, on the left, you will see something like this:

To connect using the shell:
    mongo dsXXXXXX.mongolab.com:XXXXX/Databasename -u <dbuser> -p <dbpassword>
To connect using a driver via the standard URI (what's this?):
    mongodb://<dbuser>:<dbpassword>@dsXXXXXX.mongolab.com:XXXXX/DatabaseName

It is the second URL that you need.

NOTE: The dbuser and dbpassword are not the user name and password that you use when you sign on to MongoLab. Instead, these are the database users and passwords for your database. It is an entirely separate user name and password that has nothing to do with your MongoLab account. It is the user for your MongoDb database, not for the site.

To create a database user, click on the USER tab and press Add database user.

Database User

How to Connect to mLab or a Local Database

Here is the connect.js file that handles the connection from mongoose to MongoDb database. It supports both a local database and a remote mLab instance:

/**
 * Created by charlie on 6/6/16.
 */

var mongoose = require('mongoose');

var connect = {

    connected: false,

    simpleConnect: function() {
        'use strict';
        console.log('Connecting with simple.');
        var url = 'mongodb://127.0.0.1:27017/renew';
        connect.connected = true;
        mongoose.connect(url);
        var db = mongoose.connection;
        db.on('error', console.error.bind(console, 'connection error:'));
        db.once('open', function(callback) {
            connect.connected = true;
            console.log('Opened connection to mongo');
        });
    },

    // mongodb://<dbuser>:<dbpassword>@ds049848.mlab.com:49848/elvenlab01
    mlabConnect: function() {
        'use strict';
        console.log('Connecting with mlab.');
        connect.connected = true;
        var userName = 'foobar';
        var password = 'foo';
        var siteAndPort = 'ds049848.mongolab.com:49848';
        var databaseName = 'elvenlab01';
        var url = 'mongodb://' + userName + ':' + password + '@' + siteAndPort + '/' + databaseName;
        console.log(url);
        mongoose.connect(url);

        // This part is optional
        var db = mongoose.connection;
        db.on('error', console.error.bind(console, 'connection error:'));
        db.once('open', function(callback) {
            connect.connected = true;
            console.log('Opened connection to mongo');
        });
    },

    doConnection: function(useSimple) {
        'use strict';
        var connectType = useSimple || false;
        if (connectType) {
            connect.simpleConnect();
        } else {
            connect.mlabConnect();
        }
    }
};

module.exports = connect;

This code defaults to using mLab (MongoLab) unless you specify otherwise:

var connectType = useSimple || false;

It also breaks out the various pieces of the mLab connection string into discreet pieces so that you can more easily create a valid URL:

var userName = 'foobar';
var password = 'foo';
var siteAndPort = 'ds049848.mongolab.com:49848';
var databaseName = 'elvenlab01';

Note that the goal is to create a URL of this type:

mongodb://:@ds049848.mlab.com:49848/

Ultimately, you want to change nearly the entire URL, including the number in the site and port portion. The following is not a valid URL, but it should help you see what you are trying to create:

mongodb://myusername:mypassword@ds1111.mlab.com:11111/mydatabase

Working with Mongo Data

We need to understand updating and deleting records in the database.

A general review of mongo DB.

Look at PeopleManager in CreateCharactor02

pkozlowski-opensource on github.

It is interesting to note that this a REST API, so you can simply paste URLs into the address bar of your browser, or use them as links on web pages:

More info in the MongoDriver docs:

That's not a solution for a shipping app, but it is a good tool to use when you are experimenting with a new technology.

I should add that posting your API key as a link on a public web page is a very bad idea, but you can use it on private sites.

The best way to experiement with this technology is just to take the URL and paste it into a the address field of the browser.

Getting Started

Here is a simple node program that uses the modular pattern. It will open up a MongoDb database and inserting a row of data into it:

var MongoClient = require('mongodb').MongoClient;
var format = require('util').format;

var QueryMongo = (function() {

    var url01 = 'mongodb://127.0.0.1:27017/test';
    var url02 = 'mongodb://192.168.2.19:27017/test';

    function QueryMongo() {

        // Open the test database that comes with MongoDb
        MongoClient.connect(url02, function(err, database) {
            if (err) {
                throw err;
            }

            insertCollection(database, 'test_insert', { f : 7 });

        });
    }

    var getCollection = function(database) {

        var collection = database.collection('test_insert');

        // Count documents in the collection
        collection.count(function(err, count) {
            console.log(format("count = %s", count));
        });

        // View the collection
        collection.find().toArray(function(err, results) {
            console.dir(results);
            database.close();
        });

    };

    // Will create collection if it does not exist
    var insertIntoCollection = function(db, collectionName, objectToInsert) {

        var collection = db.collection(collectionName);
        collection.insert(objectToInsert, function(err, docs) {
            getCollection(db);
        });
    };

    return QueryMongo;

})();

var q = new QueryMongo();

MongoDb Won't Start

If you try to start MongoDB, and find that it won't start, or at least won't stay running after it starts, the problem could be that you are out of disk space. MongoDB wants lots of diskspace, something like 3.5 GB. You can, however, request that it use less disk space by putting smallfiles = true at the end of /etc/mongodb.conf. Now your database will be limited in size in half a gig, but that should be plenty for our needs.

Mongoose Subdocuments

One of the most powerful features of Mongoose is the ability to work with sub-documents.

Mongoose Gyp and Bson

You may get a message about C++ BSON not being installed and so the system is falling back on the "pure javascript version". Please note this is not an error. There is nothing wrong with the pure javascript version other than it being perhaps a bit slower. That will not matter on a development machine in most cases. So I suggest simply living with the messages on your development machine, and fix them only on your production machine.

If you want to fix them, there are two issues here:

In this discussion, we will assume that our production environment is an EC2 Ubuntu server. In this scenario, on Windows, we will are only doing development and testing.

By default, our Ubuntu server does not have a C++ compiler installed. You can test if it does by typing:

gcc -v

To install the gcc C++ compiler, first connect to your EC2 instance and ensure your it is up to date:

sudo apt-get update
sudo apt-get upgrade

Then install gcc:

sudo apt-get install build-essential

More details and a reference are here

Now when you type npm install or npm install mongoose --save at the linux command prompt all will go smoothly.

On Windows, there is nothing I can see that is wrong with using the "Pure JavaScript version" of bson. It's going to be slower, but this is only our development environment.

So I would just live with the Bson messages, unless you have VS installed already. I don't really like to install Visual Studio except on a dedicated development machine or in a VM. If, however, you have it installed already, then you should only need to worry about your Python version. I have two versions installed, set up like this:

C:\Python\Python27
C:\Python\Python33

I can just change my path to point to the one I want to use at any particular time. One way to do this would be to:

Now you can just use setx to change the environment variable and then restart your windows command line session:

setx PYTHON_HOME C:\Python\Python33

You can read about setx here: