Integrating Cloud Foundry with Apache Brooklyn part 4: Putting it all together

We’re going to show you how to build a more complicated application using Node.js with MongoDB to store the product catalog, Redis to store Session data, Riak for a shopping cart, and Cassandra to keep some statistics about page visits.

Let’s take a look at the topology of those services, which we specify in the application manifest under the brooklyn section so that we can use the plugin to create them.

manifest.yml:

applications:
- name: my-app
    command: node app.js
    brooklyn:
- name: redis
    location: aws-california
    services:
        - type: brooklyn.entity.nosql.redis.RedisStore
    - name: cassandra
        location: aws-tokyo
        services:                - type: brooklyn.entity.nosql.cassandra.CassandraCluster
        brooklyn.config:
            cluster.initial.size: 3
            cluster.initial.quorumSize: 2
    - name: mongodb
        location: aws-ireland
        services:
        - type: brooklyn.entity.nosql.mongodb.MongoDBServer    
    - name: riak
        location: aws-ireland
        services:
            - type: brooklyn.entity.nosql.riak.RiakCluster
                initialSize: 1

First the Redis Session store, this is very fast in-memory storage, we’re locating this in california.  Second is Cassandra, this is good for scaling out, we only need eventual consistency since statistics are constantly changing, so we take a quorum of 2/3.  We use MongoDB for the product catalog since it has a very flexible data model allowing changes to the catalog; it also has more sophisticated querying than some of the other datastores.  Finally, the Riak Shopping cart we need high availability here, but no need for complex queries – just get the shopping cart. We set the initial cluster size to 1, but this can grow.

Setting up the services in the application

For the full source code see here.

First, get all of the service credentials from the VCAP_SERVICES variable, (if it exists):

// redis
var credentials = env.redis[0].credentials;
var brooklynAppKey = Object.keys(credentials)[0];
var redisCredentials = {
    "host": credentials[brooklynAppKey]["host.name"],
    "port": credentials[brooklynAppKey]["redis.port"]
}

In this application, we use the express.js framework to create a minimal web server. We can use express to set up a new session using Redis as the datastore.  We are going to use sessions to remember which customer is logged in.  Create a new RedisStore Object passing to it the credentials parsed earlier and submit this to the to the session method as follows:

var RedisStore = require('connect-redis')(session);
app.use(session({
    store: new RedisStore(redisCredentials),
    secret: '123456789QWERTY'
}));

As we will see later, using it is as simple as setting a variable with express automatically sending it to redis to persist.

Setup Cassandra to count page visits

We need an array of hosts for the Cassandra client, which we get from the brooklyn sensor, cassandra.cluster.nodes, but we need to remove the port first since the client expects only the host.

// cassandra
credentials = env.cassandra[0].credentials;
var clusterAppKey = Object.keys(credentials)[0];
var clusterNodes = credentials[clusterAppKey]            ["cassandra.cluster.nodes"];
var nodes = [];
for (var i = 0; i < clusterNodes.length; i++){
    var split = clusterNodes[i].split(":");
    nodes.push(split[0]);
}

To create the Cassandra client first we create a new client object passing in the cluster’s nodes, then we call connect on the client, logging any error.

var cassandra = require('cassandra-driver');
var cassandraClient = new cassandra.Client({
    contactPoints : nodes
});

cassandraClient.connect(function(err, result) {
    if(err){
        console.log(err);
    }else{
        console.log('Cassandra: OK');
    }
});

Setup MongoDB and schema for the product store

The MongoDB client expects a string containing the uri of the mongo instance, we build this up with the host.name and mongo.server.port sensors from brooklyn.

// mongo
credentials = env.mongodb[0].credentials;
brooklynAppKey = Object.keys(credentials)[0];
var mongoUri = "mongodb://"+credentials[brooklynAppKey]["host.name"] + ":" +credentials[brooklynAppKey]["mongo.server.port"];

To save writing boilerplate, we employ mongoose for object modelling which allows us to specify a client side schema for our catalog items, we simply use a name and description.

var mongoose = require('mongoose');
mongoose.connect(mongoUri + '/products');
var Schema = mongoose.Schema;
var Item = new Schema({
    name: {type: String, required: true, trim: true},
    description: {type: String, required: true, trim: true}
});

var ProductCatalog = mongoose.model('Item', Item);

When we come to query MongoDB later, we can expect the data to come back as objects with these fields.

Setup Riak for the shopping cart

We defined Riak as a cluster with an initial size of 1, so we need to drill down to that node to get the host.name and riak.webPort sensors from brooklyn.

// riak
credentials = env.riak[0].credentials;
clusterAppKey = Object.keys(credentials)[0];
var children = credentials[clusterAppKey].children;
var firstChildKey = Object.keys(children)[0];
var riakHost = children[firstChildKey]["host.name"];
var riakPort = children[firstChildKey]["riak.webPort"];

The Nodiak package gives us a client for Riak, which we use to store a user’s shopping cart.

var riak = require('nodiak').getClient('http', riakHost, riakPort);

For simplicity, our shopping cart will just be an array of product names.

How we use the services

Pages are created in the idiomatic way using express, so we focus on the datastore queries for each page of the application.

Front page

We count the visit with cassandra using a CQL query stored in the increment variable appending the page name, home.  As the Cassandra client requires a callback function we just use a simple logging function that logs any error to the console.

cassandraClient.execute(increment + "'home'", loggingCallback);

Next, we retrieve all the data from the ProductCatalog model, again we are expected to provide a callback to deal with the returned data, which is simply iterated and presented to the user.

ProductCatalog.find().lean().exec(function(error, data){
    // presentation logic here
});

Cart page

On the cart page, we get the cart for the current user, which is found in the session variable, username. Again, the presentation logic is provided in a callback.

riak.bucket('carts').objects.get(req.session.username, function(error, obj) {
    // presentation logic here
});

Login page

This login page is just a convenience to log in users through a simple GET request.  It stores the path variable in the session store, to simulate a user logging in.  You could put authentication in here, though.

app.get('/login/:name', function(req, res){
    req.session.username = req.params.name;
    res.redirect('/');
});

Counts page

A statistics page, counts, shows the page visit counts that we store in Cassandra.  Again, we query Cassandra with a CQL statement and present the results in a callback.

cassandraClient.execute('SELECT * FROM counter.page_view_counts', function(err, result) {
    // presentation logic here
});

Product page

To simplify things a little, we treat the product page as a convenience by simulating a user visiting the item and adding it to the cart, logging a page visit with Cassandra, too.  We do this by first querying MongoDB for the product with the name in the request parameter, if it is found we send a query to Cassandra to update the counter for that page, then updating the shopping cart by getting the cart from Riak for the currently logged in user, adding the item to the cart and saving it again.

app.get('/:name', function(req, res){
    ProductCatalog.findOne({ name: req.params.name}, function(error, item){
        …
        cassandraClient.execute(increment +"'"+item.name+"'", loggingCallback);
        riak.bucket('carts').objects.get(req.session.username, function(error, obj) {
            …
            obj.data.items.push(item.name);
            riak.bucket('carts').objects.save(obj, function(error, obj){ … });
        });
    });
});

Deploying the app

As with all node.js applications deployed on Cloud Foundry we need to specify our dependencies in the package.json file. Then we can use the Brooklyn plugin to push the application:

$ cf brooklyn push

Running the app

Once the application has been pushed it will be available at an endpoint provided by cloud foundry.  On the first push, however, Cassandra will not yet have the keyspace and table initialized, nor will there be any products in the catalog.  So, let’s take the endpoint provided by Cloud Foundry and store it in a variable:

$ export APP=http://my-app.10.244.0.34.xip.io

Set up the Cassandra keyspace:

$ curl -X POST $APP/keyspace

Set up the Cassandra table:

$ curl -X POST $APP/tables

Now populate MondoDB with product catalog:

$ curl -H "Content-Type: application/json" -d '{ "name": "product A", "description" : "The best Product A in the world!"}' $APP/additem -X POST