Integrating Cloud Foundry with Apache Brooklyn Part 1: Service Broker

Cloud Foundry is a platform-as-a-service making it easy to deploy, run, and scale applications on top of a host of runtime frameworks.

Apache Brooklyn is an application blueprinting and management system which supports a wide range of software and services in the cloud.

This series of blog posts covers how the two can be used together to make the wide range of Brooklyn blueprints available within a Cloud Foundry ecosystem. From Cassandra and Couchbase to ActiveMQ and Kafka, across multiple clouds and traditional IT, mixing in Chef recipes and custom HA/DR policies, the combination of Apache Brooklyn and Cloud Foundry removes all the limits on what you can automate within your PaaS.

In this post, we’ll learn about the Service Broker in the Cloud Foundry ecosystem: how this modular component bridges the gap between service providers and the CF tooling. The next post will cover a CF plugin that makes it simple to specify services in your application manifests, so you can create them and bind them to your application with one command. After that, we will give some examples, including a sharded MongoDB instance in your manifest for a truly scalable application and a Riak cluster.

The Service Broker

The Service Broker component in CF allows developers to decouple their apps from the services that they consume. The cf create-service command calls to the broker to create a new instance of a service. You can then include a reference to this service instance in your application’s manifest, or manually issue a bind-service, to associate it to one or more applications. CF will then automatically inject details of the service instance — such as the URL and credentials for consuming it — into your application’s runtime.

[ image from http://docs.cloudfoundry.org/services/api.html ]

Until now, most service brokers have dealt with a single type of service, requiring a lot of up-front investment to make a new service available in CF and limiting the degree to which these services can be customized when needed for applications.

By making Apache Brooklyn available as a service broker, however, you only need one broker to expose the broad catalog of existing Brooklyn blueprints. And adding new services is as simple as POSTing a blueprint to the Brooklyn catalog.

Let’s start by installing the Brooklyn Service Broker. You’ll need Java 8, git, and Gradle installed, and then:

$ git clone https://github.com/cloudfoundry-community/brooklyn-service-broker.git
$ cd brooklyn-service-broker
$ gradle clean build

This generates a file brooklyn-service-broker.war containing the service broker.

Next we’ll need an Apache Brooklyn server. If you don’t have one, it’s easy to install following the instructions at https://brooklyn.incubator.apache.org/v/latest/start/running.html.

Once that’s running, make a note of the brooklyn-host and brooklyn-password, and install the Brookyn service broker WAR file to /path/to/broker-war/. Choose a broker-user and broker-password, and in the YAML below replace these five tokens with the real values. Paste the resulting blueprint into the “Add Application” dialog in your Brooklyn server, and within a minute your Service Broker should be running (usually on port 8080, but see the sensors of the Tomcat if that port is already in use).

name: Brooklyn Service Broker
location: localhost
services:
-    type: brooklyn.entity.webapp.tomcat.TomcatServer
    name: Tomcat Server
    war: /path/to/broker-war/brooklyn-service-broker.war
    brooklyn.config:
        java.sysprops:
        brooklyn.uri: http://brooklyn-host:8081
        brooklyn.username: admin
        brooklyn.password: brooklyn-password
        security.user.name: broker-user
        security.user.password: broker-password
        spring.data.mongodb.host:                 $brooklyn:component("mongodb").attributeWhenReady("host.address")
    spring.data.mongodb.port: $brooklyn:component("mongodb").attributeWhenReady("mongodb.server.port")

-type: brooklyn.entity.nosql.mongodb.MongoDBServer
    id: mongodb
    name: Service Broker Mongo DB

Adding to Brooklyn Services to Cloud Foundry

Next, we’ll register this Broker with Cloud Foundry. If you don’t yet have a running instance of CF, seehttp://docs.cloudfoundry.org/deploying/, log in to it, and then continue.

First, let’s add a service to the Brooklyn catalog so we have something to access from Cloud Foundry. In the Brooklyn GUI, select the “Catalog” tab and then click the “+” and the “entity” button and add the following service blueprint:

brooklyn.catalog:
    id: mysql
    version: 1.0
    iconUrl: classpath://mysql.png
    description: MySql is an open source relational database management system (RDBMS)
name: MySQL Database
services:
- type: brooklyn.entity.basic.BasicApplication
  brooklyn.children:
    - type: brooklyn.entity.database.mysql.MySqlNode
    id: mysql
    name: MySQL Node

Now register the broker with Cloud Foundry:

$ cf create-service-broker brooklyn broker-user broker-password http://broker-host:8080/
Creating service broker brooklyn as admin...
OK
$ cf service-access
Getting service access as admin...
broker: brooklyn
     service                     plan                             access   orgs          
     MySQL Database              localhost                        none

The blueprint we added to Brooklyn is now visible in Cloud Foundry. It’s access is none, but all that remains is to make it available in the marketplace.

$ cf enable-service-access "MySQL Database" Enabling access to all plans of service MySQL Database for all orgs as admin... OK $ cf marketplace Getting services from marketplace in org cloudsoft / space development as admin... OK service plans description MySQL Database localhost MySql is an open source relational database management system (RDBMS)

To create instances of this service, simply:

$ cf create-service "MySQL Database" localhost mysql
Creating service mysql in org cloudsoft / space development as admin...
OK
$ cf services
Getting services in org cloudsoft / space development as admin...
OK

    name    service          plan        bound apps   
    mysql   MySQL Database   localhost

To deploy to new machines, all you have to do is change the location section of the blueprint. For instance if we wanted a 5-node 8GB-each Riak cluster in Softlayer instead of a MySQL on localhost, we could add this to the Brooklyn catalog (replacing XXX with your SL username and access key):

brooklyn.catalog:
    id: riak
    version: 1.0
    iconUrl: classpath://riak.png
    description:  Riak
name: riak-cluster
location:
jclouds:softlayer:
    identity: XXX
    credential: XXX
services:
- type: brooklyn.entity.basic.BasicApplication
brooklyn.children:
    - type: brooklyn.entity.nosql.riak.RiakCluster
      initialSize: 5
      provisioning.properties:
        minRam: 8GB

Refresh the catalog and Riak is now at your service.

Binding your applications

You can now bind these service instances to your applications. If your application is already deployed, use:

$ cf bind-service my-app mysql

This will make the following environment variables available to your application runtime:

"VCAP_SERVICES": {
    "MySQL Database": [
    {
        "credentials": {
            "MySqlNode:tE5W": {
                "host.address": "127.0.0.1",
                "host.name": "127.0.0.1",
                "mysql.password": null,
                "mysql.port": 3306,

…

            }
        },
        "label": "MySQL Database",
        "name": "mysql",
        "plan": "localhost",
        "tags": []
    }
    ]
}

Or, you can add it to your manifest:

applications:
- name: my-app
    path: path/to/app.war
    memory: 512M
    services:
        - mysql

And then do a cf push like normal.

Video

To wrap up this post here is a video recording of our service broker in action.

Pretty cool, eh? But what if we want to specify new services to be added to the marketplace? Well, we have a plugin just for you! Stay tuned!

Update

Check out Integrating Cloud Foundry with Apache Brooklyn Part 2: Brooklyn Plugin.