Cloudsoft AMP Tutorial – IBM Blue Box, Mesos and Daytrader

In this tutorial we will show you how to quickly and easily deploy a full Apache Mesos environment into IBM Blue Box using Cloudsoft AMP, and exercise it by deploying a WebSphere Liberty application stack to support the DayTrader benchmark application. We will also show how AMP automates scaling of the environment under load.

What you will need to get started:

Getting everything running

This section will guide you through the process of deploying AMP using Vagrant, adding a cloud location to its catalog (where you will be deploying the Apache Mesos Cluster), adding the demo application catalogs and subsequent deployment of a DayTrader Cluster to Apache Mesos.

The accompanying video illustrates the process, and gives additional insight into the workflow.

Starting AMP using Vagrant

Important: AMP requires Vagrant 1.8.0 or higher.

First, clone and cd into the demo repository:

git clone https://github.com/cloudsoft/amp-daytrader-mesos-demo.git
cd amp-daytrader-mesos-demo

Next, copy public and private keys for your Cloud Provider to the my_keys directory, these keys will be copied to the /home/vagrant/.ssh/ directory on startup so as to be available to AMP. For example if you use an OpenStack key pair to connect to your IBM Blue Box tenant you should copy them as follows:

cp ~/.ssh/openstack.pem ~/.ssh/openstack.pub ./my_keys/

Then start the Vagrant box:

vagrant up

Note that the instructions below are lifted from the README.md in your cloned amp-daytrader-mesos-demo git repository.

Configure AMP for Blue Box

  1. Make a copy of the sample-bluebox.bom in the locations folder of your cloned amp-daytrader-mesos-demo git repository. We will use this file to add IBM Blue Box as a target location for our applications
  2. Update the file, replacing the REPLACE_THIS tokens with appropriate values from your Blue Box OpenStack environment
  3. Connect to the AMP web console at http://localhost:8081
  4. From the AMP web console Catalog tab, add the updated sample-bluebox.bom to the catalog by pasting the contents into the blueprint composer and clicking Add to Catalog
  5. The location ibm-bluebox-mesos will now be available in AMP and should be used as the target location for your Mesos cluster

Deploy the Apache Mesos Cluster

  1. From the AMP web console Home tab click add application, select Apache Mesos and hit Next
  2. Supply values for the following fields, leaving the rest with defaults:
    1. Set Locations to your added location, for example IBM Bluebox San Jose CLocker Meso
    2. Give the Apache Mesos Cluster a name, for example Bluebox Apache Mesos Cluster (San Jose)
    3. Point mesos.slave.privateKey at the location of your locations private key, for example ~/.ssh/openstack.pem
  3. Hit Deploy, your Apache Mesos cluster will be ready in a few minutes
     

When we move over to the Blue Box Openstack Dashboard and refresh, you can see all nodes are now present.

Switching to the Applications tab and expanding the tree view you can find the Mesos URL on the Apache Mesos Cluster Summary page, and Marathon URL under Mesos Frameworks on the Marathon Framework Summary page.

You now have a fully functional Apache Mesos cluster, deployed and managed by Cloudsoft AMP. As an example of using AMP to deploy an application into Mesos, we’ll be deploying DayTrader, an Open Source benchmarking application.

Deploy Daytrader

In this section we will start a DayTrader cluster backed by IBM WebSphere Liberty and MariaDB.

  1. From the AMP web console Home tab click add application, select DayTrader App Cluster (WebSphere Liberty/MariaDb) and hit Next
  2. Supply values for the following fields, leaving the rest with defaults:
    • Set Locations to my-mesos-cluster
    • Give the DayTrader Cluster a name, for example DayTrader WSL MariaDB
    • Provide a WebSphere Liberty Archive URL for the webProfile7 release archive (wlp-webProfile7-8.5.5.x.zip), either hosting it yourself or via an IBM-provided source. At the time of writing a valid option is
      • https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/downloads/wlp/8.5.5.9/wlp-webProfile7-8.5.5.9.zip
  3. Hit Deploy, your DayTrader Cluster will be ready in a few minutes. While the deployment to Apache Mesos is proceeding you can navigate to the Marathon Dashboard where you can observe AMP adding application instances
  4. When the DayTrader Cluster finishes deploying (the green icon will stop flashing) you can connect to Daytrader via the mapped load balancer URI (main.uri) found on the application’s Summary tab

Autoscaling

Now that the DayTrader application has been deployed, we will use Apache JMeter to exercise it, and see that Cloudsoft AMP will manage autoscaling the application based on load.

Acquire the test suite

First, we need to acquire an appropriate JMeter test suite for use with Daytrader.

Follow the instructions on the IBM Blog post Measuring performance with the Daytrader 3 benchmark sample to download the DayTrader3Install.zip via the WebSphere Performance Page. This archive contains a sample Daytrader JMeter test suite file daytrader3.jmx.

curl -O http://www.mirrorservice.org/sites/ftp.apache.org/jmeter/binaries/apache-jmeter-2.13.tgz
tar zxf apache-jmeter-2.13.tgz
curl -O ftp://public.dhe.ibm.com/software/webservers/appserv/was/DayTrader3Install.zip
tar zxf DayTrader3Install.zip

Populate the Daytrader database

Before we can begin running JMeter against DayTrader, we will need to populate the DayTrader Database.

  1. Return to the DayTrader webpage (or open it using the main.uri on the DayTrader’s Summary tab in AMP)
  2. Navigate to the Configuration tab and click the link named (Re)-populate DayTrader Database

This will open a new, initially blank, page which will periodically update as sample stock and user data is loaded. This can take some time to complete, and updates only infrequently. You will know it is complete when the plain update text is followed by what looks like a normal configuration page.

Generate load

Now that the DayTrader database has been populated with some test date, we are ready to begin using JMeter to generate load.

  1. Start the JMeter GUI substituting the HOST and PORT values for your deployed DayTrader cluster (from the main.uri on the DayTrader WSL MariaDB Summary page) in this command:
    • ./apache-jmeter-2.13/bin/jmeter.sh \ 
    • -t DayTrader3Install/DayTrader3-EE6/Apache_JMeter_script/daytrader3.jmx \ 
    • -JHOST=MAPPED BALANCER HOST \ 
    • -JPORT=MAPPED BALANCER PORT \ 
    • -JDURATION=600
  2. On opening JMeter you should be presented with the Thread Group page, you should alter the Number of Threads (users) setting as desired. For demonstration purposes it is recommended to lower this to 10 which results in approximately 60 requests/second from an i7 MacBook Pro. The default autoscaling thresholds for both Wildfly and Websphere Liberty have been set to upper 40 and lower 20. JMeter will generate traffic that exceeds the upper bound, and Cloudsoft AMP will automatically manage scaling the application to meet demand
  3. Right click and enable Aggregate Report in the left hand tree, and when ready to start traffic click the green Play menubar item. You should immediately start to see results fill the report pane. The reqs/sec rate should approximately match that show in the Dynamic Cluster Groups reqs.per_sec.per_node sensor
  4. Expanding the DynamicCluster of WebSphere Liberty Servers entry on the Applications tab you should observe AMP provision new DayTrader nodes which are automatically added to the NGinx balancer group, examining the reqs.per_sec sensors for each node and for the Dynamic Cluster Group you can observe the traffic being distributed evenly
  5. To stop traffic click the red Stop menubar item. You can also click the sweeping brush icons (two brushes to the left of the binoculars), to clear the results window

AMP will scale the DayTrader nodes back down in response to the decreased traffic, eventually to just the original single node.

Further Exploration

Also included in your cloned amp-daytrader-mesos-demo repository are blueprints that allow you to deploy a Daytrader application on WebSphere backed by PostgreSQL, instead of MariaDB. There are also blueprints that use the WildFly 10 application server in place of IBM WebSphere, again backed by either PostgreSQL or MariaDB.

The readme in the repository has further options, advice and opportunities for exploration.

Conclusion

In this tutorial, we have done quite a bit. We have set up a Cloudsoft AMP instance, deployed a fully functional Apache Mesos environment into IBM Blue Box, deployed a DayTrader application, backed by MariaDB and IBM WebSphere Liberty into Mesos, and finally we used JMeter to show how AMP automatically scales our environment in response to load. We have shown that AMP can deploy Apache Mesos easily, and blueprints can target either a deployed Mesos environment or a public or private cloud provider, in this case Blue Box, directly. What’s next? Kubernetes of course!