Graphical wiring in application-centric blueprints

Ed: Thomas spends a lot of time making complex things easy and visual. He's one of our top cloud engineers who develops our AMP product that we use inside our AWS services, and also with customers who run applications on other clouds. In this post, he talks about a common issue: how do you visually compose applications in the cloud? Turns out, it isn't easy. This is a Cloudsoft engineering post about how we obsess about cloud applications.

Inside the Cloudsoft services that migrate, run and evolve applications on AWS, is our product called Application Management Platform (AMP). We also sell AMP directly to organizations that want to compose applications so they are easy to consistently and reliably deploy and manage at any scale. As well as wiring applications together, we can wire together anything that has an API so we also use AMP to automate complex runbooks. 

AMP has a graphical editor that allows you to create deployable blueprints for your application — it describes the components, their wiring as well as policies for in-life management. This editor is called the Blueprint Composer, which landed with AMP version 4.0.0 and is accessible with a web browser.

Blueprints are YAML files and the blueprint composer lets you generate those graphically or write them directly. It’s a two-way mapping between models so any edits made graphically will be shown in the YAML and vice-versa.

At Cloudsoft, we always strive to improve our products, and the UI/UX is a big part of it. Since the launch of the blueprint composer, we gathered feedbacks and performed usability tests to learn and stress various UI/UX hypothesis. One particular observation came back consistently from this: there is no easy graphical way to wire components together.

DSL, what now?

There is Domain Specific Language (DSL) in AMP blueprinting for wiring things together.

Think of a blueprint as a run book document that links components via DSL, and doesn't need a human to do the actions. This is applications and infrastructure as code.

DSL commands all are prefixed by $brooklyn: and support different types of operations. Let’s take a look at the following blueprint:

services:

  - type: org.apache.brooklyn.entity.webapp.tomcat.Tomcat8Server

    Brooklyn.config:

      ...

      env:

        db.jdbc: >-

          $brooklyn:formatString("jdbc:%s%s?user=%s&password=%s", $brooklyn:component("db").attributeWhenReady("datastore.url"), "visitors", $brooklyn:external("creds", "db-user"), $brooklyn:external("creds", "db-password"))

  - type: org.apache.brooklyn.entity.database.mysql.MySqlNode

    id: db

In the blueprint above I have 2 components: Tomcat and a MySQL.

On Tomcat, I deployed a war file that needs the JDBC URI to connect to the database. With AMP, I can rely on DSL commands to dynamically build this string, and store as an environment variable. In this example:

• Used $brooklyn:component("db").attributeWhenReady("datastore.url") to get the DB URL from the db component (MySQL node)
  Note: attributeWhenReady just blocks until the information is available.
• Used $brooklyn:external("creds", ...) to retrieve secrets from an external credentials store (e.g. this could be configured to use Vault).
• Incorporated the URL I got from above, plus credentials information into the JDBC pattern with $brooklyn.formatString().

By the way, these DSL commands are just a subset of what is available. If you want to know more, please check out the documentation

As you can see, it is a powerful feature of AMP but is also overwhelmingly long and complicated to read or craft. Not only do you need to know what commands are available, but you also need to know what configuration keys or sensors you can target. All components share a common set of configuration keys/sensors but they also have custom ones. Even though you can get this information via the catalogue, this is far from being easy and an ideal workflow… very far from it.

We needed to come up with a system that allows users to easily build these DSL commands graphically.

You shall not pass!

A wizard is what we need! But the opposite kind of Gandalf: easy to use and aimed at first-time users. The blueprint composer already has the necessary information (configuration keys and sensors retrieved through the REST API) based on the current blueprint, we *just* need to select the type of DSL command we want, propose the available items we can target, a couple of filters to help the user search and there you go.

How hard can it be?

Spoiler alert: quite difficult actually.

Because doing something that works is easy. However, it is hard to do something that works, and is easy to use. As we say: “If you need to explain your joke, it means it’s not funny” and guess what, it’s the same for UI/UX.

Iteration #1 - The pragmatic way

Or engineering way: there are 4 types of commands: getting a configuration key, getting a sensor, getting an entity or format a string. And so, my first iteration reflected exactly that: 4 accordions, one for each type with a list of corresponding items you could target.

Simple but not quite right. As the initial feedback pointed out:

1. Too many clicks to select an item (open accordion, search (+ filter), select, ok)
2. The user needs to know about these 4 types before even selecting something (did I say I was targeting first-time user? great work Thomas!)
3. Pattern’s arguments (formatted string) could not nest other DSL commands

Hum, ok. Back to the drawing board.

Iteration #2 - Learn from the past

This time around, I reduced the number of accordions to 2:

• One “Built-in” where I grouped all configuration keys and sensors items, from all entities, into a list. It was filterable by “type” (i.e. configuration keys, sensors or entities). Upon selection, the corresponding DSL command was created: brilliant!
• One “Pattern” which was the same as iteration #1, with a field for the main pattern and an adjustable number of argument fields.

Ok, I was going somewhere. Feedback from the team was that I reduced the number of clicks (issue #1) but didn’t do anything about #2 and #3. Also, the names of the accordions were poorly chosen: “Built-in” as confusing and “Pattern” was very engineer-oriented.

Iteration #3 - Can I really think about this?

After brainstorming with colleagues, gathering feedback and putting in a lot of thoughts, I arrived at the following conclusion: a user cares about what to target. The underlying command is just a tool that one doesn’t need to know (except power-users, but the goal is to optimise for first-time users).

To achieve this, a simple list of available items is presented to the user. One can filter by “kind” and/or the entity those items belong to, plus a search field to filter by name/description. That fixes issue #1 and #2 while also appealing to power-users, as they can explicitly search for a “kind” of items (matching the 4 types of DSL commands I was talking about earlier)

The “kind” select also includes “Formatted string” which presents the same form as earlier. However, each argument now includes the same “DSL” button to nest DSL commands within, fixing issue #3.

Wrapping up

… into a nice and shiny package, this DSL editor will be available in AMP 5.2.0. I hope you will like it and use it an anger. I would also love to hear what you have to say about it (or the UI in general) because feedback from you is more important than you know: it shapes what we will be doing tomorrow.

Ed: Visualising and composing applications in the cloud, then having something that deploys that application to the cloud AND manages its lifecycle is the secret to reducing costs, increasing uptimes, improving responses to unplanned business events and spiky demand, and exploiting the pace of cloud innovation. Cloud application automation is what makes Cloudsoft different from other cloud companies.