Nutanix Calm Runbooks & API Automation

Earlier this week Nutanix announced the general availability of Calm 3.0.0. This is an exciting release as it introduces some huge improvements to the ways application and host workflow can be managed. To name a few:

• Image creation directly from Calm i.e. create an AHV image from disk(s) attached to a Calm-managed VM
• Post-deployment update of Single VM applications
• Runbooks in Calm i.e. orchestrate tasks across multiple applications and hosts, even if they weren’t launched from a Calm blueprint

Runbooks are what we’ll look at today.

Endpoints

Within Calm, runbooks and the tasks they execute are assigned to/associated with endpoints. Think of an endpoint as an IP address or host on which the task will run. Within the Calm UI, each endpoint is configured with IP addresses and credentials – a example is shown below for a Linux VM with IP address 10.133.16.151.

Using this endpoint we can start creating runbooks that execute on that endpoint (or any other endpoint) we create. For the purposes of this article we’ll be connecting to that Linux VM using SSH on port 22, authenticated by SSH private key.

Runbooks

Once we have defined our endpoints, we can start creating the runbooks that may execute on those endpoints. The example in this article will contain the following simple workflow:

• A small script will run that returns a specific exit code.
  • Exit code 0 is interpreted as success.
  • A non-zero code is interpreted as failure. Our script can optionally return exit code 1 to simulate a failure.
• If the decision task is True (exit code 0), a small script will run that does the following:
  • Create a directory called backup in the current user’s home directory, if that directory does not already exist
  • Copies a file called /etc/haproxy/haproxy.cfg to ~/backups, if that file exists
• If the decision task is False (exit code 1), a message is printed that informs the user a non-zero exit code occurred.
• After all the above tasks have completed, successful or not, a simple exit message is shown

As with all GUI-based tasks in Calm, the creation of this workflow is extremely simple. Using the runbook editor (which I think is actually quite beautiful), we can build our runbook as follows:

It’s worth noting that this particular runbook has been configured to use the “LAMP HAProxy Server” as the default endpoint. This means the “LAMP HAProxy Server” endpoint will be used whenever an endpoint has not been otherwise specified.

Runbooks via API

In another article we’ll cover the creation of runbooks via API. For now, we’ll just look at the basics – listing and running runbooks via Nutanix Calm API. I’ll be using Postman to create and test my API requests and have shared my Postman collection at the bottom of this article.

List Runbooks

In previous articles we’ve covered a number of things that are conceptually quite similar to what we’ll look at first – listing the runbooks that exist on our Prism Central instance.

The API POST request below lists existing runbooks on Prism Central at IP address 10.133.16.228:

https://10.133.16.228:9440/api/nutanix/v3/runbooks/list

You’ll notice that this is similar to listing other entities via the Nutanix Calm APIs – blueprints, apps, etc. The only difference is that the JSON POST body will need to specify “runbook” as the kind of entity to list:

{"kind": "runbook"}

My Prism Central instance at 10.133.16.228 has a single runbook, named “HAProxy Runbook“. The response from the request above is as follows:

{
    "api_version": "3.0",
    "metadata": {
        "total_matches": 1,
        "kind": "runbook"
    },
    "entities": [
        {
            "status": {
                "last_update_time": 1591926226637012,
                "name": "HAProxy Runbook",
                "deletion_time": 1,
                "deleted": false,
                "spec_version": 11,
                "description": "",
                "creation_time": 1591842034066247,
                "state": "ACTIVE",
                "schema_version": "1.1.0",
                "entity_verison": 11,
                "running_runs": 0,
                "run_count": 13,
                "messages": [],
                "last_run_time": 1591926229971955,
                "tenant_uuid": "",
                "uuid": "8ece271f-9147-df66-5721-c93338b3f0dd"
            },
            "spec": {},
            "api_version": "3.0",
            "metadata": {
                "last_update_time": "1591926226637012",
                "owner_reference": {
                    "kind": "user",
                    "uuid": "00000000-0000-0000-0000-000000000000",
                    "name": "admin"
                },
                "kind": "runbook",
                "uuid": "8ece271f-9147-df66-5721-c93338b3f0dd",
                "project_reference": {
                    "kind": "project",
                    "name": "ntnxdemo-no-aws",
                    "uuid": "e4f8329c-bf9c-4dd1-8f33-042ce6b86eee"
                },
                "spec_version": 11,
                "creation_time": "1591842034066247",
                "name": "HAProxy Runbook"
            }
        }
    ]
}

Take particular note of my single runbook’s UUID: 8ece271f-9147-df66-5721-c93338b3f0dd. We will need that shortly when it comes to working with that specific runbook.

Once we know the UUID of the runbook we’d like to work with, getting additional information can be done with a simple GET request:

https://10.133.16.228:9440/api/nutanix/v3/runbooks/8ece271f-9147-df66-5721-c93338b3f0dd

The response contains all the information about the runbook, including scripts that are run at each stage (if any), credentials, default endpoints, and so on.

Run Runbook

Run, runbook, run! 🙂

Ok, with that terrible intro out of the way, let’s look at how runbooks can be executed using the v3 Calm APIs. So far we’ve already looked at how we can “access” a runbook via UUID. In this case, our runbook’s UUID is 8ece271f-9147-df66-5721-c93338b3f0dd.

Simple Runbook Run

At the simplest level, we can submit the runbook run request and send an empty POST body along with it. This will run the runbook and use the settings that were specified while I created the runbook in the Calm UI. This includes parameters such as endpoints and credentials.

The empty POST body is as simple as this:

{}

Note that sending the request with no POST body/payload at all will return an HTTP 500 “Internal Server Error”.

With that empty POST body specified, the full request path is as follows:

https://10.133.16.228:9440/api/nutanix/v3/runbooks/8ece271f-9147-df66-5721-c93338b3f0dd/run

The response from the successful run request is as follows:

{
    "status": {
        "runlog_uuid": "cfa369e5-fb82-4f22-97aa-610d773e502d"
    },
    "spec": {},
    "api_version": "",
    "metadata": {}
}

Advanced Runbook Run

The example above is great on its own, but what if we want to control which endpoint the runbook executes on? This is one of the most powerful features of runbooks as a feature – run them on any endpoint at any time, either from the UI or via API as we’re going to do now.

Each endpoint has a UUID, exactly the same way a runbook does. We can submit a simple GET request that will list our endpoints, as follows:

https://10.133.16.228:9440/api/nutanix/v3/endpoints/list

This request will also require that we specify “endpoint” as the kind of entity to list:

{"kind": "endpoint"}

Each endpoint will be listed in the entities array within the response, as follows. Please note I have stripped out most of this response and only left an example of an endpoint’s “main” properties:

In the screenshot above you’ll see in an endpoint named “LAMP MySQL Server” with UUID 28ce21ec-27da-6212-070a-073da9201dc6. How would we execute a runbook against that endpoint only?

The request is quite similar to the request we made earlier, only this time we need to submit an appropriate non-empty JSON payload along with the request. The JSON payload below uses default_target_reference to set the default endpoint for the request:

{
  "spec": {
    "args": [
    ],
    "default_target_reference": {
      "kind": "app_endpoint",
      "name": "LAMP MySQL Server",
      "uuid": "28ce21ec-27da-6212-070a-073da9201dc6"
    }
  }
}

Sending the request the same way we did before will produce a response that shows the runlog UUID, as expected. The difference now is the runbook’s default endpoint from the Calm UI has been overridden by the endpoint specified in our JSON payload.

Analysing Results

In the responses from our requests you’ll see one of the properties is named runlog_uuid. Similar to previously documented processes like launching blueprints (link to Part 1, link to Part 2), we can access the runlog UUID and get detailed information about what happened during the run. The GET request for this is as follows:

https://10.133.16.228:9440/api/nutanix/v3/runbooks/runlogs/cfa369e5-fb82-4f22-97aa-610d773e502d

While we can look at the response and get details on status, the only thing we’ll take notice of for now is the runlog status – please note this response is heavily truncated with only the relevant parts showing:

{
  "api_version": "3.0",
  "metadata": {},
  "status": {
    "description": "",
    "name": "",
    "state": "SUCCESS",
    "critical": false,
    "action_reference": {
      "kind": "app_action",
      "name": "HAProxy Runbook",
      "uuid": "8ece271f-9147-df66-5721-c93338b3f0dd"
    },
    "type": "workflow_action_runlog",
    "userdata_reference": {
      "kind": "app_user",
      "uuid": "00000000-0000-0000-0000-000000000000",
      "name": "admin"
    },
    "is_runlog_archived": false,
    "reason_list": [],
    "runbook_json": {
      "name": "HAProxy Runbook"
    }
  },
  "spec": {
    "name": "",
    "description": ""
  }
}

In this case we can see the runbook status property is “SUCCESS” and the name of the runbook was as expected – “HAProxy Runbook“.

Downloading Logs

In addition to viewing the JSON results of a runbook run via JSON response, we can also download the output from a runbook request. Before looking at a quick example, take a look at the screenshot below and note the following points:

• Each task has its own panel that can be viewed individually (indicated by the orange arrow)
• Start and end times for the runbook are clearly visible
• Output is logged to the runbook console

However, there may be a need for these logs to be downloaded for the purposes of storage, additional parsing or distribution. This is as simple as making a request to the runbooks/runlogs/{uuid}/output/download API:

https://10.133.16.228:9440/api/nutanix/v3/runbooks/runlogs/cfa369e5-fb82-4f22-97aa-610d773e502d/output/download

The response is a downloadable binary file, in ZIP format. Before looking at the contents of the ZIP file, let’s first look at the Calm UI visualisation of the runbook that just executed:

The output of the runbook run above, in ZIP format, is as follows:

Downloadable Postman Collection

If you’d like to try running Nutanix Calm runbooks via API, all API requests used in today’s article have been made available on GitHub. Please download the RAW JSON file and import it into Postman, making sure to set your collection variable values and authentication details before use. If you’re new to working with Postman variables, please see the article “So many variables – How I test Nutanix APIs with Postman” – it contains various walk-through steps that will explain how all that works.

What does all this mean?

The API requests themselves are not overly complex. They follow the design goals of Nutanix APIs that have been published in recent releases, i.e. entity lists are the same, as are entity details and results/runlogs. To take this one step further, we can also apply other approaches to listing these entities in very large environments – please see “The Five Hundred – 500 VMs via v3 API“. The exact same approaches apply to endpoints and runbooks.

Today’s key takeaway is that Calm 3.0.0 is probably the most important release to date. In an upcoming article I’ll cover API automation and usage of some of the other new Calm 3.0.0 features, since the way those features can fit into a development and DevOps workflow immediately increases efficiency, simplicity and overall agility.

Thanks for reading and have a great day! 🙂