One of the things that immediately drew my attention to Dispatch was the vCenter event driver. Executing an automated task based on a vCenter event has always been something really attractive to me. There are other ways to do this, but it always seemed like they were just not ideal for the environment I was working in for one reason or another. Most of the examples that I’ve seen use a vCenter alarm to run a command or send an SNMP trap.
vCenter Alarm Actions
Before diving into using Dispatch I’d like to share a couple of the issues I’ve run into when using alarm actions for automation. This isn’t to discourage anyone from using them. I’ve been using alarm actions to run scripts for several years and it works great. This is simply to provide a few reasons for my preference for an event driver like the one in Dispatch.
Alarm actions can be disabled. It’s too easy to disable alarm actions on an object in vCenter and forget about it. For example, let’s say I’m putting a host in maintenance mode so I disable alarm actions on it. I don’t want everyone to be alerted while I’m rebooting a host that is in maintenance. Unfortunately, this also disables any alarm actions that I’m using for automation. It would actually be really cool to have alarms automatically enabled/disabled based on a host’s maintenance mode status. However, if I’m using alarm actions to trigger the automation, then once alarm actions are disabled they can’t turn themselves back on!
The alarm action to “run a command” may not be officially supported on the vCenter Server Appliance. Installing anything on the vCenter Server Appliance is unsupported and generally discouraged, so running a custom script on the vCSA from an alarm action may not be technically supported. With that said, it’s relatively unobtrusive to have a bash or Python script call an external REST API or start a process on another system when an alarm action is triggered on the vCSA. This is covered in William Lam’s article, How to run a script from a vCenter alarm action in the VCSA? Also if you like Slack, be sure to check out this cool write up on vSphere Alarms with Slack and StackStorm.
The SNMP configration isn’t very flexible. The alarm action to “send a notification trap” is sent to all SNMP receivers configured in vCenter. You can’t distinguish between sending a trap for a real alert/problem and sending one just to perform some automation. Generally, this isn’t a problem, but in a larger organization where you’re sending all of your SNMP traps to an enterprise monitoring system it could be. The monitoring team will need to filter out traps for alarms that are not really problems and only exist for the sake of automation. Some common examples of this are when a new VM is created or when a host is placed into maintenance mode. For an example covering SNMP traps on VM creation see Automatically Securing Virtual Machines Using vCenter Orchestrator.
This leads me to VMware Dispatch and its vCenter event driver. Dispatch runs on Kubernetes, so if you’re like me and have been looking for something to try out Kubernetes with then here it is. I haven’t had a lot of opportunites to make use of containers while working in more traditional/conservative IT organizations and thought this would be a cool way to try out Kubernetes. If you just want to get it up and running quickly, Dispatch Solo provides a quick-start option with all of the dependencies packaged in a single virtual appliance (OVA). Their quickstart page covers deploying the OVA and a simple hello world example. This is how I’ve started out and plan to transistion to the Knative branch on Kubernetes in the near future.
Another feature of Dispatch that I really like is built-in support for Powershell Core. This makes it relatively easy to use existing Powershell/PowerCLI scripts and combine them with vCenter events. A script that I’ve had scheduled for many years is a variation of Alans Renouf’s original article, Who Created That VM?, which sets the date a VM was created and the user that created it as custom attributes. Instead of scheduling a script to run periodically and find all of the new VMs since the last time it was run, I think it’s more efficient to have it run automatically when a new VM is created. Setting VM attribbutes like this has been a common topic over the years, also being covered here, here, and in various discussions on the VMTN community forums. It’s worth noting that the VM creation date is now available on VMs created in vSphere 6.7, but you still might want to use one of the previous scripts if you’re not completely on vSphere 6.7 yet and/or if you also want to know who created the VM.
Setting up Dispatch
The Dispatch documentation has an example for hardening VMs in vSphere. Most of this will be a repeat of that process, except that I’m going to use a script I modified for Dispatch to set the VM’s attributes on creation.
Once you’ve deployed Dispatch you use the Dispatch CLI to manage it. Dispatch CLI can be downloaded and installed on macOS and Linux. On Windows 10 you can use Windows Subsystem for Linux or if you deployed Dispatch Solo you can SSH to the appliance and just run the commands there. I’m using Ubuntu on Windows Subsystem for Linux, but I’ve also SSH’d into the appliance in some cases and haven’t had any issues.
Here is a brief outline of the process to setup a PowerCLI script with the vCenter event driver:
- Create a Dispatch image with the dependencies we need (in this case VMware PowerCLI)
- Create a secret with the vSphere credentials
- Create a vCenter event driver using the previously created secret
- Create a function from the PowerCLI script
- Subscribe the new function to an event
Steps 1 - 3 only need to be performed once - as long as the dependencies and vCenter credentials don’t change these items can be reused. Steps 4 and 5 are repeated for each new function you create. Instructions for creating the PowerCLI image, vCenter secret, and vCenter event driver are in the example, Harden VMs in vSphere. Those are prerequisites for the rest of this article.
One thing to note about the payload from the event driver is that not all event types contain the ‘vm_name’. The ‘vm.deployed’ event does (as demonstrated in the hardening VMs example), but the ‘vm.created’ event does not. Here’s an example of the input received from the event driver when a VM is deployed (i.e. deployed from template):
The metadata section contains some useful info about the VM being deployed. Here is an example when a VM is created (i.e. created from an OVA/OVF or wihtout a template):
The second example applies to most other event types, such as when a VM is cloned or a host enters maintenance mode.
In the powershell script below I use the event time to retrieve events from vCenter at that time and then filter on the event message. It then has all the event information available to it and can set the “UserName” and “CreatedTime” in the VM’s “Created By” and “Created On” attributes. The other important piece here is the “handle” function, which sets some variables from the vCenter secret we provided and the input from the event driver. Dispatch looks for this function and will run it by default. You can also get this script from my GitHub repository.
Save the above code to a file named “Set-VMAttributes.ps1” or download it using
Next we’ll create a Dispatch function and verify that the function is ready.
Finally, we’ll create event subscriptions for our new function. This ties the function we just created to the vCenter events when a VM is deployed, created, cloned, or registered.
dispatch get subscription to confirm that the subscriptions are all ready.
Deploy or create a couple of VMs and confirm that the function has run:
To see all of the output from a particular run, copy the ID from the above list and use it in the command below.
This produces a lot of useful information for troubleshooting, such as the event type, the input to the function, and all of the function’s stdout and stderr.
That’s it! Now whenever a VM is deployed Dispatch will automatically set these custom attributes on the VM. My next endeavor is to run a function when hosts enter/exit maintenance mode. Hopefully I will have an article on that soon!