Jekyll2020-05-04T17:17:55-04:00https://doogleit.github.io/feed.xmlDoug Taliaferro’s IT BlogDoug is an IT pro and technologist focusing on VMware technologies, virtualization, Powershell, PowerCLI, automation and IT infrastructure.{"name"=>"", "avatar"=>"/assets/images/profile.jpg", "bio"=>"Doug is an IT professional and technologist focusing on VMware technologies, virtualization, Powershell, automation and IT/Cloud infrastructure.", "location"=>"Atlanta, GA", "email"=>nil, "uri"=>nil, "home"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>"doogleit", "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>"dougtaliaferro", "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>"virtually_doug", "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}Automate Host Maintenance with the vCenter Event Broker Appliance2019-11-29T10:00:00-05:002019-11-29T10:00:00-05:00https://doogleit.github.io/2019/11/automate-host-maintenance-with-the-vcenter-event-broker-appliance<p>The vCenter Event Broker Appliance is a new <a href="https://flings.vmware.com/vcenter-event-broker-appliance">VMware Fling</a> and open source project that was recently <a href="https://www.virtuallyghetto.com/2019/11/vcenter-event-broker-appliance-updates-vmworld-fling-community-open-source.html">released at VMworld Europe</a>. Using <a href="https://www.openfaas.com/">OpenFaaS</a> and its vCenter <a href="https://github.com/openfaas-incubator/openfaas-vcenter-connector">event connector</a> functions can be run based on vCenter events. A preview of it was also presented at VMworld US which I had the pleasure of attending. If you haven’t seen either of these sessions I highly recommend watching the recording of <a href="https://videos.vmworld.com/global/2019/videoplayer/29523">#CODE1379E “If This Then That” for vSphere - The Power of Event Driven Automation session</a>. I’m quite interested in this project and really excited to see all of the cool automation the community does with it.</p>
<p>If you haven’t deployed the appliance the <a href="https://github.com/vmware-samples/vcenter-event-broker-appliance/blob/master/getting-started.md">Getting Started</a> guide has all the details for getting it setup. Also be sure to join the <a href="https://vmwarecode.slack.com/archives/CQLT9B5AA">VMware {Code} Slack channel</a>. All of my work is based on the existing samples <a href="https://github.com/vmware-samples/vcenter-event-broker-appliance">on GitHub</a> and Opvizor’s excellent article: <a href="https://www.opvizor.com/audit-vm-configuration-changes-using-the-vcenter-event-broker">Audit VM configuration changes using the vCenter Event Broker</a>. A huge thanks to these guys for all of their work and special thanks to <a href="https://www.openfaas.com/">OpenFaaS</a> and Alex Ellis for making this all possible.</p>
<p>This example will disable alarm actions on a host while it is in maintenance mode. It is actually two functions using the same script. The first function subscribes to the <code class="language-plaintext highlighter-rouge">entered.maintenance.mode</code> event to run when a host is put into maintenance mode and disable its alarms. The second function subscribes to the <code class="language-plaintext highlighter-rouge">exit.maintenance.mode</code> event to re-enable alarms when the host exits maintenance mode. The script looks at the event type (or in OpenFaaS speak the “topic”) that called the function and performs the appropriate action to disable/enable alarms. Here is what the stack.yml file looks like with both functions:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="m">1.0</span>
<span class="na">provider</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">openfaas</span>
<span class="na">gateway</span><span class="pi">:</span> <span class="s">https://veba.yourdomain.com</span>
<span class="na">functions</span><span class="pi">:</span>
<span class="na">powercli-entermaint</span><span class="pi">:</span>
<span class="na">lang</span><span class="pi">:</span> <span class="s">powercli</span>
<span class="na">handler</span><span class="pi">:</span> <span class="s">./handler</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">doogleit/powercli-hostmaint:latest</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="na">write_debug</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">read_debug</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">function_debug</span><span class="pi">:</span> <span class="no">false</span>
<span class="na">secrets</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">vcconfig</span>
<span class="na">annotations</span><span class="pi">:</span>
<span class="na">topic</span><span class="pi">:</span> <span class="s">entered.maintenance.mode</span>
<span class="na">powercli-exitmaint</span><span class="pi">:</span>
<span class="na">lang</span><span class="pi">:</span> <span class="s">powercli</span>
<span class="na">handler</span><span class="pi">:</span> <span class="s">./handler</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">doogleit/powercli-hostmaint:latest</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="na">write_debug</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">read_debug</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">function_debug</span><span class="pi">:</span> <span class="no">false</span>
<span class="na">secrets</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">vcconfig</span>
<span class="na">annotations</span><span class="pi">:</span>
<span class="na">topic</span><span class="pi">:</span> <span class="s">exit.maintenance.mode</span>
</code></pre></div></div>
<h2 id="deploying-the-functions">Deploying the functions</h2>
<p>To deploy these functions clone the example from GitHub:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/doogleit/vcenter-event-broker-appliance
<span class="nb">cd </span>vcenter-event-broker-appliance/examples/powercli/hostmaint-alarms
</code></pre></div></div>
<p>If you’ve already created a secret named ‘vcconfig’ from one of the other PowerCLI examples and want to use the same vCenter server and credentials you can skip this next step. Note that the ‘vcconfig’ secret used in the Python tagging example is different and won’t work for these functions. Configure your vCenter details in the vcconfig.json file and create the secret:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"VC"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"vcenter-hostname"</span><span class="p">,</span><span class="w">
</span><span class="nl">"VC_USERNAME"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"veba@vsphere.local"</span><span class="p">,</span><span class="w">
</span><span class="nl">"VC_PASSWORD"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"FillMeIn"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>faas-cli secret create vcconfig <span class="nt">--from-file</span><span class="o">=</span>vcconfig.json <span class="nt">--tls-no-verify</span>
</code></pre></div></div>
<p>Finally modify the gateway in the stack.yml file with your vCenter Event Broker Appliance address and deploy the functions:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">provider</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">openfaas</span>
<span class="na">gateway</span><span class="pi">:</span> <span class="s">https://veba.yourdomain.com</span>
<span class="nn">...</span>
</code></pre></div></div>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>faas-cli deploy <span class="nt">-f</span> stack.yml <span class="nt">--tls-no-verify</span>
</code></pre></div></div>
<p>Have a host enter and exit maintenance mode to test it out. You can check the logs using either:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>faas-cli logs powercli-entermaint <span class="nt">--follow</span> <span class="nt">--tls-no-verify</span>
</code></pre></div></div>
<p>for a host entering maintenance mode or:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>faas-cli logs powercli-exitmaint <span class="nt">--follow</span> <span class="nt">--tls-no-verify</span>
</code></pre></div></div>
<p>for a host exiting maintenance mode.</p>
<p>If you use SolarWinds to monitor your ESXi hosts keep an eye out for an example on that as well. I hope to have an example on GitHub soon.</p>
<p>I was previously using <a href="https://vmware.github.io/dispatch/">VMware Dispatch</a> to experiment with running functions based on vCenter event subscriptions. If your interested you can check out some of my previous articles on using Dispatch here:</p>
<ul>
<li><a href="https://doogleit.github.io/2019/07/event-based-automation-with-dispatch/">Event Based Automation with VMware Dispatch</a></li>
<li><a href="https://doogleit.github.io/2019/07/automating-host-maintenance-with-dispatch/">Automating Host Maintenance with VMware Dispatch</a></li>
<li><a href="https://doogleit.github.io/2019/08/automating-host-maintenance-with-dispatch-part-2/">Automating Host Maintenance with VMware Dispatch Part 2</a></li>
</ul>{"name"=>"", "avatar"=>"/assets/images/profile.jpg", "bio"=>"Doug is an IT professional and technologist focusing on VMware technologies, virtualization, Powershell, automation and IT/Cloud infrastructure.", "location"=>"Atlanta, GA", "email"=>nil, "uri"=>nil, "home"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>"doogleit", "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>"dougtaliaferro", "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>"virtually_doug", "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}The vCenter Event Broker Appliance is a new VMware Fling and open source project that was recently released at VMworld Europe. Using OpenFaaS and its vCenter event connector functions can be run based on vCenter events. A preview of it was also presented at VMworld US which I had the pleasure of attending. If you haven’t seen either of these sessions I highly recommend watching the recording of #CODE1379E “If This Then That” for vSphere - The Power of Event Driven Automation session. I’m quite interested in this project and really excited to see all of the cool automation the community does with it.vRealize Easy Installer2019-11-03T12:21:00-05:002019-11-03T12:21:00-05:00https://doogleit.github.io/2019/11/vrealize-easy-installer<p>The recent <a href="https://blogs.vmware.com/management/2019/10/vrealize-suite-2019-vcloud-suite-2019-ga.html">release of vRealize Automation 8.0</a> comes with a vastly simplified installation process and architecture. The Windows based IaaS servers have been completely eliminated! The vRealize Easy Installer is a new wizard driven installer that looks similar to the installer for the vCenter Server Appliance and takes care of deploying all of the virtual appliances you’ll need, including vRealize Suite Lifecycle Manager, vRealize Identity Manager, and vRealize Automation. Starting with vRealize Automation 8.0 you have to use vRSLCM to install vRA. Migrating from an existing vRSLCM and using an existing vIDM deployment are also both options that are available in the installer. This really simplifies laying the foundation for your vRealize Suite 2019 environment. I’m going to walk through the installation process here and highlight a couple items I thought were noteworthy.</p>
<p>The first couple screens are fairly self explanatory.
<img src="/assets/images/vrei1.png" alt="vRealize Easy Installer" />
<img src="/assets/images/vrei2.png" alt="vRealize Easy Installer" /></p>
<p>Enter the vCenter Server details to deploy the virtual appliances to.
<img src="/assets/images/vrei3.png" alt="vRealize Easy Installer" /></p>
<p>Select the VM folder to deploy the virtual appliances in.
<img src="/assets/images/vrei4.png" alt="vRealize Easy Installer" /></p>
<p>Select the Compute Cluster to deploy the virtual appliances to.
<img src="/assets/images/vrei5.png" alt="vRealize Easy Installer" /></p>
<p>Select the Datastore to deploy the virtual appliances to. At least 75 GB of free space is required.
<img src="/assets/images/vrei6.png" alt="vRealize Easy Installer" /></p>
<p>Select the Network (port group) and enter the networking details.
<img src="/assets/images/vrei7.png" alt="vRealize Easy Installer" /></p>
<p>Enter the default root/admin password for this installation. Make note all of the places it will use this password. This is really convenient for using a consistent password across all of the virtual appliances. Just make sure it is a strong and secure password.
<img src="/assets/images/vrei8.png" alt="vRealize Easy Installer" /></p>
<p>Enter the details for the Lifecycle Manager virtual appliance.
<img src="/assets/images/vrei9.png" alt="vRealize Easy Installer" /></p>
<p>Enter the details for the Identity Manager virtual appliance. I highlighted the “Default Configuration Admin” here because this is the account you will need to initially use to login to vRealize Automation. Make note of the user name you enter here (I used “configadmin”). The password will be the one entered previously in step 8.
<img src="/assets/images/vrei10.png" alt="vRealize Easy Installer" /></p>
<p>Enter the details for the vRA virtual appliance and a license key. You might need to upgrade your vRealize Suite license key(s) to vRealize Suite 2019 in your VMware licensing portal for use here.
<img src="/assets/images/vrei11.png" alt="vRealize Easy Installer" /></p>
<p>Review the summary.
<img src="/assets/images/vrei12.png" alt="vRealize Easy Installer" /></p>
<p>And observe the progress and/or log file.
<img src="/assets/images/vrei13.png" alt="vRealize Easy Installer" /></p>
<p>After it’s deployed you can login to vRSLCM and configure directory authentication or deploy additional vRealize products. vRSLCM 8 offers a lot of additional management capabilities. Here’s a few other references.</p>
<ul>
<li><a href="https://blogs.vmware.com/management/2019/08/vrslcm-8-0-whats-new.html">vRealize Suite Lifecycle Manager 8.0 – What’s New</a></li>
<li><a href="https://docs.vmware.com/en/VMware-vRealize-Suite-Lifecycle-Manager/2019/rn/VMware-vRealize-Suite-Lifecycle-Manager-80-Release-Notes.html">VMware vRealize Suite Lifecycle Manager 8.0 Release Notes</a></li>
<li><a href="https://docs.vmware.com/en/VMware-vRealize-Suite-Lifecycle-Manager/index.html">VMware vRealize Suite Lifecycle Manager Documentation</a></li>
</ul>
<p>Outside of the initial deployment there are a couple of things I ran into that might be noteworthy. First, if using a wildcard certificate your domain name must have a public suffix, in other words using “.local” like I have in the lab will not work with a wildcard certificate. From the <a href="https://docs.vmware.com/en/vRealize-Automation/8.0/rn/vRealize-Automation-80-release-notes.html">vRealize Automation 8.0 release notes</a>:</p>
<blockquote>
<p>vRealize Automation 8.0 supports setting a wildcard certificate only for DNS names that match the content of the Public Suffix List (<a href="https://publicsuffix.org/">https://publicsuffix.org/</a>) For example, a valid wildcard certificate: you can use a wildcard certificate with DNS name like “<em>.myorg.com”. This is supported because “com” is part of the Public Suffix List. An invalid wildcard certificate example: you cannot use a wildcard certificate with DNS name like “</em>.myorg.local”.This is not supported because “local” is not part of Public Suffix List.</p>
</blockquote>
<p>I used vRSLCM to deploy another vRA environment and had a couple of pre-check errors.</p>
<p><img src="/assets/images/vrslm8-precheck-fail.png" alt="vRSLCM Precheck" /></p>
<p>The first one regarding disk space was a bit of an oddity. I selected the thick provisioned option and despite the target datastore having about 2 TB of free space it would not pass validation. The datastore was part of a datastore cluster, so I tried using a couple of other stand-alone datastores that were not part of a datastore cluster with the same result. I ended up using thin provisioning for now and will probably investigate this further later.</p>
<p>The second error was caused by the port group being configured to use ephemeral binding. This is not a configuration that should typically be used, but being in a test lab it didn’t really concern me. I just had to use a port group with static binding that had ports available. More info on why ephemeral binding should not normally be used can be found in <a href="https://kb.vmware.com/s/article/1022312">VMware KB 1022312 (Choosing a port binding type in ESX/ESXi)</a>. From the KB:</p>
<blockquote>
<p>Ephemeral binding</p>
<p>In a port group configured with ephemeral binding, a port is created and assigned to a virtual machine by the host when the virtual machine is powered on and its NIC is in a connected state. When the virtual machine powers off or the NIC of the virtual machine is disconnected, the port is deleted.</p>
<p>You can assign a virtual machine to a distributed port group with ephemeral port binding on ESX/ESXi and vCenter, giving you the flexibility to manage virtual machine connections through the host when vCenter is down. Although only ephemeral binding allows you to modify virtual machine network connections when vCenter is down, network traffic is unaffected by vCenter failure regardless of port binding type.</p>
<p>Note: Ephemeral port groups must be used only for recovery purposes when you want to provision ports directly on host bypassing vCenter Server, not for any other case.</p>
</blockquote>
<p>With thin provisioning selected and ports available the pre-check passed validation.</p>
<p><img src="/assets/images/vrslm8-precheck-pass.png" alt="vRSLCM Precheck" /></p>
<p>To begin setting up vRA login to the vRA appliance using the “configadmin” account created during the installation. A lot has changed in vRA 8. I recommend looking through the release notes which has links to the getting started guides and documentation.</p>
<ul>
<li><a href="https://docs.vmware.com/en/vRealize-Automation/8.0/rn/vRealize-Automation-80-release-notes.html">vRealize Automation 8.0 Release Notes</a></li>
</ul>
<p>For existing implementations you may also want to check out <a href="https://docs.vmware.com/en/vRealize-Automation/8.0/running-migration-assessment-for-vrealize-automation/GUID-C092632E-C807-459D-87DE-8E1FEBCDEAD6.html">Migrating to vRealize Automation 8.0</a>.</p>{"name"=>"", "avatar"=>"/assets/images/profile.jpg", "bio"=>"Doug is an IT professional and technologist focusing on VMware technologies, virtualization, Powershell, automation and IT/Cloud infrastructure.", "location"=>"Atlanta, GA", "email"=>nil, "uri"=>nil, "home"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>"doogleit", "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>"dougtaliaferro", "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>"virtually_doug", "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}The recent release of vRealize Automation 8.0 comes with a vastly simplified installation process and architecture. The Windows based IaaS servers have been completely eliminated! The vRealize Easy Installer is a new wizard driven installer that looks similar to the installer for the vCenter Server Appliance and takes care of deploying all of the virtual appliances you’ll need, including vRealize Suite Lifecycle Manager, vRealize Identity Manager, and vRealize Automation. Starting with vRealize Automation 8.0 you have to use vRSLCM to install vRA. Migrating from an existing vRSLCM and using an existing vIDM deployment are also both options that are available in the installer. This really simplifies laying the foundation for your vRealize Suite 2019 environment. I’m going to walk through the installation process here and highlight a couple items I thought were noteworthy.vSphere Profile-Driven Storage Errors2019-10-30T17:03:23-04:002019-10-30T17:03:23-04:00https://doogleit.github.io/2019/10/vsphere-profile-driven-storage-errors<p>This is another strange problem that I experienced recently. Restarting the Profile-Driven Storage service fixed all the errors, but diagnosing them was not so straight forward. The first error message was pretty clear that it was related to “policy based management” or PBM.</p>
<p><img src="/assets/images/pbm-error.png" alt="PBM Error" /></p>
<p>A quick search revealed several KB articles with similar PBM type errors, all of them suggesting to start or restart the profile-driven storage service.</p>
<ul>
<li><a href="https://kb.vmware.com/s/article/2118557">Cloning a virtual machine fails with the error: A general system error occurred: PBM error occurred during PreCloneCheckCallback (2118557)</a></li>
<li><a href="https://kb.vmware.com/s/article/53707">Error: “PBM error occurred during PreCreateCheckCallback: Connection refused” while creating protection group (53707)</a></li>
</ul>
<p>Restarting the service fixed the issue on the above vCenter. On a different vCenter I later found that there were a number of migration errors. Hosts were not able to enter maintenance mode due to DRS related errors.</p>
<p><img src="/assets/images/drs-unable-to-migrate.png" alt="DRS Migration Error" /></p>
<p>Trying to manually migrate VMs on any host also produced the following error during pre-validation, before you could even finish the migration wizard:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vMotion pre-validation error: "Error occurred when validating selection. Check the log files for details."
</code></pre></div></div>
<p>It didn’t immediately occur to me that these errors could also be related to profile-driven storage. We don’t use vVOLs or vSAN in this environment and I rarely think about or look at storage policies. After poking around for a bit, on a whim I decided to restart the profile-driven storage service on this vCenter as well and sure enough it fixed the problem. For future reference I’ll keep in mind that profile-driven storage is integral to many vCenter operations whether you actively use storage policies or not.</p>
<p>It may just be coincidence, but the day before all of this happened an external PSC had become unresponsive and had to be restarted. Both vCenters point to this external PSC. It seems odd that both vCenters had an issue with profile-driven storage at the same time, but I don’t really know if the problem with their PSC had anything to do with it.</p>{"name"=>"", "avatar"=>"/assets/images/profile.jpg", "bio"=>"Doug is an IT professional and technologist focusing on VMware technologies, virtualization, Powershell, automation and IT/Cloud infrastructure.", "location"=>"Atlanta, GA", "email"=>nil, "uri"=>nil, "home"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>"doogleit", "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>"dougtaliaferro", "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>"virtually_doug", "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}This is another strange problem that I experienced recently. Restarting the Profile-Driven Storage service fixed all the errors, but diagnosing them was not so straight forward. The first error message was pretty clear that it was related to “policy based management” or PBM.ESXi QFLE3 Driver Causing PSODs2019-10-29T15:08:43-04:002019-10-29T15:08:43-04:00https://doogleit.github.io/2019/10/esxi-qfle-driver-causing-psods<p>I had a couple of exciting months after upgrading all of my vSphere environments to 6.7 Update 1. There are two separate bugs in the qfle3 drivers that cause PSODs. The first PSOD didn’t occur until about a month after all the hosts were updated.</p>
<p><img src="/assets/images/qfle3-psod.png" alt="qfle3 PSOD" /></p>
<p>If the PSODs had started occuring sooner I think the issue would have been caught in a dev/test environment. After the first one several more hosts crashed in the following week, making me think that this was like a ticking time bomb. It seemed like the longer the host was running the more likely it was to crash, but I don’t have any definitive evidence of that. VMware support was quick to identify the issue and provide a resolution, which was upgrading the qfle3 driver to <a href="https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI67-QLOGIC-QFLE3-10860&productId=742">version 1.0.86</a>. This is relatively simple using update manager and you can double check the version using esxcli.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>root@esx01:~] esxcli software vib list | <span class="nb">grep </span>qfle
qfle3 1.0.86.0-1OEM.670.0.0.8169922 QLC VMwareCertified 2019-08-30
qfle3f 1.0.63.0-1OEM.670.0.0.8169922 QLC VMwareCertified 2019-08-12
qfle3i 1.0.20.0-1OEM.670.0.0.8169922 QLC VMwareCertified 2019-08-12
</code></pre></div></div>
<p>Not long after getting that updated driver rolled out we had another host crash with a slightly different PSOD.</p>
<p><img src="/assets/images/qfle3f-psod.png" alt="qfle3f PSOD" /></p>
<p>This one was caused by the qfle3f, or FCOE, driver. According to <a href="https://kb.vmware.com/s/article/71361">VMware KB 71361</a> a fix was due to be released soon and the workaround was to either disable the FCOE driver (if FCOE isn’t in use) or switch to the bnx2x driver. <a href="https://kb.vmware.com/s/article/52044">VMware KB 52044</a> may also be related to this and has the esxcli commands to disable qfle3 and enable bnx2x.</p>
<p>To disable the qfle3f driver using esxcli run the following and reboot the host.
</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>root@esx01:~] esxcli system module <span class="nb">set</span> <span class="nt">--enabled</span><span class="o">=</span><span class="nb">false</span> <span class="nt">--module</span><span class="o">=</span>qfle3f
</code></pre></div></div>
<p>To disable it on several hosts it is easier using PowerCLI. For example, the PowerCLI one-liner below will disable it on every host in “Cluster-01”. However you still need to reboot each host after running the command.</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">get-cluster</span><span class="w"> </span><span class="s1">'Cluster-01'</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nf">get-vmhost</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">(</span><span class="nf">get-esxcli</span><span class="w"> </span><span class="nt">-vmhost</span><span class="w"> </span><span class="bp">$_</span><span class="p">)</span><span class="o">.</span><span class="nf">system</span><span class="o">.</span><span class="nf">module</span><span class="o">.</span><span class="nf">set</span><span class="p">(</span><span class="bp">$false</span><span class="p">,</span><span class="w"> </span><span class="bp">$null</span><span class="p">,</span><span class="w"> </span><span class="s2">"qfle3f"</span><span class="p">)</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>To check the status of the driver on each host, showing if the driver is enabled and loaded, I used this PowerCLI one-liner.</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">get-cluster</span><span class="w"> </span><span class="s1">'Cluster-01'</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nf">get-vmhost</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nv">$vmhost</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">$_</span><span class="o">.</span><span class="nf">name</span><span class="p">;</span><span class="w"> </span><span class="p">(</span><span class="nf">get-esxcli</span><span class="w"> </span><span class="nt">-vmhost</span><span class="w"> </span><span class="bp">$_</span><span class="p">)</span><span class="o">.</span><span class="nf">system</span><span class="o">.</span><span class="nf">module</span><span class="o">.</span><span class="nf">list</span><span class="p">()</span><span class="w"> </span><span class="o">|</span><span class="nf">where</span><span class="w"> </span><span class="nx">name</span><span class="w"> </span><span class="o">-eq</span><span class="w"> </span><span class="nx">qfle3f</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nf">select</span><span class="w"> </span><span class="p">@{</span><span class="nx">l</span><span class="o">=</span><span class="s1">'vmhost'</span><span class="err">;</span><span class="nx">e</span><span class="o">=</span><span class="err">{</span><span class="nv">$vmhost</span><span class="p">}},</span><span class="w"> </span><span class="nf">Name</span><span class="p">,</span><span class="w"> </span><span class="nx">IsEnabled</span><span class="p">,</span><span class="w"> </span><span class="nx">IsLoaded</span><span class="w"> </span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>To also check the qfle3 driver version, I used the following one-liner, which admittedly is pretty ugly and probably not optimal, but it allowed me to quickly check hundreds of hosts.</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">get-vmhost</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nv">$vmhost</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">$_</span><span class="o">.</span><span class="nf">name</span><span class="p">;</span><span class="w"> </span><span class="nv">$esxcli</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">get-esxcli</span><span class="w"> </span><span class="nt">-vmhost</span><span class="w"> </span><span class="bp">$_</span><span class="p">;</span><span class="w"> </span><span class="nv">$esxcli</span><span class="o">.</span><span class="nf">system</span><span class="o">.</span><span class="nf">module</span><span class="o">.</span><span class="nf">list</span><span class="p">()</span><span class="w"> </span><span class="o">|</span><span class="nf">where</span><span class="w"> </span><span class="nx">name</span><span class="w"> </span><span class="o">-eq</span><span class="w"> </span><span class="nx">qfle3f</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nf">select</span><span class="w"> </span><span class="p">@{</span><span class="nx">l</span><span class="o">=</span><span class="s1">'vmhost'</span><span class="err">;</span><span class="nx">e</span><span class="o">=</span><span class="err">{</span><span class="nv">$vmhost</span><span class="p">}},</span><span class="w"> </span><span class="nf">Name</span><span class="p">,</span><span class="w"> </span><span class="nx">IsEnabled</span><span class="p">,</span><span class="w"> </span><span class="nx">IsLoaded</span><span class="p">,</span><span class="w"> </span><span class="p">@{</span><span class="nx">N</span><span class="o">=</span><span class="s2">"qfle3 version"</span><span class="err">;</span><span class="nx">E</span><span class="o">=</span><span class="err">{</span><span class="nv">$esxcli</span><span class="err">.</span><span class="nx">system</span><span class="err">.</span><span class="nx">module</span><span class="err">.</span><span class="nx">get</span><span class="err">(</span><span class="s2">"qfle3"</span><span class="err">).</span><span class="nx">version</span><span class="p">}}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nf">ft</span><span class="w"> </span><span class="nt">-auto</span><span class="w">
</span></code></pre></div></div>
<p>There were a few hosts where I needed to use FCOE. For these I enabled the bnx2x driver and disabled qfle3.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>root@esx01:~] esxcli system module <span class="nb">set</span> <span class="nt">--enabled</span><span class="o">=</span><span class="nb">true</span> <span class="nt">--module</span><span class="o">=</span>bnx2x
<span class="o">[</span>root@esx01:~] esxcli system module <span class="nb">set</span> <span class="nt">--enabled</span><span class="o">=</span><span class="nb">false</span> <span class="nt">--module</span><span class="o">=</span>qfle3
</code></pre></div></div>
<p>After rebooting the host I can confirm that the bnx2 drivers are listed and the version expected.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>root@esx01:~] esxcli software vib list | <span class="nb">grep </span>bnx2
net-bnx2 2.2.6b.v60.2-1OEM.600.0.0.2494585 QLogic VMwareCertified 2018-11-13
net-bnx2x 2.713.60.v60.1-1OEM.600.0.0.2494585 QLogic VMwareCertified 2018-11-13
scsi-bnx2fc 1.713.60.v60.1-1OEM.600.0.0.2494585 QLogic VMwareCertified 2018-11-13
scsi-bnx2i 2.713.60.v60.1-1OEM.600.0.0.2494585 QLogic VMwareCertified 2018-11-13
</code></pre></div></div>
<p>To wrap up this post I want to mention that VMware Skyline is making some really great progress towards proactively identifying these types of problems. I’ve since noticed that Skyline identifies and reports on findings that can cause a PSOD. You can see in the screenshot below a couple more driver issues, one of them for the qfle3i (iSCSI) driver which can also cause a PSOD.</p>
<p><img src="/assets/images/skyline-psod-findings.png" alt="Skyline Findings" /></p>
<p>In the details of each finding you can see any recommendations and helpful links.</p>
<p><img src="/assets/images/skyline-psod-details.png" alt="Skyline Details" /></p>
<p>I’m looking forward to using Skyline to more proactively address these types of issues. And now it looks like I have some more findings that need to be addressed!</p>
<p>[Update 10/29/2019] One quick note, since I originally wrote this <a href="https://kb.vmware.com/s/article/71361">KB 71361</a> has been updated with the fixed qfle3f driver. The workarounds I discussed should no longer be needed!</p>{"name"=>"", "avatar"=>"/assets/images/profile.jpg", "bio"=>"Doug is an IT professional and technologist focusing on VMware technologies, virtualization, Powershell, automation and IT/Cloud infrastructure.", "location"=>"Atlanta, GA", "email"=>nil, "uri"=>nil, "home"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>"doogleit", "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>"dougtaliaferro", "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>"virtually_doug", "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}I had a couple of exciting months after upgrading all of my vSphere environments to 6.7 Update 1. There are two separate bugs in the qfle3 drivers that cause PSODs. The first PSOD didn’t occur until about a month after all the hosts were updated.An error occurred while starting service ‘updatemgr’2019-08-24T00:00:00-04:002019-08-24T00:00:00-04:00https://doogleit.github.io/2019/08/an-error-occurred-while-starting-service-updatemgr<p>I recently upgraded an environment to vSphere 6.7 Update 1. After updating vCenter I updated a number of hosts with vSphere Update Manager before running into this problem. During the remediation pre-check it would just spin its wheels until it eventually timed out. I tested this on several hosts and it was consistent across all of them. This pre-check error was occurring on all of the remaining ESXi 6.5 hosts in vCenter. When it did eventually time out I would see this very long error message.</p>
<blockquote>
<p>Waiting for task (vim.TaskInfo) { dynamicType = null, dynamicProperty = null, key = task-116427, task = ManagedObjectReference: type = Task, value = task-116427, serverGuid = b0879725-bf73-4114-8373-59429bae84d1, description = null, name = null, descriptionId = com.vmware.vcIntegrity.RemediatePrecheckTask, entity = ManagedObjectReference: type = HostSystem, value = host-704, serverGuid = b0879725-bf73-4114-8373-59429bae84d1, entityName = esx01.local.com, locked = null, state = queued, cancelled = false, cancelable = true, error = null, result = null, progress = null, reason = (vim.TaskReasonUser) { dynamicType = null, dynamicProperty = null, userName = administrator@vsphere.local }, queueTime = java.util.GregorianCalendar[time=?, areFieldsSet=false, areAllFieldsSet=true, lenient=true, zone=sun.util.calendar.ZoneInfo[id=”GMT+00:00”, offset=0, dstSavings=0, useDaylight=false, transitions=0, lastRule=null], firstDayOfWeek=1, minimalDaysInFirstWeek=1, ERA=1, YEAR=2019, MONTH=7, WEEK_OF_YEAR=1, WEEK_OF_MONTH=1, DAY_OF_MONTH=12, DAY_OF_YEAR=1, DAY_OF_WEEK=5, DAY_OF_WEEK_IN_MONTH=1, AM_PM=0, HOUR=0, HOUR_OF_DAY=15, MINUTE=3, SECOND=9, MILLISECOND=791, ZONE_OFFSET=0, DST_OFFSET=0], startTime = null, completeTime = null, eventChainId = 1511731, changeTag = null, parentTaskKey = null, rootTaskKey = null, activationId = com.vmware.vcIntegrity.RemediatePrecheckTask } took more than 300 seconds! Try using timeout for longer tasks!</p>
</blockquote>
<p>It seemed reasonable to try restarting the Update Manager service. Unfortunately, after stopping the service it would not start back up.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@vcenter <span class="o">[</span> ~ <span class="o">]</span><span class="c"># service-control --stop vmware-updatemgr</span>
Operation not cancellable. Please <span class="nb">wait </span><span class="k">for </span>it to finish...
Performing stop operation on service updatemgr...
Successfully stopped service updatemgr
root@vcenter <span class="o">[</span> ~ <span class="o">]</span><span class="c"># service-control --start vmware-updatemgr</span>
Operation not cancellable. Please <span class="nb">wait </span><span class="k">for </span>it to finish...
Performing start operation on service updatemgr...
Error executing start on service updatemgr. Details <span class="o">{</span>
<span class="s2">"detail"</span>: <span class="o">[</span>
<span class="o">{</span>
<span class="s2">"args"</span>: <span class="o">[</span>
<span class="s2">"updatemgr"</span>
<span class="o">]</span>,
<span class="s2">"localized"</span>: <span class="s2">"An error occurred while starting service 'updatemgr'"</span>,
<span class="s2">"translatable"</span>: <span class="s2">"An error occurred while starting service '%(0)s'"</span>,
<span class="s2">"id"</span>: <span class="s2">"install.ciscommon.service.failstart"</span>
<span class="o">}</span>
<span class="o">]</span>,
<span class="s2">"componentKey"</span>: null,
<span class="s2">"problemId"</span>: null,
<span class="s2">"resolution"</span>: null
<span class="o">}</span>
Service-control failed. Error: <span class="o">{</span>
<span class="s2">"detail"</span>: <span class="o">[</span>
<span class="o">{</span>
<span class="s2">"args"</span>: <span class="o">[</span>
<span class="s2">"updatemgr"</span>
<span class="o">]</span>,
<span class="s2">"localized"</span>: <span class="s2">"An error occurred while starting service 'updatemgr'"</span>,
<span class="s2">"translatable"</span>: <span class="s2">"An error occurred while starting service '%(0)s'"</span>,
<span class="s2">"id"</span>: <span class="s2">"install.ciscommon.service.failstart"</span>
<span class="o">}</span>
<span class="o">]</span>,
<span class="s2">"componentKey"</span>: null,
<span class="s2">"problemId"</span>: null,
<span class="s2">"resolution"</span>: null
<span class="o">}</span>
</code></pre></div></div>
<p>With a little searching I found this KB article, <a href="https://kb.vmware.com/s/article/2147284">Resetting VMware Update Manager Database on a vCenter Server Appliance 6.5 or vCenter Server Appliance 6.7 (2147284)</a>. While this might seem like a bit of a sledgehammer approach, there wasn’t much to lose by reseting the DB. I would only need to recreate the ESXi 6.7 image and a few baselines. There wasn’t a lot of configuration stored in Update Manager in this case. So I proceeded to follow the article and found there were a few Python modules missing that required me to run some additional commands.</p>
<p>On the first try, running the reset-db command indicated the ipaddr module was missing:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@vcenter <span class="o">[</span> ~ <span class="o">]</span><span class="c"># /usr/lib/vmware-updatemgr/bin/updatemgr-utility.py reset-db</span>
Traceback <span class="o">(</span>most recent call last<span class="o">)</span>:
File <span class="s2">"/usr/lib/vmware-updatemgr/bin/updatemgr-utility.py"</span>, line 11, <span class="k">in</span> <module>
from vumUtils import registerWithVc, resetDB, manageConfig
File <span class="s2">"/usr/lib/vmware-updatemgr/bin/vumUtils.py"</span>, line 20, <span class="k">in</span> <module>
from cis.tools import get_install_parameter
File <span class="s2">"/usr/lib/vmware/site-packages/cis/tools.py"</span>, line 29, <span class="k">in</span> <module>
from cis.utils import <span class="k">*</span>
File <span class="s2">"/usr/lib/vmware/site-packages/cis/utils.py"</span>, line 45, <span class="k">in</span> <module>
import ipaddr
ImportError: No module named ipaddr
</code></pre></div></div>
<p>Using pip, the <a href="https://en.wikipedia.org/wiki/Pip_(package_manager)">Python package manager</a>, I installed the missing module. Before running any of the below commands make sure you take a snapshot of vCenter and have a working backup. This may not be supported, so take appropriate precautions and proceed at your own risk.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@vcenter <span class="o">[</span> ~ <span class="o">]</span><span class="c"># pip install ipaddr</span>
Collecting ipaddr
Downloading https://files.pythonhosted.org/packages/9d/a7/1b39a16cb90dfe491f57e1cab3103a15d4e8dd9a150872744f531b1106c1/ipaddr-2.2.0.tar.gz
Installing collected packages: ipaddr
Running setup.py <span class="nb">install </span><span class="k">for </span>ipaddr ... <span class="k">done
</span>Successfully installed ipaddr-2.2.0
You are using pip version 8.1.2, however version 19.2.2 is available.
You should consider upgrading via the <span class="s1">'pip install --upgrade pip'</span> command.
</code></pre></div></div>
<p>Trying to run reset-db again produced another missing module.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@vcenter <span class="o">[</span> ~ <span class="o">]</span><span class="c"># /usr/lib/vmware-updatemgr/bin/updatemgr-utility.py reset-db</span>
Traceback <span class="o">(</span>most recent call last<span class="o">)</span>:
File <span class="s2">"/usr/lib/vmware-updatemgr/bin/updatemgr-utility.py"</span>, line 11, <span class="k">in</span> <module>
from vumUtils import registerWithVc, resetDB, manageConfig
File <span class="s2">"/usr/lib/vmware-updatemgr/bin/vumUtils.py"</span>, line 22, <span class="k">in</span> <module>
from cis.cisreglib import SsoClient
File <span class="s2">"/usr/lib/vmware/site-packages/cis/cisreglib.py"</span>, line 32, <span class="k">in</span> <module>
from pyVim import sso
File <span class="s2">"/usr/lib/vmware/site-packages/pyVim/sso.py"</span>, line 27, <span class="k">in</span> <module>
from lxml import etree
ImportError: No module named lxml
</code></pre></div></div>
<p>Installing lxml revealed that yet another module was needed.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@vcenter <span class="o">[</span> ~ <span class="o">]</span><span class="c"># pip install lxml</span>
Collecting lxml
Downloading https://files.pythonhosted.org/packages/e4/f4/65d145cd6917131826050b0479be35aaccba2847b7f80fc4afc6bec6616b/lxml-4.4.1-cp27-cp27mu-manylinux1_x86_64.whl <span class="o">(</span>5.7MB<span class="o">)</span>
100% |████████████████████████████████| 5.7MB 119kB/s
Installing collected packages: lxml
Successfully installed lxml-4.4.1
You are using pip version 8.1.2, however version 19.2.2 is available.
You should consider upgrading via the <span class="s1">'pip install --upgrade pip'</span> command.
root@vcenter <span class="o">[</span> ~ <span class="o">]</span><span class="c"># /usr/lib/vmware-updatemgr/bin/updatemgr-utility.py reset-db</span>
Traceback <span class="o">(</span>most recent call last<span class="o">)</span>:
File <span class="s2">"/usr/lib/vmware-updatemgr/bin/updatemgr-utility.py"</span>, line 11, <span class="k">in</span> <module>
from vumUtils import registerWithVc, resetDB, manageConfig
File <span class="s2">"/usr/lib/vmware-updatemgr/bin/vumUtils.py"</span>, line 22, <span class="k">in</span> <module>
from cis.cisreglib import SsoClient
File <span class="s2">"/usr/lib/vmware/site-packages/cis/cisreglib.py"</span>, line 32, <span class="k">in</span> <module>
from pyVim import sso
File <span class="s2">"/usr/lib/vmware/site-packages/pyVim/sso.py"</span>, line 28, <span class="k">in</span> <module>
from OpenSSL import crypto
ImportError: No module named OpenSSL
</code></pre></div></div>
<p>This one was a bit trickier as there’s no module named OpenSSL, but a quick search found that the module I was looking for was named pyopenssl.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@vcenter <span class="o">[</span> ~ <span class="o">]</span><span class="c"># pip install pyopenssl</span>
Collecting pyopenssl
Downloading https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl <span class="o">(</span>53kB<span class="o">)</span>
100% |████████████████████████████████| 61kB 4.3MB/s
Collecting cryptography><span class="o">=</span>2.3 <span class="o">(</span>from pyopenssl<span class="o">)</span>
Downloading https://files.pythonhosted.org/packages/e6/68/50698ce24c61db7d44d93a5043c621a0ca7839d4ef9dff913e6ab465fc92/cryptography-2.7-cp27-cp27mu-manylinux1_x86_64.whl <span class="o">(</span>2.3MB<span class="o">)</span>
100% |████████████████████████████████| 2.3MB 401kB/s
Requirement already satisfied <span class="o">(</span>use <span class="nt">--upgrade</span> to upgrade<span class="o">)</span>: six><span class="o">=</span>1.5.2 <span class="k">in</span> /usr/lib/python2.7/site-packages <span class="o">(</span>from pyopenssl<span class="o">)</span>
Collecting enum34<span class="p">;</span> python_version < <span class="s2">"3"</span> <span class="o">(</span>from cryptography><span class="o">=</span>2.3->pyopenssl<span class="o">)</span>
Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting asn1crypto><span class="o">=</span>0.21.0 <span class="o">(</span>from cryptography><span class="o">=</span>2.3->pyopenssl<span class="o">)</span>
Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl <span class="o">(</span>101kB<span class="o">)</span>
100% |████████████████████████████████| 102kB 10.4MB/s
Collecting cffi!<span class="o">=</span>1.11.3,><span class="o">=</span>1.8 <span class="o">(</span>from cryptography><span class="o">=</span>2.3->pyopenssl<span class="o">)</span>
Downloading https://files.pythonhosted.org/packages/8d/e9/0c8afd1579e5cf7bc0f06fbcd7cdb954cbc0baadd505973949a99337da1c/cffi-1.12.3-cp27-cp27mu-manylinux1_x86_64.whl <span class="o">(</span>415kB<span class="o">)</span>
100% |████████████████████████████████| 419kB 2.5MB/s
Collecting ipaddress<span class="p">;</span> python_version < <span class="s2">"3"</span> <span class="o">(</span>from cryptography><span class="o">=</span>2.3->pyopenssl<span class="o">)</span>
Downloading https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting pycparser <span class="o">(</span>from cffi!<span class="o">=</span>1.11.3,><span class="o">=</span>1.8->cryptography><span class="o">=</span>2.3->pyopenssl<span class="o">)</span>
Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz <span class="o">(</span>158kB<span class="o">)</span>
100% |████████████████████████████████| 163kB 7.4MB/s
Installing collected packages: enum34, asn1crypto, pycparser, cffi, ipaddress, cryptography, pyopenssl
Running setup.py <span class="nb">install </span><span class="k">for </span>pycparser ... <span class="k">done
</span>Successfully installed asn1crypto-0.24.0 cffi-1.12.3 cryptography-2.7 enum34-1.1.6 ipaddress-1.0.22 pycparser-2.19 pyopenssl-19.0.0
You are using pip version 8.1.2, however version 19.2.2 is available.
You should consider upgrading via the <span class="s1">'pip install --upgrade pip'</span> command.
</code></pre></div></div>
<p>pyopenssl has a few dependencies which pip also graciously installed. Finally the database reset script completed and I was able to remove the patch store and start the update manager service.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@vcenter <span class="o">[</span> ~ <span class="o">]</span><span class="c"># /usr/lib/vmware-updatemgr/bin/updatemgr-utility.py reset-db</span>
Resetting vSphere Update Manager database...
Successfully reset vSphere Update Manager database
Start vSphere Update Manager service to apply the setting.
root@vcenter <span class="o">[</span> ~ <span class="o">]</span><span class="c"># rm -rf /storage/updatemgr/patch-store/*</span>
root@vcenter <span class="o">[</span> ~ <span class="o">]</span><span class="c"># service-control --start vmware-updatemgr</span>
Operation not cancellable. Please <span class="nb">wait </span><span class="k">for </span>it to finish...
Performing start operation on service updatemgr...
Successfully started service updatemgr
</code></pre></div></div>
<p>After recreating the ESXi 6.7 image and baseline I had no further issues with upgrading and patching the remaining hosts! To summarize, the missing modules were ipaddr, lxml, and pyopenssl.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip <span class="nb">install </span>ipaddr
pip <span class="nb">install </span>lxml
pip <span class="nb">install </span>pyopenssl
</code></pre></div></div>{"name"=>"", "avatar"=>"/assets/images/profile.jpg", "bio"=>"Doug is an IT professional and technologist focusing on VMware technologies, virtualization, Powershell, automation and IT/Cloud infrastructure.", "location"=>"Atlanta, GA", "email"=>nil, "uri"=>nil, "home"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>"doogleit", "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>"dougtaliaferro", "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>"virtually_doug", "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}I recently upgraded an environment to vSphere 6.7 Update 1. After updating vCenter I updated a number of hosts with vSphere Update Manager before running into this problem. During the remediation pre-check it would just spin its wheels until it eventually timed out. I tested this on several hosts and it was consistent across all of them. This pre-check error was occurring on all of the remaining ESXi 6.5 hosts in vCenter. When it did eventually time out I would see this very long error message.Automating Host Maintenance with VMware Dispatch Part 22019-08-23T00:00:00-04:002019-08-23T00:00:00-04:00https://doogleit.github.io/2019/08/automating-host-maintenance-with-dispatch-part-2<p>For part 2 of <a href="https://doogleit.github.io/2019/07/automating-host-maintenance-with-dispatch/">Automating Host Maintenance with VMware Dispatch</a> we’re going to create a function that will unmanage the node in Solarwinds when a host enters maintenance mode and re-manage it when it exits maintenance. In SolarWinds unmanaging a node is similar to putting an object in maintenance in vROps. It stops polling that node and no new alerts will trigger. An alternative is to simply mute alerts, which continues polling the node/collecting statistics, but mutes new alerts. If you’d prefer to only mute alerts I have an example of how to do that commented out in the script. However with only alerts muted the node will still turn red on dashboards and show that it is down if it is rebooted. If you have a NOC or other team monitoring for things like this it is probably preferable to unmanage it.</p>
<p>First, we need to create a secret that Dispatch can use to access SolarWinds. The SolarWinds API documentation refers to it as the <a href="https://github.com/solarwinds/OrionSDK/wiki/About-SWIS">SolarWinds Information Service (SWIS)</a>, so I will be using the “SWIS” acronym in the examples below. Create a json file that looks something like this.</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="nl">"swishost"</span><span class="p">:</span><span class="w"> </span><span class="s2">"swishost.domain.com"</span><span class="p">,</span><span class="w">
</span><span class="nl">"swisuser"</span><span class="p">:</span><span class="w"> </span><span class="s2">"<username>"</span><span class="p">,</span><span class="w">
</span><span class="nl">"swispass"</span><span class="p">:</span><span class="w"> </span><span class="s2">"<password>"</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>Fill in the values for your environment and save this to a file named swis.json. Then create the secret.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>dispatch create secret swis swis.json
Created secret: swis</code></pre></figure>
<p>Next download the Powershell script.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">curl <span class="nt">-ko</span> Set-SwisNodeState.ps1 <span class="s1">'https://raw.githubusercontent.com/doogleit/powercli-misc/master/dispatch/Set-SwisNodeState.ps1'</span></code></pre></figure>
<p>I had originally planned to use the SolarWinds Powershell module, <a href="https://github.com/solarwinds/OrionSDK/wiki/PowerShell">SwisPowerShell</a>, however it isn’t compatible with Powershell Core, so it won’t work in the powershell-base image in Dispatch. Instead I’m using the <a href="https://github.com/solarwinds/OrionSDK/wiki/REST">SWIS REST API</a> below and the native Powershell cmdlet “Invoke-RESTMethod”. This Powershell script is similar to the one used previously to disable/enable host alarm actions. I’m just going to highlight a few of the key sections below.</p>
<figure class="highlight"><pre><code class="language-powershell" data-lang="powershell"><span class="w"> </span><span class="c"># Get variables from the Swis secret</span><span class="w">
</span><span class="nv">$swishost</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$context</span><span class="o">.</span><span class="nf">secrets</span><span class="o">.</span><span class="nf">swishost</span><span class="w">
</span><span class="nv">$swisuser</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$context</span><span class="o">.</span><span class="nf">secrets</span><span class="o">.</span><span class="nf">swisuser</span><span class="w">
</span><span class="nv">$swispass</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$context</span><span class="o">.</span><span class="nf">secrets</span><span class="o">.</span><span class="nf">swispass</span><span class="w">
</span><span class="o">...</span><span class="w">
</span><span class="c"># Get the Swis node with a matching hostname</span><span class="w">
</span><span class="c"># NOTE: DisplayName in SolarWinds needs to match hostname in vCenter</span><span class="w">
</span><span class="nv">$response</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">Invoke-RESTMethod</span><span class="w"> </span><span class="nt">-Credential</span><span class="w"> </span><span class="nv">$swiscred</span><span class="w"> </span><span class="nt">-SkipCertificateCheck</span><span class="w"> </span><span class="nt">-Uri</span><span class="w"> </span><span class="s2">"https://</span><span class="si">$(</span><span class="nv">$swishost</span><span class="si">)</span><span class="s2">:17778/SolarWinds/InformationService/v3/Json/Query?query=SELECT+NodeId,+Uri+FROM+Orion.Nodes+WHERE+DisplayName='</span><span class="si">$(</span><span class="nv">$event</span><span class="o">.</span><span class="nf">Host</span><span class="o">.</span><span class="nf">Name</span><span class="p">)</span><span class="s1">'"
$node = $response.results
...
if ($event.GetType().Name -match "EnteredMaintenanceMode") {
# Unmanage the node
Write-Host "Unmanaging node $($event.Host.Name)"
$now = [DateTime]::UtcNow
$until = $now.AddDays(365)
Invoke-RESTMethod -Credential $swiscred -Method Post -ContentType "application/json" -SkipCertificateCheck -Uri "https://$($swishost):17778/SolarWinds/InformationService/v3/Json/Invoke/Orion.Nodes/Unmanage" -Body "[`"N:$nodeId`", `"$now`", `"$until`", `"false`"]"
# To supress/mute alerts instead use:
#Invoke-RESTMethod -Credential $swiscred -Method Post -ContentType "application/json" -SkipCertificateCheck -Uri "https://$($swishost):17778/SolarWinds/InformationService/v3/Json/Invoke/Orion.AlertSuppression/SuppressAlerts" -Body "[[`"$nodeUri`"]]"
}
else {
# Remanage the node
Write-Host "Remanaging node $($event.Host.Name)"
Invoke-RESTMethod -Credential $swiscred -Method Post -ContentType "application/json" -SkipCertificateCheck -Uri "https://$($swishost):17778/SolarWinds/InformationService/v3/Json/Invoke/Orion.Nodes/Remanage" -Body "[`"N:$nodeId`"]"
# To resume alerts instead use:
#Invoke-RESTMethod -Credential $swiscred -Method Post -ContentType "application/json" -SkipCertificateCheck -Uri "https://$($swishost):17778/SolarWinds/InformationService/v3/Json/Invoke/Orion.AlertSuppression/ResumeAlerts" -Body "[[`"$nodeUri`"]]"
}</span></code></pre></figure>
<p>To only mute/suppress alerts comment out the first <code class="language-plaintext highlighter-rouge">Invoke-RESTMethod</code> in each section and uncomment the second one. An advantage of using the REST API is that there’s no additional modules needed. We can use the same powershell-powercli image that I used in <a href="https://doogleit.github.io/tags/#dispatch">previous articles</a> and that is created in the <a href="https://vmware.github.io/dispatch/documentation/examples/vcenter-events">Dispatch example documentation</a>.</p>
<p>We’ll use a YAML file to describe the function and two subscriptions for the events when hosts enter and exit maintenance mode.</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">kind</span><span class="pi">:</span> <span class="s">Function</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">set-swisnodestate</span>
<span class="na">sourcePath</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Set-SwisNodeState.ps1'</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">powercli-swis</span>
<span class="na">schema</span><span class="pi">:</span> <span class="pi">{}</span>
<span class="na">secrets</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">vsphere</span>
<span class="pi">-</span> <span class="s">swis</span>
<span class="nn">---</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Subscription</span>
<span class="na">eventtype</span><span class="pi">:</span> <span class="s">entered.maintenance.mode</span>
<span class="na">function</span><span class="pi">:</span> <span class="s">set-swisnodestate</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">set-swisnodestate-entermaint</span>
<span class="nn">---</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Subscription</span>
<span class="na">eventtype</span><span class="pi">:</span> <span class="s">exit.maintenance.mode</span>
<span class="na">function</span><span class="pi">:</span> <span class="s">set-swisnodestate</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">set-swisnodestate-exitmaint</span></code></pre></figure>
<p>Now providing the YAML file to <code class="language-plaintext highlighter-rouge">dispatch create</code> will create the function and subscriptions.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>dispatch create <span class="nt">-f</span> <span class="s1">'https://raw.githubusercontent.com/doogleit/powercli-misc/master/dispatch/set-swisnodestate.yml'</span>
Created Function: set-swisnodestate
Created Subscription: set-swisnodestate-entermaint
Created Subscription: set-swisnodestate-exitmaint</code></pre></figure>
<p>Having most of the configuration in a YAML file like this also makes the cleanup rather easy. If you decide not to use this function <code class="language-plaintext highlighter-rouge">dispatch delete</code> will happily accept the same file and delete everything for you.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>dispatch delete <span class="nt">-f</span> <span class="s1">'https://raw.githubusercontent.com/doogleit/powercli-misc/master/dispatch/set-swisnodestate.yml'</span>
Deleted Function: set-swisnodestate
Deleted Subscription: set-swisnodestate-entermaint
Deleted subscription: set-swisnodestate-exitmaint</code></pre></figure>
<p>And finally if you’d like to make any changes, download a local copy to modify and simply run <code class="language-plaintext highlighter-rouge">dispatch update</code>:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>dispatch update <span class="nt">-f</span> set-swisnodestate.yml
Updated Function: set-swisnodestate
Updated Subscription: set-swisnodestate-entermaint
Updated Subscription: set-swisnodestate-exitmaint</code></pre></figure>
<p>One final note - it’s also possible to have secrets defined in a yaml file.</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">kind</span><span class="pi">:</span> <span class="s">Secret</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">swis</span>
<span class="na">secrets</span><span class="pi">:</span>
<span class="na">swishost</span><span class="pi">:</span> <span class="s">swishost.domain.com</span>
<span class="na">swisuser</span><span class="pi">:</span> <span class="s"><username></span>
<span class="na">swispass</span><span class="pi">:</span> <span class="s"><password></span></code></pre></figure>
<p>Generally speaking it’s not a good idea to store credentials in plain text. The json file above has the same problem, but you can delete the “swis.json” file after the secret is created in Dispatch. For this reason I’m keeping the secret separate, ideally stored in a secure location away from the rest of the configuration.</p>
<p>That wraps up today’s post. I’m really enjoying some of the new posibilities with Dispatch. If you’re interested, here are a few references that I used for the SolarWinds API.</p>
<ul>
<li><a href="https://github.com/solarwinds/OrionSDK/wiki/About-SWIS">About the SolarWinds Information Service</a></li>
<li><a href="https://github.com/solarwinds/OrionSDK/wiki/REST">SWIS REST/JSON Endpoint</a></li>
<li><a href="https://github.com/solarwinds/OrionSDK/wiki/PowerShell">Using the SolarWinds Information Service from PowerShell</a></li>
<li><a href="https://github.com/solarwinds/OrionSDK/wiki/Alerts#orionalertsuppression">Orion.AlertSuppression</a></li>
</ul>{"name"=>"", "avatar"=>"/assets/images/profile.jpg", "bio"=>"Doug is an IT professional and technologist focusing on VMware technologies, virtualization, Powershell, automation and IT/Cloud infrastructure.", "location"=>"Atlanta, GA", "email"=>nil, "uri"=>nil, "home"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>"doogleit", "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>"dougtaliaferro", "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>"virtually_doug", "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}For part 2 of Automating Host Maintenance with VMware Dispatch we’re going to create a function that will unmanage the node in Solarwinds when a host enters maintenance mode and re-manage it when it exits maintenance. In SolarWinds unmanaging a node is similar to putting an object in maintenance in vROps. It stops polling that node and no new alerts will trigger. An alternative is to simply mute alerts, which continues polling the node/collecting statistics, but mutes new alerts. If you’d prefer to only mute alerts I have an example of how to do that commented out in the script. However with only alerts muted the node will still turn red on dashboards and show that it is down if it is rebooted. If you have a NOC or other team monitoring for things like this it is probably preferable to unmanage it.Automating Host Maintenance with VMware Dispatch2019-07-21T00:00:00-04:002019-07-21T00:00:00-04:00https://doogleit.github.io/2019/07/automating-host-maintenance-with-dispatch<p>Following up on my previous post, <a href="https://doogleit.github.io/2019/07/event-based-automation-with-dispatch/">Event Based Automation with VMware Dispatch</a>, this one will demonstrate using Dispatch to automate actions when a host enters or exits maintenance mode. The Dispatch function we’re going to create will disable alarm actions on the host when it enters maintenance mode and re-enable alarm actions when the host exits maintenance mode.</p>
<p>Here is the Powershell script to disable/enable alarm actions on a host in vCenter.</p>
<figure class="highlight"><pre><code class="language-powershell" data-lang="powershell"><span class="kr">function</span><span class="w"> </span><span class="nf">handle</span><span class="p">(</span><span class="nv">$context</span><span class="p">,</span><span class="w"> </span><span class="nv">$payload</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="cm"><#
</span><span class="cs">.SYNOPSIS</span><span class="cm">
Disables/enables vCenter alarm actions on a host.
</span><span class="cs">.DESCRIPTION</span><span class="cm">
The 'handle' function is called by a VMware Dispatch event subscription
when a host enters/exits maintenance mode to disable/enable alarm actions
for the host.
</span><span class="cs">.LINK</span><span class="cm">
https://vmware.github.io/dispatch/documentation/examples/vcenter-events
#></span><span class="w">
</span><span class="nv">$username</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$context</span><span class="o">.</span><span class="nf">secrets</span><span class="o">.</span><span class="nf">username</span><span class="w">
</span><span class="nv">$password</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$context</span><span class="o">.</span><span class="nf">secrets</span><span class="o">.</span><span class="nf">password</span><span class="w">
</span><span class="nv">$hostname</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$context</span><span class="o">.</span><span class="nf">secrets</span><span class="o">.</span><span class="nf">host</span><span class="w">
</span><span class="nv">$eventTime</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="kt">DateTime</span><span class="p">]</span><span class="nv">$payload</span><span class="o">.</span><span class="nf">time</span><span class="w">
</span><span class="nv">$eventMessage</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$payload</span><span class="o">.</span><span class="nf">message</span><span class="w">
</span><span class="nx">Import-Module</span><span class="w"> </span><span class="nx">vmware.vimautomation.core</span><span class="w"> </span><span class="nt">-verbose</span><span class="p">:</span><span class="bp">$false</span><span class="w">
</span><span class="nx">Set-PowerCLIConfiguration</span><span class="w"> </span><span class="nt">-InvalidCertificateAction</span><span class="w"> </span><span class="nx">Ignore</span><span class="w"> </span><span class="nt">-Confirm</span><span class="p">:</span><span class="bp">$false</span><span class="w">
</span><span class="c"># Connect to vSphere</span><span class="w">
</span><span class="nf">Write-Host</span><span class="w"> </span><span class="s2">"Checking VC Connection is active"</span><span class="w">
</span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="o">-not</span><span class="w"> </span><span class="nv">$</span><span class="nn">global</span><span class="p">:</span><span class="nv">defaultviservers</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nf">Write-Host</span><span class="w"> </span><span class="s2">"Connecting to </span><span class="nv">$hostname</span><span class="s2">"</span><span class="w">
</span><span class="nv">$viserver</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">Connect-VIServer</span><span class="w"> </span><span class="nt">-server</span><span class="w"> </span><span class="nv">$hostname</span><span class="w"> </span><span class="nt">-User</span><span class="w"> </span><span class="nv">$username</span><span class="w"> </span><span class="nt">-Password</span><span class="w"> </span><span class="nv">$password</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="kr">else</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nf">Write-Host</span><span class="w"> </span><span class="s2">"Already connected to </span><span class="nv">$hostname</span><span class="s2">"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="c"># Get the event by filtering on time and message</span><span class="w">
</span><span class="nv">$eventManager</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">Get-View</span><span class="w"> </span><span class="p">(</span><span class="nf">Get-View</span><span class="w"> </span><span class="nx">ServiceInstance</span><span class="p">)</span><span class="o">.</span><span class="nf">Content</span><span class="o">.</span><span class="nf">EventManager</span><span class="w">
</span><span class="nv">$eventFilterSpec</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">New-Object</span><span class="w"> </span><span class="nx">VMware.Vim.EventFilterSpec</span><span class="w">
</span><span class="bp">$Event</span><span class="nf">FilterSpec.Time</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">New-Object</span><span class="w"> </span><span class="nx">VMware.Vim.EventFilterSpecByTime</span><span class="w">
</span><span class="bp">$Event</span><span class="nf">FilterSpec.Time.beginTime</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$eventTime</span><span class="w">
</span><span class="bp">$Event</span><span class="nf">FilterSpec.Time.endTime</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$eventTime</span><span class="o">.</span><span class="nf">AddSeconds</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w">
</span><span class="nv">$events</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$eventManager</span><span class="o">.</span><span class="nf">QueryEvents</span><span class="p">(</span><span class="bp">$Event</span><span class="nf">FilterSpec</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nf">Where-Object</span><span class="w"> </span><span class="s1">'FullFormattedMessage'</span><span class="w"> </span><span class="o">-eq</span><span class="w"> </span><span class="nv">$eventMessage</span><span class="w">
</span><span class="c"># For some reason there are always two "EnteredMaintenanceMode" events. One has an 'Unknown user' and no related events.</span><span class="w">
</span><span class="c"># We'll filter out that one and only use the event with a valid user and related events.</span><span class="w">
</span><span class="nv">$event</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$events</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nf">Where-Object</span><span class="w"> </span><span class="s1">'UserName'</span><span class="w"> </span><span class="o">-ne</span><span class="w"> </span><span class="s1">'Unknown user'</span><span class="w">
</span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="nv">$event</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nv">$alarmManager</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">Get-View</span><span class="w"> </span><span class="nx">AlarmManager</span><span class="w">
</span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="nv">$event</span><span class="o">.</span><span class="nf">GetType</span><span class="p">()</span><span class="o">.</span><span class="nf">Name</span><span class="w"> </span><span class="o">-match</span><span class="w"> </span><span class="s2">"EnteredMaintenanceMode"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="c"># Disable alarm actions on the host. "$event.Host.Host" is the MoRef of the host.</span><span class="w">
</span><span class="nf">Write-Host</span><span class="w"> </span><span class="s2">"Disabling alarm actions on host </span><span class="si">$(</span><span class="nv">$event</span><span class="o">.</span><span class="nf">Host</span><span class="o">.</span><span class="nf">Name</span><span class="p">)</span><span class="s2">"
</span><span class="nv">$alarmManager</span><span class="s2">.EnableAlarmActions(</span><span class="nv">$event</span><span class="s2">.Host.Host, </span><span class="bp">$false</span><span class="s2">)
}
else {
# Enable alarm actions on the host. "</span><span class="nv">$event</span><span class="o">.</span><span class="nf">Host</span><span class="o">.</span><span class="nf">Host</span><span class="s2">" is the MoRef of the host.
Write-Host "</span><span class="nx">Enabling</span><span class="w"> </span><span class="nx">alarm</span><span class="w"> </span><span class="nx">actions</span><span class="w"> </span><span class="nx">on</span><span class="w"> </span><span class="nx">host</span><span class="w"> </span><span class="err">$</span><span class="p">(</span><span class="nv">$event</span><span class="o">.</span><span class="nf">Host</span><span class="o">.</span><span class="nf">Name</span><span class="p">)</span><span class="s2">"
</span><span class="nv">$alarmManager</span><span class="s2">.EnableAlarmActions(</span><span class="nv">$event</span><span class="s2">.Host.Host, </span><span class="bp">$true</span><span class="s2">)
}
}
else {
Write-Host "</span><span class="nx">The</span><span class="w"> </span><span class="nx">event</span><span class="w"> </span><span class="nx">was</span><span class="w"> </span><span class="nx">not</span><span class="w"> </span><span class="nx">found.</span><span class="s2">"
}
return "</span><span class="nx">success</span><span class="s2">"
}</span></code></pre></figure>
<p>This is also available on <a href="https://github.com/doogleit/powercli-misc/tree/master/dispatch">Github</a>. Save the above code to a file named ‘Set-HostAlarmActions.ps1’ or grab it using curl:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>curl <span class="nt">-ko</span> Set-HostAlarmActions.ps1 <span class="s1">'https://raw.githubusercontent.com/doogleit/powercli-misc/master/dispatch/Set-HostAlarmActions.ps1'</span></code></pre></figure>
<p>This time to create the Dispatch function we’re going to use a yaml file. Dispatch allows you to define functions (and other things) in yaml files. When you start working with more functions this is a convenient way to define them and it makes it easier to update a function while you are testing it and making frequent changes. Here is the yaml file to define this function.</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">kind</span><span class="pi">:</span> <span class="s">Function</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">set-hostalarmactions</span>
<span class="na">sourcePath</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Set-HostAlarmActions.ps1'</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">powershell-powercli</span>
<span class="na">schema</span><span class="pi">:</span> <span class="pi">{}</span>
<span class="na">secrets</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">vsphere</span></code></pre></figure>
<p>Now to create the function we just need to reference the yaml file.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>dispatch create <span class="nt">-f</span> set-hostalarmactions.yml
Created Function: set-hostalarmactions</code></pre></figure>
<p>And to update the function we simply use <code class="language-plaintext highlighter-rouge">dispatch update</code>, referencing the same yaml file.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>dispatch update <span class="nt">-f</span> set-hostalarmactions.yml
Updated Function: set-hostalarmactions</code></pre></figure>
<p>Once the function is created we just need to subscribe it to the events for a host entering and exiting maintenance mode.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>dispatch create subscription <span class="nt">--event-type</span> entered.maintenance.mode set-hostalarmactions <span class="nt">--name</span> set-hostalarmactions-entermaint
created subscription: set-hostalarmactions-entermaint
<span class="nv">$ </span>dispatch create subscription <span class="nt">--event-type</span> exit.maintenance.mode set-hostalarmactions <span class="nt">--name</span> set-hostalarmactions-exitmaint
created subscription: set-hostalarmactions-exitmaint</code></pre></figure>
<p>You can have multiple functions in the same yaml file. There’s a more complete example in the <a href="https://vmware.github.io/dispatch/documentation/usage/functions">documentation on Functions</a>. You can also have different items in the same file, for example combining the function and subscriptions:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">kind</span><span class="pi">:</span> <span class="s">Function</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">set-hostalarmactions</span>
<span class="na">sourcePath</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Set-HostAlarmActions.ps1'</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">powershell-powercli</span>
<span class="na">schema</span><span class="pi">:</span> <span class="pi">{}</span>
<span class="na">secrets</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">vsphere</span>
<span class="nn">---</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Subscription</span>
<span class="na">eventtype</span><span class="pi">:</span> <span class="s">entered.maintenance.mode</span>
<span class="na">function</span><span class="pi">:</span> <span class="s">set-hostalarmactions</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">set-hostalarmactions-entermaint</span>
<span class="nn">---</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Subscription</span>
<span class="na">eventtype</span><span class="pi">:</span> <span class="s">exit.maintenance.mode</span>
<span class="na">function</span><span class="pi">:</span> <span class="s">set-hostalarmactions</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">set-hostalarmactions-exitmaint</span></code></pre></figure>
<p>Additionally, you can reference the yaml file over http. It appears that the function file (Set-HostAlarmActions.ps1 in this case) does however need to be on the local filesystem, but this breaks the setup down to essentially two commands.</p>
<ol>
<li>Download the function file:</li>
</ol>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>curl <span class="nt">-ko</span> Set-HostAlarmActions.ps1 <span class="s1">'https://raw.githubusercontent.com/doogleit/powercli-misc/master/dispatch/Set-HostAlarmActions.ps1'</span></code></pre></figure>
<ol>
<li>Create/update everything by referencing the YAML file:</li>
</ol>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>dispatch create <span class="nt">-f</span> <span class="s1">'https://raw.githubusercontent.com/doogleit/powercli-misc/master/dispatch/set-hostalarmactions.yml'</span></code></pre></figure>
<p>The documentation on yaml files is a little sparse at the moment, but you can retrieve the yaml for an object in Dispatch like so:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>dispatch get subscription <span class="nt">-o</span> yaml
- createdtime: 1560881481
eventtype: entered.maintenance.mode
<span class="k">function</span>: set-hostalarmactions
<span class="nb">id</span>: <span class="s2">""</span>
kind: Subscription
modifiedtime: 1561405311
name: set-hostalarmactions-entermaint
secrets: <span class="o">[]</span>
status: READY
tags: <span class="o">[]</span>
- createdtime: 1561388377
eventtype: exit.maintenance.mode
<span class="k">function</span>: set-hostalarmactions
<span class="nb">id</span>: <span class="s2">""</span>
kind: Subscription
modifiedtime: 1561405311
name: set-hostalarmactions-exitmaint
secrets: <span class="o">[]</span>
status: READY
tags: <span class="o">[]</span></code></pre></figure>
<p>Using the output as a template, take the properties and values that you would normally specify on the command line and put those in your yaml file. With a little trial and error you can figure out what’s required.</p>
<p>I’m working on another function to do essentially the same thing with nodes in SolarWinds. If you use SolarWinds for monitoring it will unmanage a host in Solarwinds when it enters maintenance mode and re-manage it when it exits maintenance. Look for a part 2 to this post, hopefully coming soon!</p>{"name"=>"", "avatar"=>"/assets/images/profile.jpg", "bio"=>"Doug is an IT professional and technologist focusing on VMware technologies, virtualization, Powershell, automation and IT/Cloud infrastructure.", "location"=>"Atlanta, GA", "email"=>nil, "uri"=>nil, "home"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>"doogleit", "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>"dougtaliaferro", "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>"virtually_doug", "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}Following up on my previous post, Event Based Automation with VMware Dispatch, this one will demonstrate using Dispatch to automate actions when a host enters or exits maintenance mode. The Dispatch function we’re going to create will disable alarm actions on the host when it enters maintenance mode and re-enable alarm actions when the host exits maintenance mode.Event Based Automation with VMware Dispatch2019-07-11T00:00:00-04:002019-07-11T00:00:00-04:00https://doogleit.github.io/2019/07/event-based-automation-with-dispatch<p>One of the things that immediately drew my attention to Dispatch was the vCenter event driver. Executing an automated task based on a vCenter event has always been something really attractive to me. There are other ways to do this, but it always seemed like they were just not ideal for the environment I was working in for one reason or another. Most of the examples that I’ve seen use a vCenter alarm to run a command or send an SNMP trap.</p>
<h2 id="vcenter-alarm-actions">vCenter Alarm Actions</h2>
<p>Before diving into using Dispatch I’d like to share a couple of the issues I’ve run into when using alarm actions for automation. This isn’t to discourage anyone from using them. I’ve been using alarm actions to run scripts for several years and it works great. This is simply to provide a few reasons for my preference for an event driver like the one in Dispatch.</p>
<ol>
<li>
<p>Alarm actions can be disabled. It’s too easy to disable alarm actions on an object in vCenter and forget about it. For example, let’s say I’m putting a host in maintenance mode so I disable alarm actions on it. I don’t want everyone to be alerted while I’m rebooting a host that is in maintenance. Unfortunately, this also disables any alarm actions that I’m using for automation. It would actually be really cool to have alarms automatically enabled/disabled based on a host’s maintenance mode status. However, if I’m using alarm actions to trigger the automation, then once alarm actions are disabled they can’t turn themselves back on!</p>
</li>
<li>
<p>The alarm action to “run a command” may not be officially supported on the vCenter Server Appliance. Installing anything on the vCenter Server Appliance is unsupported and generally discouraged, so running a custom script on the vCSA from an alarm action may not be technically supported. With that said, it’s relatively unobtrusive to have a bash or Python script call an external REST API or start a process on another system when an alarm action is triggered on the vCSA. This is covered in William Lam’s article, <a href="https://www.virtuallyghetto.com/2016/06/how-to-run-a-script-from-a-vcenter-alarm-action-in-the-vcsa.html">How to run a script from a vCenter alarm action in the VCSA?</a> Also if you like Slack, be sure to check out this cool write up on <a href="https://www.greenreedtech.com/vsphere-alarms-with-slack-and-stackstorm/">vSphere Alarms with Slack and StackStorm</a>.</p>
</li>
<li>
<p>The SNMP configration isn’t very flexible. The alarm action to “send a notification trap” is sent to all SNMP receivers configured in vCenter. You can’t distinguish between sending a trap for a real alert/problem and sending one just to perform some automation. Generally, this isn’t a problem, but in a larger organization where you’re sending all of your SNMP traps to an enterprise monitoring system it could be. The monitoring team will need to filter out traps for alarms that are not really problems and only exist for the sake of automation. Some common examples of this are when a new VM is created or when a host is placed into maintenance mode. For an example covering SNMP traps on VM creation see <a href="https://blogs.vmware.com/vsphere/2012/07/automatically-securing-virtual-machines-using-vcenter-orchestrator.html">Automatically Securing Virtual Machines Using vCenter Orchestrator</a>.</p>
</li>
</ol>
<h2 id="why-dispatch">Why Dispatch?</h2>
<p>This leads me to <a href="https://vmware.github.io/dispatch/">VMware Dispatch</a> and its <a href="https://github.com/dispatchframework/dispatch-events-vcenter">vCenter event driver</a>. Dispatch runs on Kubernetes, so if you’re like me and have been looking for something to try out Kubernetes with then here it is. I haven’t had a lot of opportunites to make use of containers while working in more traditional/conservative IT organizations and thought this would be a cool way to try out Kubernetes. If you just want to get it up and running quickly, <a href="https://github.com/vmware/dispatch/tree/solo">Dispatch Solo</a> provides a quick-start option with all of the dependencies packaged in a single virtual appliance (OVA). Their <a href="https://vmware.github.io/dispatch/documentation/front/quickstart">quickstart page</a> covers deploying the OVA and a simple hello world example. This is how I’ve started out and plan to transistion to the Knative branch on Kubernetes in the near future.</p>
<p>Another feature of Dispatch that I really like is built-in support for Powershell Core. This makes it relatively easy to use existing Powershell/PowerCLI scripts and combine them with vCenter events. A script that I’ve had scheduled for many years is a variation of Alans Renouf’s original article, <a href="http://www.virtu-al.net/2010/02/23/who-created-that-vm/">Who Created That VM?</a>, which sets the date a VM was created and the user that created it as custom attributes. Instead of scheduling a script to run periodically and find all of the new VMs since the last time it was run, I think it’s more efficient to have it run automatically when a new VM is created. Setting VM attribbutes like this has been a common topic over the years, also being covered <a href="https://psvmware.wordpress.com/2012/11/27/when-was-vm-created-how-vm-was-created-and-who-has-created-this-vm/">here</a>, <a href="http://www.vmspot.com/a-very-simple-powercli-script-to-gather-vm-creation-dates/">here</a>, and in various discussions on the VMTN community forums. It’s worth noting that the <a href="https://www.virtuallyghetto.com/2018/04/vm-creation-date-now-available-in-vsphere-6-7.html">VM creation date is now available on VMs created in vSphere 6.7</a>, but you still might want to use one of the previous scripts if you’re not completely on vSphere 6.7 yet and/or if you also want to know who created the VM.</p>
<h2 id="setting-up-dispatch">Setting up Dispatch</h2>
<p>The Dispatch documentation has an example for <a href="https://vmware.github.io/dispatch/documentation/examples/vcenter-events">hardening VMs in vSphere</a>. Most of this will be a repeat of that process, except that I’m going to use a script I modified for Dispatch to set the VM’s attributes on creation.</p>
<p>Once you’ve deployed Dispatch you use the Dispatch CLI to manage it. Dispatch CLI can be downloaded and installed on macOS and Linux. On Windows 10 you can use <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">Windows Subsystem for Linux</a> or if you deployed Dispatch Solo you can SSH to the appliance and just run the commands there. I’m using Ubuntu on Windows Subsystem for Linux, but I’ve also SSH’d into the appliance in some cases and haven’t had any issues.</p>
<p>Here is a brief outline of the process to setup a PowerCLI script with the vCenter event driver:</p>
<ol>
<li>Create a Dispatch image with the dependencies we need (in this case VMware PowerCLI)</li>
<li>Create a secret with the vSphere credentials</li>
<li>Create a vCenter event driver using the previously created secret</li>
<li>Create a function from the PowerCLI script</li>
<li>Subscribe the new function to an event</li>
</ol>
<p>Steps 1 - 3 only need to be performed once - as long as the dependencies and vCenter credentials don’t change these items can be reused. Steps 4 and 5 are repeated for each new function you create. Instructions for creating the PowerCLI image, vCenter secret, and vCenter event driver are in the example, <a href="https://vmware.github.io/dispatch/documentation/examples/vcenter-events">Harden VMs in vSphere</a>. Those are prerequisites for the rest of this article.</p>
<p>One thing to note about the payload from the event driver is that not all event types contain the ‘vm_name’. The ‘vm.deployed’ event does (as demonstrated in the hardening VMs example), but the ‘vm.created’ event does not. Here’s an example of the input received from the event driver when a VM is deployed (i.e. deployed from template):</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="nl">"category"</span><span class="p">:</span><span class="w"> </span><span class="s2">"info"</span><span class="p">,</span><span class="w">
</span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Template Windows 2016 deployed on host esx01.domain.com"</span><span class="p">,</span><span class="w">
</span><span class="nl">"metadata"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"src_template"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Windows 2016"</span><span class="p">,</span><span class="w">
</span><span class="nl">"vm_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"VirtualMachine:vm-845219"</span><span class="p">,</span><span class="w">
</span><span class="nl">"vm_name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"testvm01"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"time"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2019-06-07T17:26:29.951027Z"</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>The metadata section contains some useful info about the VM being deployed. Here is an example when a VM is created (i.e. created from an OVA/OVF or wihtout a template):</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="nl">"category"</span><span class="p">:</span><span class="w"> </span><span class="s2">"info"</span><span class="p">,</span><span class="w">
</span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Created virtual machine testvm02 on esx01.domain.com in Lab"</span><span class="p">,</span><span class="w">
</span><span class="nl">"metadata"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"time"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2019-06-07T17:30:57.93864Z"</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>The second example applies to most other event types, such as when a VM is cloned or a host enters maintenance mode.</p>
<p>In the powershell script below I use the event time to retrieve events from vCenter at that time and then filter on the event message. It then has all the event information available to it and can set the “UserName” and “CreatedTime” in the VM’s “Created By” and “Created On” attributes. The other important piece here is the “handle” function, which sets some variables from the vCenter secret we provided and the input from the event driver. Dispatch looks for this function and will run it by default. You can also get this script from my <a href="https://github.com/doogleit/powercli-misc/tree/master/dispatch">GitHub repository</a>.</p>
<figure class="highlight"><pre><code class="language-powershell" data-lang="powershell"><span class="kr">function</span><span class="w"> </span><span class="nf">handle</span><span class="p">(</span><span class="nv">$context</span><span class="p">,</span><span class="w"> </span><span class="nv">$payload</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="cm"><#
</span><span class="cs">.SYNOPSIS</span><span class="cm">
Function to set custom attributes on a newly deployed VM.
</span><span class="cs">.DESCRIPTION</span><span class="cm">
The 'handle' function is called by a VMware Dispatch event subscription
when a new VM is deployed to automatically set the custom attributes
'Created On' and 'Created By' on the new VM.
</span><span class="cs">.LINK</span><span class="cm">
https://vmware.github.io/dispatch/documentation/examples/vcenter-events
#></span><span class="w">
</span><span class="nv">$username</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$context</span><span class="o">.</span><span class="nf">secrets</span><span class="o">.</span><span class="nf">username</span><span class="w">
</span><span class="nv">$password</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$context</span><span class="o">.</span><span class="nf">secrets</span><span class="o">.</span><span class="nf">password</span><span class="w">
</span><span class="nv">$hostname</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$context</span><span class="o">.</span><span class="nf">secrets</span><span class="o">.</span><span class="nf">host</span><span class="w">
</span><span class="nv">$eventTime</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="kt">DateTime</span><span class="p">]</span><span class="nv">$payload</span><span class="o">.</span><span class="nf">time</span><span class="w">
</span><span class="nv">$eventMessage</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$payload</span><span class="o">.</span><span class="nf">message</span><span class="w">
</span><span class="nx">Import-Module</span><span class="w"> </span><span class="nx">vmware.vimautomation.core</span><span class="w"> </span><span class="nt">-verbose</span><span class="p">:</span><span class="bp">$false</span><span class="w">
</span><span class="nx">Set-PowerCLIConfiguration</span><span class="w"> </span><span class="nt">-InvalidCertificateAction</span><span class="w"> </span><span class="nx">Ignore</span><span class="w"> </span><span class="nt">-Confirm</span><span class="p">:</span><span class="bp">$false</span><span class="w">
</span><span class="c"># Connect to vSphere</span><span class="w">
</span><span class="nf">Write-Host</span><span class="w"> </span><span class="s2">"Checking VC Connection is active"</span><span class="w">
</span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="o">-not</span><span class="w"> </span><span class="nv">$</span><span class="nn">global</span><span class="p">:</span><span class="nv">defaultviservers</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nf">Write-Host</span><span class="w"> </span><span class="s2">"Connecting to </span><span class="nv">$hostname</span><span class="s2">"</span><span class="w">
</span><span class="nv">$viserver</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">Connect-VIServer</span><span class="w"> </span><span class="nt">-server</span><span class="w"> </span><span class="nv">$hostname</span><span class="w"> </span><span class="nt">-User</span><span class="w"> </span><span class="nv">$username</span><span class="w"> </span><span class="nt">-Password</span><span class="w"> </span><span class="nv">$password</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="kr">else</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nf">Write-Host</span><span class="w"> </span><span class="s2">"Already connected to </span><span class="nv">$hostname</span><span class="s2">"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="c"># Get the event by filtering on time and message</span><span class="w">
</span><span class="nv">$eventManager</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">Get-View</span><span class="w"> </span><span class="p">(</span><span class="nf">Get-View</span><span class="w"> </span><span class="nx">ServiceInstance</span><span class="p">)</span><span class="o">.</span><span class="nf">Content</span><span class="o">.</span><span class="nf">EventManager</span><span class="w">
</span><span class="nv">$eventFilterSpec</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">New-Object</span><span class="w"> </span><span class="nx">VMware.Vim.EventFilterSpec</span><span class="w">
</span><span class="bp">$Event</span><span class="nf">FilterSpec.Time</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">New-Object</span><span class="w"> </span><span class="nx">VMware.Vim.EventFilterSpecByTime</span><span class="w">
</span><span class="bp">$Event</span><span class="nf">FilterSpec.Time.beginTime</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$eventTime</span><span class="w">
</span><span class="bp">$Event</span><span class="nf">FilterSpec.Time.endTime</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$eventTime</span><span class="o">.</span><span class="nf">AddSeconds</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w">
</span><span class="nv">$event</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$eventManager</span><span class="o">.</span><span class="nf">QueryEvents</span><span class="p">(</span><span class="bp">$Event</span><span class="nf">FilterSpec</span><span class="p">)</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="nf">Where-Object</span><span class="w"> </span><span class="s1">'FullFormattedMessage'</span><span class="w"> </span><span class="o">-eq</span><span class="w"> </span><span class="nv">$eventMessage</span><span class="w">
</span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="nv">$event</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nv">$vm</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">Get-VM</span><span class="w"> </span><span class="nt">-Name</span><span class="w"> </span><span class="nv">$event</span><span class="o">.</span><span class="nf">Vm</span><span class="o">.</span><span class="nf">Name</span><span class="w">
</span><span class="nx">if</span><span class="w"> </span><span class="p">(</span><span class="nv">$event</span><span class="o">.</span><span class="nf">UserName</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nf">Write-Host</span><span class="w"> </span><span class="s2">"Setting 'Created By' = '</span><span class="si">$(</span><span class="nv">$event</span><span class="o">.</span><span class="nf">UserName</span><span class="p">)</span><span class="s1">' on VM $VM"
Set-Annotation -Entity $VM -CustomAttribute "Created By" -Value $event.UserName | Out-Null
}
if ($event.CreatedTime) {
Write-Host "Setting '</span><span class="nf">Created</span><span class="w"> </span><span class="nx">On</span><span class="s1">' = '</span><span class="err">$</span><span class="p">(</span><span class="nv">$event</span><span class="o">.</span><span class="nf">CreatedTime</span><span class="p">)</span><span class="s1">' on VM $VM"
Set-Annotation -Entity $VM -CustomAttribute "Created On" -Value $event.CreatedTime | Out-Null
}
}
else {
Write-Host "The event was not found."
}
return "success"
}</span></code></pre></figure>
<p>Save the above code to a file named “Set-VMAttributes.ps1” or download it using <code class="language-plaintext highlighter-rouge">curl</code>:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>curl <span class="nt">-ko</span> Set-VMAttributes.ps1 <span class="s1">'https://raw.githubusercontent.com/doogleit/powercli-misc/master/dispatch/Set-VMAttributes.ps1'</span></code></pre></figure>
<p>Next we’ll create a Dispatch function and verify that the function is ready.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>dispatch create <span class="k">function </span>set-vmattributes Set-VMAttributes.ps1 <span class="nt">--image</span><span class="o">=</span>powershell-powercli <span class="nt">--secret</span> vsphere
Created <span class="k">function</span>: set-vmattributes
<span class="nv">$ </span>dispatch get <span class="k">function </span>set-vmattributes
NAME | FUNCTIONIMAGE | STATUS | CREATED DATE
<span class="nt">-------------------------------------------------------------------------------------------------------------------</span>
set-vmattributes | dispatch/func-c8eb1e08-4c43-4697-b9bb-2902fb490c89:latest | READY | Fri Jun 7 17:11:48 DST 2019</code></pre></figure>
<p>Finally, we’ll create event subscriptions for our new function. This ties the function we just created to the vCenter events when a VM is deployed, created, cloned, or registered.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>dispatch create subscription <span class="nt">--event-type</span> vm.deployed set-vmattributes <span class="nt">--name</span> set-vmattributes-vmdeployed
created subscription: set-vmattributes-vmdeployed
<span class="nv">$ </span>dispatch create subscription <span class="nt">--event-type</span> vm.created set-vmattributes <span class="nt">--name</span> set-vmattributes-vmcreated
created subscription: set-vmattributes-vmcreated
<span class="nv">$ </span>dispatch create subscription <span class="nt">--event-type</span> vm.cloned set-vmattributes <span class="nt">--name</span> set-vmattributes-vmcloned
created subscription: set-vmattributes-vmcloned
<span class="nv">$ </span>dispatch create subscription <span class="nt">--event-type</span> vm.registered set-vmattributes <span class="nt">--name</span> set-vmattributes-vmregistered
created subscription: set-vmattributes-vmregistered</code></pre></figure>
<p>Run <code class="language-plaintext highlighter-rouge">dispatch get subscription</code> to confirm that the subscriptions are all ready.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>dispatch get subscription
NAME | EVENT TYPE | FUNCTION NAME | STATUS | CREATED DATE
<span class="nt">------------------------------------------------------------------------------------------------------</span>
set-vmattributes-vmcloned | vm.cloned | set-vmattributes | READY | Fri Jun 7 18:16:10 DST 2019
set-vmattributes-vmcreated | vm.created | set-vmattributes | READY | Fri Jun 7 18:16:02 DST 2019
set-vmattributes-vmdeployed | vm.deployed | set-vmattributes | READY | Fri Jun 7 18:15:53 DST 2019
set-vmattributes-vmregistered | vm.registered | set-vmattributes | READY | Fri Jun 7 18:16:18 DST 2019</code></pre></figure>
<p>Deploy or create a couple of VMs and confirm that the function has run:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>dispatch get run set-vmattributes
ID | FUNCTION | STATUS | STARTED | FINISHED
<span class="nt">------------------------------------------------------------------------------------------------------------------------------------------</span>
16d3495e-0834-4bdc-a9c6-8bfc69a0c302 | set-vmattributes | READY | 2019-06-07T18:40:00.423122825-04:00 | 2019-06-07T18:40:02.240076238-04:00
6ae02fb0-9ca1-48ae-922c-fa6ccf9df876 | set-vmattributes | READY | 2019-06-07T18:34:59.099968839-04:00 | 2019-06-07T18:35:07.330598053-04:00</code></pre></figure>
<p>To see all of the output from a particular run, copy the ID from the above list and use it in the command below.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>dispatch get run set-vmattributes 16d3495e-0834-4bdc-a9c6-8bfc69a0c302 <span class="nt">--json</span></code></pre></figure>
<p>This produces a lot of useful information for troubleshooting, such as the event type, the input to the function, and all of the function’s stdout and stderr.</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="nl">"event"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"eventType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"vm.deployed"</span><span class="p">,</span><span class="w">
</span><span class="nl">"eventTypeVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.1"</span><span class="p">,</span><span class="w">
</span><span class="nl">"cloudEventsVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.1"</span><span class="p">,</span><span class="w">
</span><span class="nl">"source"</span><span class="p">:</span><span class="w"> </span><span class="s2">"vcenter1"</span><span class="p">,</span><span class="w">
</span><span class="nl">"eventID"</span><span class="p">:</span><span class="w"> </span><span class="s2">"8a320cc1-4fe9-42bd-9e4d-5499fb826f34"</span><span class="p">,</span><span class="w">
</span><span class="nl">"eventTime"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0001-01-01T00:00:00.000Z"</span><span class="p">,</span><span class="w">
</span><span class="nl">"contentType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/json"</span><span class="p">,</span><span class="w">
</span><span class="nl">"data"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"executedTime"</span><span class="p">:</span><span class="w"> </span><span class="mi">1560534000423122825</span><span class="p">,</span><span class="w">
</span><span class="nl">"faasId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"8eb037f0-d06e-4add-bf2b-43f6ab2fb57f"</span><span class="p">,</span><span class="w">
</span><span class="nl">"finishedTime"</span><span class="p">:</span><span class="w"> </span><span class="mi">1560534002240076238</span><span class="p">,</span><span class="w">
</span><span class="nl">"functionId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"39b72f36-aa87-47e6-80f3-dcc37bff1316"</span><span class="p">,</span><span class="w">
</span><span class="nl">"functionName"</span><span class="p">:</span><span class="w"> </span><span class="s2">"set-vmattributes"</span><span class="p">,</span><span class="w">
</span><span class="nl">"input"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"category"</span><span class="p">:</span><span class="w"> </span><span class="s2">"info"</span><span class="p">,</span><span class="w">
</span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Template Windows 2016 deployed on host esx03.domain.com"</span><span class="p">,</span><span class="w">
</span><span class="nl">"metadata"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"src_template"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Windows 2016"</span><span class="p">,</span><span class="w">
</span><span class="nl">"vm_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"VirtualMachine:vm-847587"</span><span class="p">,</span><span class="w">
</span><span class="nl">"vm_name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"testvm04"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"time"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2019-06-07T18:40:00.004554Z"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"logs"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"stderr"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"stdout"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"Checking VC Connection is active"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Already connected to vcenter.nghs.com"</span><span class="p">,</span><span class="w">
</span><span class="s2">"VERBOSE: 06/07/2019 18:40:00</span><span class="se">\t</span><span class="s2">Get-View</span><span class="se">\t</span><span class="s2">Finished execution"</span><span class="p">,</span><span class="w">
</span><span class="s2">"DEBUG: 06/07/2019 18:40:00</span><span class="se">\t</span><span class="s2">Get-View</span><span class="se">\t</span><span class="s2">"</span><span class="p">,</span><span class="w">
</span><span class="s2">"VERBOSE: 06/07/2019 18:40:00</span><span class="se">\t</span><span class="s2">Get-View</span><span class="se">\t</span><span class="s2">Finished execution"</span><span class="p">,</span><span class="w">
</span><span class="s2">"DEBUG: 06/07/2019 18:40:00</span><span class="se">\t</span><span class="s2">Get-View</span><span class="se">\t</span><span class="s2">"</span><span class="p">,</span><span class="w">
</span><span class="s2">"VERBOSE: 06/07/2019 18:40:02</span><span class="se">\t</span><span class="s2">Get-VM</span><span class="se">\t</span><span class="s2">Finished execution"</span><span class="p">,</span><span class="w">
</span><span class="s2">"DEBUG: 06/07/2019 18:40:02</span><span class="se">\t</span><span class="s2">Get-VM</span><span class="se">\t</span><span class="s2">"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Setting 'Created By' = 'administrator@vsphere.local' on VM testvm04"</span><span class="p">,</span><span class="w">
</span><span class="s2">"DEBUG: 06/07/2019 18:40:02</span><span class="se">\t</span><span class="s2">Set-Annotation</span><span class="se">\t</span><span class="s2">5246050e-1666-5ef9-f687-464457e7abba</span><span class="se">\t</span><span class="s2">"</span><span class="p">,</span><span class="w">
</span><span class="s2">"VERBOSE: 06/07/2019 18:40:02</span><span class="se">\t</span><span class="s2">Set-Annotation</span><span class="se">\t</span><span class="s2">Finished execution"</span><span class="p">,</span><span class="w">
</span><span class="s2">"DEBUG: 06/07/2019 18:40:02</span><span class="se">\t</span><span class="s2">Set-Annotation</span><span class="se">\t</span><span class="s2">"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Setting 'Created On' = '06/07/2019 18:40:00' on VM testvm04"</span><span class="p">,</span><span class="w">
</span><span class="s2">"DEBUG: 06/07/2019 18:40:02</span><span class="se">\t</span><span class="s2">Set-Annotation</span><span class="se">\t</span><span class="s2">5246050e-1666-5ef9-f687-464457e7abba</span><span class="se">\t</span><span class="s2">"</span><span class="p">,</span><span class="w">
</span><span class="s2">"VERBOSE: 06/07/2019 18:40:02</span><span class="se">\t</span><span class="s2">Set-Annotation</span><span class="se">\t</span><span class="s2">Finished execution"</span><span class="p">,</span><span class="w">
</span><span class="s2">"DEBUG: 06/07/2019 18:40:02</span><span class="se">\t</span><span class="s2">Set-Annotation</span><span class="se">\t</span><span class="s2">"</span><span class="p">,</span><span class="w">
</span><span class="s2">""</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"16d3495e-0834-4bdc-a9c6-8bfc69a0c302"</span><span class="p">,</span><span class="w">
</span><span class="nl">"output"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"CEIPDataTransferProxyPolicy"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
</span><span class="nl">"DefaultVIServerMode"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
</span><span class="nl">"DisplayDeprecationWarnings"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="nl">"InvalidCertificateAction"</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="p">,</span><span class="w">
</span><span class="nl">"ParticipateInCEIP"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"ProxyPolicy"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
</span><span class="nl">"Scope"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
</span><span class="nl">"VMConsoleWindowBrowser"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"WebOperationTimeoutSeconds"</span><span class="p">:</span><span class="w"> </span><span class="mi">300</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"CEIPDataTransferProxyPolicy"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"DefaultVIServerMode"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"DisplayDeprecationWarnings"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"InvalidCertificateAction"</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="p">,</span><span class="w">
</span><span class="nl">"ParticipateInCEIP"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"ProxyPolicy"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"Scope"</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w">
</span><span class="nl">"VMConsoleWindowBrowser"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"WebOperationTimeoutSeconds"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"CEIPDataTransferProxyPolicy"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"DefaultVIServerMode"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"DisplayDeprecationWarnings"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"InvalidCertificateAction"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"ParticipateInCEIP"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"ProxyPolicy"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"Scope"</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="p">,</span><span class="w">
</span><span class="nl">"VMConsoleWindowBrowser"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"WebOperationTimeoutSeconds"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="s2">"success"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"reason"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"secrets"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"vsphere"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"services"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="p">,</span><span class="w">
</span><span class="nl">"status"</span><span class="p">:</span><span class="w"> </span><span class="s2">"READY"</span><span class="p">,</span><span class="w">
</span><span class="nl">"tags"</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>That’s it! Now whenever a VM is deployed Dispatch will automatically set these custom attributes on the VM. My next endeavor is to run a function when hosts enter/exit maintenance mode. Hopefully I will have an article on that soon!</p>{"name"=>"", "avatar"=>"/assets/images/profile.jpg", "bio"=>"Doug is an IT professional and technologist focusing on VMware technologies, virtualization, Powershell, automation and IT/Cloud infrastructure.", "location"=>"Atlanta, GA", "email"=>nil, "uri"=>nil, "home"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>"doogleit", "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>"dougtaliaferro", "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>"virtually_doug", "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}One of the things that immediately drew my attention to Dispatch was the vCenter event driver. Executing an automated task based on a vCenter event has always been something really attractive to me. There are other ways to do this, but it always seemed like they were just not ideal for the environment I was working in for one reason or another. Most of the examples that I’ve seen use a vCenter alarm to run a command or send an SNMP trap.Capturing Network Traffic on ESXi Hosts2019-04-30T00:00:00-04:002019-04-30T00:00:00-04:00https://doogleit.github.io/2019/04/capturing-network-traffic-on-esxi-hosts<p>This article shares some notes I took on packet captures (specifically using pktcap-uw) and challenges I ran into recently while troubleshooting the boot up and DHCP process for some VMs. If you only need to capture traffic for a specific VM or virtual port and are using a vSphere Distributed Switch (VDS) I’d recommend looking into the port mirroring options available on the VDS (<a href="https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.networking.doc/GUID-CFFD9157-FC17-440D-BDB4-E16FD447A1BA.html">Official VMware Docs - Working With Port Mirroring</a>), and specifically ERSPAN. A fellow blogger has published a great article on using ERSPAN: <a href="https://kacangisnuts.com/2013/12/vsphere-distributed-virtual-switch-packet-analysis-using-erspan/">vSphere Distributed Virtual Switch: Packet analysis using ERSPAN</a>. If you want or need to capture traffic on the ESXi host, then <code class="language-plaintext highlighter-rouge">pktcap-uw</code> is your tool. VMware has a nice overview in the KB article <a href="https://kb.vmware.com/s/article/2051814">Using the pktcap-uw tool in ESXi 5.5 and later (2051814)</a>.</p>
<p>The first thing to note is that pktcap-uw only captures inbound traffic by default. In most cases you’ll probably want to capture traffic in both directions. The examples below are for vSphere 5.5 - 6.5 and will start two traces, one using <code class="language-plaintext highlighter-rouge">--dir 0</code> for inbound traffic and and one using <code class="language-plaintext highlighter-rouge">--dir 1</code> for outbound traffic. This results in two separate capture files, but these can easily be merged in Wireshark. In vSphere 6.7 and later you can use <code class="language-plaintext highlighter-rouge">--dir 2</code> to capture traffic in both directions at once.</p>
<p>To capture traffice for a single VM we can specify the virtual switchport. To find it, start esxtop and hit ‘n’ for the network view. Look for the port ID in the left hand column.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[root@esx01:~] esxtop
2:39:01pm up 44 days 18:44, 1747 worlds, 42 VMs, 239 vCPUs; CPU load average: 0.76, 0.76, 0.78
PORT-ID USED-BY TEAM-PNIC DNAME PKTTX/s MbTX/s PSZTX PKTRX/s MbRX/
33554433 Management n/a DvsPortset-0 0.00 0.00 0.00 0.00 0.0
33554434 LACP_MgmtPort n/a DvsPortset-0 0.00 0.00 0.00 0.00 0.0
33554435 TST_LAG n/a DvsPortset-0 0.00 0.00 0.00 0.00 0.0
33554436 vmnic0 - DvsPortset-0 1318.80 3.65 363.00 3360.02 17.2
33554437 Shadow of vmnic0 n/a DvsPortset-0 0.00 0.00 0.00 0.00 0.0
33554438 vmnic1 - DvsPortset-0 2080.28 5.41 340.00 2033.96 3.0
33554439 Shadow of vmnic1 n/a DvsPortset-0 0.00 0.00 0.00 0.00 0.0
33554440 vmk0 TST_LAG* DvsPortset-0 86.47 0.20 301.00 34.33 0.0
33554441 vmk1 TST_LAG* DvsPortset-0 0.00 0.00 0.00 0.00 0.0
33554742 4534232:testvm1.eth0 TST_LAG* DvsPortset-0 199.64 0.42 272.00 306.45 0.3
33554744 4534241:testvm2.eth0 TST_LAG* DvsPortset-0 356.04 1.03 377.00 451.95 1.4
</code></pre></div></div>
<p>For this example let’s use ‘testvm1’ which is port ID 33554742. To capture traffice in both directions and save it to files in /tmp the command looks like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[root@esx01:~] pktcap-uw --switchport 33554742 --dir 0 -o /tmp/33554742-dir0.pcap & pktcap-uw --switchport 33554742 --dir 1 -o /tmp/33554742-dir1.pcap
</code></pre></div></div>
<p>Using the port ID is ideal on a busy host where you only want to capture traffic from a specific VM, however be aware that the port ID can change on a reboot. In my experience restarting the guest will not change the port ID, but a reset or power cycle will. If you’re troubleshooting the boot up process and might have to reset the VM this probably isn’t the best option to use. If the port ID changes the pktcap-uw process continues to run on the old port ID and is capturing nothing from that point forward. It’s also worth noting that in a DRS cluster you don’t want the VM to move to a different host when it restarts. The simplest way to temporarily disable this is to add a “VM Override” in the Cluster’s configuration and set the DRS Automation Level to Manual:</p>
<p><img src="/assets/images/vmoverrides.png" alt="VM Overrides" /></p>
<p>An alternative to the port ID is to specify the uplinks to capture traffic on. This will potentially result in a much larger capture file since it is going to capture all traffic on one or more uplinks. You’ll want to capture traffic on both uplinks in a team/LAG which results in 4 captures running simultaneously. In this scenario I ran into two issues with writing the capture files to /tmp. The first error was not really clear, simply logging <code class="language-plaintext highlighter-rouge">"Join with dump thread failed"</code> and only happened occasionally, but would result in one of the capture processes stopping prematurely while the other one continued running:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Accept...Vsock connection from port 1043 cid 2
Accept...Vsock connection from port 1044 cid 2
Dump: 385949, broken : 0, drop: 0, file err: 0Join with dump thread failedDestroying session 18
</code></pre></div></div>
<p>The other error was clearly a write error and was consistently re-occurring once the /tmp volume was full:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Local CID 2
Listen on port 45097
Local CID 2
Listen on port 45098
Accept...Vsock connection from port 1030 cid 2
Accept...Vsock connection from port 1029 cid 2
Dump: 9344, broken : 0, drop: 0, file err: 0error: Error writing 16 bytes of pkt header to file
Join with dump thread failedDestroying session 5
Dumped 10241 packet to file /tmp/vmnic0.pcap, dropped 0 packets.
Done.
</code></pre></div></div>
<p>Using <code class="language-plaintext highlighter-rouge">vdf -h</code> I could see that /tmp was out of space. It didn’t have a lot of space to begin with and it’s no surprise it was quickly filled up with the capture data.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[root@esx01:~] vdf -h
Ramdisk Size Used Available Use% Mounted on
root 32M 3M 28M 10% --
etc 28M 5M 22M 19% --
opt 32M 4K 31M 0% --
var 48M 520K 47M 1% --
tmp 256M 255M 364K 99% --
</code></pre></div></div>
<p>Changing the output files to write to a datastore not only fixed the space issue, but I didn’t have any more issues with one of the processes stopping prematurely. Here is the resulting command that I used, starting a capture in both directions on two uplinks in a LAG (resulting in a total of 4 capture processes) and writing the output to a datastore.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[root@esx01:~] pktcap-uw --uplink vmnic0 --dir 0 -o /vmfs/volumes/datastore1/vmnic0-dir0.pcap & pktcap-uw --uplink vmnic0 --dir 1 -o /vmfs/volumes/datastore1/vmnic0-dir1.pcap & pktcap-uw --uplink vmnic1 --dir 0 -o /vmfs/volumes/datastore1/vmnic1-dir0.pcap & pktcap-uw --uplink vmnic1 --dir 1 -o /vmfs/volumes/datastore/vmnic1-dir1.pcap
</code></pre></div></div>
<p>In vSphere 6.7 this should be able to be simplified to the following, but I have not tested this.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[root@esx01:~] pktcap-uw --uplink vmnic0 --dir 2 -o /vmfs/volumes/datastore1/vmnic0.pcap & pktcap-uw --uplink vmnic1 --dir 2 -o /vmfs/volumes/datastore1/vmnic1.pcap
</code></pre></div></div>
<p>To stop all of the capture processes, run the following command that is provided in the KB referenced above.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kill $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u)
</code></pre></div></div>
<p>To summarize:</p>
<ul>
<li>Use VDS port mirroring if possible</li>
<li>Don’t use <code class="language-plaintext highlighter-rouge">--switchport</code> when rebooting VMs (use <code class="language-plaintext highlighter-rouge">--uplink</code> instead)</li>
<li>Temporarily set the DRS automation level to manual if necessary</li>
<li>Save output files to a datastore with plenty of free space</li>
<li>Remember to capture traffic for all uplinks in a LAG and both directions (<code class="language-plaintext highlighter-rouge">--dir 2</code>) if necessary</li>
</ul>{"name"=>"", "avatar"=>"/assets/images/profile.jpg", "bio"=>"Doug is an IT professional and technologist focusing on VMware technologies, virtualization, Powershell, automation and IT/Cloud infrastructure.", "location"=>"Atlanta, GA", "email"=>nil, "uri"=>nil, "home"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>"doogleit", "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>"dougtaliaferro", "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>"virtually_doug", "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}This article shares some notes I took on packet captures (specifically using pktcap-uw) and challenges I ran into recently while troubleshooting the boot up and DHCP process for some VMs. If you only need to capture traffic for a specific VM or virtual port and are using a vSphere Distributed Switch (VDS) I’d recommend looking into the port mirroring options available on the VDS (Official VMware Docs - Working With Port Mirroring), and specifically ERSPAN. A fellow blogger has published a great article on using ERSPAN: vSphere Distributed Virtual Switch: Packet analysis using ERSPAN. If you want or need to capture traffic on the ESXi host, then pktcap-uw is your tool. VMware has a nice overview in the KB article Using the pktcap-uw tool in ESXi 5.5 and later (2051814).Storage DRS and RDMs2019-04-24T00:00:00-04:002019-04-24T00:00:00-04:00https://doogleit.github.io/2019/04/storage-drs-and-rdms<p>To summarize, store RDM mapping files on a dedicated datastore outside of any SDRS cluster. This is one of those things that I wanted to write about with the hope that I’ll remember it in the future or at least be able to refer back to this article. It’s very rare anymore that I have to work with RDMs, but I’m currently managing an environment where a few of the VMs still require RDMs. It’s not often that I need to do maintenance on these and even rarer that a storage vMotion is needed. While I was recently tackling a long overdue upgrade to VMFS6, I ran into this problem where I could not storage vMotion a VM with a mix of VMDKs and RDMs.</p>
<p>When trying to migrate the VM to a new volume I was getting a SDRS fault indicating that there was insufficient disk space.</p>
<p><img src="/assets/images/rdm_svmotion.png" alt="Migrate RDM" /></p>
<p>Clicking on the highlighted “View SDRS faults…” showed insufficient disk space on datastore for each of the large RDMs attached to the VM.</p>
<p><img src="/assets/images/rdm_svmotion_faults.png" alt="SDRS Faults" /></p>
<p>I knew there was enough disk space for the VMDKs and suspected that it was including the space used by the RDMs. These are physical mode RDMs and everything that I was finding online indicated that the RDM mapping files should be migrated to the new datastore with no problem. VMware has a comprehensive KB article on this subject:</p>
<p><a href="https://kb.vmware.com/kb/1005241">Migrating virtual machines with Raw Device Mappings (RDMs) (1005241)</a></p>
<p>As far as I could tell migrating the RDM pointers should not be an issue and because they are physical mode it isn’t possible to convert them to VMDKs on the destination datastore. Since the first screenshot indicates that this is an SDRS fault I decided to try moving the datastore out of the Datastore Cluster. Sure enough, this allowed me to migrate the VM, RDM mappings and all, to the new datastore.</p>
<p>After confirming it was SDRS, I found this Vmware KB article that describes the problem and resolution.</p>
<p><a href="https://kb.vmware.com/kb/2148523">Storage DRS generates only one datastore as initial placement recommendation in clusters with physical RDM (2148523)</a></p>
<p>It seems that my Google-Fu was not particularly strong on this day. If I had thought to include Storage DRS in my earlier searches I probably would have found this article before narrowing down the problem and saved some time in troubleshooting. Fortunately, it looks like this is no longer an issue in vSphere 6.7!</p>{"name"=>"", "avatar"=>"/assets/images/profile.jpg", "bio"=>"Doug is an IT professional and technologist focusing on VMware technologies, virtualization, Powershell, automation and IT/Cloud infrastructure.", "location"=>"Atlanta, GA", "email"=>nil, "uri"=>nil, "home"=>nil, "bitbucket"=>nil, "codepen"=>nil, "dribbble"=>nil, "flickr"=>nil, "facebook"=>nil, "foursquare"=>nil, "github"=>"doogleit", "gitlab"=>nil, "google_plus"=>nil, "keybase"=>nil, "instagram"=>nil, "lastfm"=>nil, "linkedin"=>"dougtaliaferro", "pinterest"=>nil, "soundcloud"=>nil, "stackoverflow"=>nil, "steam"=>nil, "tumblr"=>nil, "twitter"=>"virtually_doug", "vine"=>nil, "weibo"=>nil, "xing"=>nil, "youtube"=>nil}To summarize, store RDM mapping files on a dedicated datastore outside of any SDRS cluster. This is one of those things that I wanted to write about with the hope that I’ll remember it in the future or at least be able to refer back to this article. It’s very rare anymore that I have to work with RDMs, but I’m currently managing an environment where a few of the VMs still require RDMs. It’s not often that I need to do maintenance on these and even rarer that a storage vMotion is needed. While I was recently tackling a long overdue upgrade to VMFS6, I ran into this problem where I could not storage vMotion a VM with a mix of VMDKs and RDMs.