PowerChute 4.1 not behaving consistently or as expected with Smart-UPS X 3000 & VMware 5.0

[BillP]

I have a small two host VMware HA cluster that I am unable to get to (fully) gracefully shut down using the PowerChute VM.

I'm simulating a power outage by removing the main power lead from the UPS and the following occurs.

• PowerChute reports switch to battery power
• shutdown sequence started on hosts 1 & 2 in response to critical event: on battery
• (most of the time, but not always) my vm's start to shut down in more or less the expected order (regular vm's, then vApps)
• sometimes it will try and vmotion the VC and Chute VM's to another host at this point (despite the fact vmotion is not enabled in PowerChute), which fails due to the shutdown sequence having been initiated on the hosts.
• When all the VM's on host 1 are powered down, that host DOES go into maint. mode, UNLESS it has the VC and chute on it at the time, in which case it tries to migrate them off, and (ALWAYS) fails and the process stalls entirely for that host.
• *if* the VC and chute aren't on host 1 then generally speaking my vApp (with two VM's in it) will be on host 1 and WILL shut down the vApp and any other VM's successfully, although it has randomly started to power off the vApp VM's instead of shutting down the OS even though it is specifically defined in the vApp to shutdown the OS first for both VM's.

so then I'm left with one of either of the hosts (whichever one happens to have the VC and chute VM's on) and they sit there for a while doing nothing (checking the system logs, the Windows 2012 VC does not receive any OS shutdown request at all for some reason), THEN it gets to a point around the 30 minute mark where ChuteVM CANCELS the maint. mode of the host that still has the VM's for itself and the VC on (both still up and running), but after a couple more minutes it powers off that host without warning or maint. mode, which obviously kills both the VM's on the spot, BUT leaves the other host UP and in maint. mode and nothing further will happen without manual intervention.

When I restore power to the UPS and power the shut down host back up, it acts like you would expect after a dirty shutdown and powers up and restarts the two VM's that were already running when it went down (the VC and chute VM's), and there's where things end, no other VM's come back up and the other host stays in maint. mode even after the VC and Chute are fully up and running until I manually bring everything else up again.

I've  no AD in this environment so I have tried using a single user/pass for the chute, ups, esx hosts, vc and ilo's but it hasn't made any difference.

i've re-configured all the various shutdown timeouts both upwards and downwards by several minutes each and retested and the sequence (and failures) are always more or less the same.

the PowerChute VM has shown that as it is currently configured it IS able to shutdown VM's and vApp's, place hosts in maint. mode, shut down hosts and sometimes has shown it is able to bring it all back up again on the rare occasion most of the sequence goes correctly, but no matter what I try, it just won't do ALL of them together.

What could be going wrong?

I've attached a screenshot of the current config settings of the PowerChute VM, but I've gone as low as 0 seconds and as high as 600 seconds for all the startup/shutdown settings at this point with very little difference in the results other than the timings.

[BillP · Accepted solution]

Chris,

On 10/10/2015 8:12 AM, Chris said:

(most of the time, but not always) my vm's start to shut down in more or less the expected order (regular vm's, then vApps)

Are you configuring the priority in PowerChute or on the individual hosts? With PowerChute version 4.1 and configured with vCenter shutdown priority should be configured within PowerChute not on the individual hosts. Shutdown on the host should be off.

On 10/10/2015 8:12 AM, Chris said:

sometimes it will try and vmotion the VC and Chute VM's to another host at this point (despite the fact vmotion is not enabled in PowerChute), which fails due to the shutdown sequence having been initiated on the hosts.

 Do you have HA enabled within vCenter? If so that would be the cause of the VMs attempting to migrate.