vCenter server not shutdown cleanly by PCNS 3.1 and PCNS did not start when power was restored.

[BillP]

Hi,

I am hoping someone can help me understand why my vCenter server did not shutdown properly, and why the PCNS VM did not start when power was restored.  I will now describe our setup and observations made during a recent planned power outage.

I have 2 ESXi hosts running VMware 5.1.  They are in a HA cluster.

I am running PCNS 3.1 as a virtual appliance on the 2^nd host.  vCenter Server is also running on this host.

My setup is the same as that outlined in Setup 1 A) outlined in the APC Application Note #180.

I have configured AD single sign on as outlined in the APC Application Note #180.  The AD login can login to each host using vSphere client. I can also see this user logging into each host every 2 minutes via the Events on vSphere client. (It is annoying to see this logging so often but that is another issue!).

We had a planned power outage on the weekend and the following events occurred (from the PCNS log)

• After 10 minutes on battery power the UPS had a critical event (as planned)
• Host 1 entered maintenance mode
• Shutdown VMs on host 1
• Host 2 entered maintenance mode
• Shutdown VMs on host 2
• Then after waiting for the configured amount of time to shut down the VMs I get the following messages
  • 08/23/2014         08:16:47      vCenter Server authentication error. PowerChute may not be able to issue commands to Virtual Machines or Hosts.
  • 08/23/2014         08:17:51      Cannot connect to vCenter Server. PowerChute will not be able to perform vApp Shutdown.
  • 08/23/2014         08:17:52      Cannot connect to vCenter Server. PowerChute will not be able to perform vApp Shutdown.
  • 08/23/2014         08:21:12      Shutting down Host Host1.
  • 08/23/2014         08:21:14      Shutting down Host Host2.

From looking at the event logs on the vCenter Server it looks like it did not receive a shutdown command from PCNS.  Should I be expecting to see this graceful shutdown?

When power was restored the hosts powered up (due to BIOS settings) however PCNS virtual appliance did not start.  Host 2 was out of maintenance mode.  Host 1 was still in maintenance mode.

I was expecting the PCNS virtual appliance to be started by HA.

Any help would be appreciated.

Thanks

Janine

[BillP]

Hi Janine,

The issue with the frequent login events from PCNS in vSphere client will be addressed in the next release. It sounds like the vCenter Server VM is being shut down early along with the rest of the VMs and this is why the vCenter Server Authentication error message is appearing before the  vApp shutdown process starts. This may be occurring because PowerChute cannot locate the vCenter Server VM in the Inventory - it would explain why the vCenter Server VM is going down with the rest of the VMs and why there is no entry recorded in the PowerChute log to indicate that PCNS shut down the vCenter Server VM (the default delay for this is 240 seconds) before shutting down the ESXi hosts.

Could you provide a copy of error.log, pcnsconfig.ini and the full PCNS event Log please? These files are located in /opt/APC/PowerChute/group1 folder on the PCNS appliance.

On the VMware Host Protection page does the 2nd Host indicate that PowerChute and vCenter Server are present - there should be vSphere/PowerChute icons shown to indicate that the Host is running vCenter and PCNS as VMs? Do you see an option on the VM settings page to configure vCenter Server VM shutdown duration? Are you using Linux vCenter Server Appliance or is vCenter installed on a Windows Virtual Machine?

In relation to HA failing to start the PowerChute virtual appliance. This may be caused by HA Admission control settings - if there is only 1 Host available in the cluster (other host is in maintenance mode) HA may prevent any VMs from being started until the 2nd Host is taken out of maintenance mode.

[BillP]

Hi drehan,

Thank you for your reply.

I don't believe the vCenter Server VM (this is a Windows 2012 VM) ever received a shutdown command.  I can see the shut down command in all of my other Windows VMs event logs but not on this one.

PowerChute and vCenter Server are both present on the 2nd host on hte host protection page.  I had configured the vCenter Server VM Shutdown duration to 240 sec.  As I stated above the vCenter is installed on a Windows 2012 VM.

HA failing to start PCNS - the App Note 180 says that maintenance mode is cancelled for the host running PCNS so HA can attempt to re-start PCNS when the host powers back on.  This is what I believed has happened except PCNS did not start.  If this is the way things are going to work then it is not very useful as we want PCNS to be up to be able to start the other VMs if the power outage happens out of office hours when IT staff are not here.

Files - we are new to PCNS virtual appliance and VMware so would appreciate some guidance on how to get the files requested off the PCNS VM.  I can login to the VM and navigate the folders and list the files.  That is as far as I can get.  Also is there a way I can send them directly to you rather than post them for everyone to see?

Thanks for your help..it is very much appreciated.

[BillP · Accepted solution]

Hi Janine,

Is VMWare Tools installed on the Windows 2012 VM running vCenter Server?

On the Host Protection page do you see PowerChute/vSphere Icons for the 2nd Host? If you vMotion the vCenter Server VM to the first ESXi host does the Host Protection page update to reflect the change in location? If not then there may be an issue with locating the vCenter Server VM in the inventory.

With regard to HA failing to re-start the PCNS Virtual Appliance:

This is controlled by VMware HA Admission control. By default HA will not allow VMs to be powered on if they violate HA availability. If there is only 1 Host available (the other Host is in the cluster is in maintenance mode) then HA might not re-start the PowerChute Appliance:

It may be necessary to disable Admission Control to allow HA to re-start the PowerChute Virtual Appliance. In the Tasks and Events view for the PowerChute VM are there any errors reported in relation to HA?

As mentioned in the App Note the maintenance mode task is cancelled and the PowerChute Virtual Appliance is powered off when the host goes down so that HA can "attempt" to re-start the Appliance when the Host is powered back. HA functionality is outside of the control of PCNS. To remove the dependency on HA re-starting the appliance so that VMs can be re-started outside of office hours an option would be to install PowerChute on a physical windows machine that is powered by the same UPS protecting the ESXi hosts and setting the BIOS on that machine to start automatically when A/C power is restored.

I will see that the information above is added to the Application Note.

In relation to obtaining the files you can use WinSCP to connect to the PowerChute Virtual Appliance and transfer the files over SCP/SFTP.

WinSCP :: Free SFTP and FTP client for Windows

I have sent you a Friend request via the Forum - once you accept that you will be able to send me the files directly rather than posting them publicly. If you have any issues obtaining the files please let me know.