PowerChute 4.1 starts vCenter server before last host is going down

[BillP]

Hello all,

We decided to use last version of PCNS 4.1 in our company for shutdown VMs, hosts and UPS after switching on battery. So before using it in production we run some tests. In our test environment we can`t shutdown the UPS.

Test environment:
2 ESXi hosts in HA cluster (ESXi 5.5 Update 3b build: 3248547)
1 UPS - Symmetra RM6000 with NMC - AP9619
vCenter Server 5.5 Update 3b (build: 3252642) running on VM (OS: Windows Server 2008 R2)
PowerChute Network Shutdown 4.1 deployed on cluster as VMware Virtual Appliance. PowerChute is configured to shutdown after getting UPS event "Manual Bypass" with delay 30 seconds.

Before switching UPS to Manual Bypass two VMs (vCenter server and PowerChute) are located on second host and there are tied to second host by DRS rules.

After switching UPS to Manual Bypass all going well, all VMs gracefully shutdown except VC and PowerChute, first host switches to maintenance mode and shutdown, then vCenter server shutdowns and after some time first host gets command to shutdown, but before it powered off PowerChute starts vCenter server. After some seconds second host goes down.

We performed test twice and both times vCenter server starts before second hosts goes down.

This is a normal behavior of the procedure of shutdown? We need to start vCenter server only after start up the hosts. How to achieve this?

Time period in log files of shutdown process is 08/04/2016 11:07:54 - 08/04/2016 11:37:48

Thanks in advance!

[BillP]

Hi,

I think increasing the value below in the pcnsconfig.ini file would prevent the issue from occurring. It seems to be taking awhile for the ESXi host to shut down.

[HostConfigSettings]
delay_after_exit_maintenance_mode = 30

Try increasing the number to 60 or 90 for example.

[BillP]

Thanks Bill for the quick answer!

I will test shutdown process with new value of option delay_after_exit_maintenance_mode tomorrow and post the results. 

[BillP]

Hi,

I performed three shutdown tests with options below:
1) delay_after_exit_maintenance_mode = 60
2) delay_after_exit_maintenance_mode = 90
3) delay_after_exit_maintenance_mode = 30 and delay_after_vcsa_powered_on_and_connected = 60

and I have the same issue with starting of vCenter server before last host is powered off.

I attached log files for three tests:
1) 1_error.log, 1_EventLog.txt, 1_pcnsconfig.ini (Time period of shutdown process: 08/05/2016 08:18:12 - 08/05/2016 08:24:18)
2) 2_error.log, 2_EventLog.txt, 2_pcnsconfig.ini (Time period of shutdown process: 08/05/2016 11:14:47 - 08/05/2016 11:20:25)
3) 3_error.log, 3_EventLog.txt, 3_pcnsconfig.ini (Time period of shutdown process: 08/05/2016 14:46:56 - 08/05/2016 14:52:30)

Do you have any suggestions for solve this issue?

Thanks!

[BillP]

Up!

[BillP · Accepted solution]

Hi,

Sorry for the delay in my response. I have been on holiday. 

I think the issue is due to the locking mechanism in VirtualizationFileStore.properties and a mismatch in the Hostname/FQDN between vCenter Server and what's configured locally on the ESXi hosts.

05-08-16 08:22:00,337 DEBUG Thread-32 com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils - getHostUsingIPOrDNSName - Host IP: esxru0321cl001.mycompany.com
05-08-16 08:22:00,367 DEBUG Thread-32 com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils - Found [1] hostSystemEntites
05-08-16 08:22:00,370 DEBUG Thread-32 com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils - Found hostSystem esxru0321cl001

05-08-16 08:22:02,189 DEBUG Thread-32 com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils - getHostUsingIPOrDNSName - Host IP: esxru0321cl002.mycompany.com
05-08-16 08:22:02,224 DEBUG Thread-32 com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils - Found [1] hostSystemEntites
05-08-16 08:22:02,226 DEBUG Thread-32 com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils - Found hostSystem esxru0321cl002

While vCenter Server is online the host lock will be added to VirtualizationFileStore.properties as:

esxru0321cl001.mycompany.com
esxru0321cl002.mycompany.com

After VCSA goes offline and PCNS connects directly to the host, the ESXi host name is returned as esxru0321cl001, and esxru0321cl002 instead of esxru0321cl001.mycompany.com, esxru0321cl002.mycompany.com - so the locks get removed and PCNS incorrectly attempts to start the VCSA VM.
-
The Solution to this is to update the Hostname on each of the ESXi hosts to make sure that it uses the FQDN. Using vSphere Client go to Configuration - DNS and Routing - check that Name and Domain are correct. Also in MOB browser check that FindByIP and FindByDNSName return the FQDN. See Kbase FA241346 for assistance with MOB. 

[BillP]

Hi,

After filling Domain name in all hosts of HA cluster the issue is no longer appear. I was checked this twice with shutdown the UPS on production environment. Before that Domain name was missing on all hosts.

Thank you Bill for your help!