PowerchuteVM only shutdown the Esxi host on which is running PCNS and VCSA

[BillP]

I am trying to troubleshoot why PowerchuteVM only shutdown the VMware host on which is running. 

Here is my environment.

- Smart-UPS X 3000

- 3 VMwareHost 

- VCenter Server Appliance running on ESXI 03

- PCNS 4.3 running as a VM on ESXI 03

Here is the log

07/14/2019	19:34:32	Available runtime is sufficient.
07/14/2019	19:16:21	Powering on VMs on Host 10.0.1.40.
07/14/2019	19:16:21	Powering on VMs on Host esxi04.domain.com.
07/14/2019	19:16:20	Powering on VMs on Host esxi03.domain.com.
07/14/2019	19:15:29	vCenter Server is accessible. PowerChute will be able to issue commands to Virtual Machines or Hosts.
07/14/2019	19:13:41	Available runtime has been exceeded.
07/14/2019	19:13:41	Cannot connect to vCenter Server. PowerChute may not be able to issue commands to Virtual Machines or Hosts.
07/14/2019	19:13:14	PowerChute successfully opened TCP port 6547.
07/14/2019	19:13:14	PowerChute successfully opened TCP port 3052.
07/14/2019	19:13:14	PowerChute is attempting to open TCP port 3052.
07/14/2019	19:13:14	PowerChute is attempting to open TCP port 6547.
07/14/2019	19:13:13	PowerChute Network Shutdown version 4.3.0 monitoring started.
07/14/2019	19:13:11	Communication established with Environmental Monitor.
07/14/2019	19:13:11	Communication has been established.
07/14/2019	19:12:44	PowerChute successfully opened UDP port 3052.
07/14/2019	19:12:44	PowerChute is attempting to open UDP port 3052.
07/14/2019	17:11:07	Shutdown Host failed for Host esxi03.domain.com.
07/14/2019	17:10:47	Shutdown Host failed for Host esxi04.domain.com.
07/14/2019	17:10:27	Shutdown Host failed for Host 10.0.1.40.
07/14/2019	17:06:17	Cannot connect to vCenter Server. PowerChute may not be able to issue commands to Virtual Machines or Hosts.
07/14/2019	17:06:17	Maintenance Mode Task cancelled on Host esxi03.domain.com as VCSA VM is shutting down.
07/14/2019	17:06:16	Shutting down vCenter Server VM VMVCSA01.
07/14/2019	17:04:35	Host 10.0.1.40 has successfully entered Maintenance Mode.
07/14/2019	17:03:55	Host esxi04.domain.com has successfully entered Maintenance Mode.
07/14/2019	17:03:05	Exit Maintenance Mode on Host esxi03.domain.com.
07/14/2019	16:52:10	UPS critical event: Outlet Group Turn off initiated.
07/14/2019	16:52:10	UPS Outlet Group: vCenter-Hosts turn off has been initiated.
07/14/2019	16:52:05	Shutting down VMs on Host esxi03.domain.com.
07/14/2019	16:52:05	Starting Maintenance Mode Task on Host esxi03.domain.com.
07/14/2019	16:52:05	No vApp to shutdown on Host esxi04.domain.com.
07/14/2019	16:52:05	Starting vApp shutdown process.
07/14/2019	16:52:05	Shutting down VMs on Host 10.0.1.40.
07/14/2019	16:52:05	Starting Maintenance Mode Task on Host 10.0.1.40.
07/14/2019	16:52:04	Shutting down VMs on Host esxi04.domain.com.
07/14/2019	16:52:04	Starting Maintenance Mode Task on Host esxi04.domain.com.
07/14/2019	16:52:04	PowerChute is turning off UPS 10.0.1.10
07/14/2019	16:52:04	Shutdown sequence started on Hosts esxi03.domain.com, 10.0.1.40, esxi04.domain.com in response to UPS critical event: On Battery.
07/14/2019	16:52:04	UPS critical event: On Battery.
07/14/2019	16:50:04	UPS has switched to battery power.

[BillP]

Hello,

The log indicates there may be a permissions issue happening. When vCenter is running PCNS will communicate with the host through vCenter. Once vCenter go off line PCNS must communicate directly with the host. See page 62 of the User's Guide for assistance with creating local account that PCNS will use to communicate with he hosts.

[BillP]

Hi Bill, 

I did not add VCenter to any priority group in VM Prioritization , do you think this could be the issue.

Before the Test , I created the local user on 3 hosts  and VCenter.

[BillP]

Hi,

On 7/24/2019 12:00 PM, Juan said:

I did not add VCenter to any priority group in VM Prioritization , do you think this could be the issue.

No. vCenter is being powered down 

07/14/2019

17:06:16

Shutting down vCenter Server VM VMVCSA01.

The issue is PCNS cannot shutdown the host

07/14/2019	17:11:07	Shutdown Host failed for Host esxi03.domain.com.
07/14/2019	17:10:47	Shutdown Host failed for Host esxi04.domain.com.

if the issue is not permissions have you reviewed the error.log to see what may have caused the issue. Troubleshooting host failed to power down is listed in the troubleshooting section of the User's Guide. VMware troubleshooting starts on page 179. 

[BillP]

Bill, 

I changed the PowerchuteNS to a physical Server, all 3 host are shutting down successfully. 

Now when I restored the power , all servers turn on correctly, but only one host is able to exit out of maintenance mode, and no VMs are power on. 

I attached  the logs.

[BillP]

Here is the pcnsconfig

[BillP]

Hello,

The log shows there is a DNS configuration issue preventing PowerChute from connecting to the host e.g. a stale DNS record containing an invalid hostname/FQDN or IP address. VI SDK invoke exception:java.net.UnkownHostException

ON the Windows system add the host names and IP addresses to the etc/hosts file

The file will be found at C:\Windows\System32\drivers\etc

Below is an example of a file that I have edited and added to hosts to the file. 

#

# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.

#

# This file contains the mappings of IP addresses to host names. Each

# entry should be kept on an individual line. The IP address should

# be placed in the first column followed by the corresponding host name.

# The IP address and the host name should be separated by at least one

# space.

#

# Additionally, comments (such as these) may be inserted on individual

# lines or following the machine name denoted by a '#' symbol.

#

# For example:

#

#      102.54.94.97     rhino.acme.com          # source server

#       38.25.63.10     x.acme.com              # x client host

# localhost name resolution is handled within DNS itself.

# 127.0.0.1       localhost

# ::1             localhost

192.168.1.11 host1.vmlab.apc.com

192.168.1.12 host2.vmlab.apc.com

After editing the file you should stop the PCNS server on the server. The windows command is net stop pcns1. Go to the PCNS group1 folder. The default path is C:\Program files\APC\PowerChute\group1. Open the file VirtualizationFileStore.properties with a text editor and remove any entries that are in the file. Then restart the server net start pcns1. 

[BillP]

Do I need to add the VCSA dns name to the etc/host file ?

I removed the entries on VirtualizationFileStore.properties

Thank you

[BillP]

Hi,

You can add that info to allow PCNS to communicate with vCenter in the event that DNS is off line.

[BillP]

Hi Bill, 

I did two tests, the 3 hosts out of maintenance mode successfully. But at the moment to power on VMs from Prioritize groups some VMs wont power on. When I did the first test there were more VMs off from 2 hosts, in the second test only some VMs from one host couldn't power on automatically. 

Check the logs please

[BillP]

Hi,

According to the event log everything looks to have started properly.

08/04/2019 11:04:13 Exit Maintenance Mode on Host mx-esxi03.kwalu.com. .3.4.9.9
08/04/2019 11:04:19 Exit Maintenance Mode on Host 10.0.7.40. .3.4.9.9
08/04/2019 11:04:20 Exit Maintenance Mode on Host mx-esxi04.kwalu.com. .3.4.9.9
08/04/2019 11:05:08 Powering on VMs on Host mx-esxi04.kwalu.com. .3.4.9.9
08/04/2019 11:05:08 Powering on VMs on Host mx-esxi03.kwalu.com. .3.4.9.9
08/04/2019 11:05:08 Attempting to start vCenter Server VM MX-VMVCSA01. .3.4.9.9
08/04/2019 11:05:08 Powering on VMs on Host 10.0.7.40. .3.4.9.9

May I have an updated copy of the pcnsconfig.ini file and a copy of the VirtualizationFileStore.Properties file. Also, do you know what VMs where powered on when the shutdown process started?

[BillP]

Hello Bill, 

I see this error on error.txt (previous attached file) .

2019-08-04T11:13:29,520 ERROR pool-36-thread-2 com.apcc.m11.components.Shutdowner.VMStartupRunnable java.rmi.RemoteException: An error occurred getting a response from the connection at url https://host1:443/sdk; nested exception is:
java.net.SocketTimeoutException: Read timed out
at com.vmware.vim25.ws.WSClient.getInputStreamFromConnection(WSClient.java:201) - performVMStartUp() - RemoteException:

Also I attached the pcnsconfig and virtual properties.

Only VMs not included on High Group wont powered on in Host 1, Host 2 and Host 3 all VMs sucessfully powered on and somes VMs are Medium and Low groups.

[BillP · Accepted solution]

Hi,

I have reviewed the logs and configuration file. I can see where the start up of High, Medium, Low are being executed. I do not see a success response for all however. I also see that most all VMs for host esxi03 remained listed in the file store properties file.

I am sending the files to a colleague and will post back when I have more info.