PCNS 4.1 (vApp VM or Windows) Won't Shutdown vCenter VM

Anonymous user · ‎2021-06-29

I've been struggling with getting PCNS 4.1 to successfully shut down our virtual infrastructure when a threshold has been reached.

We have 3 hosts with vSphere 5.1 (Not Clustered/HA) running off of 1 Smart-UPS X 3000 w/ AP9631 NMC, with the PCNS vApp and vCenter VM running on separate hosts.

Once the threshold has been instigated and the shutdown commands have been issued, every VM shuts down fine except for the vCenter VM.

Once vCenter the process times out without the VM shutting down, the whole process is aborted.

I have also tried running PCNS on a windows physical machine, with the same behaviour. I've tried searching and have not been able to find any leads on why this may be happening.

Any assistance would be greatly appreciated.

Thanks!

BillP · ‎2021-06-29

Hi,

All host should power down properly. Please run the shut down test again but before doing so put PowerChute into debug mode. Kbase FA230514 provides instruction on how to put PowerChute into debug mode. When putting PowerChute into debug mode stop the service, save error.log to a new name, then restart PowerChute after editing log4j2.xml. Once the test is completed attach the PowerChute EventLog.txt, error.log, and pcnsconfig.ini files to this post for review. Also, stop the PowerChute service, switch PowerChute out of debug mode, then restart PowerChute.

See Answer In Context

BillP · ‎2021-06-29

Hi,

Have you reviewed the error.log file in the PCNS directory? It should lead you to the issue. Also, in the User's Guide there is a troubleshooting section that explains errors you will find in the error.log file.

The shutdown process is explained on page 4 of Application Note 180.

Anonymous user · ‎2021-06-29

Thanks for the reply and direction.

The following sticks out in the logs:

11-06-16 12:17:40,373 ERROR pool-5-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - getESXiHostConnection, Host host.domain.local - (NoPermission or InvalidLogin) [no details available]

11-06-16 12:20:46,131 ERROR Thread-73 com.apcc.m11.components.Shutdowner.Hosts.ESXManagedHost - checkForVCSAVMAndHostInCriticalHosts - cannot obtain HostSystem using findByIP and findByDnsName for critical host 192.168.xxx.xxx

By looking at the troubleshooting section of the provided guide, it seems to be an authentication issue.

I am using an Active Directory account to issue the shutdown commands from PCNS, so at first thought I'm guessing i have to give this account some form of permission through each host?

I've added the domain user as an administrator to each host through the permissions tab and will test during a scheduled outage in the future, but this seems to be the missing piece.

Thank you very much for your help!

Regards.

Anonymous user · ‎2021-06-29

Hi Bill,

I've since attempted to get this working but have been having no luck.

Basically what is happening is, when the critical event occurs (pulling the plug), all VM's shutdown correctly, except for the vCenter Server VM and the PCNS 4.1 VM.

As they are both on different hosts(vcenterhost.domain.local, pcnshost.domain.local), this prevents both hosts from entering maintenance mode. Our third stand-alone host (generichost.domain.local) enters maintenance mode correctly.

I resolved to creating a local user on our vCenter server, as per instructions on page 55 of the users guide, because our domain controller shut downs as part of the process, so I assumed that domain credentials shouldn't be used to issue shutdown commands.

However, it still does not work. Oddly enough, however, when power is restored and the Domain Controller comes back online, the vCenter server and its host, and the PCNS 4.1 VM and its host, shutdown completely...

Anyway, here are the logs after the critical event begins:

_{09-07-16 14:37:05,390 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - getESXiHostConnection, Host 192.168.0.223 - (RemoteException) VI SDK invoke exception:java.net.NoRouteToHostException: No route to host}
_{09-07-16 14:39:29,471 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils - getHostUsingIPOrDNSName java.rmi.RemoteException: VI SDK invoke exception:org.dom4j.DocumentException: null Nested exception: null}
_{09-07-16 14:39:31,114 ERROR pool-6-thread-1 com.apcc.m11.components.Shutdowner.ExecuteSupportedVirtualizationFeatures - startGuestVMShutdown - No shutdown. VM Shutdown Duration is zero for priority: LOW}
_{09-07-16 14:39:31,129 ERROR pool-6-thread-1 com.apcc.m11.components.Shutdowner.ExecuteSupportedVirtualizationFeatures - startGuestVMShutdown - No shutdown. VM Shutdown Duration is zero for priority: MEDIUM}
_{09-07-16 14:39:31,144 ERROR pool-6-thread-1 com.apcc.m11.components.Shutdowner.ExecuteSupportedVirtualizationFeatures - startGuestVMShutdown - No shutdown. VM Shutdown Duration is zero for priority: HIGH}
_{09-07-16 14:39:31,495 ERROR pool-7-thread-1 com.apcc.m11.components.Shutdowner.ExecuteSupportedVirtualizationFeatures - startGuestVMShutdown - No shutdown. VM Shutdown Duration is zero for priority: LOW}
_{09-07-16 14:39:31,550 ERROR pool-7-thread-1 com.apcc.m11.components.Shutdowner.ExecuteSupportedVirtualizationFeatures - startGuestVMShutdown - No shutdown. VM Shutdown Duration is zero for priority: MEDIUM}
_{09-07-16 14:39:31,566 ERROR pool-7-thread-1 com.apcc.m11.components.Shutdowner.ExecuteSupportedVirtualizationFeatures - startGuestVMShutdown - No shutdown. VM Shutdown Duration is zero for priority: HIGH}
_{09-07-16 14:39:31,655 ERROR pool-8-thread-1 com.apcc.m11.components.Shutdowner.ExecuteSupportedVirtualizationFeatures - startGuestVMShutdown - No shutdown. VM Shutdown Duration is zero for priority: LOW}
_{09-07-16 14:39:31,682 ERROR pool-8-thread-1 com.apcc.m11.components.Shutdowner.ExecuteSupportedVirtualizationFeatures - startGuestVMShutdown - No shutdown. VM Shutdown Duration is zero for priority: MEDIUM}
_{09-07-16 14:39:31,699 ERROR pool-8-thread-1 com.apcc.m11.components.Shutdowner.ExecuteSupportedVirtualizationFeatures - startGuestVMShutdown - No shutdown. VM Shutdown Duration is zero for priority: HIGH}
_{09-07-16 14:39:45,091 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils - getHostUsingIPOrDNSName java.rmi.RemoteException: VI SDK invoke exception:org.dom4j.DocumentException: null Nested exception: null}
_{09-07-16 14:41:05,099 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - java.net.UnknownHostException: generichost.domain.local: unknown error}
_{09-07-16 14:41:23,496 ERROR Thread-42 com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils - getHostUsingIPOrDNSName java.rmi.RemoteException: VI SDK invoke exception:org.dom4j.DocumentException: null Nested exception: null}
_{09-07-16 14:41:23,496 ERROR Thread-42 com.apcc.m11.components.Shutdowner.Hosts.ESXManagedHost - checkForVCSAVMAndHostInCriticalHosts - cannot obtain HostSystem using findByIP and findByDnsName for critical host generichost.domain.local}
_{09-07-16 14:42:28,436 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - java.net.UnknownHostException: vcenterhost.domain.local: unknown error}
_{09-07-16 14:42:43,527 ERROR Thread-42 com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils - getHostUsingIPOrDNSName java.rmi.RemoteException: VI SDK invoke exception:org.dom4j.DocumentException: null Nested exception: null}
_{09-07-16 14:42:43,527 ERROR Thread-42 com.apcc.m11.components.Shutdowner.Hosts.ESXManagedHost - checkForVCSAVMAndHostInCriticalHosts - cannot obtain HostSystem using findByIP and findByDnsName for critical host vcenterhost.domain.local}
_{09-07-16 14:44:47,319 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils - getHostUsingIPOrDNSName java.rmi.RemoteException: VI SDK invoke exception:org.dom4j.DocumentException: null Nested exception: null}
_{09-07-16 14:45:20,676 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - java.net.UnknownHostException: generichost.domain.local}
_{09-07-16 14:45:36,104 ERROR Thread-42 com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils - getHostUsingIPOrDNSName java.rmi.RemoteException: VI SDK invoke exception:org.dom4j.DocumentException: null Nested exception: null}
_{09-07-16 14:45:36,105 ERROR Thread-42 com.apcc.m11.components.Shutdowner.Hosts.ESXManagedHost - Unable to get HostSystem using host: 192.168.0.3}
_{09-07-16 14:46:44,123 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - java.net.UnknownHostException: vcenterhost.domain.local: unknown error}
_{09-07-16 14:47:24,667 ERROR Thread-2 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - Unable to connect to vCenter Host 192.168.0.13 - java.rmi.RemoteException: VI SDK invoke exception:java.net.SocketTimeoutException: connect timed out}
_{09-07-16 14:47:30,700 ERROR Thread-2 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - getVCenterServerConnection, Host 192.168.0.13 - . VI SDK invoke exception:java.net.NoRouteToHostException: No route to host}
_{09-07-16 14:48:07,429 ERROR Thread-42 com.apcc.m11.components.Shutdowner.Hosts.ESXManagedHost - hostShutdown - cannot obtain ESXiHostInstance with host connection str: null}
_{java.lang.NullPointerException}
_{at com.vmware.vim25.mo.SearchIndex.findByDnsName(SearchIndex.java:92) ~[vijava5120121125.jar:?]}
_{at com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils.getIPAddressFromHostname(Unknown Source) ~[pcns.jar:?]}
_{at com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils.getESXIConnStr(Unknown Source) ~[pcns.jar:?]}
_{at com.apcc.m11.components.Shutdowner.Hosts.ESXManagedHost.hostShutdown(Unknown Source) [pcns.jar:?]}
_{at com.apcc.m11.components.Shutdowner.MasterSequenceThread.run(Unknown Source) [pcns.jar:?]}
_{09-07-16 14:48:07,440 ERROR Thread-42 com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils - getHostUsingIPOrDNSName java.lang.NullPointerException}
_{09-07-16 14:48:07,441 ERROR Thread-42 com.apcc.m11.components.Shutdowner.Hosts.ESXManagedHost - Unable to get HostSystem using host: null}
_{09-07-16 14:54:12,828 ERROR main com.apcc.pcns.configservice.ConfigurationService - === Starting PowerChute Network Shutdown ===}
_{09-07-16 14:55:37,903 ERROR Thread-2 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - getVCenterServerConnection, Host 192.168.0.13 - . VI SDK invoke exception:java.net.NoRouteToHostException: No route to host}
_{09-07-16 14:56:20,407 ERROR main com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - java.net.UnknownHostException: generichost.domain.local: unknown error}
_{09-07-16 14:56:20,407 ERROR main com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - java.net.UnknownHostException: generichost.domain.local}
_{09-07-16 14:57:40,421 ERROR main com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - java.net.UnknownHostException: vcenterhost.domain.local: unknown error}
_{09-07-16 14:57:40,423 ERROR main com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - java.net.UnknownHostException: vcenterhost.domain.local}
_{09-07-16 14:59:00,430 ERROR main com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - java.net.UnknownHostException: pcnshost.domain.local: unknown error}
_{09-07-16 14:59:00,431 ERROR main com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - java.net.UnknownHostException: pcnshost.domain.local}
_{09-07-16 15:01:20,530 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - java.net.UnknownHostException: generichost.domain.local: unknown error}
_{09-07-16 15:02:40,539 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - java.net.UnknownHostException: vcenterhost.domain.local: unknown error}
_{09-07-16 15:05:22,120 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - getESXiHostConnection, Host 192.168.0.223 - (RemoteException) VI SDK invoke exception:java.net.NoRouteToHostException: No route to host}

The only error I could find information for in the troubleshooting guide was:

_{09-07-16 14:41:23,496 ERROR Thread-42 com.apcc.m11.components.Shutdowner.Hosts.ESXManagedHost - checkForVCSAVMAndHostInCriticalHosts - cannot obtain HostSystem using findByIP and findByDnsName for critical host generichost.domain.local}

Which states the cause as:

_{If the Domain name for the ESXi host running the vCenter Server VM is different for the vCenter Server Inventory compared to the Standalone ESXi host, this will cause vCenter Server VM shutdown and Host shutdown to fail.}

_{e.g. Domain is set to apcc.com in vCenter, but apcc on the ESXi host. This can be seen using the vSphere Client for each Host under Configuration > DNS and Routing > Host Identification > Domain. To avoid this ensure that the same domain name is set in vCenter Server and on the ESXi host.}

I checked and they are all added the same...

Any ideas would be greatly appreciated as I have spent more time than I would like to admit on this.

BillP · ‎2021-06-29

Hi,

Test findByIP and findByDnsName using the instructions in Kbase FA241346 Also, DNS names are case sensitive so verify that the added names match not only in spelling match case.

Anonymous user · ‎2021-06-29

Hi Bill,

I performed the test linked and passed successfully, indicating that there were no issues with the name resolution. With this in mind, I had lost confidence in running PCNS 4.1 from a VM and having it work as intended so I resolved to try running PCNS on a physical Windows machine. This proved to be more successful, although with some unexpected results.

Once a critical event had been triggered, two of the three hosts entered maintenance mode successfully, however the host running VCenter was unable to enter maintenance mode due to shutting down the VCenter VM. This then prevented the stand alone hosts from shutting down, however the VCenter host and the PCNS machine shutdown successfully. The time line of events can be seen below.

_{13/08/2016 14:48:31 UPS has switched to battery power. .3.5.1.5.4.1}
_{13/08/2016 14:48:44 The runtime remaining has dropped below the configured threshold while on Battery. .3.5.1.6.3}
_{13/08/2016 14:48:44 UPS critical event: RunTime Remaining: Below Threshold. .3.4.9.9}
_{13/08/2016 14:48:44 Shutdown sequence started on Hosts standalonehost1.domain.local, vcenterhost.domain.local, standalonehost2.domain.local in response to UPS critical event: RunTime Remaining: Below Threshold. .3.4.9.9}
_{13/08/2016 14:48:45 Starting Maintenance Mode Task on Host standalonehost1.domain.local. .3.4.9.9}
_{13/08/2016 14:48:45 Shutting down VMs on Host standalonehost1.domain.local. .3.4.9.9}
_{13/08/2016 14:48:45 Starting Maintenance Mode Task on Host vcenterhost.domain.local. .3.4.9.9}
_{13/08/2016 14:48:45 Shutting down VMs on Host vcenterhost.domain.local. .3.4.9.9}
_{13/08/2016 14:48:45 Starting Maintenance Mode Task on Host standalonehost2.domain.local. .3.4.9.9}
_{13/08/2016 14:48:45 Shutting down VMs on Host standalonehost2.domain.local. .3.4.9.9}
_{13/08/2016 14:52:45 Host standalonehost1.domain.local has successfully entered maintenance mode. .3.4.9.9}
_{13/08/2016 14:52:45 Host standalonehost2.domain.local has successfully entered maintenance mode. .3.4.9.9}
_{13/08/2016 14:52:53 Shutting down vCenter Server VM VCENTER. .3.4.9.9}
_{13/08/2016 14:52:54 Maintenance Mode Task cancelled on Host vcenterhost.domain.local as VCSA VM is shutting down. .3.4.9.9}
_{13/08/2016 14:59:14 Cannot connect to vCenter Server. PowerChute may not be able to issue commands to Virtual Machines or Hosts. .3.4.9.9}
_{13/08/2016 14:59:29 Shutdown Host failed for Host standalonehost1.domain.local. .3.4.9.9}
_{13/08/2016 14:59:34 Starting Maintenance Mode Task on Host vcenterhost.domain.local. .3.4.9.9}
_{13/08/2016 14:59:36 Host vcenterhost.domain.local has successfully entered maintenance mode. .3.4.9.9}
_{13/08/2016 14:59:36 Shutting down vCenter Server VM Host vcenterhost.domain.local. .3.4.9.9}
_{13/08/2016 14:59:38 Shutting down physical machine that PowerChute is running on. .3.4.9.9}

There was not much in the error logs, however I have included what was there below:

_{13-08-16 14:41:10,987 ERROR main com.apcc.pcns.configservice.ConfigurationService - === Starting PowerChute Network Shutdown ===}
_{13-08-16 14:51:50,750 ERROR pool-2-thread-34 com.apcc.m11.components.AutoUpdate.SoftwareUpdateChecker_HTTPS - java.net.UnknownHostException: updates.apc.com}
_{13-08-16 14:59:14,487 ERROR Thread-57 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - Unable to connect to vCenter Host 192.168.0.13 - java.rmi.RemoteException: VI SDK invoke exception:java.net.SocketTimeoutException: connect timed out}
_{13-08-16 14:59:29,510 ERROR Thread-57 com.apcc.m11.components.WebServer.util.virtualization.VMWareUtils - getHostUsingIPOrDNSName java.rmi.RemoteException: VI SDK invoke exception:org.dom4j.DocumentException: null Nested exception: null}
_{13-08-16 14:59:29,510 ERROR Thread-57 com.apcc.m11.components.Shutdowner.Hosts.ESXManagedHost - Unable to get HostSystem using host: standalonehost1.domain.local}
_{13-08-16 15:15:44,441 ERROR main com.apcc.pcns.configservice.ConfigurationService - === Starting PowerChute Network Shutdown ===}

At the end of it all, the VCenter host was shutdown, while the two other hosts were still powered on, however in maintenance mode.

I may be wrong, but I was expecting all hosts to enter maintenance mode and then shutdown, correct?

Any ideas?

Thanks!

BillP · ‎2021-06-29

Hi,

All host should power down properly. Please run the shut down test again but before doing so put PowerChute into debug mode. Kbase FA230514 provides instruction on how to put PowerChute into debug mode. When putting PowerChute into debug mode stop the service, save error.log to a new name, then restart PowerChute after editing log4j2.xml. Once the test is completed attach the PowerChute EventLog.txt, error.log, and pcnsconfig.ini files to this post for review. Also, stop the PowerChute service, switch PowerChute out of debug mode, then restart PowerChute.