PCNS did not shut down environment

[BillP]

We experienced a recent power outage where our virtual PCNS 4.3 device did not shutdown our VMware environment (we are running vSphere 7 on VCSA, the two hosts in the same cluster are running ESXi 7).  All that happened was our UPS eventually drained its battery and all the equipment powered off with no safe shutdown.

The APC UPS is a Smart-UPS SRT 5000 (SRT5KXLI) and is using the built in management card AP9537SUM.

Both the VCSA and the PCNS are on the same IP range.  I can validate the communications settings on the PCNS when I click check details, the authentication phrase is the same one used on both the PCNS and the UPS.

The ESXi hosts and the UPS device are all plugged into the same switch in the same physical location.

When I check host protection I can see the hosts.

When I look at the event log I can see random alerts that 'Cannot connect to http://x.x.x.x:80' (IP address is the UPS) but then it also alerts saying it has successfully opened TCP port 3052 and 6547.  

I also see random events that vCenter Server authentication error but then seconds afterwards it says vCenter Server is accessible

I need advice as to where to start troubleshooting and confirmation that this setup should work.

Thanks.

[BillP]

Hi,

 I would first look at the network card event log to find the date and time the UPS switched to the battery. Then you should check the PowerChute event log, error log, and debug log. The logs are in /opt/APC/PowerChute/group1. In the PowerChute event, log look to see if there is an entry posted when the UPS switched to the battery? Also, in the debug log do you see lost comm with the network card and or vCenter? 

Troubleshooting help starts on page 174 of the attached User's Guide.

[BillP]

Thank you for coming back to me.

The debug log regularly performs this routine, I assume this is normal and it is checking to see if it needs to do anything?

2020-09-15T11:48:28,190 DEBUG pool-3-thread-1 com.apcc.m11.components.Shutdowner.VirtualHostStatusTask - run() - start
2020-09-15T11:48:28,191 DEBUG pool-3-thread-1 com.apcc.m11.components.Shutdowner.ShutdownState - getCriticalShutdownTimers() - no critical timers
2020-09-15T11:48:28,191 DEBUG pool-3-thread-1 com.apcc.m11.components.Shutdowner.VirtualHostStatusTask - checkCriticalHostsShutdown() - no critical timers, host shutdown not required
2020-09-15T11:48:28,191 DEBUG pool-3-thread-1 com.apcc.m11.components.Shutdowner.VirtualHostStatusTask - checkPerformStartup() - start
2020-09-15T11:48:28,191 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - isStartupWaitsForAllHostsOnlineEnabled() - found key: startup_waits_for_all_hosts_online
2020-09-15T11:48:28,192 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - isStartupWaitsForAllHostsOnlineEnabled() - end, result: false
2020-09-15T11:48:28,192 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - isESXI() - end, result: true
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.Shutdowner.VirtualHostStatusTask - isVCSAStartupRequired() - start
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - loadProperties() - start
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - getVirtualStorePropertyFilePath() - end, result: /opt/APC/PowerChute/group1/VirtualizationFileStore.properties
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - getProperty() - VCSA_HOST = null
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - getVcsaHost() - end, result: null
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.Shutdowner.VirtualHostStatusTask - isVCSAStartupRequired() - vCenterHostIP is null, returning false
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.Shutdowner.VirtualHostStatusTask - proceedWithVirtualizationStartup() - VCSA startup is not required
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - getLockedHostSetFromVirtualizationFileStore() - start
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - loadProperties() - start
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - getVirtualStorePropertyFilePath() - end, result: /opt/APC/PowerChute/group1/VirtualizationFileStore.properties
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - getHostSetFromVirtualizationFileStore() - end, locked: false hosts: []
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - loadProperties() - start
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - getVirtualStorePropertyFilePath() - end, result: /opt/APC/PowerChute/group1/VirtualizationFileStore.properties
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - getProperty() - Witness_VM_HOST = null
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - loadProperties() - start
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - getVirtualStorePropertyFilePath() - end, result: /opt/APC/PowerChute/group1/VirtualizationFileStore.properties
2020-09-15T11:48:28,193 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - getVMAndVAppHostList() - end, result: []
2020-09-15T11:48:28,194 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - isNutanixSupported() - end, result: false
2020-09-15T11:48:28,195 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - isNutanixESXi() - end, result: false
2020-09-15T11:48:28,195 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - isESXI() - end, result: true
2020-09-15T11:48:28,197 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - isESXI() - end, result: true
2020-09-15T11:48:28,197 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.AdvancedSetupHelper - isAdvancedShutdownMode() - result: false
2020-09-15T11:48:28,197 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VMware.VMWareConnection - isvCenterServerConnectionSuccessful() - result: true
2020-09-15T11:48:28,197 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - getClusterSetWithDrsChanged() - start
2020-09-15T11:48:28,197 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - loadProperties() - start
2020-09-15T11:48:28,197 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - getVirtualStorePropertyFilePath() - end, result: /opt/APC/PowerChute/group1/VirtualizationFileStore.properties
2020-09-15T11:48:28,198 DEBUG pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualisationStatusStore - getClusterSetWithDrsChanged() - end, result: []
2020-09-15T11:48:28,198 DEBUG pool-3-thread-1 com.apcc.m11.components.Shutdowner.VirtualHostStatusTask - run() - end

At the time of the power cut the debug log mentions nothing of power loss or issuing commands, it just repeats the above every two minutes until the UPS battery was drained.

The UPS log confirms that it switched to battery so there is no fault with the UPS (could it possibly need a firmware upgrade to work with newer version of 4.3 PCNS?)  The UPS firmware is running UPS 04.0 (ID1013) and the onboard card is running application/OS version 6.5.0 and boot monitor is running 1.08 - I have tried to look up our model on the APC UPS website for updates but cannot find any?

What we have wondered today is on PCNS device going into Communication Settings does the vCenter account have to also be a local admin account on each ESXi host but also be an admin account in vSphere?  At the moment, we are using the default administrator@vsphere.local account for the vCenter login

Regardless of the above, we may have solved this because on the PCNS device Configure Events only Logging was selected.

Can you confirm that we do need to set a shutdown action, for example, Runtime remaining below threshold set to x number of seconds before it will do anything?!

🙂

[BillP]

Hi,

On 9/17/2020 11:58 AM, T said:

What we have wondered today is on PCNS device going into Communication Settings does the vCenter account have to also be a local admin account on each ESXi host but also be an admin account in vSphere?  At the moment, we are using the default administrator@vsphere.local account for the vCenter login

You should create an admin account that PCNS will use to communicate to vCenter and each host. See the videos attached to Schneider Electric FAQ FA349917.

On 9/17/2020 11:58 AM, T said:

Can you confirm that we do need to set a shutdown action, for example, Runtime remaining below threshold set to x number of seconds before it will do anything?!

If you do not configure any shutdown setting, PCNS will trigger the shutdown process when the UPS reaches low battery. See Schneider Electric FAQ FA405707

I recommend configuring PCNS to shutdown after UPS runs on battery for x time or when runtime remaining fall below x. Setting one of those will prevent the battery from draining to the low battery state. 

[BillP · Accepted solution]

Thank you for pointing me in the right direction

We have now successfully managed to shutdown our environment by implementing the following:

1. Adding local user accounts to each host + vsphere.local account to our vSphere setup with administrator access
2. Performed setup wizard again on PCNS device using port 443 to access the UPS NMC and reapplying the authentication phrase