PCNS sends signal to turn off UPS (SURTD3000RMXLT3U) but UPS never turns off

[Anonymous user]

We have a Smart-UPS (SURTD3000RMXLT3U) with an installed NMC (ap9617). We have one ESXi host with a Vcenter Server appliance, a vSphere Management Assistant VM with PowerChute Network Shutdown 4.0.0 Build 6078 installed, and 4 other VMs. (edit: So to clarify 2vapps and 4 VMs are all running on the same esxi host)

The idea for our configuration is for the VMs and then the host to shut down after 600 seconds, and then have the UPS turn off.

We pulled the plug to the UPS to test it, and after the configured 600 seconds it started shutting down the VMs as intended, and after the VMs had been shut down it initiated the host shutdown. The host powered off, but the UPS stayed on. Before we lost access to the PCNS interface due to the host shutting off, we noticed that it did record the "PowerChute is turning off UPS" event. We waited about 8-10 minutes, but the UPS was not powering off. We restored power to the UPS, and the host did not power on, presumably because UPS power was not lost. We manually powered back on the host.

Did we just not wait long enough for the UPS to turn off? The shutdown delay is set to 120 seconds. I read somewhere that the Maximum Required Delay may have an effect on how long the UPS takes to power off as well. I am just puzzled as I can see that it sent the command to turn off the UPS..

PCNS Event log from test:

5Al468vPx8tfJwhUtwy8hA%3D%3D&Rest-Api-Key=d3Bhc3F1aWw6NjM3MzU2MWEtNzYzMC00OTc5LTkyMTAtOTJjYTU0NGEwZWFl.png

Screenshots of our configuration:

jyqdq7ReWXVgY3cMBPdWlQ%3D%3D&Rest-Api-Key=d3Bhc3F1aWw6NjM3MzU2MWEtNzYzMC00OTc5LTkyMTAtOTJjYTU0NGEwZWFl.png

YnGrTk5kfPJJXA1SzrJalw%3D%3D&Rest-Api-Key=d3Bhc3F1aWw6NjM3MzU2MWEtNzYzMC00OTc5LTkyMTAtOTJjYTU0NGEwZWFl.png

UCYNkFTZ21UPF2QlMt7csA%3D%3D&Rest-Api-Key=d3Bhc3F1aWw6NjM3MzU2MWEtNzYzMC00OTc5LTkyMTAtOTJjYTU0NGEwZWFl.png

IzcOzyJUw67KBSryA4mr2w%3D%3D&Rest-Api-Key=d3Bhc3F1aWw6NjM3MzU2MWEtNzYzMC00OTc5LTkyMTAtOTJjYTU0NGEwZWFl.png

xcaQ6qlg93VPV2XfMAkmig%3D%3D&Rest-Api-Key=d3Bhc3F1aWw6NjM3MzU2MWEtNzYzMC00OTc5LTkyMTAtOTJjYTU0NGEwZWFl.png

0Lx9mF3RPIN65YkPaAHHTg%3D%3D&Rest-Api-Key=d3Bhc3F1aWw6NjM3MzU2MWEtNzYzMC00OTc5LTkyMTAtOTJjYTU0NGEwZWFl.png

4miJqe1lxHbOsDBeuyQf7A%3D%3D&Rest-Api-Key=d3Bhc3F1aWw6NjM3MzU2MWEtNzYzMC00OTc5LTkyMTAtOTJjYTU0NGEwZWFl.pnghsvH2gGjo9OtWp2G2%2BcagQ%3D%3D&Rest-Api-Key=d3Bhc3F1aWw6NjM3MzU2MWEtNzYzMC00OTc5LTkyMTAtOTJjYTU0NGEwZWFl.png

EDIT:

I read the following in application note 180:
"10. UPS waits for greater of Low Battery Duration/Maximum Required Delay (Non - Outlet Aware UPS’s) or the Outlet Group Power Off Delay .

11.    UPS turns off after the user - configurable Shutdown Delay time has elapsed or the Outlet Group turns off after the power off Delay elapses . "

If I am getting this correctly, and the case is that after all VMs/hosts are powered off, the UPS starts waiting for the full duration of whichever is longer between Low Battery Duration and Maximum Required delay, then if during my test, I would have waited 20 minutes + the 120second shutdown delay after the host shut down, the UPS would have shut down as well.
If this is correct, and if the only way to shorten the duration of the delay time of shutting off the UPS after all powered hosts are shut off is to shorten the maximum required delay/low battery duration, then this poses a problem - as it increases the duration in which the host is powered off without the UPS being powered off. Unless I am mistaken, if during that time, the power to the UPS is restored, the UPS never shuts off (and therefore will not power back on) the host will not automatically power back on either.

[BillP]

Steve,

here is the diagram for the shut down process. 

PkqhbjxiOmjPxOcgAvr2Hw%3D%3D.png

So After the delays in PCNS before the OS is commended down PCNS sends a signal to the UPS tell it to power down. The UPS will what the greater of low battery duration or Maximum required delay, wait an additional 2 minutes (as noted) and the power down. The reason for this is the max required delay is the greatest amount of time it should take for all of the systems to power down gracefully. If your system/s do not need 20 minutes to power down you can lower the setting. 

You should see an entry like this in the NMC event log

UPS: A graceful shutdown process is being used to shut down the load equipment before the UPS turns off. Initiated by PowerChute ns from 192.168.1.10.

If power is restored during the time the UPS is waiting to power down it will continue the count down / completes the power down cycle and then restarts. 

[Anonymous user]

I guess that is what I was worried about. Isn't the only way to lower the Maximum Required Delay to lower the Low Battery Duration? Is there any way to adjust the Low Battery Duration to be higher than the Maximum Required Delay? Although I guess if there was that wouldn't be a workaround either, as it chooses the higher between the MRD and LBD anyways..

Also, I guess I am confused as to what 'OS Shutdown' would be referring to in my situation. The VMs get shut down before the host, and then the host itself shuts down. Is 'OS Shutdown' referring to PCNS? Because PCNS is running on a Vapp on the host that is being shut down, and if this is the case, according to that flowchart, then that host is shut down before it can send the command for the UPS to shutdown..?

EDIT:

I found this response by another Schneider representative to a similar question "...the shutdown message gets sent before it actually shuts down. The network management card then waits a duration that is equal to "maximum required delay" + 2 minutes (this is built in and cannot be changed) + "shutdown delay."  

So maybe the in the diagram it should be noted that in situations where PCNS is running on the host it will be shutting down, it actually sends the shutdown signal to the UPS before it shuts down the host.

In any case, I believe in my test it would have shut down the UPS If I had just waited longer. But there is still the issue of not being able to adjust (shorten) the time the UPS waits to shut down after all hosts are off without adjusting the Low Battery Duration, unless I just missed something..

[BillP]

Hi,

Basically the process for your configuration is

UPS on battery, on battery delay time, VM/vAPP shut down, vCenter server shut down, PowerChute sends shutdown command to UPS, PowerChute start OS / host it is running on shutdown. Step 4 would be skipped since there is only 1 host with PCNS on it.

This info can be found on page 70 of the Users Guide  (page 73 of the PDF). We also have Application Note 180 that discusses PCNS, VMware shutdown and start up.

Here is the description of Maximum required delay from the NMC help file

Maximum Required Delay

Calculates the delay needed to ensure that each PowerChute client has enough time to shut down safely when the UPS or the PowerChute client initiates a graceful shutdown. Maximum Required Delay is the longest shutdown delay needed by any server listed as a PowerChute Network Shutdown client. This delay is calculated whenever the management interface of the UPS turns on or is reset, or when "Force Negotiation" is selected as Maximum Required Delay. "Force Negotiation" polls each client for information on the time it needs for a graceful shutdown. Two extra minutes to allow for unforeseen circumstances is then added to the calculated time. The negotiation can take up to 10 minutes. If you do not select "Force Negotiation", two minutes is used by default as the shutdown delay for all clients.

If you feel that the maximum required delay is to great yu need to change the delays set in PCNS and then run "Force negotiation" on the NMC to lower the delay time. This time is set to ensure that there is enough available runtime for everything to shut down and start up gracefully before the UPS powers off.

[Anonymous user · Accepted solution]

Thanks for the clarification, Bill. I will run another test with the lowered Maximum Required Delay and see if I get the desired results.

Edit:

Test went as intended, thanks for your help!