Single UPS multiple cluster nodes and servers PCNS shutdown

[Anonymous user]

How to properly configure sequenced shutdown of 4 servers, 2 of them are in a hyperv cluster. (S1,S2 are cluster nodes, S3 and S4 are standalone servers). Once the last server is down, I want ups to turn off to conserve battery. Once the current is recovered to the UPS, I want it (after some delay) to turn on and enable output power. The shutdown sequence is two fold: configure the events for each server and configure the shutdown behaviour. Documentation only handles the former and the latter is not mentioned anywhere. The example follows:

Let's say I want to shutdown S3 after 5 minutes on battery, S1 and S2 after 10 minutes on battery (with no migration, only guest shutdown) and S4 after 15 minutes.

I set in events->UPS on Battery, Shutdown delay: 600 s for S3, 1200 s for S1 and S2 and 1800s for S4. Then for S1 and S2 in Virtualization settings disable migration and enable VM shutdown with appropritate delay and enable VM startup with appropriate delay.

The problem that confuses me: If I set the on battery delay on S3 to 5 minutes, then, at least I hope so (no good documentation on this) if the power is returned during the delay, nothing happens. If it does not, then after the 5 minutes pass, the UPS generates 2 events UPS starts to turn off and at the same time S3 shutdown is initiated. Now if I also configure in PCNS on S3: Shutdown settings->turnoff UPS, this might lead to turning of UPS once S3 is shutdown with S1,S2,S4 still running. But the UPS shutdown event seems to be generated regarldess of shutdwon settings --- see the documentation on hyperv, chapter Hyper-V: UPS without Outlet Groups. All that the documentation covers are the delays but since the first server that starts to shutdown and after delay initiates the ups turn off, how to ensure the other servers stay alive as required until their time comes? Also which server should tell ups to shut down (the last to go or all?)?

[BillP]

Hello,

What is the delay set for VM shutdown and start-up? That will make a difference as to which PCNS client tells the UPS to power down. Also, what is the model UPS and what is the model and firmware of the network management card?

In a standard setup the the PCNS client that starts the shutdown process last (in your case S4) would be configured to tell the UPS to power down. You would configure S3, S1, and S2 with Do not turn off UPS and S4 with Turn off UPS. With a UPS without outlet groups the process is umN9vPtOTo0rkYND%2BwR4Hw%3D%3D.png

When S4 has been running on battery for 1800 seconds it will start the shutdown process. At that point PCNS will send the shutdown command to the UPS. If no command file is run the system will wait 70 seconds and the OS will be powered down. 

When the UPS receives the shutdown command it will wait the greater of Low battery duration or Maximum Required delay, then wait a non-configurable 2 minutes and then the UPS will cut power to the outlets.

Low battery duration is set in the network management card interface under Configuration - Shutdown. Also on the NMC page you will find a button to select restart when power is restored. 

[Anonymous user · Accepted solution]

Hello,

Thank you for the answer. In the meantime, I created a chart of what I expect will be going on during shutdown (I am sorry, I am now using the name within infrastructure rather than the previously used S1...S4). I have made use of this article describing various delays.:

Biwvm0TL%2FFtGTWRDKyYhQw%3D%3D.png

I have a few brief questions:

1. If there is no command file selected in the PCNS, will the delay for its execution be applied or not?
2. From what you suggested in your post, I concluded that the server that is up for the longest amount of time and the one that should tell the UPS to shut down, it should have on battery delay longer than the time of all other servers to shut down, correct?
3. In the chart above, you can see that after the last server tells UPS to shut down, takes almost half an hour for UPS to turn off, mainly due to the 'Low battery duration' (LBD) settings (in my case LBD is always equal to "Maximum required delay" (MRD), when I change LBD, MRD gets always recalculated to equal LBD). Is there any reason against lowering LBD to eg 5 minutes in the setup above when the shutdown is PCNS directed? (I believe the biggest issue here is if there are repeated power losses (power off for 8 minutes, power up for 2 minutes repeated a few times) which deplete the batteries while each time they restart the "On battery delay" counter. Could this lead to a shutdown initiated by LBD? I assume NMC would signal LBD to PCNS which would enforce shutdown and so LBD should be long enough to enable such scenario, correct?)
4. If the power comes up during "VM Shutdown delay", what happens? (where is the Point of No Return for PCNS on hyperVs with VMs shutddown and migration?)
5. Manuals on PCNS mention outlet groups. I believe our symmetra (LX 4kVA Scalable to 8kVA - SYA4K8RMI) does not support outlet groups but we are using APCs rack controllable PDUs. Can those be used in conjunction with PCNS to cut power to individual servers?

And one longer:

The most obvious issue with the setup above is if the power comes up at any time after some servers got to Point of No Return and started shutting down or have shut down but before the last PCNS issues command to UPS to shutdown, the UPS shutdown will be aborted and the infrastructure will be in partial shutdown (some servers will be down and will not go up automatically). Are there any recommendations how to handle this besides manual restart of the shutdown servers? An obvious solution that comes to mind would be that the first server to go down (in the case above "VEEAM") would tell UPS NMC to shutdown then the whole infrastructure would turn off, once a single server turns off. But this has the con that the times required to shutdown are not predictable and it could turn out that a particular server required more time and ended up without power while shutting down. Other ideas include probably scripts (but I am not sure if I can run a script once a power is restored with some delay)...

Thank you.