Network Shutdown with multiple vmware hosts connected to multiple UPSs

[Anonymous user]

We are trying to set up Network Shutdown functionality at a site that has 5 vmhosts connected to 6 UPSs (along with switches and other equipment). We had gotten it set up at one point, then it decided at some point that it had to shut everything down in the middle of the day for no reason and we uninstalled the NS server. We're not certain what happened. We were able to verify that it was the shutdown agent sending the signal, but we don't entirely understand why. Closest we could ascertain is that it determined that there was not enough runtime left to shut everything down before power failure, but we don't know why it thought it was on battery, or that there was an issue.

The hosts are connected to the UPSs in redundant fashion (with each PSU connected to a different UPS), but the connections are done at random (NOTE: we did not set any of this equipment up, we just inherited the mess from an idiot). We are not in a position where we can shut everything down and rearrange, so we are stuck working with this unless it becomes clear we have no choice. I have taken the time to trace the power cords on everything so we know where everything is plugged in.

As far as we've been able to tell in the NS client, there is no way to assign specific UPSs to specific hosts. Is this correct? Is there any work around?

The UPSs are just a group of 6 Smart-UPS RT 3000 XL with NMC1 (AP9619) cards and they seem a bit flaky. Is there a better configuration? I've discovered that these NMCs are either old or bottom end, can we upgrade these without taking the UPS offline?

Any suggestions would be greatly appreciated. If anything is not clear, I can try to elaborate. Thank you for your input!

[BillP · Accepted solution]

Hi,

On 7/24/2020 6:21 PM, Ben said:

We are trying to set up Network Shutdown functionality at a site that has 5 vmhosts connected to 6 UPSs (along with switches and other equipment). We had gotten it set up at one point, then it decided at some point that it had to shut everything down in the middle of the day for no reason and we uninstalled the NS server. We're not certain what happened. We were able to verify that it was the shutdown agent sending the signal, but we don't entirely understand why. Closest we could ascertain is that it determined that there was not enough runtime left to shut everything down before power failure, but we don't know why it thought it was on battery, or that there was an issue.

Did you configure PCNS to start a shutdown when available runtime has been exceeded? if so, that would explain the shutdown. See Schneider Electric FAQs 

FA405707 What is the Low Battery Duration setting used for?

FA241713 PowerChute Network Shutdown reports "Available runtime has been exceeded".

On 7/24/2020 6:21 PM, Ben said:

As far as we've been able to tell in the NS client, there is no way to assign specific UPSs to specific hosts. Is this correct? Is there any work around?

Are you running VMware vCenter Server or are the hosts standalone? If the hosts are managed by vCenter Server see Application Note 180 starting on page 6 for Advanced Redundant UPS configuration and Application Note 186. 

[Anonymous user]

On 7/28/2020 11:08 AM, Bill said:

Did you configure PCNS to start a shutdown when available runtime has been exceeded? if so, that would explain the shutdown.

Except that nothing was running on battery, so why would it shutdown? It was also on for 3 days before it decided to kill everything.

But yes, runtime exceeded was configured and the NMCs had the default 2 minute low battery duration.

On 7/28/2020 11:08 AM, Bill said:

Are you running VMware vCenter Server or are the hosts standalone?

Server

We actually figured all this out yesterday as we continued researching on our own, but thank you for the reply. However, this brings up a new question. NS is on a VM, which it warns about not being supported if you choose advanced configuration but it doesn't stop you from using it. Is that just due to the possibility of NS getting shutdown before everything else is done?

Oh, and can this shutdown the MSA's that hold the datastores?

I realize these questions are probably rather rudimentary, but information seems a bit scattered so we're asking questions along side our own research efforts. The input is much appreciated!

[BillP]

Hi,

On 7/28/2020 2:10 PM, Ben said:

Is that just due to the possibility of NS getting shutdown before everything else is done?

Yes. We cannot guaranty safe shutdown when PCNS is running on a VM in an advanced redundant UPS configuration.  

On 7/28/2020 2:10 PM, Ben said:

Oh, and can this shutdown the MSA's that hold the datastores?

I am not familiar with MSA datastores but if they allow for remote control via SSH you can write a script PCNS will run to command the datastores down. You will find information starting on page 79 fo the PowerChute VMware User's Guide. 

[Anonymous user]

Ok, thank you for your input. It has been very helpful.