UPS device shutdown multiple servers running Powerchute Agent

[Anonymous user]

Hi there,

I hope you can help with a predicament that i'm facing?

We have a single APC smart-UPS 5000 device installed in a rack in our local server room. The UPS is at capcity with 8 application servers connected to the UPS. Out of these servers, so far four of them have the PowerChute Network Shutdown agent running on them to manage elegant shutdowns.

The rack is powered through 2 separate mains supply for redundancy - one of which runs through a PDU (Power Distribution Unit) branch running down the side of the rack and the second directly through the UPS device. Ther servers in the rack are dual power supply, where one supply coming in to the server comes through the PDU and the other supply comes from the mains running through the UPS.

During the testing of the Powerchute agent shutdown on one of these servers, something initiated the shutdown of the other servers (that have the agent installed) as well? None of the other servers that accidently shutdown had events configured for detection of power failure, except for the server that was being tested.

The following test was geared towards a single server in this rack. The steps were specifically to:

1. Disconnect the power cable to interrupt supply coming through the PDU to this server (forcing it to depend on the supply running through the UPS)

2. Swith the mains off running through the UPS, such that the power would run off the UPS battery

3. The PowerChute agent would detect the event that the server was running off the UPS battery and initate a system shutdown.

The test server went through an elegant shutdown process when the mains were swithed off to the UPS, as expected. However, for some bizarre reason, all the other servers in the rack that had the Powerchute agent installed decided to shutdown as well. Also out of these servers, one (that has the agent on there) is not even connected to the UPS and is only powered through the PDU (due to maximum capcity reached on the UPS), it also decided to shutdown???

Our understanding is that the UPS device triggered a shutdown of all servers that had the agent installed, even though the servers that were not being tested still had a power supply running from the PDU?

I hope this will make sense to someone - would be grateful if anyone has any ideas what's causing this?

Thanks in advance...

Message was edited by: fonzarelli

[BillP]

did you let the UPS hit the low battery signaling time? what events are present on each server that shutdown in the PCNS event log?

normally, what happens is:

one server shuts down based on a special configuration you put in place, lets say shutdown after 5 minutes running on battery. that server shuts down. on each PCNS install, there is a check box under configure shutdown i believe, that says "Turn UPS off when this server shuts down" and if that is selected, it will command the UPS to turn off and go to sleep. if this box is selected and there are still other PCNS agents attached to the UPS that have not shut down yet, it will force them to shutdown.

in addition, if the UPS reaches a low battery condition, meaning no servers have anything special configured to shutdown like mentioned above, PCNS will tell everything attached, meaning PCNS clients, that it has a low battery and it is time to shutdown. therefore, any client in the PCNS clients list listed on the NMC under UPS->Powerchute->Clients will receive a unicast message to shutdown. once those PCNS clients receive a unicast message, it actually rebroadcasts that message over that network so that any clients running PCNS that are not on that list will also shutdown.

most of this is explained in [APC Application Note #20|http://www.apcmedia.com/salestools/TDOY-5UQVBV_R2_EN.pdf]. please take a look at that. additionally, there is a shutdown timeline in the PCNS agent help file.

[BillP · Accepted solution]

yes, i believe so. the rest of the servers, besides the one you specifically configured, will then shutdown when the UPS reaches a low battery, if the PCNS service is running. the default value for that would be when there is 2 minutes of runtime remaining on the UPS. so, if you have stuff that takes longer than 2 minutes to gracefully shutdown, you may want to adjust this value in the UPS Network Management Card. It is under the UPS Tab->Configuration->Shutdown-.*Low Battery Duration*

hope that helps!

[Anonymous user]

Many thanks for taking a look! - I know there's a lot to take in...

I've just checked the PCNS event logs on each of the servers and found the following:

11/04/2010 10:48:48 Communication has been established
11/04/2010 10:48:31 PowerChute Network Shutdown version 2.2.4 started
11/04/2010 10:39:30 Shutdown process started. Windows will shut down soon.
11/04/2010 10:39:30 UPS turn off has been initiated.
11/04/2010 10:39:30 Communication has been established
11/04/2010 10:38:55 PowerChute Network Shutdown version 2.2.4 started
11/04/2010 10:34:50 Shutdown process started. Windows will shut down soon.
11/04/2010 10:34:50 UPS turn off has been initiated.
11/04/2010 10:34:38 UPS has switched to battery power
11/04/2010 10:21:28 Input power has been restored: UPS has switched to On Line operation
11/04/2010 10:19:25 UPS has switched to battery power

Does this suggest that the UPS device actually did shutdown, because I'm not sure that I noticed that happening?

The thing is that I configured an event only on the server I was testing; this was under 'Configure Events' -> 'UPS: On Battery' and selected the 'Shut Down System' option.

Interestingly, I've noticed the following comment under 'Configure Shutdown', although I didn't configure anything here:

"When PowerChute shuts down your operating system, it will use these settings. Use the Configure Events page to select which events will cause the system to shut down"

The "Turn off the UPS after the shutdown finishes (Not applicable to Parallel Silcon UPS, Smart-UPS VT or Galaxy)." is checked as default?

[Anonymous user]

Just to confirm, please, if I uncheck this setting ""Turn off the UPS after the shutdown finishes (Not applicable to Parallel Silcon UPS, Smart-UPS VT or Galaxy)." under 'Configure Shutdown', I should avoid the scenario that I saw, right?

[Anonymous user]

That's great - Many thanks for your help!

[Anonymous user]

Sorry - I think that's what your post was saying above...