Redundant UPS - single UPS Failure - some servers shutdown, others do not

[Anonymous user]

Hi,

Wie have 2 SURT 10.000 XL behind 3 Bladecenters.

Using PCNS in redundant configuration

Shutdown only if UPS is on battery after 120s

Recently the batteries from 2006 failed in one of the two UPS - most Systems ignored that as expected because the other UPS still was alive.

But: 3 Servers (w2k8, W2k12) in one of the Bladecenters went into a Boot-circel:

I switched them on, OS is starting, PCNS shuts down the system.

These steps did not help:

- removing the IPs from the UPS-GUI, everytime the server came up and alive the IP showes up in the UPS GUI again and the server shuts down.

- setting the USV into Bypass-Mode

I had to remove eth-cable from the UPS and stop PCNS-Service on these three machines. I can see on the client-side, that both UPS are connected and configured.

Is there any idea what to do to prevent such behavior?

Thnx - Chris

[BillP]

Chris,

What version of PCNS is installed and what is logged in EventLog.txt

[Anonymous user]

Hi Bill,

PCNS 3.01 build 2650

Eventlog-snap:

04/14/2014	16:41:05	PowerChute successfully opened UDP port 3052.	.3.4.9.9
04/14/2014	16:41:06	PowerChute Network Shutdown version 3.0.1 started	.3.5.1.5.6.10
04/14/2014	16:41:06	PowerChute successfully opened TCP port 3052.	.3.4.9.9
04/14/2014	16:41:06	PowerChute successfully opened TCP port 6547.	.3.4.9.9
04/14/2014	16:41:08	Network Management Card apc4.xxx.com communication has been established.	.3.5.1.5.6.13
04/14/2014	16:41:08	Communication established with Environmental Monitoring Card	.3.5.1.7.20
04/14/2014	16:41:08	Network Management Card apc5.xxx.com communication has been established.	.3.5.1.5.6.13
04/14/2014	16:41:08	Communications have been established.	.3.5.1.5.6.1
04/14/2014	16:41:08	Communication established with Environmental Monitoring Card	.3.5.1.7.20
04/14/2014	16:41:08	Available runtime has been exceeded.	.3.5.1.6.1
04/14/2014	16:41:25	Shutdown process started. Windows will shut down soon.	.3.5.1.4.3

It is indeed configured to shutdown when runtime is exceeded - but runtime is only on "apc5.xxx.com" exceeded - "apc4xxx.com" is not affected.

[BillP]

Hi,

If one of the 2 NMCs report runtime has been exceeded PCNS will start OS shutdown.

This is from the PCNS help file

Available runtime has been exceeded:

This event is always logged and, on a Redundant-UPS configuration, event actions are carried out even if it occurs on only one Network Management Card.

To correct the issue you will need to resolve the reported issue.

Two configuration conditions can cause this event:

  When the command file time on the Configure Shutdown page plus 10 seconds is greater than the Low Battery Duration field value. (A low battery condition always causes a shutdown, so when it occurs the UPS must have enough time available to complete shutdown activities). To fix this, either decrease the command file time, or increase the Low Battery Duration value. (To change the Low Battery Duration, on the Network Management Card user interface select the UPS tab and Configuration - shutdown).

  When the UPS runtime is less than the sum of the shutdown delay time configured for the UPS: On Battery event plus the command file time on the Configure Shutdown page, minus two minutes. This condition can be caused by having too much load on the UPS, or by the UPS batteries not being fully charged. If the UPS batteries are fully charged, to fix this a) remove some equipment from the UPS, or b) decrease the shutdown delay time for the UPS: On Battery event, or c) decrease the command file execution time on the Configure Shutdown page.

[Anonymous user]

Thank you Bill.

Although it seems as "works as designed" this "design" makes not any sense to me:

If I configure a "redundant UPS setting" I exactly expect that it is redundant in any way - but if a shutdown is always initiated when a "runtime exceeded" failure is noticed on ONE of my "nodes" the whole configuration is not to be called "redundant" - if in an clustered server enviroment any node logs a failure and the whole cluster would shut down nobody would need to setup a cluster.

And: What is the correct solution to avoid that shutdown-circle? I thought to put the failed UPS in Bypass-mode or to remove the Server-IP from the UPS, but both of these things did nod help and I cannot imagine that I always have to plug of the NMC or the cable to stop all the servers to shutdown permanently until new RBCs arrive.

[BillP · Accepted solution]

Hi,

The logic is, if 1 of the 2 UPS cannot support the entire load by itself then it is deemed not safe and since you have selected shutdown for runtime exceeded action should be taken. The solution is to correct the issue as described in the help file. You can also configure the UPS to not provide power until the batteries have charged properly and that will prevent the looping.

In your case since the UPS may or may not charge properly so the runtime will always be low I would stop the PCNS service, edit pcnsconfig.ini on the affected systems and deselect shutdown due to runtime is exceeded (event_RunTimeExceeded_enableShutdown = false)  until the batteries are replaced.