Symmetra PX-80 going to Battery when a single Power Module failed

[Anonymous user]

We have a pair of PX-80's redundantly powering one of our two core campus data hubs. Had an unexpected thing happen, which I was hoping someone from 3 Phase support could comment on:

Our "B" ups experienced a lone power module failure around noon Monday, and instead that module being taken offline automatically, the UPS went to battery.... and stayed that way until I had raced over to the site to intervene....when I arrived it was indeed showing an "on battery" led on the front panel, red fault led on, and charge levels had dropped to around 47% I had a few minutes to spare. The failed power module's fans were still spinning. I had to physically remove the failed module from the slot it was in before the UPS went back online and started charging again.  According to the MIM/RIM logs that module experienced a PFC failure....there was nothing in the logs to state the module had been offlined other than an entry saying the battery charger was shutdown, and that module was unable to work on Mains. Usually I see something about failed modules being Locked out.

The UPS had 8 power modules before this happened....now at 7.

So, does this mean there are certain circumstances where module failures cannot be completely isolated and they will affect the UPS overall operation ?

What would have happened if I had left it as is and the batteries drained completely ... would it see that it still has normal input voltage and go to bypass, or would it drop the load ?

...right now the unit is set to "If UPS fails ... Go to Bypass" in the Power/Utillity Line config...would this event / situation constitute "a failure of the UPS" ?

MIM log is below...

Thanks,

Dave

0100D53E IM    D UPS is below min acceptable runtime      14/07/21 13:40:39 561

0208D43E PM 08 D PFC fail. Unable to work on Mains        14/07/21 13:37:24 405

0900A05E IM    A IM oper. mode: Shift to ON LINE          14/07/21 13:37:23 957

0100D32E IM    D Power Module 8 present                  14/07/21 13:37:23 946

0100A53E IM    A UPS is below min acceptable runtime      14/07/21 13:31:12 804

0208A43E PM 08 A PFC fail. Unable to work on Mains        14/07/21 13:16:45 300

0900A06E IM    A IM oper. mode: Shift to ON BATTERY      14/07/21 13:16:29 497

0100D57E IM    D Inverter sync. error                    14/07/21 13:16:29 309

0208A05E PM 08 A CHG oper. mode: Shift to OFF            14/07/21 13:16:25 738

0208A03E PM 08 A PFC oper. mode: Shift to BATTERY        14/07/21 13:16:25 738

0208A20E PM 08 A Mains synch fail                        14/07/21 13:16:25 729

0100A57E IM    A Inverter sync. error                    14/07/21 13:16:24 966

0100D57E IM    D Inverter sync. error                    14/07/21 13:15:38 762

0100A57E IM    A Inverter sync. error                    14/07/21 13:15:37 183

0100D57E IM    D Inverter sync. error                    14/07/21 13:04:16 442

0100A57E IM    A Inverter sync. error                    14/07/21 13:04:14 948

0100D57E IM    D Inverter sync. error                    14/07/21 13:03:35 259

0100A57E IM    A Inverter sync. error                    14/07/21 13:03:31 637

0100D57E IM    D Inverter sync. error                    14/07/21 13:03:08 649

0100A57E IM    A Inverter sync. error                    14/07/21 13:03:03 592

[BillP · Accepted solution]

Hi Dave,

I did some consulting. This is by design. In your log, you see "unable to run on mains" AKA cannot run online which is a fault you can also see on LX coincidentally too but the behavior differs on LX vs PX. I'll focus on PX since that is what you are asking about.

When this issue happens, the UPS switches the PFC to battery power to keep the module working so that the inverter can still be used to power the load but the battery supply will drop slowly. I think the idea is that we want the inverter to be available, even on battery, since redundancy can vary from customer to customer (meaning we don't know every customer is necessarily running with N+1 or N+2, etc).

And also, it would go to static bypass if the battery capacity was depleted.

Lastly, this is the only type of module failure that would cause a a shift in operational state (unless of course the system became overloaded due to a power module failure which it would also go to bypass for).

[Anonymous user]

Thanks, Angela,    ...that is a relief to know i would not have had load issues had I not been around to intervene. The only thing I am still unsure of is what would happen if I tried to put it in manual static bypass (because it almost came down to that). The battery capacity did not exactly drop "slowly" though; it was down to below 50% capacity after only 20 minutes or so, so a discharge of about 1/2 the normal rate but still quite a heavy drain. The batteries are probably 5 years old now but still in good condition. Cheers!

[BillP]

Hi again

The team here I asked does not see a reason why it wouldn't have gone into static bypass if you did it manually. And the rate at which the modules drain would really depend on the number of batteries and number of power modules installed because thinking about it, if you had more power modules, you have more chargers, and hopefully less of a drain. And like you said, the individual batteries may respond differently too based on their age/capacity.

Let me know if you have any other questions.