Symmetra LX 16000 reports all batteries failed

[Anonymous user]

I'm having a bit of a panic attack. We have 2 Symmetra LX 16000 and one of them suddenly shows all 9 battery modules as failed and reports Runtime Remaining: 0 minutes. I tried disconnecting and reconnecting all battery modules but after a few minutes all batteries are reported as failed. I also noted that in the overview you can see "Battery Capacity: 99%" which confuses me even more.

Is there anything that can be done? Can 9 battery modules fail simultaneously?

[BillP]

Hi Carlos. Very sorry for the delayed response. Typically, under normal circumstances if all the batteries are being used the same and are from the same batch (same age) then it is not impossible for them to all fail at once. It is not as common, but I have seen it before. How old are the batteries in question? Have they ever been replaced before? The event logs and a screenshot of the battery system from the NMC page showing serial numbers and mfg dates of the batteries would be helpful. 

[Terry_Kennedy_apc]

I am not familiar with the LX, but in the Classic and RM Symmetra models, all communication happens over a single I2C bus and anything "hanging" that bus will generate lots of scary messages.

I'm just another user (I don't work for APC) but here is what I would do (assuming you don't have on-site APC support available):

Assuming your input power is clean enough and management policies allow it, I'd put the UPS in manual bypass, unseat all of the modules (MIM/RIM/PM/BATT) and unplug any communication cables from extended run cabinets (battery bus cables can stay connected). Then bring up a minimum configuration of MIM, 1 PM and 1 BATT. If the management interface indicates that this is a working config, then add items one at a time (first the RIM, then the PMs, then internal BATs, then external comms then external BATTs) until you either get the failure notice again or everything is working properly. Sometimes a simple remove/insert is enough to clear the problem.

Note that the minimum configuration is sufficient to start the UPS but is probably not enough to support your load - don't come out of bypass until you have everything reconnected and verified as working or you risk dropping the load.

A second note is that this procedure is ONLY for recovering from this specific condition. If you have a report of a single failed battery, don't remove and re-seat it in the hope it will get better. On the older Symmetra models the unit will mark it as good when it re-appears and you'll have the same thing the next time it does a self-test. On at least some newer Symmetra models, the UPS rewrites the SEEPROM in the battery module marking it permanently bad.

[Anonymous user · Accepted solution]

What is your Actually Battery Dc bus voltage?  Also, do you have any internal temp faults in your logs?