Symmetra PX battery thermal runaway, how?

[ojfsdi]

I have noticed several times when our UPS reported a battery module failure via its email alerts ("

UPS: The internal battery temperature exceeds the critical threshold.

", that it's not just one battery cartridge, but two, that are quite hot.

I examine with a flir camera before touching, and usually see an overheating PAIR, not just one.

It's always been an AB or CD pair too (never BC).

I'd like to understand, electro-chemically, why this happens in a pair.

I know that the frame pairs up A with B and C with D.

There's a 192v low, high, and neutral bus, with neutral between B and C, and the low and high on either side of the frame.

So, is the problem that one of the batteries in cartridge B fails short circuit, and that causes the others in A and B to over charge, which produces heat and eventually causes another to fail short circuit from the heat, further raising the voltage across the remaining cells ad infinitum?  (Eventually placing 192v across effectively one battery?)

Can the symmetra PX automatically isolate an AB or CD pair of cartridges when it starts to overheat?  (Does it have a couple relays on each shelf for instance?

I hate the idea of thinking that if one of these modules goes bad, it could end up causing a fire if there's nobody on site to manually remove it.  I've measured 190F on some of the internal batteries before.  High enough that I park the hot packs in front of a CRAC for a couple hours before opening them to reduce the risk of running into hot acid.  I'd also be a little surprised if they had any liquid left in them being so hot for so long they might be all dried up.

[al_buffington]

Hi,

The PX does not have the ability to isolate any individual batteries (only battery cabinets through the DC disconnect - and even then this will not activate based on battery temperature). If a battery is reading 190F then it likely has been in alarm for quite some time, as this is well over the threshold for declaring a fault. The Symmetra PX includes a network management card which would notify the user as soon as a battery reaches the upper temperature limit.

[ojfsdi]

I'm a little confused how the PX battery cabinet can know the temperature of the batteries actually....

I've looked inside the modules.  I generally like their design, but they have a circuit board at one end, separated by an inch and some plastic from the batteries themselves.  There are no temperature probes per se that are not on the circuit board itself. 

Every battery cartridge that I have observed high temps on, the highest temps were in the innermost batteries, far from the circuit board and any thermometers on them.

If you number the batteries from front (the handle) to back:

[Handle] 0 1 2 3 [1cm spacing] 4 5 6 7 [Circuit board]

It' s 3 and 4 that get the hottest.  Very often cells 1-3 and 4-6 melt together, with 3 and 4 so deformed that I feel like PPE is mandatory to remove them (though I have yet to actually see free acid).  I've seen several times where 3 and 4 were so bloated that they were in contact with each other.  I let the whole pack cool off in front of a CRAC unit for several hours before touching it when they're that bad.

Often 0 and 7 show little sign of damage, even when 3 and 4 are melted way down.

So, how could the PX know the battery temps without a temp sensor on the batteries?

Does the PCB infer cell temps via the circuit through them or does it just use a thermistor on the PCBoard? 

And to my original question, how does the thermal runaway happen, internally i the batteries.  I realize battery chemistry might be a little beyond a UPS forum.  But maybe not, given how central batteries are to the concept of a UPS.

[al_buffington]

Hi,

I am not an expert in the area of battery chemistry where I can comfortably address that specific portion of your question. From my experience it's usually when batteries overheat due to things like storage conditions, overcharging, lack of maintenance, environmental conditions etc.  I can better help you with how the temperature is sensed. As you have noted, in the back of each battery module there is a PCB.  That board measures the temperature, as well as the type of battery (vendor) and reports it back to the Intelligence Module. So, it's not measuring each  of the 8 batteries inside the module individually, but reporting one single module temperature back to the IMs. Hope this helps.

[ojfsdi]

Interesting!  I looked at the pcboard more closely, and I didn't actualy see any components  (thermistors) on it that looked like a temp sensor.  Of all the components that were sticking up in the air it was mostly resistors, which I guess could infer temperature but it's not very common.

Perhaps the temp sensors are built into the shelves themselves, pointing down into the battery modules?

How would something sense the type of battery (vendor) ?

The ones that were in our Symmetra originally were "Apex" brand, and the new ones are Interstate Battery.  Neither of them have any communications pins, or rfid that I'm aware of.

I'll take one of the pcboards out and get a photo of both sides of it.  Maybe the thermister is on the back side, but that would be even more isolated from the actual battery temps.

[al_buffington]

Hi,

You should see a reference designation labeled TH1 on the board. You should see a wire (blue I believe) connected at this point which runs to the connector on the outside of the module.  Regarding the sensing of the battery type, this is done with a resistor(s), the value of which tells the UPS what battery is installed. As a result there are a few different versions of the PCB. The batteries you are currently using (Interstate) and the prior batteries (Apex) are not qualified for use in this UPS. It's quite possible that this is contributing to your overheating situation. 

[ojfsdi]

I found it!  Little green thermistor on the PCB indeed!

There's no way that thing is getting a good temp read on the batteries.  There's just too much distance, no airflow, and the one on the end by the circuit board is never the hottest one when they overheat.  Better than nothing, but far from perfect.

I found the following wiring in my battery module. 

[Blue]----<thermistor th1>-----[brown] -----<resistors>-----[yellow]

[Double-black wires]------<Fuse>-----[single-black-wire]-----<battery string>-----[Double-red wires]

There's space for three paralleled resistors on the board.  I have two installed, either 110 Ω ±2% or 2 kΩ ±1%.  I'm not sure what direction to read the bands in.  So it doesn't 'detect' the capacity of the batteries.  It 'reports' it, based on how the pack is configured by the resistors installed at the factory.

Seeing the blue, brown, and yellow wires are about 12 AWG thickness, I'm surprised they're merely used for signalling.

How does the resistance correlate to the ampacity of the cartridge?  Is there a table somewhere of what value resistors to use for different ampacity batteries?

[al_buffington]

Hi,

The resistance does not correlate to the ampacity of the battery. There is only one battery size (capacity) used in the PX 10-80k. The resistance correlates to what vendor's battery is installed inside the module. 

[ojfsdi]

What vendors does/has Schneider use/used?