Symmetra RM 6000 problems

[gcc_apc]

Hello,

I am running several RM 6000 UPS in my data center, and I hope to resolve some problems with failing batteries.  When batteries are are getting near the end of their life, they tend to not hold their charge.  Our UPSs seem to charge them harder at some point, which leads to overheating, swelling, and leaking batteries.  We usually get SNMP alerts to battery overtemp conditions.  But if we don't pull out the failing battery pack soon after the first message, the UPS continues to overcharge it, and within a matter of hours, the batteries become destructive.  This is a problem when the condition occurs on weekends.  Is there a way to configure the UPS to not keep cooking the batteries once it's detected an abnormal battery condition?

We use SNMP to catch messages from the UPS.  We can read an internal temperature from the UPS this way, but we get no information on battery temperature.  Is there a way to get that information?

thanks,

gcc

[gcc_apc]

Hi Rob,

Thanks for your reply and information.

> At what voltage is the UPS charging the battery modules?

I don't know what voltage the batteries get charged at; it's hard to measure with a voltmeter as the terminals are in the back of the module, and when installed, can not be accessed.  The telnet interface shows battery voltage shows 138.7 VDC.  Is there some way to see the charging voltage?

> What is the firmware version of the main intelligence modules of the systems?

Here are the information from the IM:

	IM Status	: On & OK	Master Firmware Rev : 143
	Serial Number	: JD0735013202	Slave Firmware Rev  : 226
	Manufacture Date : 08/31/07	Hardware Rev	: NA

> As lead acid batteries reach end of life, they can sometimes enter thermal runaway as you have seen.  When the UPS detects this, you will receive an internal temperature alert and the UPS will halt charging.

Well, none of my Symmetra RM6000 units are stopping the charge after they've reported battery overtemp conditions.  I'm running 6 of them, and everyone of them have behaved the same way; they report a battery module high temperature condition, and by the time I get to them, the batteries are hot.  If I take more than a few hours to get to them, they actually swell up from the overcooking, making it difficult to remove from the chassis.

The most recent occurrence was found when I went into the computer room to do something else and I smelled burning odors.  I found the UPS had overcooked the batteries to the point that they swelled up to fill the entire inside of the module, and what I was smelling was actually boiled acid.  The module case was too hot to touch; I had to wear gloves to pull it out enough to disconnect it from the chassis.  The inside of the module got pretty eaten up by the leaking acid.  The UPS did not report an overtemp condition that time.  This seems like an error condition.  Is there a way to diagnose this failure?  I'm guessing the battery temp sensor is in the module, and the UPS checks it through the interface.  If the UPS does not detect that condition, it must think everything is OK, and continues to try to charge it.  I can't imagine what would have happened if I had not found it by accident.

So, why are these things continuing to charge the batteries even after they've detected abnormally high temperatures?  And with the last problem, the obvious over temperature condition was not detected or not reported.  I need help fixing these problems.

[gcc_apc]

Hi Rob,

Good to know at least the battery voltage looks normal.

In checking my records, the worst cases where the batteries swelled up and leaked acid were when the overtemps occurred on a weekend, and it took us at least a day to get to them.  When we got to them, the batteries are usually too hot to touch, and most of them have leaked acid through their venting process.  The hot acid actually burned through the nylon insulator jackets surrounding the spade lugs that push onto the battery terminals.  This tells me that the UPS had continued to cook the batteries even after it had detected the overtemp condition.  Is it the MIM that controls charging?  It's obviously not working properly in any of my units.  Is there a firmware fix that prevents this from happening?  How do I upgrade the firmware?

> I don't think you need to fear as historically the UPS has proved extremely safe.

A few years ago, I was still using the step-down transformers to convert the 208VAC output of the UPS to 125VAC to supply my equipment.  One day, a whole rack of equipment shut down, and the room was filled with smoke.  I found the source of the problem in the transformer enclosure.  I found that the 30A (10 gauge) wires coming from the UPS were being fed to the toroidal transformer through the slide-on spade lugs of its power factor correction capacitor.  These spade lugs were friction fit, and through time and use, must have gotten loose or dirty.  One of them developed enough resistance to heat up to the point of melting the lug and the wire, and burning a lot of insulation in the process.  I thought that was a horrible design, to push 30 amps of current through two 1/4 inch friction fit spade lugs on each side.  My fix was to connect the input wires from the UPS directly to the transformer using ring lugs held together with stainless steel fasteners, and tapping off of that with a pair of wires to connect to the capacitor.  I'm now powering my equipment with 208VAC, bypassing the step down transformers.  So maybe the UPS is extremely safe, but its associated equipment started a fire during mission critical operations.

[gcc_apc]

Hi Rob,

I do not have a tech support case for this problem, as I do not have any kind of maintenance contract, and I'm pretty sure all my UPSs are out of warranty.

I checked the logs from the PV front panel, and there was nothing unusual in there.  In fact, this particular unit has been silent for a long time, with the exception of running and passing its weekly self-tests.  I can't seem to see the log through the network interface.  Is that another function lacking in this version?

So this unit is too old to have flash upgradable firmware.  Can you tell me at what firmware level can be flash upgraded by the user?  Is it just a matter of a newer firmware to provide this functionality, or are there other hardware changes?  If the former, is there a firmware ROM I can replace in the MIM to get the latest version?

Here is a picture I took of the transformer that was damaged:

The connector in the foreground was welded to the spade lug it was pushed onto, and the 10G wire that was connected to it was melted off.  The capacitor puked some corrosive fluid out, and most of the inside was covered with soot.  I've cleaned most of it off.  You can see my fix in the upper left with the ring connectors bolted together, and wrapped in 2 layers of heat shrink tubing.  I also had to get a new capacitor to replaced the one that melted.

This unit usually runs between 35% and 70% load:

	Status of UPS: On Line, No Alarms Present
	Last Transfer: Due to software command or UPS's test control
	-----------------------------------------------------------------------
	Input Voltage	: 206.1 VAC  at 60.01 Hz
	Output Voltage	: 211.3 VAC  at 60.04 Hz
	Output Current	:  14.7 Amps
	Output VA	: 03.10 kVA
	Output VA Percentage	:   051 %kVA at n+0 redundancy
	-----------------------------------------------------------------------
	Battery Capacity : 100.0 %  Runtime : 9 minutes

[gcc_apc]

On 12/12/2014 5:43 AM, Rob said:  Thanks for the load information.  At this load level charging should
operate as I described above.  I'm not sure I can tell you more about
what might have happened without the logs.

An event log should be present on all models and versions of network
management cards.  A data log should be present on all network cards
except for AP9606.  Are the logs empty or can you not locate the place
where they would be?  The location of things will vary depending on
network card revision so it is difficult for me to tell you exactly
where they are located.

The only firmware that can be flash upgraded on the Symmetra RM
platform is the display firmware and network management card firmware.
Intelligence module firmware could only be changed by purchasing and
installing a new SYMIM3.

Hi Rob,

Sorry for the long delay.  When I tried to respond to http://forums.apc.com/message/65679#65679, I was told the page no longer existed.  Then I could not log in with my account and password to respond to this thread, and had to reset my password.

But here's my original response to your questions:

I found the log display on the network card (the option shows up on
some of the menus that I've never paid attention to).

Here are the relevant entries:

     12/07/2014  12:38:15  UPS: The load exceeds 100% of rated capacity.
    12/07/2014  12:38:04  UPS: Number of Power Modules increased.
     12/07/2014  12:37:35  UPS: Number of Power Modules decreased.
     12/07/2014  12:13:06  UPS: In bypass for maintenance.
     12/03/2014  09:04:25  UPS: Passed a self-test.
     12/03/2014  09:03:51  UPS: Started a self-test.
     11/27/2014  01:00:28  System: Set Date or Time.
     11/19/2014  02:40:22  UPS: Passed a self-test.
     11/19/2014  02:39:48  UPS: Started a self-test.
     11/13/2014  01:00:28  System: Set Date or Time.
     11/04/2014  20:16:16  UPS: Passed a self-test.
     11/04/2014  20:15:42  UPS: Started a self-test.
     10/30/2014  01:00:25  System: Set Date or Time.
     10/21/2014  15:45:39  UPS: The number of batteries increased.
     10/21/2014  15:44:57  UPS: The number of batteries decreased.
     10/21/2014  15:44:21  UPS: The number of batteries increased.
     10/21/2014  15:42:38  UPS: The number of batteries decreased.
     10/21/2014  15:41:59  UPS: The number of batteries increased.
     10/21/2014  15:41:34  UPS: The number of batteries decreased.
     10/21/2014  13:52:15  UPS: Passed a self-test.
     10/21/2014  13:51:40  UPS: Started a self-test.
     10/16/2014  01:00:25  System: Set Date or Time.

   - Exit, - Refresh, - Next, - Back, - Delete

     10/13/2014  15:21:17  UPS: The number of batteries increased.
     10/13/2014  09:36:20  UPS: The number of batteries increased.
     10/13/2014  09:35:23  UPS: The number of batteries decreased.
     10/13/2014  09:35:07  UPS: The number of batteries increased.
     10/13/2014  09:34:52  UPS: The number of batteries decreased.
     10/13/2014  03:54:08  UPS: The internal battery temperature no longer
                           exceeds the critical threshold.
     10/13/2014  03:52:06  UPS: A faulty battery no longer exists.
     10/13/2014  03:52:05  UPS: A battery fault no longer exists.
     10/13/2014  01:54:13  UPS: Started a self-test.
     10/13/2014  01:54:10  UPS: The internal battery temperature exceeds the
                           critical threshold.
     10/13/2014  01:54:10  UPS: At least one faulty battery exists.
     10/13/2014  01:54:08  UPS: A battery fault exists.

So what's significant are these:

On 10/13/2014, we had a battery overtemp incident, where one of the
modules overheated, swelled up, and puked acid.

On 10/21/2014, I replaced all the batteries with new batteries, as all
the old batteries were the same age, even though only one of the modules
had overheated.

Everything seemed fine until 12/07/2014, when I discovered the burning
smell in the hardware room, and found the same module that had
overheated on 10/13/2014 (and replaced on 10/21/2014) had massive
overheating and swelling.  This was when the module got too hot to
touch, that I just pulled it out enough to disconnect it from the UPS,
and left it to cool off.  Throughout this time, there were NO messages
from the UPS, until I started to remove modules to inspect them.


It looked like the UPS was just continuously charging the batteries,
and not monitoring their state.  As a result, the brand new batteries
in that module got cooked.  I've also installed batteries from the
same lot into the other two modules in the same UPS, and they're doing
fine.  So it seems this battery module has gone bad.  Is there a way
to test it other than to install new batteries, and watch them cook?

Can you tell me how battery charging works on the RM-6000?  I assume
that it must monitor battery voltage and temperature from sensors on
the module to at least decide whether the battery is healthy.  How does
it actually control charging?  Can it stop charging to each individual
module, or are they all charges together?

[BillP]

Hi Eugene,

Rob is on vacation this week so I'll fill in. Both of us are out next week though 

Can you clarify if you replaced the entire APC part number SYBT2 or did you open up the SYBT2 modules and replace the individual (10) 12V batteries inside of it? Just want to clarify since you said module in a few spots but I think you really were referring to the battery bay (where the SYBT2 modules slide into).

The UPS charges all battery bays in parallel and cannot stop/start charging individual modules in individual bays. What you said is accurate - it keeps them at a float voltage by trickle charging. If they are completely discharged, it will overcharge it for a time period (using 147V for no longer than 5 hours) and then switch to trickle charging (at 137V) to get the batteries to 100%. If there is a battery overtemp detected by the UPS in the SYBT2 module, then it would stop charging until the condition clears.

Assuming you replaced the entire SYBT2 module, we'd deduce there was a problem with that individual SYBT2 module (manufacturing defect within a cell, etc).

I hope this helps.

P.S. sorry for the trouble with the URL - we upgraded the forum this past weekend. I will have to see what is going on with the URL mapper as the posts I think were supposed to redirect you.

[BillP · Accepted solution]

Hi. I have the same problem with battеries on symmetra 6000

Do you know the reason оf overcharging?

Please help