SmartUPS10000 Battery problem

[mlewis_apc]

On a Smart UPS 10000 RT unit that has been running just fine, I decided to change all of the batteries. I read that I could pull each tray, one at a time, replacing each with new batteries, without having to turn the system off or do anything else for that matter.

Since doing so, the UPS is now flashing a red light (very bottom) and beeping every 5 seconds or so.

In the UPS status window, I see;

Smart-UPS RT 10000 RM XL
Critical Alarm Battery Disconnected
The battery is not installed properly.

I've checked each of the four packs and each has voltage.
In the Quick Status, it shows 100% Full Battery but that's a mistake.

Battery
Capacity: 100.0 %
Battery Voltage: 216.0 VDC
External Batteries: 1

I am also receiving email notices of a battery charger fault since then.

Code: 0x012A
Critical - UPS: A battery charger fault exists.
Code: 0x012B
Informational - UPS: A battery charger fault no longer exists.

I keeping trying to find answers by searching Google but am not sure what is wrong.
Could someone help me on this.

Thanks very much.

Mike

[BillP]

Are you able to double check the battery connections? That would be the first idea... They can be tricky to fully connect.

[mlewis_apc]

That's what I'm planning to do this afternoon. I don't recall what total voltage I should be seeing from all 4 battery packs but if I knew that, I would have a better clue as to what I should be looking for. Or even, what total power each pack should have. Can't recall but have to assume the batteries are 12V each and I think there were 8 per pack? That's some 96VDC I should be seeing per pack.

[BillP]

Yeah, you should be seeing about 100v per battery pack (about 90v minimum, or else it won't recognize them) if you're measuring with a voltmeter, though the web interface should show about 217v.

[mlewis_apc]

Yes, it shows 216VAC I think I posted originally, can't recall at the moment.

[mlewis_apc]

Ok, make that 214.9 VDC now :).
Either way, I checked them all, two are around 140VDC, two are around the 100VDC mark, if not dead on.

Two of the ones I pulled still show 100VDC so had to guess that was correct.
I replaced the two 140VDC packs with the old ones which are still showing 100VDC but that made no difference.

PS: The beeping is more like everything two seconds, not five. All of the battery LEDs are off.

Message was edited by: mlewis

[BillP]

Two of the RBC44 modules were reading 140v? That doesn't seem right... Though I would also keep a close eye on the fault messages, since you mentioned there was a battery charger fault briefly. Had you seen that message before replacing the batteries? Have you seen it since? Actually, you might want to just paste the event & data logs available via the NMC.

The beeping & LED symptoms are consistent with a battery disconnect issue, but the battery charger fault and measuring 140v on one battery pack are suspicious anomalies.

[mlewis_apc]

Two of the RBC44 modules were reading 140v?

Something around that, not 100VDC as the two others. So now all 4 read around 100VDC.

I would also keep a close eye on the fault messages

There aren't any other messages other than Battery Disconnected and the following.

04/05/2010 15:49:53 UPS: A battery charger fault no longer exists.
04/05/2010 15:49:52 UPS: A battery charger fault exists.
04/05/2010 15:33:06 UPS: A battery charger fault no longer exists.
04/05/2010 15:33:04 UPS: A battery charger fault exists.
04/05/2010 15:19:09 UPS: A battery charger fault no longer exists.

These seem to have stopped since I changed those two battery packs. But the Disconnected error remains.

Had you seen that message before replacing the batteries? Have you seen it since?

Yes, and that's in fact why I changed out the batteries.

Actually, you might want to just paste the event & data logs available via the NMC.

Attached. I don't see anything much in there that gives anything away though.

[mlewis_apc]

This morning I look at the device and the battery power is 214.9VDC but I have new errors in the log.

04/06/2010 05:55:20 UPS: The battery is now installed.
04/06/2010 05:55:03 UPS: The battery is not installed properly.
04/06/2010 05:54:46 UPS: The battery is now installed.
04/06/2010 05:54:28 UPS: The battery is not installed properly.
04/06/2010 05:54:12 UPS: The battery is now installed.

So now it's gone from a charger fault to a battery installed/not installed fault.

What is going on with this crazy thing?

[BillP]

Thanks for the logs. Looks like they were cleared recently so you're right, not a lot in there, except seeing the battery charger faulting/clearing repeatedly. Could you also post the data logs? They're in the same area as the event logs, different tab on the left though. Also, what's the serial number on this unit?

So the battery charger faults have gone away since you replaced the 140v batteries with the 100v ones? Not sure how the batts were 140v - were these APC batteries or third party? The only reason you should have to replace the batteries is from the replace battery LED / error message, not for anything else. If there's a legitimate issue with the battery charger, new batteries won't fix it.

After all you've described, it's starting to sound like there's an issue with the UPS. Have you tried any other troubleshooting steps thus far?

[mlewis_apc]

Could you also post the data logs?

Oops, didn't even notice those, I've attached them here.

Also, what's the serial number on this unit?

Model: Smart-UPS RT 10000 RM XL
Position: RACK
Serial Number: YS0451110605
Firmware Revision: 476.13.M
Manufacture Date: 12/13/04

So the battery charger faults have gone away since you replaced
the 140v batteries with the 100v ones?

Yes, and now I'm getting no errors at all. Looking on the UPS, it is now saying No Alarms Present and 'UPS is online'.

Not sure how the batts were 140v - were these APC batteries or third party?

Half I got in the box from an APC dealer and the others I got a deal on eBay. The eBay batteries are the ones which show 140VDC.

The only reason you should have to replace the batteries is from the replace
battery LED / error message, not for anything else.
If there's a legitimate issue with the battery charger, new batteries won't fix it.

Everything seemed to be ok, even with the charger errors. It's been doing that since day one. I decided to change the batteries thinking maybe they were not holding a charge or something. Mind you, I test the UPS now and then, pulling the power so that it goes to battery until the generator kicks in.

After all you've described, it's starting to sound like there's an issue with the UPS.
Have you tried any other troubleshooting steps thus far?

No, because I'm nervous about doing that. I don't want to accidentally take the load down or worse, do something that would send the load something weird and mess up all of the hardware.

Mike

[BillP]

Heh well now I'm confused. Has the unit been showing battery charger faults since day 1, when you purchased it in 2004 or whenever (based on the manufacture date), or since before you replaced the batteries, or what? If it's been showing the faults for so long, why are you only concerned about them now?

And if the faults have cleared, since it's showing online with no failures, what's the current issue? Is it just that they are short faults (lasting only 1 second) so they don't show up on the overview status page?

[BillP]

Sounds like those battery charger faults are definitely false alarms then. Maybe it had something fishy to begin with and that's why the previous owners wanted to get rid of it. 😛

There aren't any circuit boards in the batteries and I'm not sure how they would ever have voltage that high. I could understand if they were the ebay batteries, since who knows what people do to refurb batteries, but the APC ones certainly shouldn't be. Do the batteries list a model and serial number on them anywhere?

[mlewis_apc]

Heh well now I'm confused.

Sorry, I was in a rush and wasn't detailed.

Has the unit been showing battery charger faults since day 1, when
you purchased it in 2004 or whenever (based on the manufacture date),

Since I purchased it used so about one or two years now, lost track.

If it's been showing the faults for so long, why are you only concerned about
them now?

It's been on and off. I'll get charger notices, then they go away even when the load hasn't changed at all.

And if the faults have cleared, since it's showing online with no failures
what's the current issue? Is it just that they are short faults (lasting only
1 second) so they don't show up on the overview status page?

Since changing the two battery packs, after a couple of hours or so, it went into normal mode.

Load
Load Power: 49.0 %Watts
Apparent Load Power: 41.0 %VA
Load Current: 19.71 Amps
Battery
Capacity: 100.0 %
Battery Voltage: 216.0 VDC
External Batteries: 1

Another weird thing is that the two packs which are showing 140+VDC aren't the eBay ones, it is the APC ones. That makes me wonder if the circuit cards which connect between two batteries in each pack might be messed up?

Mike

[mlewis_apc]

APC RBC44 Replacement Battery Pack 12V 5ah batteries
From Gruber Power Services

The circuit card I am talking about is in 6 of the battery packs. I pulled two pack out, replaced those and replaced the batteries in the two other packs. It is a small board that goes between one set of batteries instead of like the others where it's just a direct cable.

[BillP]

Any serials? Did you buy new batteries to replace the 140v ones or did you just replace the internal batteries on those as well?

[mlewis_apc]

Any serials?

Serials, for?

Did you buy new batteries to replace the 140v ones or did you
just replace the internal batteries on those as well?

I was going to buy all 32 batteries in one shot but came across someone selling replacement packs for a good price, but he only had two. The other 16 batteries came in boxes from Gruber. Those are the ones which have the weird voltage and they are blue.

[mlewis_apc]

Too weird. Still getting notices, same as before I changed the batteries.

UPS: Started a self-test.
UPS: Passed a self-test.
Refused a self-test; UPS is overloaded.
UPS: A battery charger fault exists.
UPS: A battery charger fault no longer exists.
UPS: Passed a self-test.

Now, perhaps this result is since the last test, when there was a fault at the time I was having the problems a week ago or so?

[BillP]

Serials to see how old the batteries were. That whole procedure is still kind of throwing me off. If you had 16 separate batteries from Gruber, I'm assuming you bought the battery cartridges rather than the entire modules. So those cartridges must not have the right voltage for these packs. APC recommends replacing the whole battery module rather than the internal batteries, especially since replacement cartridges (which APC doesn't sell) may not be the right ones.

Now, perhaps this result is since the last test, when there was a fault at the time I was having the problems a week ago or so?

I'm not sure, there aren't any timestamps - they should show in the log though.

[mlewis_apc]

Serials to see how old the batteries were.

Half are installed now, I'd have to pull them out.

That whole procedure is still kind of throwing me off.
If you had 16 separate batteries from Gruber, I'm assuming you
bought the battery cartridges rather than the entire modules.

Sorry, I thought I posted that. I bought 16 batteries from Gruber, they arrived in a box. The others, I got with the cartridges, already installed. I just pulled the old ones and put the new ones in.

So those cartridges must not have the right voltage for these packs.

Not sure where the problem is. I only know that I pulled all 4 packs, replaced two packs, and replaced the batteries with gruber batteries on two others.
The voltage was off on the two packs with the Gruber batt's in them. I replaced the two packs with the originals and the system went back to normal.
However, it was still throwing errors now and then but always, the last errors were that it was back to normal.

Message was edited by: mlewis

[mlewis_apc · Accepted solution]

So the problem has resurfaced, when I needed it most of course. We had a power failure over the weekend and things went ugly.
I left it as we last talked because while there seemed to be an error, it seemed to be self correcting, perhaps only a logging problem. But sure enough, it went down after all.

Back at square one again, needing to figure out why the battery packs are different and seem to be problematic.

07/10/2010 04:54:45 UPS: The output power is now turned on.
07/10/2010 04:54:41 UPS: The output power is turned off due to a low battery.
07/10/2010 03:40:00 UPS: A discharged battery condition no longer exists.
07/10/2010 03:39:57 UPS: The output power is now turned on.
07/10/2010 03:39:57 UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately.

Message was edited by: mlewis