Smart-UPS 3000 RM XL Not Using Batteries

[Anonymous user]

Hi everyone,

We have an older 3000 RM XL, manufacture date of 2004. For at least the past 6 months, it hasn't been flipping to battery when we have a power issue. Our building is rock solid with power and has been for a long time, so we've never really noticed this until recently. There's only a few things plugged into this device so I can't imagine it's over loaded. I don't spend a ton of time logged into it's network interface, it's a bit out of sight and out of mind.

We've had some construction in our office floor and the workers have accidentally messed with the power a couple of times in the past few months. Every time they do, even if it's just for a moment, bammo, the whole APC goes down, just like it was a dumb surge protector and you flicked the power switch. The device then beeps and requires us to manually power it back on. This last happened yesterday morning, again related to power issues with a construction crew on our floor. All the other APCs in our racks coped with the power issues properly.

When we lose power, we don't see anything in it's log files. Just "System: Coldstart." It doesn't report that it transferred to battery power.

We have an external battery pack attached to the device which shows up properly in the APC Network Management Card. This external battery pack was installed in 2008.

Oddly, looking at the network interface website now while writing this, it's showing a "runtime of 1 hour 38 minutes", which doesn't make sense as the APC has been running for at least a day.

I am seeing this on the Network Management card interface:

Battery Capacity:	100.0	%
Battery Voltage:	55.05	VDC
Number of  External  Batteries:	001
Self-Test Result:	Passed
Self-Test Date:	05/09/2012
Calibration Result:	Unknown
Calibration Date:	Unknown

Here's the serial number and firmware:

Model:	Smart-UPS 3000 RM XL
Firmware Revision:	93.14.D
Manufacture Date:	08/06/04
Serial Number:	QS0432222781

Any ideas on what I could try or things to do in order to bring this device back up to speed?

[BillP]

Hello

The runtime of 1 hour 38 minutes is your estimated battery back up time - not the uptime which is what I think you thought it was.

Anyway, have you had the chance to pull the plug or test it yourself at all by simulating a power failure to check LEDs on the front panel? It is helpful to know if during an outage if just the load turns off or if the UPS itself also turns off completely.

Also, on your network management card, can you go to the Help->About system menu and let me know what you see there? I'd like to know what firmware its on (your screenshot is the UPSs firmware - not network management card's). I can tell its older by your screenshot though.

Anyway, I am thinking we should just get the event.txt and data.txt logs via FTP either way from the management card. I know you didnt see anything but we might be able to see something that was overlooked.

Lastly, have you visually verified that the stuff turning off is definitely connected to the UPS and someone didnt plug it into a wall outlet or something and didnt move it back? It'd be nice to see the load percentage reported by the management card (which will also be in data.txt if you can get it and also depending on network management card firmware).

Thanks!

[Anonymous user]

Plochner:  You say the UPS is from 2004 and you added an external battery in 2008. If the original battery module (inside the ups) was never replaced prior to adding the external pack, there is the possibility you've got an unstable mix of newer (5 yrs old) and very old (9 yrs ) batteries...?  Ideally all the batteries should be the identical age. If there is indeed a mix of new and old the UPS could crash during an actual power outage due to a sudden dip in the DC bus voltage caused by the weak batteries acting as a load instead of an energy source.

[Anonymous user]

Thanks for the reply. Yes, I have visually verified that the systems are plugged into the APC itself. I have not simulated a power failure in a long time on this device, only that I do know that any tiny blip of power issues causes the entire APC to turn off and start beeping, as that has happened several times in the past few months due to the construction and whatnot. Basically, it behaves like there's no battery installed at all. I have to push the power button on the front of the APC to turn it back on when this happens. There are no LEDs lit up when it happens.

I have not done a yank-the-power test on the device yet to see how it behaves, but I have suspicion that it'll behave the same as it does during those power blips (ie shut down completely). All of the other newer APCs in the racks handle these few-and-far-between power blips without a hitch.

The system is reporting as you said approx. 90 minutes of runtime at current load. None of our power issues in the past couple of months have lasted longer than 90 minutes, more like 60 seconds.

I'll see if I can FTP into the card to grab those text files. FTP was disabled on the device, I assume that means just the access interface, and data.txt is still being written somewhere to the APC?

Here's the results of the About menu:

Hardware factory information
Model Number:	AP9617
Serial Number:	ZA0436005671
Hardware Revision:	A10
Manufacture Date:	09/02/2004
MAC Address:	00 C0 B7 70 2C 9B
Application module information
Name:	sumx
Version:	v2.5.1
Date:	02/26/2004
Time:	11:51:05
APC OS(AOS) information
Name:	aos
Version:	v2.5.0
Date:	02/24/2004
Time:	19:36:02

[Anonymous user]

Unfortunately neither events.txt or data.txt provide any smoking guns as to the problem. They are the same as what I see in the web interface. From yesterday's power blip:

06/05/2013    07:58:51    UPS: Turned on.    0x0113

06/05/2013    07:35:59    UPS: Communications established.    0x0101

06/05/2013    07:35:50    System: Network service started. System IP is 172.24.0.5 from manually configured settings.    0x0007

06/05/2013    07:35:43    System: Coldstart.    0x0001

06/05/2013    07:21:47    System: Set Date or Time.    0x0021

06/05/2013    05:20:47    System: Set Date or Time.    0x0021

06/05/2013    08:25:34    117.6    117.6    117.6    40.5    60.00    100.0    55.65    101.6  

06/05/2013    08:15:34    116.3    117.6    117.6    41.0    60.00    100.0    55.65    99.1  

06/05/2013    08:05:34    115.7    118.9    116.3    42.6    60.00    100.0    55.05    95.1  

06/05/2013    07:55:34    118.9    119.6    0.0    0.0    60.00    100.0    55.65    101.6  

06/05/2013    07:45:34    118.9    119.6    0.0    0.0    60.00    100.0    55.65    100.0  

06/05/2013    07:27:26    117.0    118.3    117.6    43.6    60.00    100.0    55.05    95.1  

06/05/2013    07:17:26    117.6    118.3    117.6    42.6    60.00    100.0    55.65    103.2  

06/05/2013    07:07:26    116.3    118.3    117.6    42.6    60.00    100.0    55.05    103.2  

Note that above, it was set to Set Date or Time from NTP every hour, which is a bit overkill and was flooding the logs with that. I have since asked it to check it's time against NTP every 48 hours.

[Anonymous user]

Re: Smart-UPS 3000 RM XL Not Using Batteries              

Plochner:  You say the UPS is from 2004 and you added an external battery in 2008. If the original battery module (inside the ups) was never replaced prior to adding the external pack, there is the possibility you've got an unstable mix of newer (5 yrs old) and very old (9 yrs ) batteries...?  Ideally all the batteries should be the identical age. If there is indeed a mix of new and old the UPS could crash during an actual power outage due to a sudden dip in the DC bus voltage caused by the weak batteries acting as a load instead of an energy source.

The event / data you post is showing that the ups crashes - the Coldstart at 07:35 followed by UPS turned on means the management card lost power momentarily (meaning the UPS crashed/shut down internally) then the ups turned itself on when power was restored or the blip was over.

[BillP]

OK a few things.

1.) So when the UPS is turned off, you said its beeping - please describe the beeping. We have beeps and chirping. No LEDs on it when you walk over during the beeping either? I am wondering if what you hear is the battery disconnected chirp which would make sense when we have old batteries that the UPS cannot see until it tries to transfer.

2.) What is the last column in the data log?

3.) I don't think its going to "fix" the problem but if it happens again while we are working on this, we'll get better logging. The firmware can be updated which will change the interface. I recommend doing it just because your card is so behind and working with the card will be a lot easier. I know it will make it easier for me to help you as well .. We are at AOS 3.7.3/ sumx 3.7.2 for that card. I'll attach the firmware to this post and in our knowledge base, http://www.apc.com/site/support/index.cfm/faq/ article ID FA156047 for instructions on updating. This works via FTP as well so you'll need that on to do an upgrade unless you can do it via SCP.

[Anonymous user]

Thanks again. Now that you mention the difference between a "chirp" and a "beep", I can't quite remember. Usually I'm rushing into there to power it back on. I do know it's a sequence of tones, ie beep-beep-beep, followed by some silence, then repeating the sequence. Unlike say my desktop APC UPS'es which, when they have battery issues, deliver me a solid "beeeep" tone.

When I power it on I am pretty sure I have no LEDs lit up, of any color, on the front of the APC.

The legend for the data log file is:

Date    Time    Vmin    Vmax    Vout    %Wout    Freq    %Cap    Vbat    TupsF

Looks like the last entry in the data log file is thus for Internet Temperature in deg F.

Will the firmware upgrade cause an outage on the APC?

[BillP]

Ok thanks for the info.

That temperature on the UPS is very high!! We should also check the internal battery pack on the UPS to ensure batteries are not swelling or even leaking because of that heat, especially if they are from 2004. I suggest you do that immediately. Doing that (disconnecting the battery) would not cause a power disruption normally.

Also, upgrading firmware on the network management card will NOT disrupt output power on the UPS either.

[Anonymous user]

I'm used to seeing things in celcius but 102 / 103 F is high. That and the fact that Plochner reports there are "no front panel LED's on of any kind/colour" when he powers it up makes me believe this ups has suffered an internal hardware failure.... the high temperatures can mean failed internal fans ( I have replaced many dead fans in ups newer than that) and failed fans could mean cooked circuitry.

[Anonymous user]

Might have mistaken his comment about LEDs ...probably meant that when it crashes and at the time he goes to power it back up that's the time that every LED is off - IE prior to him hitting the 1/0 button.

[Anonymous user]

I mean to say when the device is powered off it has no LEDs lit up on it. Currently, with the device powered on and everything running fine, it has a series of green LEDs all lit up. The battery LED is showing fully lit up with green, the LED "bar" on the left is showing halfway lit up (I assume this is the load bar), and the middle LED to the left of the power switch is showing green.

When the device is powered off during the above described incidents, and I enter the NOC to turn the UPS back on, nothing is lit up on the UPS, but I do hear beeping (or maybe it's chirping). I just wanted to verify that when the device is in it's powered off state, I see no LEDs, red, green, blinking, flashing, anything. Just powered entirely off.

[Anonymous user]

I have upgraded the firmware successfully on the device. The web interface now looks similar to all the other APCs in my NOC.

I am not seeing anything alarming in any of the tabs, although from the time I started typing this message the temperature has gone from 95deg to 100deg. Seems to be hovering at about 100deg F right now.

Status
Last Battery Transfer:	No transfers have occurred
Internal Temperature:	95.9°F
Runtime Remaining:	1 hour 41 minutes
Power
Input Voltage:	117.0 VAC
Output Voltage:	117.0 VAC
Frequency:	60.0 Hz
Load
Load Power:	34.3 %Watts
Battery
Capacity:	100.0 %
Battery Voltage:	55.05 VDC
External Batteries:	1

[Anonymous user]

A few minutes ago, I used the web interface and initiated a Self Test and had the power failure issue. Ran the self test with the web interface and the device shut itself down immediately, all power lights off, absolutely no battery was used. Powered it back on as usual. I suspect the core unit's batteries are faulty, especially with the internal heat increase.

Is it possible to unplug the core unit's batteries and only use the external battery pack? Meaning, remove the bezel on the front of the core unit and unplug it from there? I don't have the maintenance window to physically remove the core unit and replace it for another couple of weeks, so I'm hoping to just unhook it until then.

If I can unhook the core unit and only use external batteries, I assume I need to shut down the APC first, which will require a brief outage. Any concerns about pulling the physical battery plug on that device considering it probably hasn't been pulled for nearly 10 years?

Thanks again for all the help.

[BillP · Accepted solution]

Yes, I think you definitely need to disconnect the internal batteries on the unit or at least see what condition they are in before doing anything else. If they are leaking, you may need some baking soda to neutralize any acid but hopefully they are not doing that yet. Based on that high temperature and age, it is likely that they are potentially swollen or leaking though.

I asked my Smart UPS counterpart and are are not sure if just the XL battery pack will work (it sounds like maybe not based on your current situation) because internal battery pack may be required to complete the battery bus circuit. You should test it but I tend to think since the internal batteries sound already as good as dead, it may not work since XL batt pack is connected - but those batteries are several years old now and may not be doing well either.

I don't have any concerns on pulling the internal battery connector specifically - more so just the condition of the batteries and them being swollen. If they are and won't come out, let them cool (which hopefully they will when not connected to UPS/charging any longer) and maybe if you have a fan around to help expedite the cooling it would help.

[Anonymous user]

Thanks for all your help. We're going to just replace both this APC (and the external battery pack) with a new Smart-UPS RM SMT3000RM2U. Amazon has them for $1200ish, sadly without the management card though so that'll be another $200ish. With Prime shipping we should get them early next week and we'll just have to chance it with our dying APC for another 4 or 5 days.

Any chance the existing management card (with it's now new firmware) will work on the new APC? I know, it seems weird putting 2004 network hardware in a 2013 device but if i can save $250, it may be worth it.

[Anonymous user]

The AP9617 card in your old UPS will not work in the SMX/SMT series....there are actually tabs on the UPS chassis smartslot which prevent the older cards from even being able to be installed. You could get an SMX3000RMLV2UNC model which includes the network card so you'll save a few bucks with the bundle. Good luck with your replacement UPS!