Smart UPS 3000 RMXL - Inverter Fault error

[Anonymous user]

Hello,

Just wanted to get some advice here regarding one of my Smart UPS 3000 RMXL units.

Last weekend, I started receiving email notifications (good, notifications are working) that UPS1 is in bypass mode, refusing to perform a self-test because the UPS is overloaded (not true, less than 50% load)

Refused a self-test; UPS is overloaded.

An inverter fault exists.

In bypass in response to an internal hardware fault.

I confirmed the unit was still connected to A/C power and left it alone. However, on Memorial Day, I received an email saying that a graceful shutdown process is being used... (good that PCNS is working...but not what I wanted...the servers are all plugged into UPS2, not UPS1)

I came into the office and discovered that UPS1 had completely drained its batteries, despite being plugged into a working A/C power source. A slight burning smell was detected and UPS1 exhaust air was a little warm. I switched all devices to a power strip that is plugged directly into A/C power; I then shut down UPS1 and unplugged it from A/C power . I booted up my servers (plugged into the working UPS2) and brought the network back up, then went home.

The next day, I came into the office and turned UPS1 back on with no load attached to it. The battery LED was flashing, indicating the battery was very low. Within 10 minutes, I detected the burning smell again and the warm exhaust temperature. I went to do some research online and came back to find the UPS had turned itself off. I unplugged UPS1 from A/C power just to be safe.

I called APC support, and Andi suggested a few troubleshooting steps. I turned UPS1 back on and started noticing the smell again, I looked inside of the exhaust vents and saw orange colored glowing coming from inside the unit (i believe it was the heat sinks). Since Andi wanted to see if the battery would still charge, I left it plugged in. I checked up on it again a bit later and discovered that UPS1 had turned itself off again. This time, I left it plugged into A/C power (the glowing and burning smell did not occur when the UPS was turned off)

At the end of the day, I turned UPS1 back on and discovered that the battery was now fully charged. The burning smell and glowing was not occurring this time, so I felt comfortable to leave UPS1 powered on and plugged into A/C power.

Andi asked me to discharge the batteries and keep an eye on things, so I connected a desktop computer to UPS1 and ran it off batteries overnight (unplugged the A/C), at 8AM this morning, I received notifications that the battery was low (I excluded UPS1 from PCNS this time) and that the UPS would turn off when the battery was too low. UPS1 is now plugged in to A/C power and charging the batteries, with no burnt smell or glowing heatsinks. The UPS appears to be working fine/normally.

Based on this chain of events, I have a few questions:

Is it safe to assume this UPS is actually ok now? (Battery seems to be discharging and recharging ok, no more inverter fault error messages, no more burnt smell or orange glowing heatsinks, etc.)

Was it normal for the heatsink to be glowing orange inside the UPS unit?

Was it normal to smell a slight burning smell when the battery was low and plugged into A/C power?

Why did the UPS unit shut itself off? overheating temperature? (the data log did not show high temperatures, despite the glowing orange heatsinks)

Should I request an RMA anyways and get it replaced under warranty?

Is it normal for these UPS units to emit a buzzing sound when running normally? (both UPS1 and UPS2 have been doing this ever since they were installed, even before UPS1 started throwing errors)

I sent Andi my initial logs already. I can re-attach or email them again.

Model:	Smart-UPS RT 3000 RM XL
Firmware Revision:	805.3.D

Hardware Factory

Model Number:	AP9631
Hardware Revision:	05
Manufacture Date:	02/13/2013

Application Module

Name:	sumx
Version:	v6.1.1
Date:	Mar 7 2014
Time:	12:05:01

APC OS (AOS)

Name:	aos
Version:	v6.1.1
Date:	Mar 7 2014
Time:	11:52:49

APC Boot Monitor

Name:	bootmon
Version:	v1.0.2
Date:	Jan 21 2010
Time:	13:35:57

Thanks!

Added firmware version and NMC specs

[Anonymous user]

Hello Angela, thank you for looking into this. It was your frequent and knowledgeable replies to other customer posts that prompted me to post on the forums here.

It appears the data logs have been overwritten due to it being a week since the initial event. However, the event logs still show the correct sequence of events.

In regards to the graceful shutdown initiated from Serial Port 0, I assume that is the default behavior of the NMC/PCNS? There is nothing connected to the UPS unit other than the built-in A/C cord and the network cable going to the NMC. I did use the special serial cable to program the NMC initially, but that was only connected at the time of initial setup and then disconnected afterwards.

As for the LED indicator it was the opposite, more like:

0

0

0

0

1 - Blinking

The blinking LED eventually turned solid, and is now the battery currently shows that it is fully charged:

1

1

1

1

1

During the discharge and recharge cycles, I did notice some inconsistencies in the amount of battery capacity remaining...like percentage "drops" between each 10-minute reporting period. You won't be able to see it in the newer data logs, but the older data logs sent to Andi, should show what I'm talking about...this could be in-line with a partly failed battery? An example of what I'm talking about is the initial event on 5/24, you will see the battery capacity drops from 100% to 59%, then back up to 64% within 30 minutes.

05/24/2014	21:02:42	118.8	119.5	118.8	5.44	29.0	59.96	64.0	205.2	21.8	21.0
05/24/2014	20:52:42	118.8	118.8	118.8	5.37	29.0	60.00	59.0	204.1	22.4	21.0
05/24/2014	20:42:42	118.0	118.8	118.8	5.44	29.0	60.02	59.0	204.1	22.9	21.0
05/24/2014	20:32:42	118.8	119.5	119.5	5.37	29.0	60.01	100.0	218.1	22.4	21.0

Unfortunately, I did not capture the data logs from the recharging cycle, but I can discharge the battery again and recharge to re-capture that if necessary.

As it stands right now, the UPS unit is plugged into A/C with only one desktop test machine plugged into its output power. Status remains normal and no indications that the UPS is having any problems. However, I do not know what will happen once I connect my networking equipment back onto the UPS unit and there is some load on the UPS output. (i'm a bit scared to try, but know this will be the only way to fully test the unit)

Thanks for looking into this.

[BillP]

Hi again!

Thank you for the feedback

I discussed with my Smart-UPS counterpart here, since my forte is the NMCs and I suspected we'd have to replace it, and he said that the glowing heatsink is concerning and that we should go ahead and replace it so we're not wondering if something is going to happen again. Plus, it does sound like there is something wrong possibly based on all you've shared with us so far.

What I'll do in the case number that I found (10906967) is put a link to this thread and put some notes to OK a replacement. Do you have Andi's contact information where you can reference the case number and ask that she (?) process the replacement with approval from me? If there is any issue with that, let me know and I'll ask someone else to work with you to get it replaced.

Also, don't forget to swap your NMC over to the new UPS and let me know if those serial port 0 messages keep popping up.

[Anonymous user · Accepted solution]

Hello,

Just wanted to get some advice here regarding one of my Smart UPS 3000 RMXL units.

Last weekend, I started receiving email notifications (good, notifications are working) that UPS1 is in bypass mode, refusing to perform a self-test because the UPS is overloaded (not true, less than 50% load)

Refused a self-test; UPS is overloaded.

An inverter fault exists.

In bypass in response to an internal hardware fault.

I confirmed the unit was still connected to A/C power and left it alone. However, on Memorial Day, I received an email saying that a graceful shutdown process is being used... (good that PCNS is working...but not what I wanted...the servers are all plugged into UPS2, not UPS1)

I came into the office and discovered that UPS1 had completely drained its batteries, despite being plugged into a working A/C power source. A slight burning smell was detected and UPS1 exhaust air was a little warm. I switched all devices to a power strip that is plugged directly into A/C power; I then shut down UPS1 and unplugged it from A/C power . I booted up my servers (plugged into the working UPS2) and brought the network back up, then went home.

The next day, I came into the office and turned UPS1 back on with no load attached to it. The battery LED was flashing, indicating the battery was very low. Within 10 minutes, I detected the burning smell again and the warm exhaust temperature. I went to do some research online and came back to find the UPS had turned itself off. I unplugged UPS1 from A/C power just to be safe.

I called APC support, and Andi suggested a few troubleshooting steps. I turned UPS1 back on and started noticing the smell again, I looked inside of the exhaust vents and saw orange colored glowing coming from inside the unit (i believe it was the heat sinks). Since Andi wanted to see if the battery would still charge, I left it plugged in. I checked up on it again a bit later and discovered that UPS1 had turned itself off again. This time, I left it plugged into A/C power (the glowing and burning smell did not occur when the UPS was turned off)

At the end of the day, I turned UPS1 back on and discovered that the battery was now fully charged. The burning smell and glowing was not occurring this time, so I felt comfortable to leave UPS1 powered on and plugged into A/C power.

Andi asked me to discharge the batteries and keep an eye on things, so I connected a desktop computer to UPS1 and ran it off batteries overnight (unplugged the A/C), at 8AM this morning, I received notifications that the battery was low (I excluded UPS1 from PCNS this time) and that the UPS would turn off when the battery was too low. UPS1 is now plugged in to A/C power and charging the batteries, with no burnt smell or glowing heatsinks. The UPS appears to be working fine/normally.

Based on this chain of events, I have a few questions:

Is it safe to assume this UPS is actually ok now? (Battery seems to be discharging and recharging ok, no more inverter fault error messages, no more burnt smell or orange glowing heatsinks, etc.)

Was it normal for the heatsink to be glowing orange inside the UPS unit?

Was it normal to smell a slight burning smell when the battery was low and plugged into A/C power?

Why did the UPS unit shut itself off? overheating temperature? (the data log did not show high temperatures, despite the glowing orange heatsinks)

Should I request an RMA anyways and get it replaced under warranty?

Is it normal for these UPS units to emit a buzzing sound when running normally? (both UPS1 and UPS2 have been doing this ever since they were installed, even before UPS1 started throwing errors)

I sent Andi my initial logs already. I can re-attach or email them again.

Model:	Smart-UPS RT 3000 RM XL
Firmware Revision:	805.3.D

Hardware Factory

Model Number:	AP9631
Hardware Revision:	05
Manufacture Date:	02/13/2013

Application Module

Name:	sumx
Version:	v6.1.1
Date:	Mar 7 2014
Time:	12:05:01

APC OS (AOS)

Name:	aos
Version:	v6.1.1
Date:	Mar 7 2014
Time:	11:52:49

APC Boot Monitor

Name:	bootmon
Version:	v1.0.2
Date:	Jan 21 2010
Time:	13:35:57

Thanks!

Added firmware version and NMC specs