Smart-UPS RT 6000 RM XL inverter failure, what to do next?

BillP · ‎2021-06-29

Having an issue identifying a failure on our Smart-UPS RT 6000 RM XL. This error has occurred two times in the past two weeks. The basic scenario is we walk into the office in the morning and find all hardware connected to the Smart-UPS RT 6000 RM XL powered off. The UPS does display some error lights on the front interface. Unfortunately I've never been first on site and cannot provide any insight to these lights. Due to the nature of the shutdown no email messages can get sent - the UPS cuts power to all attached equipment including the router/switches. To further complicate the issue my notifications setting intervals were set to 2 minutes for a SMTP failure. The log file was filling with errors so quickly ("I can't contact your SMTP server!" x500 times) I cannot see the genesis of the shutdown. I did however see the event clear in the log. I've since changed the notification intervals and expanded the logs to their maximum capacity so I can have a comprehensive recording of this even should it occur again. I expect it to. Below is the log entry for the event clearing.

06/06/2014 05:39:13 UPS: Passed a self-test.

06/06/2014 05:39:02 UPS: No longer on battery power.

06/06/2014 05:38:58 UPS: On battery power in response to an input power problem.

06/06/2014 05:38:58 UPS: The output power is now turned on.

06/06/2014 05:38:57 UPS: Started a self-test.

06/06/2014 05:38:01 UPS: An inverter fault no longer exists.

The inverter fault clearing seems to be the tell tale error for me. It is my understanding that I can put the battery backup into bypass mode to take the inverter out of the equation (but the batteries won't charge). We have three Smart-UPS RT 192V RM Battery Packs attached to the Smart-UPS RT 6000 RM XL. What are my next steps to troubleshoot? Do I need to get a repair company involved? Two batteries failed and were replaced (sulfur smell, hotter than the surface of the sun) about 12 months ago. Is there any diagnostics I can run to narrow this issue down without experiencing a failure again? I would assume I'm out of warranty with a July 2008 manufacturing date and October 2008 purchase. Thanks.

BillP · ‎2021-06-29

Sorry for the delay, just getting back from vacation.

If they are around six years old, I would just consider replacing them. As I understand that may be easier said than done for multiple reasons, like funding perhaps, I do recommend you at least start budgeting for replacements solely based on the age (which with this technology is required) and yes, do replace them all at the same time. Old batteries can damage new ones if they are all on the same DC bus. As far as a test, a prolonged power outage like you had is a good test to start. A self test just compares the voltage drop while on battery with a load against a table of stored values. A runtime calibration or power outage does this for a longer time and is a good load testing. The other thing you could do is check each module with a voltmeter but that means taking the battery modules apart and being six years old, I wouldn't even waste the time. So, I think a power outage is a good test beyond the self test. I know another user who initiates a runtime calibration for just a few minutes too as a "longer" self test to help predict bad batteries.

On replacing the batteries, there isn't much to it as you've found in relation to the type of UPS you have. I can provide you a little general information on lead acid batteries here -> http://www.apc.com/whitepaper/?wp=30 - there is an ideal storage temperature operating environment which you've likely had since the batteries haven't failed until now (some info here -> Battery Discharge During Storage | FAQs | Schneider Electric US - otherwise, just make sure you replace them all at the same time and yes, update your battery replacement date and never store the UPS with the batteries connected because they'll discharge. This UPS takes qty (2) RBC44 in the UPS and each battery pack does as well.

Let me know if you still have further questions or I overlooked anything. Happy to clarify.

See Answer In Context

BillP · ‎2021-06-29

I think the LEDs you've been unable to catch are going to be the best option here in addition to the event log and data log from the Network Management Card. In my experience recently, I am finding that the UPS reporting is somewhat poor on this type of UPS and multiple issues get reported as an inverter fault. With the event log and the events before/after it and the data log, we might be able to figure out what else it could be based on the other information there, versus just going off of inverter fault. The LEDs will likely report something helpful too. It may be a real inverter fault too, though.

The faults are lost on the UPS itself when it is powercycled too

I think the logs from when and if it happens again are going to be your best bet. I can't tell much yet personally without seeing them or hearing about the LED combination. I would also be wondering if there are other UPSs around in the area that tell you if there was an outage - like is this happening when the UPS has to go to battery power and this UPS fails and cannot? Thus, maybe we could replicate it somehow.

For now, you could try a UPS self test and be warned that it could potentially trigger a problem. I know it's reporting that it passed one though so that might not help. The other thing to do would be to manually put the UPS on battery.

Lastly, when you replace one set of batteries, you should be replacing all of them. I know some folks might not be able to because of cost but adding new batteries to a UPS with existing/older batteries could and likely will cause the older or even faulty batteries to damage the new ones. This could also end up being a battery issue on some level is why I mention it.

Hope this helps. Let me know if you have any further questions.

BillP · ‎2021-06-29

Thanks for the response. Holding my breath waiting for another failure. I'd much rather replace all the batteries than the $6k+ UPS unit. Even if the batteries are $4k I have to replace them in bulk at some point anyways. I'll have the first people on site snap a picture of the LEDs next time and go from there. Hopefully the log file has some useful data next time around. Cringing at the thought though. It usually happens on the weekend.

BillP · ‎2021-06-29

Angela,

Thanks for your guidance thus far. Figured I'd continue in this thread since it has a lot of info in it. We've *knock on wood* been error free for a few weeks now. We did however experience a prolonged power outage a few days ago. I was remoted into the network during the outage and monitoring the %battery / time remaining stats during the outage. We began with around 2:30 of battery time and it appeared closely track that estimate throughout the outage which lasted around 2 hours. I was hopeful power would return or the generator would be turned on (which it finally was after some mechanical issues). I did however notice that once the estimate of battery time remaining got to around 60 minutes it dipped from 60 to 45 to like 11 very quickly, a matter of 2 or 3 minutes. Normally the servers would have shut down due to rules configured in PCNS but surprise! all those settings get erased when you replace your NMC. We had recently replaced with factory installed NMC with the NMC2 for some better SMTP options for notifications. Which brings me to my actual question. How the hell do we replace the batteries / verify their condition? They're around 6 years old and other than two failing and being replaced we've had zero indication that they need replaced.

Can a test be run to verify the health of the batteries / ups as a whole? Outside of a self test (which the unit always passes) what can we run?
If replacement is required where can I find a best practices or some documentation on the steps? All my reading has basically turned up "replace them all at once, change the battery replacement date on the NMC."

Thanks!

BillP · ‎2021-06-29

Sorry for the delay, just getting back from vacation.

If they are around six years old, I would just consider replacing them. As I understand that may be easier said than done for multiple reasons, like funding perhaps, I do recommend you at least start budgeting for replacements solely based on the age (which with this technology is required) and yes, do replace them all at the same time. Old batteries can damage new ones if they are all on the same DC bus. As far as a test, a prolonged power outage like you had is a good test to start. A self test just compares the voltage drop while on battery with a load against a table of stored values. A runtime calibration or power outage does this for a longer time and is a good load testing. The other thing you could do is check each module with a voltmeter but that means taking the battery modules apart and being six years old, I wouldn't even waste the time. So, I think a power outage is a good test beyond the self test. I know another user who initiates a runtime calibration for just a few minutes too as a "longer" self test to help predict bad batteries.

On replacing the batteries, there isn't much to it as you've found in relation to the type of UPS you have. I can provide you a little general information on lead acid batteries here -> http://www.apc.com/whitepaper/?wp=30 - there is an ideal storage temperature operating environment which you've likely had since the batteries haven't failed until now (some info here -> Battery Discharge During Storage | FAQs | Schneider Electric US - otherwise, just make sure you replace them all at the same time and yes, update your battery replacement date and never store the UPS with the batteries connected because they'll discharge. This UPS takes qty (2) RBC44 in the UPS and each battery pack does as well.

Let me know if you still have further questions or I overlooked anything. Happy to clarify.