SUM1500RMXL2U constantly going into self test

[Anonymous user]

Hi folks,

I recently replaced the batteries on my SUM1500RMXL2U and have had a series of issues since.

After allowing the new batteries to charge for a full 24 hours, I started the calibration procedure. I initiated the calibration via the web interface on the AP9619 management card and was watching the battery slowly discharging via the web interface. After a few hours of calibration I noticed that the AP9619 had lost the current time and defaulted to a date in 2000 (this had occurred during the calibration). I attempted to force a NTP time sync, and then a manual time but the updated time didn't stick. After the calibration finished, I shut down and restarted the UPS. The UPS came back up, but the AP9619 disappeared off the network. I inspected the status LED on the mangement card and it was blinking suggesting that the network wasn't configured. I logged in to the UPS via the serial console, and indeed the mangement card had lost it's network settings. I attempted to re-apply them and restart, but the settings would not take.

I had to leave the UPS for a few days to do other off-site activities, but was informed by site personnel two days later that the UPS was "beeping and clicking", but I was unable to do remote diagnostics due to the AP9619 being off the network. I returned to the site the following day and found that the UPS front panel was dark and appeared to be in bypass mode, but was periodically emitting a faint beeping sound corresponding with the LEDs on the front panel illuminating for a fraction of a second. I attempted to log in via the serial console, but could not do so. I then measured the voltage from the battery pack and found them to be a staggeringly low 24V. The periodic beeping and front panel LED blinking seemed to correspond with the UPS attempting (and then failing) to start. I removed and then charged each of the battery packs up to 54.4V with a bench power supply (that was current limited to 750mA), confirmed each of the battery cells inside the pack were within the expected voltage range (13.4V-13.7V) and re-installed the battery pack. I powered on the UPS from the front panel and for about 5 minutes everything looked normal. I then began to hear an intermittent soft beeping sound (such as you would hear when pressing buttons on the front of the UPS), and then the UPS switched to be on battery. This cycle repeats every few seconds. I logged in to the serial console and saw that the UPS was switching to self test corresponding with the on battery LED on the front panel. The cycle kept repeating over and over, and gradually drained the batteries ~20% percent before I intervened. I then attempted to shut the UPS down from the front panel, and the UPS turned off, but then immediately turned on again seemingly on it's own. I tried this several times. I ultimately had to disconnect the batteries to power the UPS down fully.

Thinking that this was related to the earlier issue with the mangement card, I removed the AP9619, inspected it and did not see any obvious issues. I then removed the CR2032 memory backup battery and checked the voltage (it looked ok at 3.23V). I also cleaned the contacts inside the battery holder which looked a little oxidised. I reinstalled the battery, reinstalled the AP9619, reconnected the batteries and powered the UPS back up. I again logged in via the the serial console and set the network parameters. This time the settings took, and I was able to log in via the web interface and set the remaining properties (including time). Unfortunately, after approximately five minutes, the soft beeping started, and the management card logs showed that it was going in to self test over and over:

05/02/2016	13:46:18	UPS: Passed a self-test.
05/02/2016	13:46:02	UPS: Started a self-test.
05/02/2016	13:45:54	UPS: Passed a self-test.

I issued a shutdown via the web interface of the management card, which normally works, but this time it just power cycled the UPS just like when I pressed the power off button on the front panel the previous time. I again disconnected the batteries so as not to cause further damage.

I have all but exhausted any attempt at further troubleshooting and would appreciate any suggestions you might have.

[BillP]

Hi Brian,

Thanks for the background. FYI, the coin cell on the AP9619 NMC itself is only used to keep date and time when the NMC is without power. But, I am not 100% sure if that could contribute to the problem you described with not being able to set date/time properly..maybe.

Anyway, the first thing I think of when I see random/repeated self tests is a dangling USB or serial cable connected to the rear of the unit. Or, if it is connected something like a PC/laptop, that it is somehow getting erroneous information which is triggering the UPS to do something. It could also be a faulty cable either way.

Can you verify what is connected to the rear of the UPS and remove any USB and/or serial cables connected? The serial port on this UPS is essentially like a dry contact so shorting a few of the pins could be causing this. If there is no cable attached, just make sure nothing is stuck in that port too to be sure.

I also would be interested in the full event log to see if there are any other clues in there. This knowledge base (http://www.schneider-electric.us/en/faqs/FA156131) has details on how to download the log files. Event.txt and data.txt will be most helpful for something like this.

[Anonymous user]

Thanks Angela,

That seemed to do the trick to resolve the constant self test issue. I had the serial cable (940-1524D) plugged in to a USB to RS-232 converter that wasn't attached to anything and after removing it from the UPS, the problem has not reoccurred.

Unfortunately a new issue has cropped up that I cannot readily explain. I reconnected all the battery packs (one in the main unit and two in a SUM48RMXLBP2U) and expected the battery voltage to equalize (they were at slightly different discharge levels due to earlier testing) and start charging. However, the voltage started progressively dropping as though the UPS was not on line power (although the status LEDs and management card web interface indicated that the UPS was not on battery). It took almost a full day for the batteries to discharge (which would have been a matter of hours if the UPS was truly on battery). I currently have the UPS offline and am currently bench charging all the battery packs together so there is no potential issue with equalization. I can provide the event.txt and data.txt logs from the management card once the batteries are charged. In the mean time, I have attached some graphs from the NMS showing the UPS running during that period of time.

[BillP]

Hi Brian,

You're welcome. We'd need to know the exact state of the UPS. I know that usually bad batteries and good batteries mixed would cause a problem. But, I was also wondering if the UPS was in its bypass mode. This model has a bypass switch on the power module and I believe it also has an auto-bypass (I'd have to double check that auto-bypass part). If it was in this mode - either by the switch or if the auto-bypass mode exists, I have a feeling it wouldn't charge.

Did the web interface say the UPS was Online (you noted it said not on battery) but it would've said bypass though you probably would've noticed that as part of what you explained.

[Anonymous user]

Thanks Angela,

The front panel LED did show the UPS was online and the bypass LED was not illuminated. I also didn't see anything in the logs suggesting it was going in to bypass mode. The only alert was due to the battery being in a discharged state. I suspected that the low runtime remaining value was due to the fact that calibration procedure needed to be performed again since everything was reset when I was troubleshooting the issue with the management card as no previous calibration date is shown.

[Anonymous user]

After charging all three battery packs (one main and two in an external SUM48RMXLBP2U) up to 54.8V with a bench supply, I tried the test again and unfortunately had the same results (the battery packs gradually discharged when the UPS on line power). I was able to do two short load tests by disconnecting line power (one for 10 minutes and one for 20 minutes) this time and the UPS immediately switched over to battery and seemed to be working properly. However, upon re-applying line power, the battery voltage/estimated charge slowly climbed up a little, but then eventually started going down again.

One item of note that I had not mentioned before: The fans in the power unit seem to be running constantly and I don't recall this happening before the recent issues. After I do a UPS shutdown and remove line power the louder fan stops, but a quieter fan still seems to be running and only ceases once the batteries are all disconnected. 

I have attached the data.txt and event.txt as requested.

[Anonymous user]

Any thoughts on this?

[BillP]

Hi Brian,

Sorry for the delay. I am not confident in what the problem is unfortunately. It seems like the UPS is just not charging or is not capable of doing so. I do not see any faults reported in the event log to back that possibility. And the data log just seems to be reflecting what you already identified - the battery voltage and capacity going down all of the time.

You can try one thing which I don't think will work but it is free. I think between all of your testing, you may have also done it already. What I mean is braindeading the UPS - turning off any load attached, turning the UPS off itself, and unplugging it. Then, pressing and holding the off button (o) and waiting for the UPS to click and flash its LEDs. This is a sort of reset. I think that pulling the power module itself out would essentially do this but I am not 100% sure. Leaving the UPS disconnected from AC power for more than 8-10 minutes would also do this. Same if you pulled the batteries too and let it sit with no AC input.

In an ideal world, I'd try another power module if you could (part # SUPM1500) but I can't be sure specifically where the problem is. The chassis of the UPS itself supposedly only has some relays in it for the outlets but I've seen it be the culprit too. I'd only try a power module as a possible cause if you had a spare one around or could find it for cheap. I don't want to recommend it for certain and you find out it is something else and the SUPM was useless.

Either way, based on what you've described to me, I do not see how this could be you doing something wrong or a configuration problem (since we already checked the bypass). I am not a battery expert either as to what types of problems they could cause but you said these were new batteries. They were purchased from a reputable place I assume and you have no reason to believe it could be them? Plus, they charge up OK in your external charger so probably not..

[Anonymous user]

Thanks again Angela,

I fully bench charged the batteries again and performed the "braindeading" procedure, and don't see any difference with how the UPS is performing as compared with the previous attempt.

Unfortunately I don't have a spare SUPM1500 laying around to try. I did, however open up the power module and the main UPS chassis to look for telltale signs of component failure and was unable to locate anything suspect in either.

Yes, the batteries were new when this all began, but the abuse they suffered when the UPS was constantly going in to self test for the better part of two days does leave an open question if they were irreparably damaged in the process. From my limited testing, they look like they have recovered: They follow a normal SLA battery charging profile and hold the charge voltage, but the true test requires putting each of the cells under load and monitoring the voltage during discharge. If all else fails I can order another battery pack, disconnect the SUM48RMXLBP2U, test with the new pack in the SUM1500RMXL2U and see if it resolves the problem. If not, the problem must be with the chassis or power module. That said, what options do I have as far as sending the UPS in for repair?

[BillP]

Hi Brian,

We don't offer repair on these model UPSs - either under warranty or not. We just replace - so in this instance, the SUPM1500 is what would be replaced. The only other option is CoastTec which is an authorized repair partner of ours which does this stuff. Worst case scenario, it is possible to evaluate the Trade-UPS program to compare it (trading in the broken UPS for credit on a new one) to the repair cost through CoastTec if you were to pursue that.

[Anonymous user]

Thanks Angela,

I have put a bit too much time into this already, and am pursuing repair through CoastTec.

[BillP · Accepted solution]

I hope they can get it working!