Smart-UPS RT 8000 XL powers off outlets claiming battery is low when input id distorted

[Anonymous user]

Late last night one of our Smart-UPS RT 8000 XL UPS units running lightly loaded at about 39% briefly turned off all outlets and then turned them back on again 7 seconds later. The 2 emails I received from this UPS read:

http://192.168.1.2
http://FE80::2C0:B7FF:FEA0:5CA4
Serial Number : 5A1338T09604
Date : 01/27/2021
Time : 23:33:57
Code : 0x0114

Critical Events - UPS: The output power is turned off.The output power is
turned off due to a low battery.

and:

http://192.168.1.2
http://FE80::2C0:B7FF:FEA0:5CA4
Serial Number : 5A1338T09604
Date : 01/27/2021
Time : 23:34:04
Code : 0x0113

Informational Events - UPS: The output power is now turned on.

No other messages were sent from this UPS and there's no other information in the logs. However, at the same time another of our UPS units, a later model Smart-UPS SRT 8000, sent the following 2 alerts (but did not dump its loads):

http://192.168.1.10

http://[FE80::2C0:B7FF:FEDA:78F6] (Local)
http://[]

Serial Number : 5A1644T33122
Device Serial Number : AS1645271165
Date : 01/27/2021
Time : 23:33:55
Code : 0x020F

Critical - UPS: Bypass not in range ; low voltage.

and:

http://192.168.1.10

http://[FE80::2C0:B7FF:FEDA:78F6] (Local)
http://[]

Serial Number : 5A1644T33122
Device Serial Number : AS1645271165
Date : 01/27/2021
Time : 23:33:55
Code : 0x0210

Informational - UPS: An input voltage or frequency problem no longer
prevents switching to bypass mode.

Both UPSs are wired as single phase units but are using different phases of the same 3 phase supply. My question is: why did the first UPS dump the loads when there was nothing wrong with the batteries? (The four RBC 140 cartridges were actually replaced on January 8th). Surely the UPS is supposed to continue providing power to the attached loads regardless of input fluctuations, instead of causing a major IT outage overnight? Output voltages are set to 230v and input is around 237-239v by the way.

Incidentally, we have 9 other 8 KVA UPS units like these all connected to the same supply and none of them sent alerts or powered down their outputs.

Andy

[Malvineous_apc · Accepted solution]

It sounds like the batteries could be bad.  I know you said they are new, but have you tested them (e.g. with a calibration) to make sure they aren't faulty?

Also make sure you don't accidentally overload the UPS in a power interruption.  For example if you have a bunch of machines with dual power supplies and you only put one supply on a given UPS, when the power goes out on the other feed, those machines will switch all their load over to the power supply connected to the UPS, causing a sudden increase in load that wasn't there when the last UPS calibration run was done.  If you have enough machines do this the load on the UPS will increase, which means the amount of current drawn from the batteries will increase, reducing their capacity and possibly (if the batteries are in poor condition) causing the battery voltage to sag enough to cause the UPS to drop the load with a low battery condition as the voltage is too low to maintain stable AC output.  I'm sure you're already aware of this but I mention it just in case, and for the benefit of future readers.

As for the reliability aspect, in my own limited experience, UPS devices cause more outages than they prevent.  Nowadays I prefer to use Automatic Transfer Switches (ATS) between the load and the UPS so that when (not if) the UPS craps out, the ATS will switch the load over to another supply until the UPS is fixed.

EDIT: Also, you might want to consider setting the output voltage to more closely match the input.  It doesn't matter so much with a dual conversion/always-online/RT UPS which is looks like you have, but for the benefit of anyone else reading this, in a standby UPS when the power goes out and the inverter starts up, the connected load will suddenly see the voltage drop from 240V to 230V.  As a result it will suddenly draw more current to keep the total watts the same.  This sudden surge could be enough to tip the balance between keeping the load powered and dropping it because the batteries couldn't handle the brief surge.  If the input and output voltages more closely match there will be much less of a peak during switchover.

For RT units where the inverter is already running there won't be a surge, but running at a higher output voltage will lower the amperage running through the cables which will produce less heat and be slightly more efficient.  The difference may not be huge, but if you have a large data centre it may be a measurable improvement.  It's the same reason why the electricity company uses higher voltages like 11 kV or 132 kV for power distribution.  230 vs 240 V is the same thing just at a much smaller scale.

[Anonymous user]

Thanks very much for your detailed and very helpful reply. No, I did not calibrate the UPS after the new batteries were fitted since it was online and powering a lot of small single-PSU servers when I did this and I wanted to avoid dropping the loads (the last time I attempted a calibration - admittedly this was around 15 years ago on an old SU 5000i UPS - it powered off & rebooted!). So I have been wary of doing these calibrations after battery replacements ever since. However, because the quality of the incoming AC supplies to this server room have been deteriorating lately, it looks like we'll be having some scheduled downtime soon for the engineers to investigate and fix the building's incoming supply so this would be a good time to re-calibrate all of our UPS units.

You've made a good point about raising the UPS output voltage so I will adjust these up to 240v. I'm in the UK which some years ago supposedly lowered its standard AC line voltage from 240v to 230v to match the European mainland voltage; in the server room it's usually around 230v although lately it's crept up to 237-239v, possibly because with nearly everyone working from home owing to Covid-19, the building is empty and AC demand is low. (I say 'supposedly' since at my home the AC line voltage reported by my own APC UPS's and by an accurate DVM is 245 -253 volts and has been like that for over 30 years!)  

[Malvineous_apc]

I've had a similar experience with calibrations.  It works when the batteries are good, but once they get to the point where they can't deliver enough current to power the connected load, the calibration fails.  Because in a stand by UPS this involves switching the power over to the inverter (with a short interruption to the output power as the relays click over), then the UPS realising there is no power available from the inverter, then switching back to the mains (with another interruption), the combined time without output power is long enough that most things connected to the UPS shut down or reboot.  In extreme cases (with completely dead batteries) I have also seen the UPS completely power off as the batteries couldn't even provide enough power for the UPS to switch back to the mains.

I suspect the RT units might be better in this regard as they are already running the inverter and they have large capacitors in them, so when they do a calibration and switch off the mains input there is no interruption, however I haven't yet investigated enough to learn whether they start drawing power from the mains again before or after the inverter completely shuts off, so I don't know whether there is still an interruption at the end of the calibration process (or at the start, if the batteries are bad).

I'm in Australia and within the last year the power standard for the area I'm in has also changed from 240 V to 230 V, with the reason being so much rooftop solar is being installed that during the day it is pushing the grid voltage so high it's close to going out of spec.  However the new spec is defined as 230 V +10%/-6% giving a range of 216-253 V, which means all the existing transformers that are delivering 240 V don't need to be adjusted as 240 V is still within the spec.  So most places here are still getting 240 V too.