Smart UPS SRT 5000 Commanded itself off

[Roger.Prichard]

    Hello.

We are employing an SRT-500 that recently seems to have commanded itself off.  Trying to understand why.  I saw some similar issues on the forum, but they were quite old.  Hoping to get some answers on how this happened and how to prevent it from happening again.

A snapshot of the event logs is pasted below.  The debug and other logs are attached.  Any help or advice is certainly appreciated.

Roger

    04/22/2024  06:44:32  Device  UPS: The power for switched outlet group 2, Outlet Group 2, is now turned off.

     04/22/2024  06:44:32  Device  UPS: The power for switched outlet group 1, Outlet Group 1, is now turned off.

     04/22/2024  06:44:32  Device  UPS: The output power is turned off.

     04/22/2024  06:44:32  Device  UPS: The graceful shutdown period has ended.

     04/22/2024  06:44:32  Device  UPS: Switched outlet group 2, Outlet Group 2, has been commanded to turn off, sequenced.

     04/22/2024  06:44:32  Device  UPS: Switched outlet group 1, Outlet Group 1, has been commanded to turn off, sequenced.

     04/22/2024  00:56:27  System  Web user 'apc' logged out from 10.30.10.60.

     04/22/2024  00:52:01  apc     Web user 'apc' logged in from 10.30.10.60.

[Teken]

Based on all of the great logs you provided it appears the unit when first commissioned (2022) was overloaded at the outputs. Regardless of that, the system has tracked the unit sees dirty and unstable AC power which continues to this day. 

This causes the unit to go into battery only mode. At some point the system indicates the batteries are nearing the define threshold of depletion and thus begins the shutdown sequence. 

If we ignore the batteries for now you can adjust the upper & lower transfer voltages along with the power quality sensitivity.

Changes to these two settings will force the unit to accept a (wider) range of frequency, voltage, and THD.

As it relates to the batteries you’ll want to validate they are truly fine and do provide the expected operational runtime based on the connected load (watts) vs the APC runtime calculator.

Why?!? 🤔

Because the UPS has been running on battery only mode for a very long time. The average service life of new cells are 3-5 years. This is based on usage, temperature, and environmental factors. 

Given this unit has been using the batteries more often than expected your service life is more than likely half the stated service life of 1-2 years.

You can easily test this now and see the results. Unplug the unit and time how long it continues to operate. If it shuts down immediately or seconds later - Replace! 

If the unit continues to operate for X minutes compare it to the APC Runtime Calculator. If they say Y load equals 60 minutes but you see 10-15 minutes - Replace! ☝️

If you see 35-45 minutes you need only schedule a battery cartridge purchase in the near future. 👍

Start by changing the power quality (sensitivity) frequency, and upper / lower transfer voltages.

Follow up with a Lights Out Test as outlined above.

Let me know what you see and observe and appreciate all the logs! 👍🍺

[Roger.Prichard]

- Thank you for the quick response.  The advice about adjusting the upper and lower transfer voltages (power quality) settings is something we will dive into right away.  

- Regarding the batteries sensing that they are reaching the defined threshold of depletion, that seems to conflict with the data that we are collecting from the unit.  Over the last month, where, despite the date time being incorrect, the batteries never reported below 99% battery remaining.  Perhaps I am misunderstanding.  And again, thank you for the great response.

Roger

[Teken]

I would not take what the system indicates as to the battery capacity as very accurate or factual.

A true test of the operational runtime of the unit is to complete a (Lights Out) validation. Remove the AC Mains from the wall and time how long the system operate for.

There are going to be three scenarios from instant shut down, marginal runtime, to acceptable runtime as I outlined above.

You can also initiate a battery calibration with a minimum of 30% attached load. In both cases the system will continue to run until it has reached the defined depletion (time) you have setup in the unit.

Either procedures should be performed once a year to validate the UPS is fully capable of meeting the expected runtime during a grid down event / dirty power.

Both validation tests are extremely tough on the battery cells and reduces its service life which is 260 full depletion cycles.

Run either tests and let me know how the unit behaves and operates! 👍

Awesome SNMP graphs / line charts! 🤟🍺🍺

[Roger.Prichard]

Unfortunately, I cannot easily run the UPS to a point where it shuts down due to depletion of the batteries. I can potentially perform the battery calibration, but it will have to be a coordinated event.  An uncontrolled shutdown of the supported devices would be disastrous to our client since the unit is supporting revenue collection.  Additionally, we have had difficulty recovering from uncontrolled shutdowns. 

Some additional information:  This site was installed and commissioned on 3/21/2024 which involves running on battery power for 1 hour while under full load (we must prove that capability).  One hour for us is more than enough time considering we have a backup generator that takes around 17 seconds to come online after utility power is  lost.  

Bottom line, I understand why the test is important, however, I am confused about how it is possible that the batteries were depleted considering (1) that the utility power is backed up by a generator in cases of utility power loss (2), that the data we are collecting from the UPS shows no downward trend of battery life immediately preceding the shutdown event, (3) that the unit was proven as recently as 3/21/2024 to sustain power on full load, and finally, (4) that most of the "on battery power" events since 3/21/2024 seem only momentary (seconds, and not even minutes) and not to full depletion of the batteries.  

Despite that, I can explore the possibility to perform some test of some length of time or the battery calibration.

Thanks again, your help is much appreciated.

Roger

[dpierce4776]

@Roger.Prichard As far as testing the runtime of the batteries doing the calibration test should be fine. It runs on batteries until the batteries say (or the unit thinks) are at 20% capacity. So there is little chance (unless a battery is toast completely) of dropping the load with that. This also helps the UPS "know" or estimate how long the batteries will last. I always run one of these test after replacing the batteries in a unit.

I do question though as you stated that you stated "difficulty recovering from uncontrolled shutdowns" is that due to the UPS not providing power or more of a server/device issue?

To your "confused about how it is possible that the batteries were depleted considering"
As @Teken stated it appears from the logs that the unit was overloaded at the outputs at some point and has tracked dirty and unstable AC power which continues to this day. 

To this the fact that you have a generator may not come into play as the dirty power may not be coming from outside the building per say. It could be things like

1. a transformer inside the building or converting from the normal 480 3 phase (feed from generator or building power is going bad.
2. An upstream distribution panel having a bad phase rotation
3. An upstream distribution panel has an incorrect bonded or lose neutral.
4. A loose hot or neutral wire from the panel to the UPS.
5. Improper earth ground in the panel or at the UPS.
6. If you have an external battery pack that is not grounded to the UPS.
7. If the circuit feeding the UPS is not dedicated to the UPS other devices such as motors, heater, and even florescent lights can cause noise or feed back. 
8. In rare cases if there are several larger motors connected to the distribution panel that feeds the UPS even on a direct dedicated circuit to the UPS the feedback/noise can cause an issue  

All of these can cause the UPS to switch to batteries and the generator never know or fire up. Because they are all internal of the building.

On top of that the generator is not going to fire up due to dirty power unless it is "dirty enough". UPSs from almost any manufacturer are generally more sensitive than the generator or building ATS is. Generators typically only look at the voltage on each phase and unless it drops low enough they do not kick on. Dirty and out of phase really doe not come into play here unless the noise causes the generator/ATS to see a drop in voltage or it falls out side of a preset range.   

To your #2 Typically when you switch to battery power the first batteries in line get hit and take the load. As the outage continues the batteries start to "balance" the load across each other. This is why if you watch sometimes the "remaining time" will drop down like 20% then climb back up a few percentage points. Looking at your "APC Percent Battery Remaining 4.9 - 5.7.jpg" upload it looks like you have a bad cell in one of the batteries. It will pass voltage but when a load is placed on it it will fail open. This causes the UPS to not see any batteries as the circuit is open like you disconnected the batteries from the UPS. This can cause the UPS to shut down.

To your #3 and #4 The funny thing about batteries is that as @Teken said they might hold a load when you test them one time. But as they continue to be hit they finally go bad right after that. Murphy's law and all

Personally I would be interested to see what happens in a calibration test. Because if it is a bad cell in one of the batteries it may not present itself at the initial drop in power to the UPS. As the load spread across the other batteries though (over a few seconds) that cells can not sustain the load and fails. A calibration test would/should pick that up.

Something else I have seen with dirty power and/or sometimes the logics internal of the UPS gets messed up and fails. I have seen UPSs report 100% batteries and even with brand new tested batteries. Once a load is put on the UPS "thinks" the batteries are depleted and shuts down. Case in point I have had a UPS from APC and Eaton do this. We put new batteries in and tested with load. I took the batteries out and put them into another UPS of the same size and make. Then tested with the same load and it held like normal.

Dirty power and all the things I listed above not only play havoc with batteries. These smaller rack mountable UPS also get hammered as well. Their internals are not hardened or as able to take the repeated hits like say a symmetra or a larger building UPS can.