Battery runtime Incorrect?

[BStein_apc]

We wanted to test the settings in the PCNS for shutdown a VSphere environment.

Starting the test by removing the UPS showed remaining battery for 1h xx.

Removing the Power on the UPS triggered the PCNS to shutdown the VMs after 9min.

The set shutdown sequence (Vm shutdown, vcentershutdown, etc) is accumulated to 20min. The low battery setting on the UPS was on 25min.

The UPS itself showed at time of runtime exceed Msg in PCNS 47min remaining...

Digging around and testing brought following screenshots on the UPS (see attached document).

Within several minutes the UPS show remaining times between 2h and 20min with a constant load (in my opinion). Is there a bug in the battery calculation (that would explain the runtime exceed) ?

The log shows:

02/14/2024 02:31:15 Device UPS: A low battery condition no longer exists. 0x0110
02/14/2024 02:31:13 Device UPS: The battery power is too low to continue to support the load; the UPS will shut down if input power does not return to normal soon. 0x010F
02/14/2024 02:31:11 Device UPS: Internal pack, cartridge 2 is installed. 0x0130
02/14/2024 02:31:11 Device UPS: Internal pack, cartridge 1 is installed. 0x0130
02/14/2024 02:31:11 Device UPS: Internal pack, cartridge 2 is not installed. 0x012F
02/14/2024 02:31:11 Device UPS: Internal pack, cartridge 1 is not installed. 0x012F
02/14/2024 02:31:03 Device UPS: A low battery condition no longer exists. 0x0110
02/14/2024 02:31:01 Device UPS: The battery power is too low to continue to support the load; the UPS will shut down if input power does not return to normal soon. 0x010F
02/14/2024 02:31:01 Device UPS: Internal pack, cartridge 2 is installed. 0x0130
02/14/2024 02:31:01 Device UPS: Internal pack, cartridge 1 is installed. 0x0130
02/14/2024 02:30:59 Device UPS: Internal pack, cartridge 2 is not installed. 0x012F
02/14/2024 02:30:59 Device UPS: Internal pack, cartridge 1 is not installed. 0x012F
02/14/2024 02:30:49 Device UPS: A low battery condition no longer exists. 0x0110
02/14/2024 02:30:47 Device UPS: The battery power is too low to continue to support the load; the UPS will shut down if input power does not return to normal soon. 0x010F
02/14/2024 02:30:46 Device UPS: Internal pack, cartridge 2 is installed. 0x0130
02/14/2024 02:30:46 Device UPS: Internal pack, cartridge 1 is installed. 0x0130
02/14/2024 02:30:45 Device UPS: Internal pack, cartridge 2 is not installed. 0x012F
02/14/2024 02:30:45 Device UPS: Internal pack, cartridge 1 is not installed. 0x012F
02/14/2024 02:26:24 Device UPS: On battery power in response to rapid change of input. 0x0109
02/14/2024 02:26:24 Device UPS: Bypass not in range ; distorted waveform. 0x020F
02/14/2024 02:26:24 Device UPS: Main input bad. 0x01B0

...

These errors are also pretty weird.

The UPS is a:

Network Management Card AOS v2.0.0.6
Smart-UPS APP v2.0.0.5
Model: Smart-UPS SRT 8000
Firmware Revision: UPS 12.2 (ID1021)

Do we have a problem here on the UPS site and PCNS was triggered wrong?

[dpierce4776]

A couple of things

1. Why does the log show that battery packs were disconnected and reconnected? Reading your post IMHO it looks like you were disconnecting batteries while the UPS was trying to run on them.

2. Do you have any external battery packs? If not and the card gives the options under "configuration > UPS" to set the number of battery packs make sure it is set to one. if it does then make sure that setting is set to 1+ the number of whole towers (not the battery cartridge).

3. Have you done a battery calibration test? If not you can find this under "Test > UPS". The batteries must have 100% charge when starting this test.

All in all the logs look like you removed building power from the UPS for the test. Then started disconnecting internal battery packs from the UPS and reconnecting them. That is not a good idea. The way the UPS is wired is to spread the load across all batteries and the time remaining is calculated using all batteries. Removing batteries, especially when they are being used, could damage the batteries and/or the UPS. Also it puts more load on the remaining batteries draining them faster. Once the batteries are reconnected the load is re-spread across all batteries, but since the batteries are at different levels of charge. Causes not only the load but the lesser charged batteries to start pulling on the higher charged batteries to balance back out. This pulls them down faster than normal as well. All causing much shorter runtime than the UPS is calibrated to have since it is calibrated using all batteries at the same time and spreading its load across them evenly.

 

[BStein_apc]

1. I don't know at that's a reason why I believe there is something wrong in the UPS. And of course I was NOT manipulating anything on the Batteries during the test.

2. No there are only internal Battery packs installed (2 of them) both showing ok and online

3. The batteries showed a 100% load before testing started

Again: The batteries were not removed during the test physically.

[Teken]

Just to add some clarification on the above. The battery disconnect errors seen in the logs are (A Typical). We have hundreds of these units in the field and all of them show the same disconnect / reconnection error.

This in my opinion is a serious bug that APC refuses to address! Part of the problem is the SRT UPS line polls the battery packs. My belief is this polling isn’t done correctly or in a manner that makes sense. 🤢 Thus, when the system does complete the polling the battery shows connected.

Regardless, a few things should be called out and understood. The runtime called out by APC is based on new cells that are 100% charged. It is also based upon a fixed minimum load. In the real world you won’t have that exact load, fresh batteries, temperature, battery SOC.

As such the runtime will not be the same. As the other member noted a battery calibration can help the unit reflect a more accurate runtime and displayed (time). Battery calibration should be done only once a year as this process is extremely tough on the cells. The average service life is 3-5 years based on ideal environmental usage and temperature. These cells are rated for only 260 cycles. 

If the unit is in an environment where the utility power is very poor. The battery service life will be a lot shorter. If the environment is very hot this too will shorten the service life of the battery cartridges.

As part of the UPS maintenance every cell must be inspected. More often than not you’ll find one cell that is weak / poor. Just a single weak cell can result in shorter runtime or cause the pack to fail prematurely as it drains the other cells!

Every site should have a calibrated Fluke multi meter, carbon pile load tester, variable resistive load, and conductance meter. Using these tools will provide you 100% insight as to the state of health of the cells. Having the ability to measure the internal resistance of each cell is paramount in knowing its true state of health. So it can be compared to the manufacturers data sheet.

Having them on hand allows you to test the individual packs (offline) and outside. Thus, not impacting the (uptime) and operational service of the companies network.

[BStein_apc]

Hej Teken, thank you for the response related to the battery disconnect errors. It doesn't help but make the statement clear...

Anyhow it is not clear to me why the battery remaining power is changing in such a short manner (as documented in the word-file) and if this explains the PCNS to shutdown the whole environment after short period of time.

In the meantime I was also checking the FW status and yes I would be a new one available (installed 12 new 15). Unfortunately it's not visible to me if such a kind of bug is fixed in the new FW and the update must be done on power off on this model 😞 In short words: it still is not clear if the UPS works as it should or if there is a serious problem. It's a bad situation to relay on a UPS if one is not sure it works....

[Teken]

How old are the batteries in this unit? What does the UPS indicate as the connected load percentage? If the site sees a lot of power quality issues this too will shorten the lifespan of the batteries.

The average service life we see is three years at some target sites. Whereas others that have excellent power quality the batteries last five years or more.

All of the sites are scheduled for battery replacements at the three year mark. Sites that have proven to have no issues and excellent power quality we change out at the four year mark. Regardless of what any field testing shows to insure UL certification and insurance coverage.

If the batteries at your site are three years or more in age - it’s time to replace them. If you review the APC Runtime Calculator for your specific UPS model you’ll get a general idea of what operational runtime should be seen based on a defined load.

If for example the APC chart says 250 watts equals 60 minutes and you see 5-10. It’s time to replace those cartridge’s. If on the other hand you see 30-45 minutes. It’s time to notify senior management to budget finances for new batteries for the system. 

Let us know how old the batteries are and the connected load (watts). 

NOTE: All of our units are running the latest firmware on the UPS and NMC. None of that has changed the battery disconnect error messages we see. 🙁 The read me file on the various iterations of firmware doesn’t call this out so it’s obvious this problem continues to exist. 😡

[dpierce4776]

Also would you happen to have or know the age of the UPS? As @Teken stated batteries are generally only good for 3 to 5 years. I have seen some go longer but that is not usual.

Also I have seen older UPSs (around 10 years start "eating batteries". But since you said that you are not touching the batteries then it sounds like you have at least one battery going bad. Generally what happens is when the USP polls or test the batteries one cell can not old the load or it is weak and drags the others down. I have also sen a few times that a unit reports that it is 100% charged but when a test is done it drops to 0% in a couple of seconds. When you start getting the disconnected reconnected alerts that usually means a battery is going bad. And Yes APC is horrible and battery alerts 

[BStein_apc]

The UPS is 1,5 years old, as well as the batteries inside. So they hopefully are not bad.

Then the only way is to do a battery recalibration test?

But anyhow it's quite bad to have such a situation without any accurate reporting what's going on..

How can we relay on such a thing?

[Teken]

I would urge you to contact APC Technical Support to determine if there is a problem. They may very well just send you out replacement batteries if they believe they aren’t performing correctly. 👍

Regardless, I haven’t read what the connected load is on the unit while a (lights out) test is underway. If you intend to perform the battery calibration test there must be at the minimum 30% connected load on the UPS. So if for example the unit can sustain 1000 watts a 30% load would be 300 watts. If the output is only 100 watts and you run the battery calibration the runtime and all displayed battery levels will be incorrect.

Technically you don’t even need to run the battery calibration! You could just rip the AC Mains out of the wall and see what the actual runtime is! 

There are several reasons (Why) you would do this though and it hinges on dozens of environmental variables and your technical knowledge and experience.

If say you had an old (APC) unit you can get away with just ripping the cord out of the wall and using a watch / timer to see how long it lasts (runtime). The reason you can / could do this is older models were less than smart and didn’t have much intelligence to control the main unit.

Hence why the LED / LCD never ever matched the actual runtime! Keeping in mind even in 2024 the display / Power Chute runtime is never accurate. 🤦‍♂️ 

Newer units have a little more checks and balances as it relates battery SOC, temperature, resistance, etc. Even then they are still not very accurate as it relates to estimated runtime! 😡

At the end of the day a real world (Lights Out) test is the only way to know (IF) a UPS will provide X vs Y runtime. Keeping in mind a battery calibration or a lights out test is extremely tough on a SLA battery which only offers 260 full cycles based on the CSB data sheets.

In contrast LiFePo4 cells offer thousands of cycles and can be used and drained from 100 - 0% without harm. Whereas a SLA battery can not be drained past 30% without impacting its service life! 

This is why the APC default is set to a minimum shutdown period. This attempts to limit the battery from being over discharged and impacting its overall service life. 

This leads to something you can play with to see the cause and effect. You can set the unit to turn off sooner / later in (minutes) and observe the results. You’ll see the runtime be extended / reduced simply by changing when the UPS should shut down. 

Obviously you would need to compensate when the PCNS engages! 🤣 Otherwise you’ll find out the hard way of what an unplanned outage looks like! 🤦‍♂️😢

[Teken]

I had taken a lot of time to write a thoughtful reply offering some solutions. A moderator thought it would be fun to delete my reply and mark it as SPAM?!? 🤦‍♂️👎

Sent who ever the moderator is for this forum asking for the reason they did this. Of course I never received a response much less a considerate explanation of why!

Theres literally nobody in this forum stepping up and taking their personal time to help out the forum members such as @

Dpierce4776 and myself.

This is why nobody comes here because it’s just crickets of silence! 

Im out . . . 

[dpierce4776]

First I would do the calibration and watch the time remaining and charge. Of the batteries. If it starts dropping fast you have a weak battery. 

Now just an FYI it is not unusual to see the batteries drop 10 to 10 percent at first, but then as the batteries level out and spread to load amongst themselves then it should hold there for awhile and may go back up some. 

This happens because of the way batteries work. When the load is first applied the first battery takes the hit and tries to old the old after a few seconds on short strings (longer the more batteries you have) the frist one will start to pull on the next in line and so on until all batteries level out and start working together.. think of 4 tanks connected together when you start draining the fist tank in line the take drain quicker until water is flowing from all the tanks.

[BStein_apc]

We updated all the environment to the latest free versions (UPS and NMC) and then ran a test again.

Conclusion: The automatic state about remaining runtime is nonsense (it is still "flapping" around). There are still Msgs in the log that the battery is not sufficient to hold the load even the battery has enough load remaining. The only value that could be trusted in is the remaining percentage of the battery capacity. Unfortunately this is no triggerpoint to PowerChute!!

We ran the environment for about 30min on battery and the load dropped from 100% down to 60%. During this time there were 2 calls out that the load is too high and the Groups will go off (nothing happend in fact). Also the msg that the batteries disconnected and reconnected is visible during this time. So the only trustfull triggerpoint is the "UPS is on battery" with a manual CALCULATED time to start the shutdown.

This is a pitty (and maybe worth to check out with other products)!!

[dpierce4776]

@BStein_apc there is a few things to consider as well. There is really no way to "accurately estimate runtime" as loads change constantly as demands change, batteries go bad or get weak, heat changes can effect battery life and charge, and the like. The estimated runtime on any UPS is always going to differ some from the actual runtime. As the estimated runtime is usually done under normal loads and usually when the batteries are fresh.

Also if you have a weak battery in the string. It can report that it is good but when a load is put on it it will not hold the load. This spreads the load to the other batteries draining the faster. Sorry I know this is not a "great answer" nor is it a defense of APC. But I have worked with Toshiba, Liebert, Mitsu, and APC UPSs of various sizes and they all are pretty much the same in this regard.