Smart-UPS SMT-1500IC very low runtime remaining

[klosz007]

Hi,

The following scenario:

Power to the UPS is cut.

Outlet group 1 is switched off after 20 mins by PCNS running on server (Proxmox) host (which shuts itself down).

Then after another 10 mins main outlet group (Ethernet switch etc.) is switched off by load shedding timer (set to 30 mins after power failure).

At this point NMC 2 says that battery charge is at 65% and runtime remaining is ~2 hours. Eventually UPS turns off, display goes off.

Everything works as expected to this point.

When power is restored, UPS wakes up, UPS initially starts counting down power on delay of 180 secs (UPS Turn On by SYS in...).

But after a while it stops and counter returns to initial 180 seconds.

When I connect to the NMC then, it says that output power cannot be turned on because "The battery power is too low to support the load".

And indeed, on the UPS status page, it says that battery charge is 40% and remaining runtime is just 2 seconds (!). Main outlet group is conifgured to enable output when UPS has sufficient charge for 600 seconds.

There are two things I do not understand here:

- how is this possible that when main output group was powered down, battery charge was 65% and when input power got restored and I could connect to NMC again, battery charge is only 40% (what ate 25% of battery when outlest were powered off and there was no power on the input ?)

- how is this possible that with 40% charge, remaning runtime (with both outputs powered off) is 2 seconds. Note that with full battery charge and full load (main output ang group 1 powered on, runtime is around 1 hour, with group 1 powered off - around 2 hours).

Eventually, at certain battery charge level, remaining runtime goes back to 'normal' value (such as 24 hours with no load) and outputs get enabled then. 

I ran battery calibration with artifical load (two bulbs) and it completed successfully but did not help with this issue at all. UPS was purchased December 2023 and battery pack is still the original one.

Dying batter pack is the only idea I have for now but to my taste it's far too new to be worn out.

Thanks,

Zbignew

[Teken]

The average service life of the RBC is 2-5 years based on usage, temperature, and environmental conditions.

The cells are designed to endure 260 full cycles before degradation.

Based on the limited information you have provided the RBC is near EOL. Also, the system is (NOT) properly calibrated to know the actual operational runtime.

As the values you have stated are the mythical default values of 24 hours. The Battery calibration must be performed with at least 30% or greater load and based on your reply this wasn’t the case.

As it relates to percentage of charge this value and operational runtime are estimates and are not based on any real time and factual data.

SLA chemistry based cells do not provide the same output curve or can sustain a DOD below 50% vs Lithium based cells.

Questions Ask . . . 👍

[klosz007]

Hi,

I used to have worn out batteries and the behavior was different - they discharged much quicker than NMC (and me) expected. That's not happening here. Drop from 100% to say 60% during 30 minutes of my load is perfectly expected.

In another SMT-1500IC original batteries are 3 years old and still good. These batteries here are like 20 months only. Even 3rd party CSB batteries typically survive 3 years.

What surprises me that batteries (as per NMC) keep discharging when outlets and UPS are already turned off (and they do it then actually much quicker than when outlets are on). I'm going to experiment a bit more but that makes me suspect that UPS might be faulty.

24 hours of runtime is shown only when outlets are on but with no load connected. With load connected, main outlet group enabled and switched group 1 disabled it's 2,5 hours, with both outlet groups (my full load) enabled - it's around one hour.

I executed some more test and the statement I made previously is incorrect.

What happens is that:

When power returns, NMC claims runtime is 2 seconds even if I wait for that long so that batteries are fully recharged to 100%. Still at 100% UPS refuses to enable outlets because these are configured to enable when runtime is sufficent for 600 seconds (and two is less than 600).

Available runtime updates to 'expected' value only when I manually reconfigure required runtime on the outlets to 0 seconds. Then main outlet gruup gets enabled first (0 seconds is less than 2 seconds) and out of the blue, once it get enabled after power on delay, available runtime changes from 2 seconds to 2,5 hours !

Another suspicion I have is that, becuase the NMC 2 card is 10 years old (I bought second hand one), the battery on it got discharged (not sure what this small button battery is used for).

[Teken]

You’ll need to provide the following logs for review as no factual data has been given. As such there is no reference baseline to form an root cause analysis.

Provide the following logs in their entirety: via the NMC: Event, Data, UPS, Diagnostics.

Once the logs have been provided the next steps are to complete a static / dynamic test and document the results.

• The coin cell is to retain RTC, Logs. In older NMC versions the coin cell could be removed and replaced from the holder. Later on the cell was tac welded in place due to some make belief guise of safety?!? 🤦‍♂️

If you’re very careful and handy you can remove the tacked cell and replace with the same high quality one. 

Questions Ask . . . 👍

[klosz007]

Hi,

I have run another test yesterday.

I reorganized cabling a bit so that Ethernet switch does not get powered down when main outlet group goes off (so that I can see what's going later on via NMC).

UPS switched off both outlet groups in sequence as expected, then it ran for another 10 minutes dispalying "Waiting for AC". At this point NMC was still powered on from battery pack and it said that remaining runtime was 24 hours (no surprise with all outlets off, prior to switching off main outlet group it said it was like 2,5 hours). Battery charge was like 80% at this point (due to lesser load it lost only 20% of charge when running on battery).

Eventually, after these 10 mins, UPS LCD went off and contact with NMC was lost.

Then I waited for like 30 minutes then and restored AC power to UPS.

Then same story again - NMC claiming remaining runtime 2 seconds and charge level only 55%. 

So during that 30 minutes when display was off, battery lost another 25% (from 80% to 55%) and remaining runtime dropped from 24 hours to 2 seconds.

Obviously UPS refused to power on the outlets again (insufficiuent runtime).

At this point I cut AC power, removed battery connector at the back and removed NMC.

I measured voltage on NMC button battery and it was like 3.1V so battery seems to be good, even though NMC is 10 years old (as I read BR2032 batteries were designed to last up to 15 years in contrast to CR2032 which are supposed to work for 5-10 years). Yes, battery is soldered to the PCB but for me it would be no problem to replace it if needed.

Then, I decided to reconnect battery without reinserting NMC and enabled AC power again.

To my surprise UPS started counting down to enable Main outlet group. So the issue with 'insufficent runtime' was gone when UPS got powered on without NMC inserted.

As far as I remeber, I let UPS enable Main outlet group after 180 secs (there was nothing connected to it anyway) but I cut AC power again before Outlet Group 1 got powered on (group with NAS servers, Proxmox server etc.). I disconnected battery again, reinserted NMC, reconnected battery, reconnected AC.

And UPS started again counting down to power on Main outlets.

I connected to NMC and found out that battery charge was at 85% again and remaining runtime was again 24 hours.

The small mistake I made was allowing UPS turn on Main outlet group before reinserting NMC. Forcing UPS to enable Main outlet group (by setting required runtime to 0) was what healed the situation previously, without removing NMC.

Long story short, some kind of magic is happening here.

I decided to buy cheap (100 USD) second-hand (untested, from return) NMC 3 and will give a try.

If 'new' NMC 3 works fine and issue is gone then probably old NMC 2 was at fault.

If either 'new' NMC 3 is bad or the same issue is still there with NMC 3 then possibly UPS is at fault and I will have to contact Schneider support (UPS is on warranty).

If someone wants to review attached logs and help me, you are more than welcome.

Thanks !

[Teken]

After reviewing the logs I'll affirm once again some facts that you need to accept and understand and take next steps as outlined listed below. The average service life of the RBC is 2-5 years based on Usage, Temperature, and Environmental Conditions. 

Your hardware was produced in 38 week of 2023 so you just missed the 2 year warranty coverage for the RBC. You can always submit a RBC warranty claim to see if they will provide you one as asking is free. 

The event and data logs show you have dirty power present for the short period of time the logs are available. Some of this could be due to you testing the unit with a Lights Out / Grid Down event and battery calibration. As the voltage, frequency for the period covered are fine and well within the accepted margins.

The data logs do not show the RBC at 100% SOC and assume you exported the logs before the system was fully charged.

This stands out because the last reading was 98.53% SOC, 26.78 VDC, and the internal RBC temperature was climbing to 38.XX plus 'C. 

Self Test: The self test is a very crude diagnostic test performed by the system which places a defined load on the RBC. It also completes a basic POST on key components within the unit such as communications, hardware, and electrical. It can not be relied upon to determine if the RBC is actually fine given the crude testing methodology used to determine the RBC state.

By the time the Self Test actually declares a RBC as being bad the cartridge was long expired. You can search this forum and the Internet to affirm the same as to the validity of the self test. 

The gold standard used by serious companies measure voltage, resistance, and conductance which is referenced to the battery makers data sheet at a given ambient temperature. All serious lithium based systems use a BMS to monitor individual cells for over / under voltage, temperature, balance, current, and internal resistance, SOC.

Lithium based cells are designed to endure 100% (DOD) depletion over thousands of cycles unlike a sealed lead acid chemistry based cell that's used in this UPS.  

As stated earlier to identify and narrow down possible root cause requires a combination of Static / Dynamic testing which requires basic to advanced tools and time . . . Both tests can also be performed in a simple manner but leaves a large margin of error / unknowns if the proper diagnostic tools are not available / used to validate the same.

A basic static and dynamic test can be conducted on the system now without any extra tools. But, again the results are not to be taken as gospel if other factors are not taken into consideration as doing so will provide you with false positives / false negatives!☝️ 

Static Test: Disconnect the RBC Anderson connector from the system. The system should continue to run and declare a battery disconnected fault / error. If this is true great - if not and the system immediately shuts down or acts erratically there is an internal hardware fault.

Keeping in mind if there is dirty power present this will result in a false positive. As the system may try to switch to battery only power vs AVR mode and if the RBC isn't present it will obviously shut down! 😖 

Dynamic Test: Insure the RBC is at 100% SOC and there are no power quality issues present. Disconnect any communication cables prior than the AC Mains from the system. There must be 30% or greater load connected to the units outputs. The unit should immediately declare a loss of AC Power and switch over to battery only mode.

If the system operates as outlined above - great . . .

Now time how long the system continues to operate for before it shuts down. Compare this operational runtime with the APC Runtime Calculator. Keeping in mind this is based on a new RBC with a defined load at a specific ambient temperature.

If the system turns off immediately or provides only 1-2 minutes of operational runtime - Replace RBC. If the operational runtime is half of what is expected its time to order a RBC for replacement as its nearing EOL. 

NOTE 1: A dynamic test or one that mimics a (lights out / grid down event / battery calibration) is extremely tough on the RBC and goes against the 260 cycles the cell is designed to provide. As such a battery calibration should only be done upon first RBC replacement and on a annual basis to validate the state of health of the cartridge. 

The correct steps and procedure during a RBC replacement are as follows:

Install Date: Update the battery installation date as this resets internal counters and adjusts charging tables if applicable via LCD Menu, NMC, PCSS.

Self Test: Manually initiate a self test via LCD Menu / NMC / PCSS as this allows the system to complete the POST and load test the RBC to affirm operating state.

Battery Calibration: This procedure must be done with a minimum of 30% attached load or higher on the outputs. Doing so will allow the system to learn the capacity of the RBC. This will help the system reflect the operational runtime more accurately on the LCD, NMC, PCSS, SNMP.

Keeping in mind the values displayed are estimates based on a crude table and do not reflect reality. 

Dynamic Load Testing:  This is why a dynamic load test is performed, timed, and compared to the APC Runtime Calculator. To provide a baseline and gauge as to operational runtime as referenced up above.

NOTE 2: The most accurate method to determine a cells state of health is to measure the internal resistance and measuring its conductance with a calibrated and certified meter and referencing the information the battery makers data sheet based on a defined ambient temperature.

Measuring the voltage of the RBC or single cell is not to be relied upon!

Regardless of the above a crude method if advanced tools are not present is to measure the voltage at the Anderson connector of the RBC while connected to the system. This will measure the float voltage from the UPS and documented.

Next step is to disconnect the RBC and measure the surface charge which will be the same as the float voltage until it dissipates - write it down. Leave the RBC disconnected for 72 hours and measure the voltage. A drop in 0.1 ~ 0.3 VDC is normal and expected.

A drop in 1 ~ 3 VDC indicates a marginal to failing RBC . . 

More often than not you will find one or more cells that are dragging down the pack. Any obvious signs of bulging, cracks, leaks, corrosion indicate a failed cell. Any cell(s) that are hot are failed along with any cells that drop voltage in the 1-3 VDC range.

When the system is connected to a NMS you can easily see several metrics that indicate a failing RBC / Charger fault. As you will see rising battery temperatures, erratic float voltage, and operational run time counters that drop from established baselines.

Any cheap carbon pile load tester can be used for validating individual cells keeping in mind this goes against the 260 cycle life while doing so. A load test must be referenced to the battery makers data sheet as to load and duration and recovery.

Questions Ask . . . 👍 

[klosz007]

Hi,

Thanks for such deep insight into my problem !

Answers to some of the items:
I checked SmartConnect and it says UPS 3-year warranty expires on 21 Sept 2026 so if RBC is covered by 2-year warranty then I still have a few days left on RBC warranty.
Thanks for information anyway, I thought warranty for RBC was 1 year only 🙂

"The data logs do not show the RBC at 100% SOC and assume you exported the logs before the system was fully charged."
Very likely yes.

Self test was successful but as you said it is very basic. I had cases of RBC dying where self test was successful 🙂
I did not know such thing as Static Test exist. Will try it.

Instead I ran Calibration test. I disconnected all 'computer' loads and connected two tradiitonal bulbs (rougly 200Watts total) instead.
I do not remember how long it ran but something around 30 minutes so plausible for good RBC and that amount of load.
It did not end very quickly such as 5 minutes.

I did not satisy 30% load though, rather 20%. Probably I would need at least 300+ Watts.

For now nothing tells me that RBC is at fault.

When outlets are already switched off by sequencing but UPS still displays 'Waiting for AC power' on LCD and NMC is still powered on, remaining runtime (as reported by NMC) is 24 hours (no surprise with outlets off).
When outlets were still on the runtime was like 2 hours.

Problem begins when when UPS turns off completely waiting for AC power and LCD goes black (this happens 10 minutes after all outlets got turned off).
When UPS is left in this 'dark' state for a while (I tried like 30 minutes) and AC power eventually returns, then NMC claims then remaining runtime is only 2 seconds and RBC SOC is much lower than it was before UPS got self turned off.
Exactly as if somewthing was draining power quickly from RBC when all outlets are off and UPS/LCD is off.
UPS refuses to turn the outlets on then (insufficient runtime - outlets are set to be turned on when runtime exceeds 600 seconds).
But when I force the outlets to get turned on (by changing required runtime to 0), then out of the blue, NMC returns to displaying remaining runtime from 2 seconds to 24 hours (I think with no load the maximum value it can show is 24 hours).
If load is connected then obviously remaing runtime is lower (depending on load it can be say 2 hours).

Nevertheless I will be testng it with 'used' but 'new' NMC 3 tomorrow.

Thanks,
Zbyszek

[klosz007]

An update.

I contacted Schneider support, they confirmed RBC was still covered by warranty and offered a free RBC replacement.

It arrived and I replaced it on Monday. Unfortunately in order to achieve full 100% charge on the new RBC I had to wait for almost two days. It reached 98% relatively quickly but from 98% to 100% was like another 24 hours. Not sure if this is normal or not. Without full 100% charge NMC refused to start calibration test.

Then I ran calibration test with 50% (Watts) load (5x100W traditional bulbs) so above 30% recommended level. It took 15 minutes to discharge the 'new' RBC.

Tomorrow I will run another simulated AC failure test and I will see if RBC replacement helped with my problem.

[klosz007]

Unfortunately RBC replacement did not help. Sadly as I expected, given the fact that calculated runtime (after calibration) did not reveal any signs of significant RBC wear on the original RBC. It must be something with electronics then.

After consulting it with support again, they are going to ship a new UPS to me.

I executed similar test with another SMT1000I (10 year old) and it behaves perfectly well.

After 30 minutes of complete shutdown, when AC returns again, RBC charge and remaining runtime are exactly where they were before shutdown (let's say 84% and 5 hours).

On the problematic SMT1500IC, during 30 minutes of full shutdown RBC charge drops quickly (by 20%) and remaining runtime drops to 2 seconds. However it returns to pre-shutdown levels when switched output group 1 turns on.

I'm waiting for new SMT1500IC to arrive, replace it and will retest.

[klosz007]

This seems to be some systemic issue with these "C" UPSes...

Both the original SMT1500IC, the replacement SMT1500IC they sent me and 6 year old SMT1000IC I was able to obtain for testing have the same issue. I have one more SMT1500IC at my colleague's company office but he's travelling abroad so I cannot test it now.

Only my 10 year old SMT1000I is not prone to this behavior.

When:

AC power goes off (power outage)

All UPS outlet groups are sequentially turned off (it can be PCNS+NMC but even without NMC installed and with Load Shedding configured then it is still the same problem)

UPS turns itself completely off (LCD goes off, NMC powers down if installed) - fixed 10 minutes after all outputs are off

(at this time battery charge is let's say 85% and Runtime Remaining is let's say 1 hour)

And now when AC outage continues and lasts sufficently long (10 minute not sufficient, 30 minutes is sufficent) and AC power returns, then UPS 'forgets' its Reaining Runtime and resets it to 2 seconds.

If you have Min Return Time configured on outlet groups, then UPS will refuse to start powering on the outlets.

One has to change Min Return Time to 0 (either via LCD or NMC if present), then UPS will sequence powering on the ouptuts.

UPS starts displaying 'correct' Runtime Remaining again (the value that was there before LCD went off completely, such as 1 hour) only when Outlet Group 1 eventually gets turned on by sequencing.

Thids seems to me like some design issue, firmware problem (but these UPS'es I have tested have different IDs and different (latest) firmwares for their IDs) and all C's are affected the same way.

Only my very old SMT1000I (non-C, without green Ethernet port) does behave correctly in such scenario. Even if it is powered down for an hour or two after LCD went off, when AC returns it still remebers what was the Runtime Remaining (say 1 hour) and sequentially enables outlets, even if Min Return Time is not set to zero. It does not reset Runtime Remaining to 2 seconds if outage lasted too long.

I'm surprise no one noticed this problem before. Test your UPS'es. You do not have longer power outages ?

[Teken]

Default this new system back to a factory state. Do not connect anything to outputs of the UPS.

Insure the NMC is not installed while this test is performed.

Make a video of this problem and relay a link for review. 

Call out the UPS ID / Firmware for this system under test.

Questions Ask . . . 👍

[klosz007]

I have tested three UPSes:

SMT1500IC (ID1060 UPS 2.0) - 1,5 year old (it's where I first noticed this), new battery sent by APC (recalibaration run after replacement) but new battery did not help

SMT1500IC (ID1027 UPS 18.0) - brand new with brand new battery, sent by APC/SE as replacement to the one above (will be returned since same problem as with the original UPS so original UPS not at fault)

SMT1000IC (ID1015 UPS 18.0) - 6 year old with 6 months old battery.

All three have the same issue.

Only on my 10 year old SMT1000I (ID18 UPS 15.1) I was not able to recreate this problem so far.

NMC is not needed for the issue to happen. Whether it is installed or not then same issue.

Configure from LCD menu Main Outlets and on Outlet Group 1 as follows:

Power Off Delay = 300s

Reboot = 8s

Power On Delay = 300s

Min Return Time = 600s

Additionally for Outlet Group 1 set Load Shedding to power off when Power Failure lasts longer than 180 seconds.  For Main Outlets set it to 600 seconds.

Battery fully charged and some load connected to outputs (I used 60W bulb).

Disconnect AC power (simulate grid failure).

Outlet group will switch itself off after 3 mins (by Load Shedding) from grid failure, Load Shedding will switch off Main group  after 10 mins from grid failure. (It seems like without NMC it is not possible to disable "Skip Outlet off delay" option - delay is always skipped if NMC not installed).

Before Main output is cut off, read from the menu battery charge level and remaining runtime (in my case it was let's say 85% and 2 hours - load was rather low ).

After another 10 minutes UPS will turn off completely (LCD goes black).

Wait. I waited for 30 minutes during testing to recreate the issue. If I waited like 10 minutes, issue did not happen.

Reconnect AC power.

UPS will start counting down to enable Main Outputs starting from 300 seconds (Main outlets startup delay).

When problem is encountered it reaches around ~270 seconds, then resets back to 300 seconds and stops.

Go into the menu and read battery charge level and remaining runtime. In my case it would be much lower percent than recorded before UPS shutdown (so let's say 65% rather than 85%) and remaining runtime would be "<1min" (from LCD below 1 min is always "<1min"; if NMC is installed the in NMC GUI it will be more precise such as 2 seconds).

(Here you could wait for hours for battery to reach near 100% but remaining runtime will slowly rise to only like 5 seconds and outputs will never get enabled.)

Via menu change Min Return Time for both outlet groups to 0s.

UPS will now starting counting down from 300s, will enable Main outlets after 300 seconds, then Outlet group 1 after another 300 seconds.

Only when Outlet group 1 gets enabled, battery level will instantly return to the ~value that was recorded before UPS got turned off (so 85%) and Remaining Runtime will jump from 2 seconds to what is applicable to the connected load (so let's say 2 hours, 24h if no load etc.).

As said, whether NMC installed or not - the same problem so it's not NMC-dependent.

I described all of that to APC support and they will be investigating/escalating it internally.

[Teken]

You literally ignored and didn’t follow the action items requested?!?! 🤦‍♂️

[klosz007]

Hmm, I have provided all information required to recreate the problem, incl. ID's and firmware versions.

Not sure what else is needed ? A video ? What is this going to add to the whole picture (unless a hard proof if you just do not believe me it happens ?). 

APC support did not ask for videos and thanked me for very extensive description of the problem (they received less information than my posts here). They will be investigating it deeper but not ETA for solution. For sure it's not a hardware fault since three different pieces have exactly the same problem. Hopefully it's only a firmware bug that can be corrected.

I spent a couple of days investigating and the result is still the same. If it is powered down for too long, remaining runtime is reset to 2 seconds and UPS refuses to turn the outlets on if Min Runtime Remaining condition is set. I'm just too tired with this issue to record video to prove the issue is there. It would probably be an hour long to record the whole set of events.

My last post was just FYI - check you UPS because it may be also affected and I consider this as serious issue. I no longer expect to get any solution here since I did not get an answer so far and even APC support was not able to provide it without further investigation.