Smart-UPS SMT-1500IC very low runtime remaining

[klosz007]

Hi,

The following scenario:

Power to the UPS is cut.

Outlet group 1 is switched off after 20 mins by PCNS running on server (Proxmox) host (which shuts itself down).

Then after another 10 mins main outlet group (Ethernet switch etc.) is switched off by load shedding timer (set to 30 mins after power failure).

At this point NMC 2 says that battery charge is at 65% and runtime remaining is ~2 hours. Eventually UPS turns off, display goes off.

Everything works as expected to this point.

When power is restored, UPS wakes up, UPS initially starts counting down power on delay of 180 secs (UPS Turn On by SYS in...).

But after a while it stops and counter returns to initial 180 seconds.

When I connect to the NMC then, it says that output power cannot be turned on because "The battery power is too low to support the load".

And indeed, on the UPS status page, it says that battery charge is 40% and remaining runtime is just 2 seconds (!). Main outlet group is conifgured to enable output when UPS has sufficient charge for 600 seconds.

There are two things I do not understand here:

- how is this possible that when main output group was powered down, battery charge was 65% and when input power got restored and I could connect to NMC again, battery charge is only 40% (what ate 25% of battery when outlest were powered off and there was no power on the input ?)

- how is this possible that with 40% charge, remaning runtime (with both outputs powered off) is 2 seconds. Note that with full battery charge and full load (main output ang group 1 powered on, runtime is around 1 hour, with group 1 powered off - around 2 hours).

Eventually, at certain battery charge level, remaining runtime goes back to 'normal' value (such as 24 hours with no load) and outputs get enabled then. 

I ran battery calibration with artifical load (two bulbs) and it completed successfully but did not help with this issue at all. UPS was purchased December 2023 and battery pack is still the original one.

Dying batter pack is the only idea I have for now but to my taste it's far too new to be worn out.

Thanks,

Zbignew

[Teken]

The average service life of the RBC is 2-5 years based on usage, temperature, and environmental conditions.

The cells are designed to endure 260 full cycles before degradation.

Based on the limited information you have provided the RBC is near EOL. Also, the system is (NOT) properly calibrated to know the actual operational runtime.

As the values you have stated are the mythical default values of 24 hours. The Battery calibration must be performed with at least 30% or greater load and based on your reply this wasn’t the case.

As it relates to percentage of charge this value and operational runtime are estimates and are not based on any real time and factual data.

SLA chemistry based cells do not provide the same output curve or can sustain a DOD below 50% vs Lithium based cells.

Questions Ask . . . 👍

[klosz007]

Hi,

I used to have worn out batteries and the behavior was different - they discharged much quicker than NMC (and me) expected. That's not happening here. Drop from 100% to say 60% during 30 minutes of my load is perfectly expected.

In another SMT-1500IC original batteries are 3 years old and still good. These batteries here are like 20 months only. Even 3rd party CSB batteries typically survive 3 years.

What surprises me that batteries (as per NMC) keep discharging when outlets and UPS are already turned off (and they do it then actually much quicker than when outlets are on). I'm going to experiment a bit more but that makes me suspect that UPS might be faulty.

24 hours of runtime is shown only when outlets are on but with no load connected. With load connected, main outlet group enabled and switched group 1 disabled it's 2,5 hours, with both outlet groups (my full load) enabled - it's around one hour.

I executed some more test and the statement I made previously is incorrect.

What happens is that:

When power returns, NMC claims runtime is 2 seconds even if I wait for that long so that batteries are fully recharged to 100%. Still at 100% UPS refuses to enable outlets because these are configured to enable when runtime is sufficent for 600 seconds (and two is less than 600).

Available runtime updates to 'expected' value only when I manually reconfigure required runtime on the outlets to 0 seconds. Then main outlet gruup gets enabled first (0 seconds is less than 2 seconds) and out of the blue, once it get enabled after power on delay, available runtime changes from 2 seconds to 2,5 hours !

Another suspicion I have is that, becuase the NMC 2 card is 10 years old (I bought second hand one), the battery on it got discharged (not sure what this small button battery is used for).

[Teken]

You’ll need to provide the following logs for review as no factual data has been given. As such there is no reference baseline to form an root cause analysis.

Provide the following logs in their entirety: via the NMC: Event, Data, UPS, Diagnostics.

Once the logs have been provided the next steps are to complete a static / dynamic test and document the results.

• The coin cell is to retain RTC, Logs. In older NMC versions the coin cell could be removed and replaced from the holder. Later on the cell was tac welded in place due to some make belief guise of safety?!? 🤦‍♂️

If you’re very careful and handy you can remove the tacked cell and replace with the same high quality one. 

Questions Ask . . . 👍

[klosz007]

Hi,

I have run another test yesterday.

I reorganized cabling a bit so that Ethernet switch does not get powered down when main outlet group goes off (so that I can see what's going later on via NMC).

UPS switched off both outlet groups in sequence as expected, then it ran for another 10 minutes dispalying "Waiting for AC". At this point NMC was still powered on from battery pack and it said that remaining runtime was 24 hours (no surprise with all outlets off, prior to switching off main outlet group it said it was like 2,5 hours). Battery charge was like 80% at this point (due to lesser load it lost only 20% of charge when running on battery).

Eventually, after these 10 mins, UPS LCD went off and contact with NMC was lost.

Then I waited for like 30 minutes then and restored AC power to UPS.

Then same story again - NMC claiming remaining runtime 2 seconds and charge level only 55%. 

So during that 30 minutes when display was off, battery lost another 25% (from 80% to 55%) and remaining runtime dropped from 24 hours to 2 seconds.

Obviously UPS refused to power on the outlets again (insufficiuent runtime).

At this point I cut AC power, removed battery connector at the back and removed NMC.

I measured voltage on NMC button battery and it was like 3.1V so battery seems to be good, even though NMC is 10 years old (as I read BR2032 batteries were designed to last up to 15 years in contrast to CR2032 which are supposed to work for 5-10 years). Yes, battery is soldered to the PCB but for me it would be no problem to replace it if needed.

Then, I decided to reconnect battery without reinserting NMC and enabled AC power again.

To my surprise UPS started counting down to enable Main outlet group. So the issue with 'insufficent runtime' was gone when UPS got powered on without NMC inserted.

As far as I remeber, I let UPS enable Main outlet group after 180 secs (there was nothing connected to it anyway) but I cut AC power again before Outlet Group 1 got powered on (group with NAS servers, Proxmox server etc.). I disconnected battery again, reinserted NMC, reconnected battery, reconnected AC.

And UPS started again counting down to power on Main outlets.

I connected to NMC and found out that battery charge was at 85% again and remaining runtime was again 24 hours.

The small mistake I made was allowing UPS turn on Main outlet group before reinserting NMC. Forcing UPS to enable Main outlet group (by setting required runtime to 0) was what healed the situation previously, without removing NMC.

Long story short, some kind of magic is happening here.

I decided to buy cheap (100 USD) second-hand (untested, from return) NMC 3 and will give a try.

If 'new' NMC 3 works fine and issue is gone then probably old NMC 2 was at fault.

If either 'new' NMC 3 is bad or the same issue is still there with NMC 3 then possibly UPS is at fault and I will have to contact Schneider support (UPS is on warranty).

If someone wants to review attached logs and help me, you are more than welcome.

Thanks !

[Teken]

After reviewing the logs I'll affirm once again some facts that you need to accept and understand and take next steps as outlined listed below. The average service life of the RBC is 2-5 years based on Usage, Temperature, and Environmental Conditions. 

Your hardware was produced in 38 week of 2023 so you just missed the 2 year warranty coverage for the RBC. You can always submit a RBC warranty claim to see if they will provide you one as asking is free. 

The event and data logs show you have dirty power present for the short period of time the logs are available. Some of this could be due to you testing the unit with a Lights Out / Grid Down event and battery calibration. As the voltage, frequency for the period covered are fine and well within the accepted margins.

The data logs do not show the RBC at 100% SOC and assume you exported the logs before the system was fully charged.

This stands out because the last reading was 98.53% SOC, 26.78 VDC, and the internal RBC temperature was climbing to 38.XX plus 'C. 

Self Test: The self test is a very crude diagnostic test performed by the system which places a defined load on the RBC. It also completes a basic POST on key components within the unit such as communications, hardware, and electrical. It can not be relied upon to determine if the RBC is actually fine given the crude testing methodology used to determine the RBC state.

By the time the Self Test actually declares a RBC as being bad the cartridge was long expired. You can search this forum and the Internet to affirm the same as to the validity of the self test. 

The gold standard used by serious companies measure voltage, resistance, and conductance which is referenced to the battery makers data sheet at a given ambient temperature. All serious lithium based systems use a BMS to monitor individual cells for over / under voltage, temperature, balance, current, and internal resistance, SOC.

Lithium based cells are designed to endure 100% (DOD) depletion over thousands of cycles unlike a sealed lead acid chemistry based cell that's used in this UPS.  

As stated earlier to identify and narrow down possible root cause requires a combination of Static / Dynamic testing which requires basic to advanced tools and time . . . Both tests can also be performed in a simple manner but leaves a large margin of error / unknowns if the proper diagnostic tools are not available / used to validate the same.

A basic static and dynamic test can be conducted on the system now without any extra tools. But, again the results are not to be taken as gospel if other factors are not taken into consideration as doing so will provide you with false positives / false negatives!☝️ 

Static Test: Disconnect the RBC Anderson connector from the system. The system should continue to run and declare a battery disconnected fault / error. If this is true great - if not and the system immediately shuts down or acts erratically there is an internal hardware fault.

Keeping in mind if there is dirty power present this will result in a false positive. As the system may try to switch to battery only power vs AVR mode and if the RBC isn't present it will obviously shut down! 😖 

Dynamic Test: Insure the RBC is at 100% SOC and there are no power quality issues present. Disconnect any communication cables prior than the AC Mains from the system. There must be 30% or greater load connected to the units outputs. The unit should immediately declare a loss of AC Power and switch over to battery only mode.

If the system operates as outlined above - great . . .

Now time how long the system continues to operate for before it shuts down. Compare this operational runtime with the APC Runtime Calculator. Keeping in mind this is based on a new RBC with a defined load at a specific ambient temperature.

If the system turns off immediately or provides only 1-2 minutes of operational runtime - Replace RBC. If the operational runtime is half of what is expected its time to order a RBC for replacement as its nearing EOL. 

NOTE 1: A dynamic test or one that mimics a (lights out / grid down event / battery calibration) is extremely tough on the RBC and goes against the 260 cycles the cell is designed to provide. As such a battery calibration should only be done upon first RBC replacement and on a annual basis to validate the state of health of the cartridge. 

The correct steps and procedure during a RBC replacement are as follows:

Install Date: Update the battery installation date as this resets internal counters and adjusts charging tables if applicable via LCD Menu, NMC, PCSS.

Self Test: Manually initiate a self test via LCD Menu / NMC / PCSS as this allows the system to complete the POST and load test the RBC to affirm operating state.

Battery Calibration: This procedure must be done with a minimum of 30% attached load or higher on the outputs. Doing so will allow the system to learn the capacity of the RBC. This will help the system reflect the operational runtime more accurately on the LCD, NMC, PCSS, SNMP.

Keeping in mind the values displayed are estimates based on a crude table and do not reflect reality. 

Dynamic Load Testing:  This is why a dynamic load test is performed, timed, and compared to the APC Runtime Calculator. To provide a baseline and gauge as to operational runtime as referenced up above.

NOTE 2: The most accurate method to determine a cells state of health is to measure the internal resistance and measuring its conductance with a calibrated and certified meter and referencing the information the battery makers data sheet based on a defined ambient temperature.

Measuring the voltage of the RBC or single cell is not to be relied upon!

Regardless of the above a crude method if advanced tools are not present is to measure the voltage at the Anderson connector of the RBC while connected to the system. This will measure the float voltage from the UPS and documented.

Next step is to disconnect the RBC and measure the surface charge which will be the same as the float voltage until it dissipates - write it down. Leave the RBC disconnected for 72 hours and measure the voltage. A drop in 0.1 ~ 0.3 VDC is normal and expected.

A drop in 1 ~ 3 VDC indicates a marginal to failing RBC . . 

More often than not you will find one or more cells that are dragging down the pack. Any obvious signs of bulging, cracks, leaks, corrosion indicate a failed cell. Any cell(s) that are hot are failed along with any cells that drop voltage in the 1-3 VDC range.

When the system is connected to a NMS you can easily see several metrics that indicate a failing RBC / Charger fault. As you will see rising battery temperatures, erratic float voltage, and operational run time counters that drop from established baselines.

Any cheap carbon pile load tester can be used for validating individual cells keeping in mind this goes against the 260 cycle life while doing so. A load test must be referenced to the battery makers data sheet as to load and duration and recovery.

Questions Ask . . . 👍 

[klosz007]

Hi,

Thanks for such deep insight into my problem !

Answers to some of the items:
I checked SmartConnect and it says UPS 3-year warranty expires on 21 Sept 2026 so if RBC is covered by 2-year warranty then I still have a few days left on RBC warranty.
Thanks for information anyway, I thought warranty for RBC was 1 year only 🙂

"The data logs do not show the RBC at 100% SOC and assume you exported the logs before the system was fully charged."
Very likely yes.

Self test was successful but as you said it is very basic. I had cases of RBC dying where self test was successful 🙂
I did not know such thing as Static Test exist. Will try it.

Instead I ran Calibration test. I disconnected all 'computer' loads and connected two tradiitonal bulbs (rougly 200Watts total) instead.
I do not remember how long it ran but something around 30 minutes so plausible for good RBC and that amount of load.
It did not end very quickly such as 5 minutes.

I did not satisy 30% load though, rather 20%. Probably I would need at least 300+ Watts.

For now nothing tells me that RBC is at fault.

When outlets are already switched off by sequencing but UPS still displays 'Waiting for AC power' on LCD and NMC is still powered on, remaining runtime (as reported by NMC) is 24 hours (no surprise with outlets off).
When outlets were still on the runtime was like 2 hours.

Problem begins when when UPS turns off completely waiting for AC power and LCD goes black (this happens 10 minutes after all outlets got turned off).
When UPS is left in this 'dark' state for a while (I tried like 30 minutes) and AC power eventually returns, then NMC claims then remaining runtime is only 2 seconds and RBC SOC is much lower than it was before UPS got self turned off.
Exactly as if somewthing was draining power quickly from RBC when all outlets are off and UPS/LCD is off.
UPS refuses to turn the outlets on then (insufficient runtime - outlets are set to be turned on when runtime exceeds 600 seconds).
But when I force the outlets to get turned on (by changing required runtime to 0), then out of the blue, NMC returns to displaying remaining runtime from 2 seconds to 24 hours (I think with no load the maximum value it can show is 24 hours).
If load is connected then obviously remaing runtime is lower (depending on load it can be say 2 hours).

Nevertheless I will be testng it with 'used' but 'new' NMC 3 tomorrow.

Thanks,
Zbyszek

[klosz007]

An update.

I contacted Schneider support, they confirmed RBC was still covered by warranty and offered a free RBC replacement.

It arrived and I replaced it on Monday. Unfortunately in order to achieve full 100% charge on the new RBC I had to wait for almost two days. It reached 98% relatively quickly but from 98% to 100% was like another 24 hours. Not sure if this is normal or not. Without full 100% charge NMC refused to start calibration test.

Then I ran calibration test with 50% (Watts) load (5x100W traditional bulbs) so above 30% recommended level. It took 15 minutes to discharge the 'new' RBC.

Tomorrow I will run another simulated AC failure test and I will see if RBC replacement helped with my problem.