UPS Test - Runtime calibration

[hsyip]

Hello, 

We did a UPS Test on Runtime calibration via NMC web management for SRT10kVA with 10 external Battery Pack. The UPS Self Test passed without issue. However, the Runtime calibration result was aborted and indicate the cause as internal fault. 

We did a check on all the connection there was no issue with the cable connection to each of the Battery Pack. Here is the screen shot and kindly assist in the resolution.

Thank you.

YIP HS

[Teken]

Was there at least 30% load connected while the test was underway?!? Were the batteries 100% charged prior to conducting the load test? How old are the batteries and if they are more than 3 - 5 years old they may be end of life.

[hsyip]

Hello, 

The connected load is more than 30 % and the batteries are 100% charged. The batteries are 1.5 year old and this alarm came about when we performed manual calibration test to check the runtime. During this runtime calibration, the test fails to complete and aborted. Alarms was generated and send out via email.

However, at some instances the runtime test was successful and all was good. 

Kindly advise what was the cause of this intermittent issue. Thanks.

[Teken]

If you review the APC FAQ they call out some conditions which must be met to complete the battery calibration. It also calls out some of the issues that may cause the test to fail. It goes without saying the entire UPS System must be installed correctly per the maker’s instructions such as connected directly to the power source. All units & UPS must be properly grounded and connected to a single point Earth ground. The UPS System must be contained and operated in a room where the ambient air temperature falls within the specifications outlined in the installation guide.

All of the battery packs are securely interconnected with the supplied fasteners and grounded to one another. If the environment is very dusty disconnect, clean, and visually inspect all connectors to insure they are sound and fully mated to one another.

I would gather these basic installation requirements are met. So would look at other possible causes. But, before doing so I would suggest you simply perform a real world lights out test and remove AC power from the wall. ☝️

Doing so will provide you real world results as to how the system operates and the battery condition. You’ll obviously need to track and monitor how long the unit operates until it shuts down. That the UPS does in fact operate as programmed by sending out emails, SNMP Traps, Power Chute shutdown commands that everything works as expected.

If it does you can focus on why the battery calibration didn’t work. As a starter I would insure both the NMC & UPS are on the latest firmware. If not do so, if current verify all UPS settings are correct.

As an aside I can’t see the image you posted so don’t know what the error is.

[hsyip]

Hello Teken, 

Please find here in the status on UPS Test performed manually 

In addition, please find here in the event log for your review and analysis.

Thanks.

[Teken]

Appreciate the log file as it provides a lot of insight especially what the declared fault is:

Runtime Calibration aborted, internal UPS fault.	0x0124

I would reach out to APC Support and request more information as to what this error means along with possible next steps. Please share what this error message indicates as APC refuses to share this basic information to help IT Support staff understand and troubleshoot. 🤦‍♂️👎

In the interim I would stick to the basics and do the following to help isolate and identify the root cause. The goal is to identify if the problem is a UPS hardware issue or simply a battery failure, misconfigured UPS, loose connections, etc.

- Backup the UPS Configuration file.

- Default the UPS to a OEM state from the LCD panel.

- Disconnect all the extended battery packs.

- Run a battery calibration with 30% load attached just with the main UPS. Insure the main system is 100% charged and remove none critical loads during this test.

If the automated battery calibration is successful on the main unit. Connect another extended battery pack insuring its seen by the UPS / NMC. At no time should the UPS declare a cartridge is disconnected once properly connected to the main unit.

This would be a really good time to validate all battery connectors are clean, free of obstruction, and all Anderson connectors are firmly seated when connected (front & back). 

NOTE 1:

The error logs call out lots of disconnected battery cartridge messages. If they were caused by you during trouble shooting no worries. If not, identify which extended battery pack is throwing the error and remedy.

The reason to default the UPS to a OEM (Out of the box) state is the logs show lots of misconfigured settings such as transfer etc.

Performing battery calibrations is extremely time consuming and hard to the batteries! The batteries have a rated 260 cycle life. As such completing such tests reduces the service life of the cells.

NOTE 2: 

A less invasive and industry standard to determine the state of health of a cell is to use a conductance meter / tester. This measures basic to critical metrics such as the internal resistance of the cell. If a new battery is specified to have a 20 mOhm resistance and the meter indicates some crazy value / dead short etc. You’ve identified (quickly & efficiently) that one or more cells are taking down a pack or system. Using this method does not harm the cell or reduce its service / cycle life! It saves time waiting for a pack to drain. It avoids any downtime as you let the UPS run while you remove the hot swap packs and test on the bench. 👍

Given there are 10 extended battery packs you will need to decide what next steps makes sense for you and the company. If we assume you completed all the steps above (reset default ups settings) and confirmed all connections are clean, secure, and fine.

You could perform a real world *Lights Out* test by removing AC Mains from the wall. You would have 30% load connected and let the UPS drain and turn off.

If it offers the stated runtime with the attached load with all 10 packs. This affirms the UPS & Extended battery packs are working. If it doesn’t provide the expected runtime this means there are one or more bad cells present that need to replaced. Keep in mind it’s NOT recommended to replace just a single cell in a pack! ☝️

Either buy directly from APC / Authorized reseller if the Main UPS is still under warranty. If the unit is NOT under APC warranty you can replace the individual cells within a pack. CSB is the primary battery used by APC in all of their systems. Avoid buying 3rd tier no name brands from Amazon etc.

Interested to see what your next steps are and the results of resetting the UPS to a default state and just load testing the main UPS for battery calibration and those results! 👍