Disable Outlet Group Shutdown

[aviator4460]

Hello! We've had some unusual behavior from a UPS in one our buildings where when a power outage occurs the device seems to switch to battery power normally, but when power is restored it will send commands to turn off the Outlet groups which then powers off our devices unnecessarily. I'm not sure if there's something more at play or if there's a misconfigured setting. At best I think I'd like to just disable the outlet command functionality if needed.

I did capture the last series of logged events during the outage (most recent at top):

07/06/2024	12:29:23	System	Network service started. System IP is 10.75.*.* from DHCP server 10.75.*.* with 691192 second lease.	0x0007
07/06/2024	12:24:29	Device	UPS: Restored the local network management interface-to-UPS communication.	0x0101
07/06/2024	12:24:10	System	Network service started. IPv6 address FE80::2A29:86FF:FE25:C694 assigned by link-local autoconfiguration.	0x0007
07/06/2024	12:24:00	System	Network Interface coldstarted.	0x0001
07/06/2024	12:10:10	Device	UPS: A low battery condition no longer exists.	0x0110
07/06/2024	12:10:10	Device	UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately.	0x0107
07/06/2024	12:10:10	Device	UPS: The output power is turned off.	0x0114
07/06/2024	12:10:10	Device	UPS: The graceful shutdown period has ended.	0x014F
07/06/2024	12:10:10	Device	UPS: No longer on battery power.	0x010A
07/06/2024	12:10:09	Device	UPS: Switched outlet group 3, Outlet Group 3, has been commanded to shutdown with on delay.	0x0174
07/06/2024	12:10:09	Device	UPS: The power for switched outlet group 3, Outlet Group 3, is now turned off.	0x0135
07/06/2024	12:10:09	Device	UPS: Switched outlet group 2, Outlet Group 2, has been commanded to shutdown with on delay.	0x0174
07/06/2024	12:10:09	Device	UPS: The power for switched outlet group 2, Outlet Group 2, is now turned off.	0x0135
07/06/2024	12:10:09	Device	UPS: Switched outlet group 1, Outlet Group 1, has been commanded to shutdown with on delay.	0x0174
07/06/2024	12:10:09	Device	UPS: The power for switched outlet group 1, Outlet Group 1, is now turned off.	0x0135
07/06/2024	12:03:36	Device	UPS: The battery power is too low to continue to support the load; the UPS will shut down if input power does not return to normal soon.	0x010F
07/06/2024	12:03:30	Device	UPS: A low battery condition no longer exists.	0x0110
07/06/2024	12:03:30	Device	UPS: The battery power is too low to continue to support the load; the UPS will shut down if input power does not return to normal soon.	0x010F
07/06/2024	10:09:07	Device	UPS: On battery power in response to distorted input.	0x0109

12:10 seems to be when power is restored but the outlet shutdown commands happen anyway.

I also collected logs from another UPS that recorded the same power outage, for the same length of time. They have identical configurations, but this one did NOT issue an outlet shutdown command:

07/06/2024	12:24:15	Device	UPS: No longer on battery power.	0x010A
07/06/2024	12:22:44	Device	UPS: Restored the local network management interface-to-UPS communication.	0x0101
07/06/2024	12:22:35	Device	UPS: On battery power in response to none.	0x0109
07/06/2024	12:22:35	Device	UPS: Restored the local network management interface-to-UPS communication.	0x0101
07/06/2024	12:22:14	System	Network service started. IPv6 address FE80::2A29:86FF:FE5F:A877 assigned by link-local autoconfiguration.	0x0007
07/06/2024	12:22:08	System	Network Interface restarted.	0x0002
07/06/2024	10:09:28	Device	UPS: On battery power in response to low input voltage.	0x0109

Both UPSs are SMX3000RMLV2UNCs. However, they do have different firmware versions. The one that issues outlet commands is FW v 07.4 (ID1003), and the other UPS is v 08.1 (ID1003).

Has anyone seen this behavior before? Thanks in advance for any input!

[Teken]

You’ll want to review the (Read Me) file as to what changes / updates have been made in the latest 15.XX UPS Firmware.

It goes without saying if the unit is still under APC Warranty. You’ll want to contact APC Technical Support to walk you through the upgrade process! ☝️

Should anything go side ways they may assist in a break fix / RMA.

If the hardware is NOT in warranty you have a better than 50% chance of bricking the unit! 🤦‍♂️

As such ensure you follow all the best practices outlined in the APC Firmware Upgrade FAQ. 

As it relates to the current problem you’ll really want to validate how PCBE / PCSS is programmed relating to outlet management / load shedding if enabled.

If the batteries / cartridge are more than 3-5 years old. They need to be inspected (physically) and via measurement of voltage, resistance, and capacity.

If you complete a manual (Lights Out) test to mimic a black out. You’ll want to compare the operational runtime against the APC Runtime Calculator based on a similar load / wattage.

If the expected runtime is 60 minutes but you only see 5-10 the cartridge is due for a replacement! Obviously, if the unit shuts down immediately or within a few minutes it needs to be replaced ASAP! ☝️

As it relates to programming ensure this specific unit is defined to turn back on when AC Mains is restored. Keeping in mind if the battery is shot or teetering on dying it will cause the unit to react and behave erratically regardless of the programming because the load can’t be sustained by the current battery state of charge.

Meaning, the battery is too low message (active fault) overrides everything else and will cause the outputs to turn off / unit to shutdown.

[dpierce4776]

@aviator4460 From these logs it looks like you had power coming and going (flickering).

as @Teken stated I would check and calibrate the batteries. You can do this by going under Test > UPS > Runtime calibration.

Also from your logs it appears that the UPS lost building power. ran for a while and when the batteries got low shut the outlets off and then shut itself down. Just and FYI depending on your settings, this is a process started but not controlled by the runtime the UPS thinks it has. Which is why calibrating is important. In other words once the process is started unless power is restored it will follow all the steps regardless of runtime unless power is restored to the unit.

At 12:10  we do see the message that the message appears that "A low battery condition no longer exist" The issue there is that right before that we see "Output power turned off" In other words The UPS turned off its outlets. This causes the UPS to have no load and think it can now last longer on batteries. This does not mean that power was restored, and as I said before since the power was not restored then the process of shutting down continues.

When power is restored you will get a message saying "No longer on battery power". 

[aviator4460]

Both of the units in question, I believe, are 10+ years old, and I am unsure of the last battery replacement so they could very well be due.

I am planning on some self-test and runtime calibrations when I can schedule a maintenance window and we'll see if that helps. But I agree that the batteries may simply be due for replacement as you are both correct that dying batteries cause weird symptoms in UPSs (or at least in APC UPSs lol).

I'm pretty sure neither is in warranty and I've never done a firmware upgrade of the UPS itself, so I'm going to take that off the table.

Thank you both for your input!