Power on Group-Delays fail when AC power is restored

[getgray]

MODEL : Smart-UPS X 1500
FIRMWARE : COM 03.8 / UPS 03.8
USB to Linux Server

No NMC in this model avaialble AFAIK.   Problem:

AC fails
UPS directs server to shutdown - works
Final part of shutdown, server tells UPS to shutdown - works
   I understand apcuspd is using  --killpower command
UPS gets command and starts group shutdown sequence.  Turns off group 3, then group 1, then 2, as configured with timedelays - works

UPS powers off.
AC restored
UPS comes on, After just a few seconds, turns on all 3 power groups simultaneously.  - FAIL.

If I power down from control panel, the power on from control panel all groups sequence correctly.  Only when it has been shutdown via USB command does it fail.  I've debugged this till my plugs are sore and cannot find a workaround.

I mean this is a fundamental feature of the model.  Sequenced outlets.  And it has USB comms because that's what every user on the planet has available.  I'd even do RS2322 if it helped.  It's crazy that it behaves this way.  I've dug for a solution and I can't tell that one exists, hoping to find something different here.

Anyone know of a fix?  OR someone, some place that could help?  Without adding 3rd party solutions like network controlled sequence strips.  Power is down, network is off, time to gracefully shut everything don and be able to gracefully bring it back up, which in my case means sequentially.

Thanks,

Scott

[Teken]

This sounds more like a application problem vs hardware. Question is has it ever worked successfully with APCUPSD?

If not, test and validate how the unit behaves using Power Chute (PCSS) and NUT.

Theres better odds one or both applications will operate as expected as both are actively developed vs the other has been abandoned years ago.

If both fail to operate than it’s more likely a UPS firmware issue as your unit is sixteen years old and stuck on the last release.

Which doesn’t incorporate more than a dozen bugs & fixes.

Let me know the results using the two alternate software solutions.

Questions Ask . . . 👍

[getgray]

Maybe buy I contend its the UPS.  I had opened a case and support confirmed it was expected this morning basically said "sorry, know it's wrong but not gonna be a fix".  In trying to find the solution I've found several occurrences of it out there, not just on my model, others with same issue.  

FWIW I came to to add for posterity...
When the UPS gets shut down via software signal, it goes into a different off state.  It's front panel gives a Grp1 Stayoff bu USB for: [and a seconds countdown].  At the end of that the LCD turns off but, the 4 LEDs under the power button blink in a left right left etc pattern and stay that way.  Unlike powering off via the power button when it goes off.  So it's in a sleep or hibernate state I'd venture.  Coming out of that state it wakes poorly.  I have not tested what happens i fth ebattery run down completely.  That might clear that hibernate state and it wake up properly. 

[getgray]

Forgot to add, I installed NUT and tried it, it made no difference, same behavior.  Can't use Powershute as the base OS is slackware (Unraid 7.3 OS).

[Teken]

Does the unit have a NMC? I try to see if the test units in the lab behave the same way on both v3.8 vs v15.X firmware from different production years.

Questions Ask . . . 👍

[getgray]

Apologies missed your reply.  No I do not have the NMC.  And I cannot get the RJ50 serial port to respond either.  Tried a known good PC with a serial port I use for FW upgrades, tried the bran reset (remove AC, remove battery, let it drain, restore), I did not have the UPS cable attached, but I can get nothing out of that port, nto even noise for a terminal.  So USB is all I can get to communicate right now.  

[Teken]

Hi,

I was able to book sometime in the lab to see how the SMX Series with different production years and firmware behaved.

Short Version: The problem you raised is valid and this seems to impact older legacy UPS ID 11 with 3.8 firmware and newer models that span 2012 ~ 2022 that I have in the lab. All of the newer SMX Series span 750, 1000, 1500, 2000, 2200 KVA running the latest UPS Firmware 15.X

Long Version: I completed a few basic and simple validation tests which encompassed the following in no specific order of importance or relevance.

The outlet delay was programmed via front LCD, NMC, Power Chute to see if there was an inherent difference in behavior.

There were no differences using any of the above methods to program the Outlet delays.

Next, I connected three loads to the rear of the machine for confirmation as to activation / power on / power off status. I then disconnected the RBC on a UPS running 3.8 firmware. The power was removed using a power bar. I waited approx. a minute to insure the capacitors were depleted in the system. I then restored AC power via the power bar.

The unit powered up and all of the front status LED pulsed on / off. The rear outputs pulsed on / off with the front LED?!?! In this state with no RBC present which also mimics a real world dead RBC. This condition would cycle power to the connected output endlessly.

Obviously if this was a server corruption of the OS / Data set is the outcome. 😖👎

The following test had the RBC connected which was 100% charged. Turning the system back on via the power bar and the 100% charged RBC. The system immediately activated all the outputs as I could see all three test lights turning on immediately with no delay. 🤢 

The same tests was conducted on several newer SMX Series UPS that span 1000, 1500, 2000 KVA. All of these units are running the latest 15.X UPS firmware. They span in production year of 2012 ~ 2021. Performing the same identical tests of no RBC connected had a different result.

Upon AC power being applied via the power bar the outputs never turned on or cycled (pulsed). The front LED Status only briefly came on and remained unlit. The outputs never came on with the RBC not connected to the system.

Testing the various systems with the RBC in place and then applying AC power also did no cause the outputs to come on in a staged manner??? 😡

Using the front lcd to invoke a delay shutdown and restart did in fact display a countdown timer and toggled between the various outputs. At that point the outputs did in fact come on in a staged manner?!?! 🙄

So this problem isn't related to UPS 3.8 firmware . . . This problem exists on the latest generation of SMX Series using the newest UPS firmware of 15.XX. I've sent a group mail to the entire IT Department to ask those site holders to confirm if their units do stage (delay) the outputs when powering up.

This problem obviously is (NOT) related to USB . . .

As an aside I have seen many SMT, SMX, SRT, SURT, SURTA, ULTRA'S systems that did in fact operate with a staged (delay) output. So, I am at a loss as to why this problem has been ignored for so long. Given APC Technical Supports reply to you they know this problem exists and obviously refuse to fix an obvious production bug!

Lets see if someone from SE / APC care enough to investigate and release a new UPS Firmware Release to address this issue! For you who is on 3.8 firmware there is only one solution assuming SE / APC ever addresses this problem and that is to purchase a newer or used model built after 2013.

As they can accept the new firmware vs the 2011 hardware is locked at 3.8X

Questions Ask . . . 👍