Firmware Bug in Back-UPS BX1300G Causes Reboot Loops

[Anonymous user]

Hi,

I have two APC BX1300G UPSes, and for the longest time one of the two units would always go into a "reboot loop" any time the utility power was gone for longer than I had configured the shutdown period in the software to be.  So long that the utility power would leave, and then return before the timeout was up, the UPS would hold the load up. This was incredibly frustrating, as the unit with the looping problem was attached to my Linux machine and it would be halfway through booting up, when the 1300G would decide to go on battery again (even though utility power was clean and available) and then subsequently drop the load, triggering a filesystem check once the UPS had finally "stabilized" itself enough to continue running. The 1300G hooked to my Windows machine would have no such issues. So, I did what any logical person would do, and swapped the Linux attached UPS for the Windows attached UPS, to see if the problem would follow the unit. The behavior was exactly the same, the problem stayed with the machine, and did not follow the UPS. At the time, I chalked it up to some marginal incompatibly with the power supply in the Linux machine and just lived with it, grumbling every time the power went out about "that junk APC" rebooting my box all the time. I could not figure out why my Windows machine had no problems, so I just wrote it off.

Fast forward a good time later, and I happened to be looking at the changelog for apcupsd (the linux software for APC UPS's) and something in particular caught my eye:

r2120 | adk0212 | 2014-03-28 22:31:45 -0400 (Fri, 28 Mar 2014) | 3 lines

Workaround for UPS firmware bug causing killpower to execute repeatedly in a loop

So installed this latest version of apcupsd, and lo and behold my "reboot loop" problems are gone. I could not be happier with the new found reliability in the 1300G I had grown to loathe over the previous months. Of course, I wanted to dig into *why* this fixed my problem.

The basic gist is that there appears to be a bug in the firmware of the BX series of APC UPSes that causes it to re-execute the last command that it received via USB once the USB is "reconnected", either a pulled USB cable is reinserted after the PC is booted up, or by the USB chips getting initialized by the system BIOS (that's right, even before the OS is booted).  If PowerChute on Windows sends the "kill the ups" signal to the UPS due to software timeout, PowerChute keeps on checking the status of the UPS by sending a "check status" operation down the USB cable.  So the power gets killed, and the last thing in the firmwares memory is "check status", a totally benign command.

Now, using apcupsd (previous to version 3.14.12), once the timer has run out and the software sent the "kill the ups" signal, it would send no more operation codes down the USB line.  Everything would shut off as you would expect, but now the last thing in the memory of the UPS is the "kill the ups" signal, and not a harmless "check status" signal.  So, when the power would be restored, the 1300G would come up, the USB lines would get power again (from the PC), and it would execute the last command it received, namely "kill the ups". It would cheerfully do so once again (even with good power), and then drop my Linux box mid-boot, much to my chagrin.

This was fixed in apcupsd version 3.14.12 by mimicking the PowerChute behavior on Windows by sending some additional "check status" signals after the "kill the ups signal", so that the bug that still exists in the UPS would run a harmless command instead of triggering a "reboot loop" like it would before. I'm pleased to report that it now works wonderfully.

So, my question is, has APC looked into this behavior in the BX (and possibly other) lines in regard to the "re-execution" of commands previously sent via USB? While the software is now able to work around the bug in the firmware, it still stands that there is a defect in the product, and I would think that would be of interest to the firmware engineers at APC (or at least I would certainly hope so!) I have always had a lot of faith in APC products, and I would really like to ensure that future products I may buy from them would not have this bug present in the product.

The gritty details of how all this was discovered is at the following link:

Apcupsd UPS control software / Mailing Lists

[BillP]

Hello, sorry for the delay - I was researching.

We are aware of this issue and it is definitely on the radar. We have fixed issues recently revolving around this type of thing it seems from my research but just to make sure we are on the same page with what is fixed when, could you provide your UPS's firmware version (which should be available via Apcupsd)?

Thanks!

[Anonymous user]

HI Angela,

It is good to hear that you've identified this issue internally. Both of the BX1300G's have the same firmware version:

FIRMWARE : 864.L5 .D USB FW:L5

Thanks

[BillP]

Hi again,

We're going to do some validation testing here regarding the issue and what's fixed and what's not and that team doing it has agreed to make sure it is completed between 9/10-9/12 at this time so I hope to report back right around then.

[BillP]

Sorry, I forgot to respond here. I checked shortly after 9/12 to see what engineering found out and they still wanted to do another few tests after they did identify something. They needed to review the USB traffic it seemed like. Will check in the end of this week.

[BillP]

Hello,

Some good news. It looks like the firmware update to fix this issue will be worked in later this year after we confirmed the scope of the issue internally. Thus, by Q1 2015 units should be available via distribution so ultimately, the entire unit(s) would need to be replaced since these Back-UPS are not field upgradable. Or, use the latest version of apcupsd that has a bug fix within it to sort of bandaid the problem via the software.

[Anonymous user]

Hi Angela,

That is good news. For me personally, I'm OK with just using the patched version of the software, as the units work perfectly fine otherwise. I do wish the firmware was field upgradable, but it's not worth doing an RMA considering there is an acceptable workaround.

Thanks for all your assistance.

[Anonymous user]

Has there been an actual firmware update released for the BX1300G now?

I have a bunch of APC units, some have the 864.L5 firmware as well.

I believe I've seen 864.L8 on newer units, but there might be newer too.

[Anonymous user]

Has there been an actual firmware update released for the BX1300G now?

I have a bunch of APC units, some have the 864.L5 firmware as well.

I believe I've seen 864.L8 on newer units, but there might be newer too.

[BillP]

Martin,

The issue has been resolved in BX1500G firmware 863.L6.D that is shipping on new BX1500 now. The issue will be corrected in BX1300G firmware 922.L9.D that should start shipping on BX1300Gs in July. 

Unfortunately the BX model UPS do not have upgradeable firmware. 

[Anonymous user · Accepted solution]

That's too bad, I have a BX1300G that was signaling my PC to shut down via the USB cable

every time there was a quick power outage. After shutting down the UPS and removing

the battery, it works fine again. Probably until the next full battery drain power outage.

I guess there wouldn't be any way of fixing that, replacing f/w chip, UPS board, jtag, etc ...