APC UPS SMX1500 firmware update failure - and fixed

[BillP]

This question was originally posted by Colin on APC forums on 2/9/2018

---

Hello,

This is posted in the hope that it helps other people with resolving firmware update failures that can otherwise leave the UPS "bricked".

I have a SMX1500RMI2UNC (SMX1500 240v, with network card at f/w 6.5.0), ID=20, UPS was at f/w 9.9, now at f/w 10.1.

Attempting to update from 9.9. to 10.1 through the NMC failed. The UPS was left in a "restart cycle", with LED1 off, LED2 yellow, LED3 blinking red and LED4 permanently red and the message "firmware update in progress"  on the front panel display. Previous updates have worked as intended via the NMC.

Event log messages during the update attempt and failure were as follows (bottom up, most recent at top):

02/09/2018 14:02:39 Device UPS: Lost the local network management interface-to-UPS communication. 0x0102

02/09/2018 14:02:19 System File 'SMT20UPS_10-1.enc' transfer failed. 0x000F

02/09/2018 14:02:19 Device UPS: Firmware transfer to UPS failed 0x0178

02/09/2018 14:02:19 Device UPS: Finished UPS firmware transfer  [Xmodem Cancel] 0x0177

02/09/2018 13:58:21 Device UPS: Transferring firmware to UPS. 0x0176

02/09/2018 13:57:48 Device UPS: The output power is turned off. 0x0114

02/09/2018 13:57:48 Device UPS: The graceful shutdown period has ended. 0x014F

02/09/2018 13:57:47 Device UPS: The power for switched outlet group 3, OG3-C14x4-desk, is now turned off. 0x0135

02/09/2018 13:57:47 Device UPS: The power for switched outlet group 2, OG2-C14x2-ds10, is now turned off. 0x0135

02/09/2018 13:57:47 Device UPS: The power for switched outlet group 1, OG1-C14x2-xw8600, is now turned off. 0x0135

From these messages, it appears that the NMC loads the firmware into the UPS with the xmodem file transfer protocol. The failure appears to be when the NMC transfers the firmware file internally to the UPS using xmodem over the internal serial interface between the NMC and the UPS.

The UPS was left in an unresponsive state. To reset the UPS, disconnect the batteries, disconnect the mains input, remove the NMC and leave the UPS for several minutes. It can also be useful to hold down the power button for 10 to 15 seconds to drain the capacitors etc.

On powering up the UPS, attempting to connect with a USB2 cable was unsuccessful. The firmware update wizard could find the UPS, but could not communicate with it.

Connection through the serial port using a terminal emulator was successful. Port settings are 9600 baud, 8bit, 1 stop bit, no parity, xon/xoff flow control.

On power-up the UPS output was as follows, in a repetitive loop with a beep every restart of the download manager:

APC(R)2011 Download Manager

Version 07.05 - 0014

CCCCCCCCCCCCCCCCCCCC

Error XModem retry count exceeded

APC(R)2011 Download Manager

Version 07.05 - 0014

CCCCCCCCCCCCCCCCCCCC

Error XModem retry count exceeded

APC(R)2011 Download Manager

Version 07.05 - 0014

CCCCCCCCCCCCCCCCCCC

Error XModem retry count exceeded.

At this point, the UPS is "bricked", unless firmware can be loaded using the download manager as a "failsafe firmware loader" mechanism, which requires using the xmodem file transfer protocol over the serial connection.

I used xmodem (included with extraputty, based on the putty terminal emulator) to transfer the firmware file (10.1) to the UPS. This was successful. While the transfer is in progress, the LEDs flash alternately green (led1) and red (led3).

When the transfer completed, the UPS screen went blank with a beep and the UPS turned off.

On power up, the UPS worked, passed the self test and showed 10.1 as the firmware version.

After re-installing the network card, the UPS is back and working, without having lost its configuration data (the UPS specific configuration elements and the network card configuration elements).

The conclusion is that the firmware update process from 9.9 to 10.1 has a fault under some as yet unknown set of circumstances, which can cause it to fail and leave the UPS in an unserviceable state, until the firmware is reloaded through the serial port with xmodem.

Cheers, Colin.

[voidstar_apc · Accepted solution]

This was originally posted on APC forums on 2/10/2018

---

Nice! Depending on UPS, you can often re-attempt the transfer via the NMC. Doesn't look like that was the case with yours but luckily you're handy with RS232.

[ChromeTO]

I just stopped by to say that this worked for me!

I bricked my SMT750RM2U, by unplugging the switch that my laptop was using to send the firmware upgrade to the UPS. Lots of lessons learned, sleep deprivation is not a good thing, don't do a UPS firmware upgrade via the NMC on this UPS. Use the USB cable.

So, what I did to get my "dead" UPS working again. Basically followed the same instructions as above. Note the following:

1. I had to leave the USB cable connected to the computer the UPS seemed more responsive with the USB cable constantly connected.

2. I did have to find an AP 940-0625A cable. This unit has an SERIAL RJ45 port... and the other end SHOULD go into a serial (DB-9) port on a computer/laptop.

3. I tried making my own cable with what I had on hand, and just couldn't get it going, so, I bought one.

4. Once the serial cable was connected, I used Extra Putty for the session. I didn't get anything on the session screen, the UPS didn't want to communicate.

5. I sent the xmodem transfer ANYWAY, with the same firmware I used to update the UPS originally.

6. The "connecting" message just stayed there, it seems like forever, but it was probably more like a minute, then it started transferring.

7. A few minutes later, the UPS was back to life...and all my setting retained. Crazy, but it worked.

Some after thoughts, maybe the cables that I Macgyvered, did work, but I was expecting to see some kind of communication in Putty. There was nothing, so, I assumed the cables didn't work, and never tried an xmodem transfer with my Macgyvered cables.

Thank you @BillP 

[AndyO]

Ok, also saved by this post.

APC SMT1500I UPS - bricked during a firmware update. Symptoms for me were no information on the LCD screen, LED status lights not lit - basically a slightly warm oversized brick.

Powering off the UPS, disconnecting the battery (at the back), reconnecting the battery & powering on brought up all the lights, which then went out one by one over a couple of minutes.

Unable to get any sense out of it via USB cable (Firmware update utility recognised it but failed to communicate). 

I've the added complexity of using mostly MacOS, and not having a serial port....

So... I picked up a RS232 / serial to USB adapter. I went for a StarTech jobbie using an ftdi chipset, which may or may not have helped. This was connected to the OEM AP 940-0625A cable that came with the UPS.

Broadly I got no feedback from the UPS, or xmodem, other than 

Retry 0: Timeout on sector ACK

Retry 0: Timeout on sector ACK

Repeating. Slowly.

Quite a lot of faffing followed with various flavours of xmodem, both natively and via a Windows VM on Parallels... 

Ultimately I realised the latest comment from @ChromeTO mentioning USB cable helping - when I connected the USB cable AS WELL AS the Serial to USB setup, I found a different behaviour on the USB LED's (two LED's lit permanently).

Using native Mac:

• Minicom installed using Homebrew
• lrzsz from here: https://ohse.de/uwe/software/lrzsz.html
• and some config from here:https://botbench.com/blog/2020/05/10/rc2014-cp-m-and-xmodem-on-macos/ 
• xon/xoff flow control is the same as software flow control (thanks for the consistency minicom...)
• using "ls /dev/cu.*" on the Mac gave me the weird port ref for the USB serial adapter (in the format "/dev/cu.usbserial-AV0NNIzz"), this was configured in minicom etc.

it kicked into action. with the serial port settings described earlier.

The transfer was over the USB to Serial setup - IIUC this won't work over the USB to USB cable, but I had to have this connected to make progress....

During the upgrade, I got the flashing lights described by @BillP & the update took a few minutes to transfer over the serial cable.

I've since updated the firmware further using both the USB cable and Serial to USB setup - the USB cable did not play well with Parallels / VM - the USB device reported as 'unrecognised' by Windows 11 at 15% and failed (oh how I laughed....)

Using the Serial to USB setup, I just had to leave the USB to USB cable physically connected, but not passed through to the VM, and tell the APC upgrade utility that I'd select the port manually, choosing the only option (COM3 for me) which was the USB to Serial - the firmware update then went through fine.

Thanks again to both @BillP & @ChromeTO for making me think this was possible, and providing all the information!!!

[Marfi]

Hello all together,

Unfortunately, this is the first time that I am at a loss and am beginning to despair.
I now have 2 defective SMT2200RMI2U here.
They were running perfectly, I just wanted to update the firmware of the UPS.
I first tried this with the "AP9630" network module via network.
(According to the instructions, I checked the ID and downloaded it, according to interface ID18).
This update was started, but after sending the update, the update message remained on the display

of the UPS, so I waited overnight.

Unfortunately, the web interface showed the next day that no connection to the UPS is possible,

also this update message was still on the display and the UPS did not react,

so I restarted the UPS (batteries disconnected and power).

Then it didn't switch on any more

(display stays black, but the backlight is visible and the 4 LEDs sometimes all light up).

Now I thought, that was a unfortunate run.

So I tried to update my second UPS in a completely different way using the tool:

"Smart-UPS_Firmware_Upgrade_Wizard_v4.3.1" and via serial cable.
This was recognised normally, according to the tool I also had to charge the battery to 50% and then

it wanted to carry out the update.
Unfortunately, the same thing happened again. According to the software, the update was successful.

It only said that the UPS was still completing the update. On the display of the UPS,

the same message was shown again that the update was now being applied (see Photo).
Unfortunately, this remained overnight and caused the same problem as with the first UPS.
I now have 2 defective UPS after firmware updates!
I have never experienced anything like this in my entire life.

Especially because I tried to update the firmware in 2 different ways and with 2 different versions

(newer and older versions, both ID18).
The tool at the second attempt also directly found the correct firmware (ID18)

that I had previously, To be on the safe side, read out (Both UPS have ID18).

For this purpose, I had removed the AP9630 card before the update.

Unfortunately, I can't get a connection to the UPS via serial with the option described above (@BillP & @ChromeTO ).

Is there a way to completely reflash the UPS?

I can solder and do everything that is necessary to get the thing running again.

Thank you very much for your effort and answer.

I hope someone can help me.
I do not have a UPS that is now running.
I now have 2 defective UPS that were running perfectly before the update.