APC UPS SMX1500 firmware update failure - and fixed

[BillP]

Hello,

This is posted in the hope that it helps other people with resolving firmware update failures that can otherwise leave the UPS "bricked".

I have a SMX1500RMI2UNC (SMX1500 240v, with network card at f/w 6.5.0), ID=20, UPS was at f/w 9.9, now at f/w 10.1.

Attempting to update from 9.9. to 10.1 through the NMC failed. The UPS was left in a "restart cycle", with LED1 off, LED2 yellow, LED3 blinking red and LED4 permanently red and the message "firmware update in progress"  on the front panel display. Previous updates have worked as intended via the NMC.

Event log messages during the update attempt and failure were as follows (bottom up, most recent at top):

02/09/2018 14:02:39 Device UPS: Lost the local network management interface-to-UPS communication. 0x0102

02/09/2018 14:02:19 System File 'SMT20UPS_10-1.enc' transfer failed. 0x000F

02/09/2018 14:02:19 Device UPS: Firmware transfer to UPS failed 0x0178

02/09/2018 14:02:19 Device UPS: Finished UPS firmware transfer  [Xmodem Cancel] 0x0177

02/09/2018 13:58:21 Device UPS: Transferring firmware to UPS. 0x0176

02/09/2018 13:57:48 Device UPS: The output power is turned off. 0x0114

02/09/2018 13:57:48 Device UPS: The graceful shutdown period has ended. 0x014F

02/09/2018 13:57:47 Device UPS: The power for switched outlet group 3, OG3-C14x4-desk, is now turned off. 0x0135

02/09/2018 13:57:47 Device UPS: The power for switched outlet group 2, OG2-C14x2-ds10, is now turned off. 0x0135

02/09/2018 13:57:47 Device UPS: The power for switched outlet group 1, OG1-C14x2-xw8600, is now turned off. 0x0135

From these messages, it appears that the NMC loads the firmware into the UPS with the xmodem file transfer protocol. The failure appears to be when the NMC transfers the firmware file internally to the UPS using xmodem over the internal serial interface between the NMC and the UPS.

The UPS was left in an unresponsive state. To reset the UPS, disconnect the batteries, disconnect the mains input, remove the NMC and leave the UPS for several minutes. It can also be useful to hold down the power button for 10 to 15 seconds to drain the capacitors etc.

On powering up the UPS, attempting to connect with a USB2 cable was unsuccessful. The firmware update wizard could find the UPS, but could not communicate with it.

Connection through the serial port using a terminal emulator was successful. Port settings are 9600 baud, 8bit, 1 stop bit, no parity, xon/xoff flow control.

On power-up the UPS output was as follows, in a repetitive loop with a beep every restart of the download manager:

APC(R)2011 Download Manager

Version 07.05 - 0014

CCCCCCCCCCCCCCCCCCCC

Error XModem retry count exceeded

APC(R)2011 Download Manager

Version 07.05 - 0014

CCCCCCCCCCCCCCCCCCCC

Error XModem retry count exceeded

APC(R)2011 Download Manager

Version 07.05 - 0014

CCCCCCCCCCCCCCCCCCC

Error XModem retry count exceeded.

At this point, the UPS is "bricked", unless firmware can be loaded using the download manager as a "failsafe firmware loader" mechanism, which requires using the xmodem file transfer protocol over the serial connection.

I used xmodem (included with extraputty, based on the putty terminal emulator) to transfer the firmware file (10.1) to the UPS. This was successful. While the transfer is in progress, the LEDs flash alternately green (led1) and red (led3).

When the transfer completed, the UPS screen went blank with a beep and the UPS turned off.

On power up, the UPS worked, passed the self test and showed 10.1 as the firmware version.

After re-installing the network card, the UPS is back and working, without having lost its configuration data (the UPS specific configuration elements and the network card configuration elements).

The conclusion is that the firmware update process from 9.9 to 10.1 has a fault under some as yet unknown set of circumstances, which can cause it to fail and leave the UPS in an unserviceable state, until the firmware is reloaded through the serial port with xmodem.

Cheers, Colin.

[voidstar_apc · Accepted solution]

Nice! Depending on UPS, you can often re-attempt the transfer via the NMC. Doesn't look like that was the case with yours but luckily you're handy with RS232.

[ChromeTO]

I just stopped by to say that this worked for me!

I bricked my SMT750RM2U, by unplugging the switch that my laptop was using to send the firmware upgrade to the UPS. Lots of lessons learned, sleep deprivation is not a good thing, don't do a UPS firmware upgrade via the NMC on this UPS. Use the USB cable.

So, what I did to get my "dead" UPS working again. Basically followed the same instructions as above. Note the following:

1. I had to leave the USB cable connected to the computer the UPS seemed more responsive with the USB cable constantly connected.

2. I did have to find an AP 940-0625A cable. This unit has an SERIAL RJ45 port... and the other end SHOULD go into a serial (DB-9) port on a computer/laptop.

3. I tried making my own cable with what I had on hand, and just couldn't get it going, so, I bought one.

4. Once the serial cable was connected, I used Extra Putty for the session. I didn't get anything on the session screen, the UPS didn't want to communicate.

5. I sent the xmodem transfer ANYWAY, with the same firmware I used to update the UPS originally.

6. The "connecting" message just stayed there, it seems like forever, but it was probably more like a minute, then it started transferring.

7. A few minutes later, the UPS was back to life...and all my setting retained. Crazy, but it worked.

Some after thoughts, maybe the cables that I Macgyvered, did work, but I was expecting to see some kind of communication in Putty. There was nothing, so, I assumed the cables didn't work, and never tried an xmodem transfer with my Macgyvered cables.

Thank you @BillP 

[AndyO]

Ok, also saved by this post.

APC SMT1500I UPS - bricked during a firmware update. Symptoms for me were no information on the LCD screen, LED status lights not lit - basically a slightly warm oversized brick.

Powering off the UPS, disconnecting the battery (at the back), reconnecting the battery & powering on brought up all the lights, which then went out one by one over a couple of minutes.

Unable to get any sense out of it via USB cable (Firmware update utility recognised it but failed to communicate). 

I've the added complexity of using mostly MacOS, and not having a serial port....

So... I picked up a RS232 / serial to USB adapter. I went for a StarTech jobbie using an ftdi chipset, which may or may not have helped. This was connected to the OEM AP 940-0625A cable that came with the UPS.

Broadly I got no feedback from the UPS, or xmodem, other than 

Retry 0: Timeout on sector ACK

Retry 0: Timeout on sector ACK

Repeating. Slowly.

Quite a lot of faffing followed with various flavours of xmodem, both natively and via a Windows VM on Parallels... 

Ultimately I realised the latest comment from @ChromeTO mentioning USB cable helping - when I connected the USB cable AS WELL AS the Serial to USB setup, I found a different behaviour on the USB LED's (two LED's lit permanently).

Using native Mac:

• Minicom installed using Homebrew
• lrzsz from here: https://ohse.de/uwe/software/lrzsz.html
• and some config from here:https://botbench.com/blog/2020/05/10/rc2014-cp-m-and-xmodem-on-macos/ 
• xon/xoff flow control is the same as software flow control (thanks for the consistency minicom...)
• using "ls /dev/cu.*" on the Mac gave me the weird port ref for the USB serial adapter (in the format "/dev/cu.usbserial-AV0NNIzz"), this was configured in minicom etc.

it kicked into action. with the serial port settings described earlier.

The transfer was over the USB to Serial setup - IIUC this won't work over the USB to USB cable, but I had to have this connected to make progress....

During the upgrade, I got the flashing lights described by @BillP & the update took a few minutes to transfer over the serial cable.

I've since updated the firmware further using both the USB cable and Serial to USB setup - the USB cable did not play well with Parallels / VM - the USB device reported as 'unrecognised' by Windows 11 at 15% and failed (oh how I laughed....)

Using the Serial to USB setup, I just had to leave the USB to USB cable physically connected, but not passed through to the VM, and tell the APC upgrade utility that I'd select the port manually, choosing the only option (COM3 for me) which was the USB to Serial - the firmware update then went through fine.

Thanks again to both @BillP & @ChromeTO for making me think this was possible, and providing all the information!!!

[Marfi]

Hello all together,

Unfortunately, this is the first time that I am at a loss and am beginning to despair.
I now have 2 defective SMT2200RMI2U here.
They were running perfectly, I just wanted to update the firmware of the UPS.
I first tried this with the "AP9630" network module via network.
(According to the instructions, I checked the ID and downloaded it, according to interface ID18).
This update was started, but after sending the update, the update message remained on the display

of the UPS, so I waited overnight.

Unfortunately, the web interface showed the next day that no connection to the UPS is possible,

also this update message was still on the display and the UPS did not react,

so I restarted the UPS (batteries disconnected and power).

Then it didn't switch on any more

(display stays black, but the backlight is visible and the 4 LEDs sometimes all light up).

Now I thought, that was a unfortunate run.

So I tried to update my second UPS in a completely different way using the tool:

"Smart-UPS_Firmware_Upgrade_Wizard_v4.3.1" and via serial cable.
This was recognised normally, according to the tool I also had to charge the battery to 50% and then

it wanted to carry out the update.
Unfortunately, the same thing happened again. According to the software, the update was successful.

It only said that the UPS was still completing the update. On the display of the UPS,

the same message was shown again that the update was now being applied (see Photo).
Unfortunately, this remained overnight and caused the same problem as with the first UPS.
I now have 2 defective UPS after firmware updates!
I have never experienced anything like this in my entire life.

Especially because I tried to update the firmware in 2 different ways and with 2 different versions

(newer and older versions, both ID18).
The tool at the second attempt also directly found the correct firmware (ID18)

that I had previously, To be on the safe side, read out (Both UPS have ID18).

For this purpose, I had removed the AP9630 card before the update.

Unfortunately, I can't get a connection to the UPS via serial with the option described above (@BillP & @ChromeTO ).

Is there a way to completely reflash the UPS?

I can solder and do everything that is necessary to get the thing running again.

Thank you very much for your effort and answer.

I hope someone can help me.
I do not have a UPS that is now running.
I now have 2 defective UPS that were running perfectly before the update.

 

[JoeNetwork]

THIS PROCEDURE WAS AN ABSOLUTE LIFE-SAVER!  THANK YOU SO MUCH

I bricked an SMX3000 trying to do a firmware update via the NMC; this process worked like a champ!

I used a Debian 12 server with an old school DB9 serial port, of course using the special APC RJ45 <-> DB9 cable.  Never thought I would be using Xmodem at 9600 for anything in the year 2023.  LOL

[Hiren-Karsun]

Hi, 

We do have SMX3000LVNC & during UPS firmware upgrade process from NMC we bricked it. Now its in constant reboot loop & only message on screen is "Firmware Upgrade in process", after some minutes/interval it reboot & come back to same place. LED status & everything else is same as BillP. We tried serial cable ExtraPutty & xmodel but no transfer of firmware file. We used our Cisco switch serial cable which we are using regulary & work fine, and it has same one end RJ45 & other end DB9. Does APC serial cable look like same but internally different ? Does our Cisco cable is compatible with APC ? We also have USB-to-RJ45 serial cable that also work on Cisco switches but not working with APC in this case. Any help in this matter will be apprciated.

[Teken]

The proprietary APC Serial cable is wired completely different than a standard one. In almost all cases a standard serial cable if attached to the main UPS (RJ50) port will cause it to shut down.

You can purchase a APC Serial cable, buy a third party, or make one. 👍

====================
====================

As it relates to recovering a bricked UPS. Connect the USB cable at the same time the RJ50 -> DB9 Serial cable is in place.

Launch the firmware update wizard (FUW) again and let it detect the COM port. (If) it detects the COM port and passes this step. 

The odds of the firmware upgrade process completing is very high! 👍 Once the firmware has completed the LCD will return to a working state. 

It’s imperative that you reflash the unit again to insure the firmware has been properly loaded. Once complete verify the unit operates fine via the LCD panel and confirm the firmware in the About tab.

If the unit has a NMC (reinstall it) and login to the web portal and confirm all tabs are present and not greyed out. Go to the About tab and confirm both the UPS / NMC display the correct firmware and related stats. Change whatever you need to in the NMC as it relates to the UPS and NMC.

================
================

At the end of the day here are some key takeaways to do, and not to do! 🤦‍♂️

NMC: Remove the NMC from the unit and never attempt to use it to load UPS firmware. This is the primary reason a UPS is bricked! 👎

USB: Remove this cable to insure other applications and services do not conflict with the firmware load such as Power Chute, SNMP, etc.

Battery: The unit must have a healthy battery that is 100% charged and capable of holding a load.

Outputs: Remove all connected loads from the rear outputs. Turn off the output via the LCD panel. Even (IF) your model UPS supports attached loads do NOT leave the output in the (On) position as this is one of the major factors that cause a failed upgrade! ☝️

Serial cable: You must use the supplied APC proprietary RJ50 -> DB9 Serial cable or one you’ve made that is validated to work on a APC UPS. 

Power: The UPS should be connected to another fully operational UPS as silly as this may sound. Doing so will prevent another major factor impacting the upgrade process: AVR, Transfer

Computer: A clean laptop that has all firewall, antivirus disabled and not connected to the network either hardline / wireless. Turn off all power management that relates to the screen, HDD / SSD, USB, CPU, Sleep / Hibernate. 

Following all of the above steps will ensure a successful firmware update. Lastly connecting the USB cable with the Serial cable at the same time has allowed me to recover hundreds of bricked UPS’s! 👍

[Hiren-Karsun]

Finally I have resolved the bricked UPS issue. Thanks to BillP for correct solution. But I'm wondering why APC does not post or suggest such solution to customer when having issue. Is it APC just want to sell more product ?

[stpenn123]

Hello

I have the same problem with my SMT2200RM2U. Unit appears bricked. Only get the backlight from the LCD and no LEDs

After a firmware upgrade using the wizard that stated it completed successfully.

Attempting to xmodem the firmware using extraputty and my hacked together RJ45 to DB9 and all I ever get is "Connection..."

Also don't get any output in the terminal whatsoever. I've tried with and without the USB also connected with the serial cable.

Does any body have any advice for me? Or did I just waste a bunch on a new battery for this UPS?

[Pieter]

Found this thread, same happened to my new SMX1500RM2U, trying to update from factory 9.8 to current 15.0, update tool kept failing and timing out at 7%, used NMC2 card, got XMODEM error, now no firmware.

Tried manual XMODEM update, stops intermittently, resumes, powers back on still in update mode.

Ref: https://community.se.com/t5/APC-UPS-Data-Center-Enterprise/SMX1500RM2U-Firmware-update-from-9-8-to-1...

Maybe I should try an older firmware first, where to get them, or any other advise?

[KrabKley]

I downloaded extra putty. I turned on the deceased after firmware via NMC - SMT3000i. Connected the AP 940-0625A cable. Have you selected the port settings as you said and I do not understand what to do next? 1. How do I understand if there is a connection with ups? 2. I saw a file being uploaded to me via xmodem, but there doesn't seem to be a connection.

[Tekmanagement]

Thank you for this fix!

How I proceeded:

Disconnected power and batteries, pressed the power button, removed NMC card, waited 10 minutes.

Hooked up USB (A to B) and a USB to serial port cable from ATEN, to the 940-0625A cable (note that I first used a custom one that had previously worked on a APC PDU but it didn't work here, so I tracked down a legit 0625A cable instead as I wasn't prepared to give up).

Installed Extra Putty and selected my ATEN COM port # that was displayed in device manager, verified the COM baud default settings matched what was posted above.

Plugged in the UPS to power, at which point I immediately started getting that CCCC... message, I chose the "file transfer > xmodem > send" option and pointed it to the SMTSMX20UPS_15-0.enc file that matched my ID20 SMX1500RM2U. It started sending and eventually completed. 

The UPS turned itself off, after turning it on everything came online as expected. I shut it down and hooked it all back up like before (NMC, extra battery etc.) and it has been working since with the latest firmware.

You saved my UPS! I can't believe APC doesn't test this stuff. These NMC and supporting infrastructure wasn't cheap and these are business grade UPS units, they shouldn't be so careless with their firmware update procedure. Just FYI, I was on 10.2 of the UPS firmware, so I wasn't that far out of date.