Firmware update failed on SMT2200RMI2U

[FrankAusNRW]

Hi there,

I've recently purchased a APC UPS SMT2200RMI2U for my homelab after my old UPS died.

Well, battery pack was dead, batteries replaced and the UPS could be startet.

In addition, this UPS is having a NMC 9630v2 that I was able to reset and login to the WebUI.

The FW on the NMC was outdated, so I updated the NMC 9630v2 to v7.1.2 sucessfully.

Next step was to update the FW on the UPS as this was still the 06.5 version. I've pivked the ID18 "SMT18UPS_15-0.enc" and started the FW update from the WebUI of the NMC to the UPS with no calbles connected to the UPS except the LAN cable to the NMC.

The FW was started to be uploaded to the UPS and nothing happens for the next 30+ minutes. The WebUI was still "uploading", the front panel of the UPS was showing "Upgrade in progress" (German language).

Since I was able to connect to the UPS via FTP client, I've copied the FW file "SMT18UPS_15-0.enc" to the "/tmp" folder, hoping that the UPS will start the upgrade automatically upon reboot, but nothing happens - but the FW file might still be there in the "/tmp" folder.

Non of the keys on the front panel was causing any reaction, so finally I decided to power off the UPS and unplug the battery pack for a while. Plug the battery pack again and power on the system was not restarting the upgrade process or doing anythig else.

The four LEDs on the front are all ON, the display empty. The NMC was telling me that the UPS can not be found, but the WebUI is still working.

Later on I've read that this process was is not ideal, but there was no way back now. 

I've then removed the NMC and tried to get access to the UPS with a console cable and got no answer from the device at all. Console cable was tested and is working fine.

Are there any ideas how to get the UPS up and running again? The appliance is dated back to 2012, so RMA is not an option here.

Somebody was writing to use "xmodem" to establish a connection to the UPS. Any experience on this recommendation, as I have never used xmodem yet?

Thanks in advanced,

Frank

[Teken]

Hi Frank,

Did you turn off the outputs on the system as instructed to in the APC Firmware FAQ?!?

How did you confirm the serial cable operates correctly? Are you able to SSH into the system?

If I assume you did turn off the output as outlined. You have a valid serial cable on hand and it does allow you to SSH into the UPS.

You may try the following steps and see what happens. 

Power: Disconnect the battery and AC Mains.

NMC: Remove the NMC from the unit 

USB: Connect a USB cable to the UPS and the target computer that has the FUW (Firmware Update Wizard).

Serial: Connect the RS-232 to your computer and the RJ50 directly to the back of the UPS.

Power: Reconnect the battery cartridge and apply AC Mains to the system.

Launch the FUW and confirm it finds the connected UPS. If you follow the prompts  it will indicate the (Only) valid firmware for the system on hand! ☝️

When the software indicates to proceed with the firmware process it may complain / cry about the USB cable. Ignore it for now and press continue than disconnect the USB cable.

You may also try to connect the USB cable while the serial cable is attached. Both steps (sometimes) forces the firmware to proceed.

Let me know what you see and observe on the computer and UPS LCD (LED). 👍

[FrankAusNRW]

Hi Teken,

yes, the outputs have been turned off as described and all devices pluged-off too. The NMC has also been removed.

The serial cable has been tested and is working on another device (a Sophos appliance) perfectly and the COM-port is assigned to it.

Allthough, the last time I was able to SSH into the UPS was during the upgrade process, that lateron failed. 

Currently no connaction via SSH or Telnet can be established.

Battery pack and the power cord have been off now for three days.

So I did what you said and put the USB-A/USB-B cable to the UPS and the computer, same with the serial-2-USB cable mentioned above, then reconnected the battery-pack and the power-cord.

After the battery pack was reconnected all 4 LED's on the panel went "ON", the LCD panel left blank.

I've startet the FUW (as admin) and let it go to check the COM ports. When it gets to the serial-2-USB cable the cable indicates some traffic, but finally no connection could be established.

Message from the FUW: "Communication could not be established with the device from this computer. Possible causes: ..." You may know the rest of this message.

The LCD panel is still empty, the LED's are all "ON".

Is there some kind of pins on the PCB to perform a BIOS reset when this pins are bridged or something similar? I haven't seen any kind of BIOS battery so far to force a "brain dead" for the UPS - just an idea ...

Thanks & kind reagrds,

Frank

[Teken]

Hi Frank,

Did you try to add the USB cable vs just starting the FUW with the USB cable in place?

X-Modem is really the next step in trying to recover the unit. There is no battery in the system as it relates to the MCU.

The only possible way is to have an identical system on hand and dismount the chip, read it, than write the same to the bad one.

Lastly, before you go forward with the other method. Insert the NMC while both the serial & USB are in place.

Go to the UPS firmware upgrade tab and see if you can load the file indicated by the FUW.

If the system allows you to select and load firmware there’s hope. Obviously if there is a resume button present you’re golden.

As stated earlier play around with the USB cable while doing so. As an aside, you may find several threads in this forum about simply leaving the UPS for days while the firmware is loading via NMC.

I call this out because there’s a less than 2% that could happen. 

Let me know what happens either way. 👍

[FrankAusNRW]

Hi Tecken,

I've tried what you recommended to do. Anyhow, the USB-cable was of no help so far, neither the UART serial-cable. At least, the UART serial cable is providing a COM port, but this COM Port doesn't help to run a FW update with the FUW.

With the NMC in place I got again the message, that there is no UPS connected, but I was able to establish a FTP connection (SSH failed) to the NMC - or even the UPS? Don't know ... I've uploaded the FW package to the "/upsfw" folter ... waiting ... waiting ... waiting ... just let it go for a couple of hours, but nothing happened.

So my next apptempt was to shut the UPS down (power cord and battery pack) for an hour and try to trigger the FW update with a restart and the file previousely loaded to the "/upsfw" folder. Again, even after a couple of hours nothing happened.

During the entire process all LEDs on the front have been "ON", and the display remain empty.

Turning in circles, I'm running out of ideas. Don't have a second, identical UPS around. Seem to be bricked during the initial FW upgrade started from the NMC panel. It's weired, that so many people like me are trying this way, as it seem to me a standard procedure, but at the end the appliance can hardly be restored.

Any more ideas? Would be very much appreciared ...

Thanks,

Frank

[FrankAusNRW]

Here the log from the NMC from the initial FW upgrade. The file was sucessfully transfered and the connection interupted. Ok, if the UPS is starting the upgrade process, that was confirmed on the front display, the connection to the NMC get lost. So far everything was fine.

[Teken · Accepted solution]

Hi Frank,

This is probably your only method to recover the system. You’ll quickly understand why the system was bricked! 🤦‍♂️👎

As those in APC, are releasing firmware that have zero common sense. As to firmware validation and carry over requirements.

https://www.tarball.ca/posts/unbricking-an-apc-smt3000-ups-after-a-failed-firmware-update/

[FrankAusNRW]

Hi Teken,

thank you for the link, I will give it a try at a later point of time. Currently busy with some other topics.

Probably I will need a bid time on my own, and the SMT32 link utility, not yet here on stock.

I will mark your suggestion as "solution".

Appreciate your patience and time you've spend here, even on a 12 year old device - good enough for a second life in my homelab. Thumbs up!

Cheers mate,

Frank