SMT3000 "Bricked" Trying To Update The Firmware?

[BillP]

Hi Folks, I have an SMT3000 that was running on firmware SMT18UPS_09-1.enc and I attempted to upgrade the firmware to SMT18UPS_09-3.enc via the installed NMC firmware update mechanism.  I powered down the UPS and logged into the NMC, located the correct firmware will under the "Configuration/Firmware Update" drop down and proceeded.  The front panel of the SMT indicated that it was accomplishing the firmware update and the screen on the NMC web interface said that it would restore after the firmware update was successful.  The NMC screen restored and I attempted to turn the UPS back on and now it appears to be dead with a constant tone coming from the UPS' alarm.  I have verified that the battery packs are good.

The NMC interfaces shows the following:

HOME:

Smart-UPS 3000: Unknown      2 Critical Alarms Present An internal UPS communication fault exists. The battery power is too low to support the load; if power fails, the UPS will be shut down immediately.    2 Warning Alarms Present The power for the main outlet group, UPS Outlets, is off. The power for switched outlet group 1, Outlet Group 1, is off.

Recent Device Events   Date Time Event   01/10/2016 11:19:34 UPS: The power for switched outlet group 1, Outlet Group 1, is off.   01/10/2016 11:19:34 UPS: The power for the main outlet group, UPS Outlets, is off.   01/10/2016 11:19:34 UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately.   01/10/2016 11:19:34 UPS: An internal UPS communication fault exists.   01/10/2016 11:19:34 UPS: Restored the local network management interface-to-UPS communication.

STATUS:

UPS Status Smart-UPS 3000
Last Battery Transfer:	None
Battery Temperature:	0.0°C
Runtime Remaining:	0min
UPS Input
Input Voltage:	0.0 VAC		@ 0.0 Hz
UPS Output
Output Voltage:	0.0 VAC		@ 0.0 Hz
Load Current:	0.0 Amps
Output VA:	0.0 %
Output Watts:	0.0 %
Output Efficiency:	output off
Output Energy Usage:	711.90 kWh
Battery Status
State of Charge:	0.0 %
Battery Voltage:	0.0 VDC

I can no longer attempt a firmware update because the NMC thinks the UPS is turned on:

Firmware Update
Note: The UPS is not off. It must be turned off to update the firmware.
Last Update Result:	Refused
UPS Firmware
Current Version:	ID18UPS 09.1

Does anyone have any ideas on how I can proceed or need any additional information that may be helpful?

Any help will be greatly appreciated and thanks much!

[BillP]

Unplug the mains and battery packs, this should clear any residual voltage but leave it for 30 minutes and try to power it back up again.

[BillP · Accepted solution]

Hi Gareth.  Thanks for your response.  No joy.  I left it powered off unplugged with batteries disconnected for the evening.  I am afraid that the UPS, although itself less than 3 years old, is nothing more than a boat anchor at this point.

APC does not provide repair services nor will they sell parts.  The only aftermarket repair solution won't touch it either.  $1,100 down the drain at this point.

Lessons learned?  Don't update firmware on these units unless absolutely necessary and specifically told to do so by APC tech support.  These units are not designed with robust firmware fail-safe features in case of a disaster.

[drasche]

Hey community

just yesterday I ran into a very similar situation. I tried to update my SMT3000RMI2U with the latest firmware. It was running on a very old version. Version I tried to flash was "SMT18UPS_09-8.enc". the UPS reported itself as (ID18), so I assume that is the correct firmware  The unit is equipped with a NMC2 AP9630, so I used the WebInterface to update to the disired firmware. NMC by the way is already on the latest release v7.0.8 

The firmware update started as supposed to but never finished 😞 After aprox. 2 hours in the status "Firmware upgrade in progress..." I switched of the UPS completey (maybe shouldn´t have done this...), but it wasn´t working anyways.

Now it is dead 😞 NMC states "No UPS found" and all tests to start some kind of "recovery flash" didn´t work. I tried the Firmware Update utility, I tried the serial cable through the NMC and also tried the serial connection that is directy wired on the mainboard of the UPS itself.

Is there maybe any other method to flash the firmware of the UPS? The NMC is working without problems, but it seems that the UPS itself has completely lost its firmware and wont start anymore.

Cheers

Dirk

[elchi]

Hey,

just run Yesterday to the same situation as drashe did. The Diffrence is, I've tried the Firmwareupdate via the Serial Firmware Update Tool. The Tool stated a successfull Update. But the UPS hangs at Installing Firmware....... After 2Hours I've tried to reboot the UPS. Now we've a useless conterwight in the rack. It seems there is an error at the Firmwareupdate procedure, but I've also the feeling that apc is not interested to solve these problems with these old UPS. Mi suspicion is, that the SMT 3000 counts to the UPS where an update is not possible. On the Website the SMT 2200 is stated an not able to be updated. Perhaps this is also true for the 3000? In the Readme for the Firmwarefile they stated, that it is for the 1500...... But why the Updatetool did not recognize it?

Kind & Frustraded Regards  

Micha

[drasche]

Hi Micha,

sad to hear that you ran into the same situation 😞 And I fully agree, the software should have taken care!!

I just wrote in another threat that I got hold of a second UPS and I will try to extract the firmware to inject it into the broken one 🙂 Maybe I can share the result with you. 

Currently I am looking for the service manual for this type of UPS, perhaps someone is reading this threat and can share that with me.

Cheers 

Dirk 

[JonPro]

Hi @drasche and @elchi,

I am really sorry to hear about the issues that you've experienced.  If you have used the UPS Firmware Upgrade Wizard v4.3.2, as what the FAQ article states,

"IMPORTANT: IT IS NOT RECOMMENDED TO USE LOCAL CONNECTION VIA USB CABLE WITH THE CURRENT VERSION OF THE FIRMWARE UPGRADE WIZARD. It is recommended to use local connection via Serial cable. The serial cable supplied with the UPS must be used. USB support will be re-added to the next version of the Firmware Upgrade Wizard.
Note: UPS models with only USB communication port will have to wait for the next release of the FIRMWARE UPGRADE WIZARD which is expected to launch very shortly."

How do I update the firmware of my CSH, SCL, SMC, SMT, SMTL, SMX, SRT, SRTL, XP and XU series Smart-... 

Is this the version that you've used?

[elchi]

Hi JonPro,

I've used Version 4.3.1 of the Firmware Upgrade Wizard. That was the Latest Version I found on the APC Website (sometimes it's a little bit difficult to find the right links on the website).  As far as I remember this was the Version also mentioned and linked on the  FAQ at the time of the download. I know I've found a newer Firmware ZIP but no according updater so I've used 4.3.1 and the containing firmware for the attempt. And I Used an APC Serial Cable.  The Updater has also found the UPS via the serial cable and detected the right model (and showed the matching firmware as far as I remember). The Updater did his work (the progression bar fills up and the ups showed "Firmware Update in Progress..." in the display). The Updater then showed that the firmware is successfull flashed, but the UPS still showed "Firmware Update in Progress..." and the Green, the Yellow and the right Red LED are lit. As the Updater say it will take some time I waited over two ours (perhaps nearly 3), than I tried to reboot the UPS. I have to disconnect power and battery to quit this state.  That was perhaps the wrong decision, but before I did that I've tried to start the Update Wizard again, but there was no connection possible over the serial cable.

Thanks for your reply

Kind Regards

Micha

[JonPro]

Hi Micah,

Have you tried reaching out to our Tech Support team to further check the warranty of the UPS and what else they can possibly do?

https://www.se.com/ww/en/work/support/country-selector/contact-us.jsp 

[elchi]

Hi JonPro,

no I haven't tried this, because our UPS left the warranty period long time ago. 😉 But is there a chance that the support team cares for such an veteran? If yes, I would be gleadly invest the time to contact the support team.

Kind Regards

Micha

[elchi]

Hi JohnPro,

I opened an support ticket for that some weeks shortly after your post. The ticket is still open, I've received no response, nothing 😞 That is a little bit disappointing for me. What's on at APC? Even a "sorry, we can't help you, your UPS is too old" would be ok, not the best answer, but an anwer.

Regards Micha

[elchi]

Ok, replying to myself.

The ticket get closed without any questions or notifications. Some month after I opened the ticket. This is a bit disappointing and not what I tend to knew as a good service.

@JonPro this was not the way I expected, how it would come out. we're now at the evaluation phase, to switch to another manufacturer for the UPS we sell to our customers......

Regards Micha

[Romaschka]

Hi Micha!
I have the same problem: after a try to update the firmware on my SMT and waiting for a long time with "updating firmware" on the screen I've rebooted the UPS and now I can't get any connection to it. Did you possible found any solution for this or is the only option to recycle this pease of hardware?

Roman. 

[Lukasha]

@Romaschka @elchi @BillP I did the same as you. I broke my smt3000 due to the update. Were you able to fix these devices?

[Romaschka]

Hi Lukasha,

no I have no solution for this. As somebody said, we have a good boat anchor now. It’s unbelievable, that hardware with that price have a such bad support and update management.

Roman.

[Lukasha]

@Romaschka @elchi @BillP 

And I'm pleased to say that today I managed to start the UPS - I uploaded the firmware directly to the chip.

A full description can be found online (I don't know if it's allowed to post links here)
Type in Google:
Unbricking an APC SMT3000 UPS after a failed firmware update

I hope this helps

[Romaschka]

Oh! Wow! So fast!

Did you have had an usb programmer to do this?

Thank you very much for your answer, I’ll try my luck too, but it will take some time.

Roman.

[Lukasha]

Yes, I used the programmer: ST-LINK. I uploaded firmware ver. 15. The UPS started and worked normally. However, I also wanted to update the MCU, UBL, MBL - that's why I downgraded the firmware to 0.94 (changing the firmware via USB cable using the official tool - not the programmer). This process updated the MCU, UBL, MBL. Then I uploaded 0.96. This also updated the MCU, UBL, MBL. Finally, I uploaded firmware 15 (of course also using the official tool). After these operations, I have versions: UPS 15.0, MCU 20.0, UBL 08.5, MBL 20.0. Good luck!

[Romaschka]

Hallo Lukasha,

I am really grateful for your link regarding the possibility of restoring the UPS. Thanks to your advice, I have also restored my device and can use it again.

Thank you very much!!!