AP7900 failed firmware upgrade, bricked my unit -> how to reset?

[Anonymous user]

Hi - I tried a firmware upgrade to 3.73 via TFTP download. First time the AOS download failed but I was able to get back in and try again, the second time it seems to have worked (I got a message indicating the file transfer was successful). At that point the unit was unresponsive and my telnet session froze up. I tried to connect to the serial console, nothing, not even a prompt. So I was never able to download the app image. Tried the password reset procedure but I never get blinking LEDs, just two solid amber LEDs. I also tried powering up with the reset button pressed to see if that would reset the unit, didn't work.

File names are:
apc_hw02_aos_373.bin -> this seems to have downloaded successfully
apc_hw02_rpdu_373.bin -> never downloaded this

Is there any way to reset the unit to factory defaults? I'd think there must be a way to recover from corrupted downloads. Please help!

Thanks!

[BillP]

Thank you for sharing with us. That sounded familiar and I thought I had it indicated in a knowledge base.

I had it here (how to upgrade a single card) -> How do I upgrade the firmware on an APC Network Management Card? | FAQs | Schneider Electric US

I will go ahead and add that same note to the kbase we have on mass upgrades (How do I perform a mass firmware upgrade on APC network enabled products? | FAQs | Schneider Electri...). Had you referred to that?

Or basically, where can I document it besides your post to assist other users that may run into this issue? Just trying to think how a user might come across this to prevent issues.

[Anonymous user]

Thank you for the response Angela and for your help with the support case.

We searched around via Google, and via the APC support knowledge base, and via this support forums but could not find the information about the 11 character username and password limit in the NMC Upgrade Utility.  We didn't know we were looking for a username/password truncation issue so we were looking for issues about the firmware upgrade failing halfway through the process.

At the time we were searching via Google for

   "AP7900 firmware upgrade fails"

   "AP7900 firmware Could not login"

and that is how we ended up at this article

If there were an KB or FAQ article with a title something like "NMC Upgrade Tool firmware update fails with Could not Login on username or password longer than 11 characters" we might have had more luck finding it.

Of course just fixing the problem of truncation in the NMC Upgrade Tool would be great - the funny thing is the tool's first login to FTP works, it is the login after the reboot that fails, so there is some part of the codebase that can handle longer than 11 characters. Barring that another thing that might help is if the NMC Upgrade Tool gave a warning about the username/password truncation issue (and/or detected if user was about to have the problem) and/or if the NMC Upgrade Tool shipped with a text file of known limitations/issues. There are also release notes with the firmware that we did read through that might have mentioned the issue.

Seeing the FAQ articles the information on username/password truncation is definitely there on the FAQ about upgrading:

*In NMC1 Upgrade Utility v1.2 or higher, a username or password longer than 11 characters will be truncated. Thus, you will need to change the password to 11 characters or less or choose another firmware upgrade method. Reference the picture below on how to check your NMC Upgrade tool version.

Note: *In NMC1 Upgrade Utility v1.2 or higher, a username or password longer than 11 characters will be truncated. Thus, you will need to change the password to 11 characters or less to use upgrd_util.exe. This could be modified using a mass configuration first, outlined in knowledge base FA156117. Reference the picture below on how to check your NMC Upgrade tool version.

Thanks again Angela.

-Michael

[Anonymous user]

Raising this thread from the dead because it helped me un-brick my AP7900.

Upgrading to 3.74 looked good; the installer detected old enough firmware to know it had to do two upgrades; first one worked, second one didn't (Status LED flashing green/amber).

Power failed, Status green, but Overload LED with 0 amps (nothing plugged in). Could ping AP7900 but not Telnet or web to it. Was able to FTP in with apc/apc (whew!) and saw apc_hw02_aos_374.bin, but no apc_hw02_rpdu_374.bin, similar to what OP found above. FTP put apc_hw02_rpdu_374.bin and Overload LED immediately changed to OK!

Web'd in using apc/apc, reset everything the way it was and all seems to be working ok.

Just paying it forward, hope this helps someone!

[BillP]

hi brian,

can you tell us what LEDs we have now around the RJ-45 network jack on the PDU? I hope that I can help you with this it may just take some tinkering.

can you tell me what the old firmware version was that you were trying to upgrade from?

can we FTP to the card? if not, you can try power cycling the PDU for 5-10 seconds and giving it a few minutes to boot up. if its in production, see if you can press the "reset" button on the PDU display with a paper clip to reboot the management interface only.

if you can FTP, do an ls command and tell me what .bin files you see listed.

thanks!!

[Anonymous user]

When I power on the two LEDs on the RJ45 are solid amber (orange) and they never change colors. The LED near the display indicates Overload, this also never changes unless the unit is powered off. Doesn't matter if the RJ45 is connected or if the console is connected. When I press the reset button with a paper clip the Overload LED stays on but the two amber LEDs go off. As soon as I release the reset button the amber LEDs come back on and nothing else happens.

I think the old AOS version was either 2.64 or 2.65 and the app version was either 2.64 or 2.65. Can't say with 100% confidence, but it was definitely greater than 2.02.

I can't even ping the unit, let alone telnet or FTP. There are no blinking LEDs on the switch I have it connected to... the link is totally down and never comes up. I have power cycled many times and waited plenty of time. The PDU appears to be doing nothing, it's not booting, and there's no console output. I have tried 9600 and 2400 bps many times.

I think I'm hosed without some kind of recovery procedure built into the hardware. For example on a Cisco 871 router you can power cycle with the reset button pressed (with a paperclip) for 30 seconds and the router will clear it's flash memory and dump you to a boot loader prompt. I tried this with the PDU and kept the reset button pressed for more than 2 minutes. There seems to be no boot loader prompt.

[BillP · Accepted solution]

yeah there is no procedure like that on these. on our new PDUs due out shortly and our Network Management Card 2 based devices, there is a recovery mode.

if you have power cycled it and waited for several minutes before reapplying power, then i do think it is hosed. while the card was writing to its SRAM, something my have happened.

power cycling was the greatest possibility of trying to get it to a state where we could at least FTP to it and sort of finish the upgrade.

can you tell me the first several digits of your serial number and we'll see if its in warranty? something like ZA0732 it should start with?

[Anonymous user]

Hi - I tried a firmware upgrade to 3.73 via TFTP download. First time the AOS download failed but I was able to get back in and try again, the second time it seems to have worked (I got a message indicating the file transfer was successful). At that point the unit was unresponsive and my telnet session froze up. I tried to connect to the serial console, nothing, not even a prompt. So I was never able to download the app image. Tried the password reset procedure but I never get blinking LEDs, just two solid amber LEDs. I also tried powering up with the reset button pressed to see if that would reset the unit, didn't work.

File names are:
apc_hw02_aos_373.bin -> this seems to have downloaded successfully
apc_hw02_rpdu_373.bin -> never downloaded this

Is there any way to reset the unit to factory defaults? I'd think there must be a way to recover from corrupted downloads. Please help!

Thanks!