NMC2(AP9630) 6.2.1 to 6.4.0 Firmware Update Failure

[BillP]

The update utility was used to update 12 AP9630's from 6.2.1 to 6.4.0.  One of the cards failed the update.  The logs show the following failure info:

Transferring firmware file (2/3): OS
[11/06/16 12:35:09] 150.102.210.201: ERROR --> OS transfer failed

The card was reseated to reset it.  The web interface then would not display the homepage and referred to the logfile.  I then tried to FTP the update files starting with the bootmonitor which transferred ok.  The AOS file was then FTP'd.  The hash marks stopped after a few lines and hung never displaying a transfer complete error.  When I closed, reopened, and Telnet'd to the card again, it seems to connect but commands aren't displayed.  It will remain like this until I reseat the card again.  I tried this a couple of times.  A dir of the card shows just the old AOS 6.2.1 file but no SMUX file.  Is there a way to recover the card?

Thanks,

Chris

[voidstar_apc]

Your card seems fine so worst case you can use xmodem from the boot monitor (a bit on that in a second)

If the FTP process didn't finish, I'd worry about interference from a network middlebox or firewall. Try swapping the card into a UPS + network connection where the upgrade succeeded. Make sure to FTP in binary mode.

If that doesn't work, connect serially to the console port at 57600 baud, 8 data bits, 1 stop bit, no flow control. Keep hitting enter while reseating the card. You should get a BM> prompt. Type "xmodem" and enter, then send your file via xmodem and when it finishes type boot. Repeat for each file, aos first, then sumx.

[BillP · Accepted solution]

The UPS with the problem is in the same rack and another on the same subnet.  While the card was removed, I ping'd to make sure there wasn't another device using the same IP.  I may change it to another IP to see if that makes a difference since there are rare instances of devices on our network that do not respond to pings.  I'm also thinking about hard resetting it which formats the flash from what I have read.  I'll also give the console port method a shot if all else fails.

[BillP]

Hello Chris,

In relation to what voidstar has posted, I would also recommend doing a factory reset or format on the network management card before attempting a firmware upgrade. Below is our FAQ on how to use BootMonitor to recover or upgrade an Network Management Card 2. The factory reset can be found on Steps 4 and 5.

Using Network Management Card 2 (NMC2) bootmonitor to recover/upgrade firmware

To format the network card, Press and hold the pinhole reset button down for about 20 seconds. If this is being performed correctly, you should see the status LED "pulsing" green. When the process is actually complete, the status LED will change its color and "pulse" orange. This is when it is time to let go. The NMC will continue to "pulse" orange a few times and then the status LED will go off, and the NMC will reboot.

[BillP]

I tried all suggestions so far.  The bootmon transfers fine using either the upgrade utility, FTP, or XMODEM but the AOS transfer fails either way.  Here is what the system command shows.  It seems it can't save anything since even the bootmon shows Invalid Module Header! even after a successful transfer.

apc>system
E000: Success
Host Name Sync: Disabled
Name:           apc4D8832
Contact:        Unknown
Location:       Unknown
Message:
DateTime:       08/19/2009:00:01:37
User:           apc
Up Time:        0 Days 0 Hours 1 Minute
Stat:           P+ N4+ N6+ A-
Bootmon:        Invalid Module Header!
AOS:            Invalid Module Header!
App:            Invalid Module Header!

apc>

[BillP]

I'll give it a shot.  The same files were used for all 12 of the AP9630's that were upgraded and this one is the only one that had problems.

[BillP]

Just tried it again with the file you attached.  The results were the same with the Firmware Upgrade Utility.  Bootmon transferred with no problem but the OS transfer failed.  I have another board to try.  If it works, the current board will go in the scrap bin.  If not, I'll order a replacement.  This has eaten up a lot of time I don't have.  Thanks for trying.

[voidstar_apc]

Agreed, trying the AOS again was a long shot. I've never seen the hardware fail that way but if xmodem didn't work and the file is good, I can't think of any other possibilities. Hopefully it's not a problem with the UPS itself, which you'll find out by testing the other board.

[BillP]

This has been a strange problem.  I think only recall one other similar problem in the past with a firmware upgrade.  For now, I found an old AB9617 from a Symmetra, replaced the AB9630 with it, loaded the firmware for a SmartUPS, configured it, and it is online.