PDUs corrupted by ITE firmware updates to RPDU2G v706, AOS v704

[GJK]

I have several PDUs (AP8641) that were corrupted by the EcoStruxure ITE upgrade process (4 of 6 failed miserably).  My issue now is I can't remotely recover any of the units - although I can SSH to the units and get basic info.  All attempts to FTP the .bin files fails - they appear to upload correctly but never appear in dir and still don't load. I've tried 6.9.6 and 7.0.6.  Anyone know why Windows CMD FTP would not work?  I followed the process here: https://www.apc.com/us/en/faqs/FA156047/#FTP.  And yes, turned on hash and put into bin mode.

Proper Unit Upgrade: 

Model Number: AP9538
Serial Number: 5A2052E0xxxx
Hardware Revision: 05
Manufacture Date: 01/09/2021

Application Module
---------------
Name: rpdu2g
Version: v7.0.6
Date: Sep 28 2021
Time: 15:10:16

APC OS(AOS)
---------------
Name: aos
Version: v7.0.4
Date: Jul 26 2021
Time: 15:26:26

APC Boot Monitor
---------------
Name: bootmon
Version: v1.0.9
Date: Mar 27 2019
Time: 16:23:06

Failure 1: 

Model Number: AP8641
Serial Number: 5A2120E0xxxx
Hardware Revision: 02
Manufacture Date: 05/22/2021
MAC Address: 28 29 86 49 34 E2
Management Uptime: 0 Days 0 Hours 1 Minute

Network Management Card
---------------
Model Number: AP9538
Serial Number: 5A2120E0xxxx
Hardware Revision: 05
Manufacture Date: 05/21/2021

Application Module
---------------
Invalid Module Header!

APC OS(AOS)
---------------
Name: aos
Version: v6.9.6
Date: Nov 5 2020
Time: 08:31:17

APC Boot Monitor
---------------
Name: bootmon
Version: v1.0.9
Date: Mar 27 2019
Time: 16:23:06

Failure 2:

Model Number: AP8641
Serial Number: 5A2120E0xxxx
Hardware Revision: 02
Manufacture Date: 05/22/2021
MAC Address: 28 29 86 49 35 0C
Management Uptime: 0 Days 0 Hours 2 Minutes

Network Management Card
---------------
Model Number: AP9538
Serial Number: 5A2120E0xxxx
Hardware Revision: 05
Manufacture Date: 05/22/2021

Application Module
---------------
Name: rpdu2g
Version: v7.0.6
Date: Sep 28 2021
Time: 15:10:16

APC OS(AOS)
---------------
Name: aos
Version: v6.9.6
Date: Nov 5 2020
Time: 08:31:17

APC Boot Monitor
---------------
Name: bootmon
Version: v1.0.9
Date: Mar 27 2019
Time: 16:23:06

[skendric]

The last couple days, I've been upgrading  AP8841, AP 8858NA3, and AP8981 from various flavors of 6.8.x and 6.9.x to RPDU2G 7.0.6 & AOS 7.0.4 , using the NMC Firmware Update Utility.  Mostly worked fine.

However, a couple of AP8981 resisted:  the Update Utility reported success ... but in fact, the upgrade hadn't occurred.  [Well, the bootmon image succeeded.]  I've been doing it manually, i.e ftp'ing first the AOS image and then the overlay image

apc_hw05_bootmon_109.bin [Skipped this, since the Update Utility delivered it successfully)

apc_hw05_aos_704.bin
apc_hw05_rpdu2g_706.bin

And hit various flavors of failure

(1) The AOS upgrade succeeded ... but when I upload the overlay image, the NMC reboots (of course) but reboots back into the previous 6.8.x or 6.9.x versions

(2)  The AOS upgrade succeeded ... but subsequent efforts to upload the overlay image fail with "550 STOR requested action not taken." (remotely rebooting the NMC got me past this, i.e. ssh into the CLI, type 'reboot')

(3) The AOS upgrade and the overlay upgrade succeeded! ... I can see this gain via the ssh session ... but a few minutes later, the card reboots and when it returns to service, it has downgraded to the previous 6.8.x or 6.9.x version

Here is an example of what that last experience looks like, from a CLI and a log perspective

apc@dc-705-pdu's password:

Schneider Electric Network Management Card     AOS                      v7.0.4
(c) Copyright 2021 All Rights Reserved                 RPDU 2g APP       v7.0.6
-------------------------------------------------------------------------------
Name : dc-705-pdu                                                                   Date : 06/24/2022

[...]

2022-06-24T15:45:02-07:00 dc-705-pdu.corp.alleninstitute.org Network Interface restarted. 0x0002
2022-06-24T15:45:56-07:00 dc-705-pdu.corp.alleninstitute.org Rack PDU 1: Remote RPDU 2 (SN: 5A1950E01287) firmware version does not match. 0x3438

Case #1 above has finally succeeded ... after a handful of repeated efforts

Case #2 succeeded after the NMC reboot

Case #3 continues to fail (to auto-downgrade) despite a handful of repeated efforts

--sk

Stuart Kendrick

[GJK]

Stuart, sounds like you are having the same issues as me.  There is no rhyme or reason on why one update works, and then others fail.  All the PDUs I tried to update are the same model, bought at the same time, and all started at the same firmware - a couple went through and four others died using the same processes.  The SE/APC update process continues to be a problem, with little hope of resolution unless you go onsite with a laptop and console cable. 

[skendric]

Yes, I haven't discerned a pattern yet.  And yes, physical intervention is how I have overcome the troubled units I describe above

- From standing in front of them, I notice that generally, once PDU #1 is upgraded ... a few minutes later, PDU #2 announces that its 'Host has initiated an upgrade', and it upgrades as well, presumably via the interlink cable between the two.  This is a successful trajectory

- But in some cases, PDU #1 announces 'Host has initiated an upgrade' ... recall that this is the PDU which I just upgraded, the one which is clearly labeled '1' on its console (i.e. '1' means 'Host') ... and it reboots twice and ends up back at the old revision, i.e. specifically, the revision which PDU #2 is still running and presumably has just pushed to PDU #1

- In general, for troubled PDUs, I succeed by separating them (unplugging their interlink cable), plugging them both into the Ethernet/IP network, upgrading them independently, then dasy-chaining them again with the interlink cable

But, then, I have the luxury of being able to reach these units via a short walk

--sk

[Guinea_Pig]

I have seen issue with upgrades for several months when using the firmware upgrade utility.

At some point in the past the firmware utility bundled with the firmware updates was changed (upgraded?) and since then I have had quite a few problems with FW upgrades.

The bootmon always upgraded successfully.

The AOS file usually uploads all the way (and I believe is loaded by the NMC/PDU)

The firmware utility fails to upload the APP file (but reports as successful)

If you try to log in using FTP manually at this point the FTP server on the NMC/PDU fails to respond.

We found that this could be resolved by one of the following methods:-

• Hard reboot from the pushbutton  (this sometimes works)
• Logging in via CLI then disabling FTP, rebooting, logging in again, enabling FTP then rebooting again. (More often than not this works)
• Just Wait!  Not sure how long for as it seems variable - sometimes a couple of minutes, sometimes much longer.

I also found that if a device failed to accept a firmware upgrade via the Utility then it would be unlikely to next time - however if you try enough times it might eventually work.

Also note that if the utility fails then you probably cannot upload the firmware FTP manually either until the FTP server is back up and running.

I actually do not use the utility anymore - I always use FTP manually and this has ever failed so far.