AP9640 3.1.1.1 Upgrade Issues

[TechJen]

Hello tech friends, I have a conundrum I'm dealing with here and hoped someone here might have more insight to prevent these issues.

I manage a fleet of 3500+ UPS's with AP9630 and AP9640 webcards, installed all over the US and CAN. We manage this fleet through DCE. We do not have highly technical people on-site to plug into these using the console port, we rely on them to do simple resets and troubleshooting when needed.

Previously we used to upgrade the firmware using the WinSCP File Transfer program, as I had negative experiences using the Schneider Utility to upgrade the firmware, it would not resume upgrades after network interruptions, resulting in a "bricked" webcard that needed to be reset to defaults with a user on-site or replaced. Because of this we avoided using the Schneider Utility, and just used WinSCP which resumes transfers, and had a very low rate of failure when upgrading NIC firmware. 

We have begun to use DCE to upgrade our fleet, as doing 1400 devices one-by-one using WinSCP was not practical. Up until now, doing upgrades through DCE has allowed file resuming and has not bricked many cards.

Recently, we have been trying to get all the NIC firmware up to date. I went through and upgraded all SY and SUMX AP9630 and AP9631 firmware to 7.1.2 with a fair amount of success, there were 1-2 failures which required a webcard replacement or reset. 

I then went through and upgraded our SY firmware from 2.3.1.1 to 3.1.1.1 without incident. 

I am now attempting to upgrade SU 1.3.3.1 and 2.3.1.1 to 3.1.1.1 and did a test run of 71 devices. Of those 71, 12 did not take the update and "bricked" in the sense that the IP no longer pinged. Having an end-user reboot and re-seat the webcard did not fix the issue. We found the only solution was to have the user reset the webcard to defaults via the reset button, and then reprogram the IP Subnet and Gateway. This of course wipes all our configurations, however it did take the upgrade and I was able to reconfigure from there using DCE.

Obviously I have concerns when doing the rest of the 1400 devices that I need to upgrade to 3.1.1.1. I am approaching this by using WinSCP to upgrade 1.3.3.1 firmware to 2.3.1.1 firmware, then from there will jump to 3.1.1.1, however I would like to avoid a high number of these resets, since it requires our support team to work on these with end-users, which can take days or weeks to complete, all the while the NIC is offline during this situation.

Does anyone have any more advice to avoid these issues? Would a step-increase to specific versions be a better option, say 2.3.1.1 to 2.5.0.6 to 3.0.0.11 to 3.1.1.1 or something similar? I would like to avoid causing our Support team more headaches with these resets.

Thank you for the help in advance!

[Teken]

A stepped upgrade is very time consuming but reduces the incident of a bricked NMC.

As it relates to upgrading over the WAN you’re simply playing with fire! ☝️

Anything to do with firmware upgrades on any APC hardware must be done locally or the very least via the LAN.

In 2024 APC / Schneider continue to use 1990 technology and has zero fail over protection relating to firmware protection / recovery! 👎

Almost every industrial device and security hardware we have incorporates a secure partition area. That allows the end user to recover / roll back to the previous firmware and working state.

All of these devices validate the firmware for compatibility, validity, and progress. 

If at any time the firmware should fail. The unit will simply stop, roll back to the previous firmware’s - good working state! 

In 2024 APC still doesn’t offer a high memory save, store, reload?!? Even if you were too cheap to incorporate this hardware you could do so in software!

As you noted in WinSCP . . . 

How can a piece of software written more than 15 years ago have the ability to RESUME, yet the actual maker (APC) not incorporate the same in their Wizard???

Answer = Lazy and incompetent! 🤦‍♂️👎