Are there any known issues with firmware upgrades on rack PDU's?

[DCIM_Support]

We have had several occasions where upgrading the firmware from DCE on rack PDU's results in a total loss of comms to the PDU via the network interface.

Also we have noticed that sometimes after a firmware update via DCE, the PDU will show 'application error, when accessed via the web interface.

Another attempt to upgrade the firmware will sometimes bring these PDU's back to correct operation, but usually only for a short period of time (could be a few hours or maybe a few days).

On other occasions another attempt will bring the PDU down for good with total loss of comms.

Is this an issue with the PDU's (i.e. they are faulty) or is this an issue with the firmware upgrades?

 From a reliability point of view, is it better to upgrade firmware via DCE or using the firmware upgrade utility?

(CID:105465303)

[DCIM_Support]

Hi Garry,

Although I have seen incidents where firmware upgrades have issues, I have never seen it to be a hardware issue specifically. If you are seeing any hardware issues however, that could cause upgrade issues  but not in the numbers you seem to be inferring ....check the front screens of the PDUs. There could have been issues with specific PDUs or specific firmware revisions that I'm not fully aware of but I would need more specifics about your setup to look  at that. 

Before we can even start saying what is happening, we'd have to look at the PDU itself after the firmware upgrade to see what state it was left in. Did the unit lose it's IP? Did the firmware upgrade only go thorough part of the way? Was it recoverable? Could you still get to the PDU's IP, etc.

The firmware upgrade utility does the same thing in the background that DCE does. One thing I would suggest is that if you're seeing issues to make sure you upgrade the firmware in smaller batches. This will do 2 things, it will cause less traffic on the DCE server and the connected network, it will also allow you to see if a device or group of devices is having issues. 

Steve

(CID:105465350)

[DCIM_Support]

Thanks Steve. The number of PDU's is probably less than 0.25% of those we upgrade using DCE. However I have not really tracked this potential issue thoroughly so that is more an estimated value than definitive. I have already been performing the upgrade in smaller batches to try to reduce network load but I did not really see any difference in network traffic. It seems to try to upgrade the same number of device 'in parallel' whether you choose 20 or 100 devices to upgrade. It does not appear to be specific to a particular version of FW, but my feeling is that it happens more on Gen 2 devices than Gen 1. I suppose what I really should have asked though is what sort of failure rate is expected during firmware upgrades. I know 0% would be ideal, but probably not what we would get in the real world.

(CID:105465498)

[DCIM_Support]

Hi Garry, If you are having 1/4 of 1% seeing this issue (or did you mean 25% instead of 0.25%), I doubt it's a firmware issue. Since you're also saying you're not able to connect it to a specific firmware, that too makes it difficult to pin it down. I have seen issues where a PDU that has issues with it's display may take a little longer to load each firmware component. Different firmware revisions may take longer than others. Network issues could also cause delays. If these all happen at once, maybe the AOS (APC operating system) layer of the firmware adds but the app (Application) layer may not get pushed. You never mentioned what the exact issue was but not loading both parts of the firmware is about the only thing I could think of. What we do within DCE is basically the same thing that is done by the firmware's executable. We FTP the first file, wait, then ftp the second file. If you've got SCP enabled instead, that too could cause slower firmware pushes. If you can upgrade the firmware outside of DCE though, we can pretty much rule out hardware issues. I don't have any numbers on firmware update failure rates. Yes, agreed, 0 would be best but that's not likely. I also know environment plays a part. DCE itself is part of the environment so if you have many devices, are running reports, backing up the server, gathering surveillance, there's much more chance of issues than if you just have 100 SNMP devices and you're polling them once every 5 minutes. I can check with engineering to see if they have any numbers for expected firmware upgrade failures but I can't promise anything. Steve

(CID:105465540)

[DCIM_Support]

Hi Garry, I spoke with engineering and they indeed have no numbers for expected firmware upgrade failure rates. I'd be more than happy to try to troubleshoot a specific PDU or 2 if you want to provide logs, model, firmware start/finish, etc. Let me know. Steve

(CID:105465561)

[DCIM_Support]

Hi Steve. I did mean 1 quarter of 1 percent yes. Pretty low but not good if it renders a £1K PDU useless. I have 2 earlier models coming back that failed, and if I can get into them I will post the logs etc here if you wish.

(CID:105465630)

[DCIM_Support]

Hi Steve. An old thread I know but I got one of the PDU's back. I went to site to swap it out and the display was locked up. It is an AP8853 and was on V6.0.9 firmware so a known issue. I tried on site to update the firmware using a USB Key and also serially but with no luck. After getting the PDU back to our offices I powered it back up and it worked fine on the V6.09 software for a couple of hours before I upgraded to 6.4.0 using a USB key. I have another AP8853 PDU on site and this was originally on 6.10 (I think - I know it was not on 6.09). This stopped responding during an update of the firmware to 6.40 using the Firmware update utility. It is not local to me so all I know is that it has failed to come back up online after the upgrade - I think it managed to upload the bootloader file and then failed at that point. I have yet another that came back from a customer site and simply shows a blue LCD screen with no text on the screen and also shows an orange light. It is a switched PDU (AP8953) and the outlets never turn on. I have no idea what firmware it is on but would guess older than 6.09. I cant get into this PDU serially at 9600 baud or into BM at 57600. Any ideas on how to recover this one? Thanks

(CID:106207910)

[DCIM_Support]

Hi Garry, I'm not really a PDU guy. I know some of the issues but I'm not as up to date potentially as some of the tech support guys that do support the PDU. If you don't get someone answering here, I suggest calling the guys in support. The last one you mentioned sounds like it's toasted but I'm not 100% sure. Support will know better. Steve

(CID:106207916)

[DCIM_Support]

Hi Guys - please assist I am in process of upgrading the firmware on our 1phase and 3phase PDU's. So 1phase run seamlessly however the 3 phase i am struggling with. See attached error I am upgrading via serial to USB cable. Version AP7855 A ( install file - apc_hw02_aos392_rpdu392) Any ideas ?

Error 3phase.JPG

(CID:110007879)

[DCIM_Support]

Hi Yusef, https://community.exchange.se.com/t5/Critical-Power-Cooling-and-Racks/ct-p/critical-power-cooling-ra... may be a better place to post PDU posts. Since the original post was DCE related, it was also good to post here. This is a different question. Judging by the error, it looks like the firmware upgrade utility could not log into the card / device to perform the upgrade. Please re-verify the user name and passwords are correct and that FTP is actually turned on in the card / device. Steve

(CID:110007908)

[DCIM_Support]

I have had an issue where upgrading to firmware 3.7.4 where I lose all network connectivity with the units. I haven't been able to determine what happens yet. I had to serial into the device and downgrade the units manually.

If I ever figure out what happened I will update this post.

(CID:105465570)

[DCIM_Support]

Hi John, looks like I might also have another 2 coming back soon, that failed during firmware upgrade and have also lost serial. I'll check them out when they get back to me to see if I can get them working and post back here.

(CID:105465629)

[DCIM_Support]

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.