AP8881 management constantly reboots

[peterskovgaard_apc]

We have an AP8881 PDU that constantly restarts its management interface. I have managed to grab a debug zip from the PDU, it is attached. Any clues as to what is happening?

When the management interface is up for a short while, the lastrst command yields "05 Failsafe Reset". The firmware versions are:

Bootmon:        bootmon:v1.0.2

AOS:            aos:v6.1.3

App:            rpdu2g:v6.0.9

[peterskovgaard_apc]

Hi again. Sorry for the wait, I have been on holiday.

The config.ini was posted directly from the device without modifications. I have performed a format of the device, but the result is the same, it continuously restarts. We have more than 50 devices, of which a few have started exhibiting these problems. This may be linked to a firmware upgrade that was performed months ago. All the devices are under control of StruxureWare, if that makes any difference.

I'll upload a configuration dump from a device that is working, maybe you can see some sort of difference.

Thanks!

[BillP]

Thanks! We are investigating a few issues of this type. A few possible guesses at this time are NTP or something related to StruxureWare Data Center Expert. Can you tell me, are the devices that are working/not working on the Data Center Expert public network interface or private LAN? What other devices are on the same network as these PDUs (APC devices, non-APC devices, etc)?

I don't know if on one that's not working, you want to see if disabling NTP does anything different. If you do disable it, it was recommended to me to still put 0.0.0.0 in the server field to rule out a possible issue of a valid IP (yet inactive) causing a problem.

[BillP]

Did you reboot the management card after? And when you disabled NTP, did you put 0.0.0.0 in the field afterwards?

[peterskovgaard_apc]

Disabling NTP didn't make any difference. The management card still reboots regularly with the reason "05 Failsafe Reset".

[peterskovgaard_apc]

We have an AP8881 PDU that constantly restarts its management interface. I have managed to grab a debug zip from the PDU, it is attached. Any clues as to what is happening?

When the management interface is up for a short while, the lastrst command yields "05 Failsafe Reset". The firmware versions are:

Bootmon:        bootmon:v1.0.2

AOS:            aos:v6.1.3

App:            rpdu2g:v6.0.9

[BillP]

Let me check into it. Based on the frequency, I think something is corrupted but I would like to ask someone else to review the logs and then report back to you.

[peterskovgaard_apc]

Hi Angela,

Thanks, that would be very helpful. I suspect that it may be the management processor itself that is faulty somehow.

Peter

[BillP]

99.9% of the time it is a firmware thing - like a particular setting combination, memory issue, etc. Sort of like mini Windows BSOD in a certain sense (assuming it is not am expected reboot).

Let net me check in on this to see where we are at with the logs. Usually if there is nothing very obvious, we suggest "formatting" the PDU (the Network Card inside it technically) and seeing if it continues to do it with the same configuration in case something became corrupted. If it still does it, then we format and reconfigure slowly to see if a particular setting triggers it and then we go from there. Before I had you do that though, I just wanted to have someone look at the logs to make sure I did not miss anything since I have seen this on a few other PDUs recently and it peaked my interest.

[BillP]

Hello again. So from what we can tell, one of the "tasks" in the operating system is not checking in properly. I need to try and replicate that. I am going to load your config.ini from your PDU onto my simulator and see if I can re-produce it. Did you remove any settings from config.ini before posting that file? From what I can see, you have an almost factory default setting and are using DHCP - is that fair to say?

In the meantime, I'd suggest that the quickest thing will be to do what I suggested and see if any improvement on your end - "format" the device and see if it reacts differently at all and stops rebooting or if it starts doing the same thing. This will wipe every setting/item on the PDU beyond the firmware so it'd have to have its TCP/IP settings set up again but it looks like you just use DHCP anyway. You could issue the format command via CLI and follow the instructions it gives you after entering it to begin the process.

Let me know if you have any questions before you get started and if you can confirm what I said on config.ini so I can try to see if my simulator will do the same thing.

Thanks!

P.S. - do you have any PDUs in the same network location with similar or different network configurations that are behaving properly? And did you recently do anything, like upgrade the firmware or anything that rings a bell?

[robertsa_apc]

Was a cause or solution ever figured out for this? I have the the same problem with my PDUs even after upgrading from 6.1.3 to 6.3.2. 

[BillP]

The versions you're mentioning are the AOS or APC Operating System version. We did identify a problem with a PDU application version outlined here -> 

[robertsa_apc]

I had 6.0.9 I went through part of the process. I was hoping I wouldn't have to power cycle the PDU. I will do a power cycle and see how it goes.

I also have another problem that the firmware update didn't fix. I am not able to connect to the PDUs web interface through from a different network. I can ssh, telnet, and ftp to the PDU from a different network but not connect to the web interface. I have two different networks 10.150.150.x and 10.149.149.x they are able to communicate just fine.

I had a PDU that was still running firmware 5.1.9 on 10.149.149.x, and could connect to it from 10.150.150.x. As soon as I upgraded to 6.1.3 it wouldn't connect to the web interface anymore from 10.150.150.x. I would have to be on 10.149.149.x to connect to the web interface. I replicated this test again except going from 5.1.9 to 6.3.2 and came out with the same results. I have already tried reseting to factory defaults, and clearing all the logs. 

Also with ftp I am not able to GET or PUT things from a different network I have to be on the local network. 

[Terry_Kennedy_apc]

This sounds like the MTU issue described in Knowledge Base article FA235265. I'd post a link to it, but the APC web site is rather unhelpful in providing direct links to articles. You can view it in PDF form

[BillP]

I agree with you Terry on it being this MTU issue. Here is the link format I have to use and just replace the kbase ID at the end: 

[robertsa_apc]

I ended up not power cycling the PDU. Instead, I cleared all the log and data files off of the PDU and that along with the upgrade to 6.3.2 fixed the restarting issue. Setting the MTU to 1382 allowed me to access the web interface, but ftp download still doesn't work. That would be great if you could get me the beta to try. What is the one tweak?

[BillP · Accepted solution]

It is something over my head but something inside of the TCP/IP stack that we modified.