AP9731 network card dropping connection

[Anonymous user]

Our company has 4 x AP7931 Rack PDU's with network issues. All issues are all the same. We have well over a dozen of these PDU's, but only 4 are giving trouble. It seems that I can get them all to connect to the network, update their firmware or whatever and then after a few seconds/minutes/hours they will just stop responding to the network. I can go to them and reset them by pushing the mini reset button and they will work again for any amount of time. It seems randomly they will just stop responding. This does not affect the actual use of the actual PDU but we can no longer remote an unstable network card. Are these cards now faulty? I tried updating the firmware, which did install but did not help the disconnect issue. We have multiple more PDU's that are still responding to the network etc so I'm unsure what to do. Please let me know if anyone can help.

[BillP]

can you provide a little bit of the event log from any of the PDUs (or all of them if there are different events)? what type of events are you seeing on it if you cant provide the event log?

also, when the PDU drops connection, what color/state (flashing or solid) LEDs do you see on the management card for the Status and Link LEDs when you go to push the reset button?

do these PDUs have anything in common that your others don't - for instance - same gateway, physical switch, subnet, etc

answering these questions can at least get us started 🙂

[Anonymous user]

Ok, so first off... I went to one of the branches today. All of our working and non working ones are at separate branches around our county.

Branches are set up like this (Only an example) 192.168.(branch number).xxx

All branches then connect to our main campus branch. I noticed for starters today that all the working PDU's were manufactured about 2006 to present. The one that I went to see today was manufactured in 2004. If I was able to visit the other 3 branches I would but they are way out in the county and it's not worth the drive right now to go there just for this.

Here are some Event logs from the last time I visited it and got it back up and running again. This is filtering nothing so everything should show up.

08/27/2008 11:55:55 System: Configuration change. Event log web display time selection.
08/27/2008 11:55:07 System: Web user 'apc' logged in from *(IP Address)*.
08/27/2008 11:55:00 System: Configuration change. DHCP lease date.
08/27/2008 11:55:00 System: Set Date or Time.
08/27/2008 11:52:43 System: Console user 'apc' logged out from serial port.
08/27/2008 11:52:09 Switched Rack PDU: Communications established.
08/27/2008 11:52:09 System: Network service started. System IP is *(IP Address)* from DHCP server *(IP Address)* with 43200 second lease.
08/27/2008 11:52:09 System: Configuration change. DHCP lease date.
08/27/2008 11:52:06 System: Console user 'apc' logged in from serial port.
08/27/2008 11:51:53 System: Warmstart.
06/04/2008 14:24:05 Switched Rack PDU: Communications established.
06/04/2008 14:23:57 System: Network service started. System IP is *(IP Address)* from current DHCP lease.
06/04/2008 14:23:57 System: Warmstart.
06/04/2008 14:11:18 System: Web user 'apc' logged out from *(IP Address)*.
06/04/2008 14:08:07 System: Configuration change. Event log web display time selection.
06/04/2008 14:07:33 System: Web user 'apc' logged in from *(IP Address)*.
06/04/2008 14:07:25 System: Web user 'apc' logged out from *(IP Address)*.
06/04/2008 14:04:35 System: Web user 'apc' logged in from *(IP Address)*.
06/04/2008 14:04:03 Switched Rack PDU: Communications established.
06/04/2008 14:03:55 System: Network service started. System IP is *(IP Address)* from current DHCP lease.
06/04/2008 14:03:55 System: Warmstart.
06/04/2008 13:50:19 System: Console user 'apc' logged out from serial port.
06/04/2008 13:47:07 System: Console user 'apc' logged in from serial port.
06/04/2008 13:47:07 System: Web user 'apc' logged out from *(IP Address)*.
06/04/2008 13:45:43 System: Configuration change. Event log web display time selection.
06/04/2008 13:40:19 System: Web user 'apc' logged in from *(IP Address)*.
06/04/2008 13:39:03 System: Web user 'apc' logged out from *(IP Address)*.
06/04/2008 13:34:41 System: Configuration change. DHCP lease date.
06/04/2008 13:34:41 System: Set Date or Time.
06/04/2008 13:34:02 System: Web user 'apc' logged in from *(IP Address)*.
06/04/2008 13:33:39 Switched Rack PDU: Communications established.
06/04/2008 13:33:38 System: Network service started. System IP is *(IP Address)* from DHCP server *(IP Address)* with 43200 second lease.
06/04/2008 13:33:38 System: Configuration change. DHCP lease date.
06/04/2008 13:33:23 System: Warmstart.



Here is what I did when I got there today.

1. Looked at the lights. Solid green and never flickered. Amber (link) flickered intermittently but never flickered fast.
2. Connected via serial port. Nothing when I used putty.
3. Reset PDU with paper clip and lights flickered like a normal DHCP connection was trying to be established.
4. Connected via putty and saw that device was now working.
5. Connected via web browser and logged in.
6. Checked link lights and they appeared to flicker EXACTLY the same as before I touched it.
7. I did some minor browsing because it has cut out before just browsing around the device.

If we need to coldstart it I can do this, but of course only after branch hours. Hope this is all helpful enough! Thanks again if anyone can help me.

Message was edited by: Bosnian Child

[BillP]

hi-

thanks for all the info. event log looks pretty normal. powercycling never hurts so if you can do that, i'd say go for it.

which firmware did you reflash the PDUs to? AOS 3.5.9 and rpdu APP 3.5.8 is the latest version.

also, if you are able to, maybe you can try resetting one of them to defaults to monitor to see if it does the same thing.

[Anonymous user]

I did a factory reset already sadly. It's almost like something goes to talk to it, DNS or DHCP and it just stops responding. If i connect a serial port to it now it'll just sit there. Any bad batches maybe for this year? I'm hoping that the others I see are frm 2004 also or something and they are just reaching end of life.

Well poo, at 6 hours about 10 seconds ago while I was typing this it died. I'll go tomorrow and see the logs again. I did have it once stop after a few minutes though. Sigh.

So tomorrow we shall see. I was pinging the device and piping it to a txt file, but since it died i guess it was almost 6 hours if not exactly 6 hours to the second this time.


Firmware is version:

Hardware Factory

Model Number: AP7931
Serial Number: **
Hardware Revision: B2
Manufacture Date: 06/25/2004
MAC Address: **
Management Uptime: 0 Days 5 Hours 58 Minutes

Application Module

Name: rpdu
Version: v3.5.7
Date: 03/17/2008
Time: 13:24:04

APC OS (AOS)

Name: aos
Version: v3.5.8
Date: 03/17/2008
Time: 11:51:00

[Anonymous user]

Yeah, that's about the last thing I've tried, power cycling I mean. I was hoping to make it a last resort. But this is definitely something that can be done before or after hours. Thanks a ton again for all the help! Before I updated to this firmware it was having problems, that is why I updated, but you NEVER KNOW. I know how pc's work, they are like cars. Kicking can help, but in the end it's always something you gotta say DUH to.

[BillP]

yeah i hear ya..

only thing is because there are multiple ones doing this, i feel like we are missing something that they have in common. as I mentioned, i think we can rule out a "bad batch" or anything like that. next, my bet would be firmware unless there is something with the networking - which i doubt since you have plenty of PDUs that are fine, but who knows. let us know how you make out!

[Anonymous user]

So I'm bumping this back up because I ended up having to remove the power strip.

First off, when I did a cold reboot of the PDU it never came back online. It just showed on the device that it was overloaded constantly. So I finally just swapped out the PDU with a different for temporary.

When I was finally able to get it to work, I realized that I could only log in to it via console. All the ports were set to the OFF setting. I set them all to ON and presto, away we go. I reset it back to defaults, upgraded to the newest firmware and waited. After a few hours it stopped communicating still (sad).

I have now set it to not use DHCP (manual) and it has lasted through the night so far... BUT.... this is what it's doing in the log and I wasn't sure if this is common or maybe it can help track down the problem. Every 20 minutes it's basically rebooting. This is just for the past few hours, but the log has been showing this all through the night rebooting every 20 minutes. So far though it is maintaining communication which is good technically. I'm guessing it would belly up before when I was logged in to the device and after 20 minutes if I was logged in to it, it would belly up... maybe? When set to manual it severs my connection but after a few seconds of reboot i can reconnect.

Oh and it's set to get date and time every 1 hour.

Thanks again for all the help!

09/10/2008 09:30:03 System: Web user 'apc' logged in from IP Address.
09/10/2008 09:29:31 Switched Rack PDU: Communications established.
09/10/2008 09:29:22 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 09:29:22 System: Warmstart.
09/10/2008 09:16:17 System: Web user 'apc' logged out from IP Address.
09/10/2008 09:10:02 System: Web user 'apc' logged in from IP Address.
09/10/2008 09:09:29 Switched Rack PDU: Communications established.
09/10/2008 09:09:20 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 09:09:20 System: Warmstart.
09/10/2008 09:08:31 System: Web user 'apc' logged in from IP Address.
09/10/2008 09:05:44 System: Web user 'apc' logged out from IP Address.
09/10/2008 09:03:28 System: Set Date or Time.
09/10/2008 09:00:22 System: Web user 'apc' logged in from IP Address.
09/10/2008 08:49:27 Switched Rack PDU: Communications established.
09/10/2008 08:49:19 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 08:49:19 System: Warmstart.
09/10/2008 08:29:26 Switched Rack PDU: Communications established.
09/10/2008 08:29:17 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 08:29:17 System: Warmstart.
09/10/2008 08:09:24 Switched Rack PDU: Communications established.
09/10/2008 08:09:15 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 08:09:15 System: Warmstart.
09/10/2008 08:03:23 System: Set Date or Time.
09/10/2008 07:49:21 Switched Rack PDU: Communications established.
09/10/2008 07:49:13 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 07:49:13 System: Warmstart.
09/10/2008 07:29:20 Switched Rack PDU: Communications established.
09/10/2008 07:29:11 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 07:29:11 System: Warmstart.
09/10/2008 07:09:18 Switched Rack PDU: Communications established.
09/10/2008 07:09:09 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 07:09:09 System: Warmstart.
09/10/2008 07:03:17 System: Set Date or Time.
09/10/2008 06:49:16 Switched Rack PDU: Communications established.
09/10/2008 06:49:08 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 06:49:08 System: Warmstart.
09/10/2008 06:29:15 Switched Rack PDU: Communications established.
09/10/2008 06:29:06 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 06:29:06 System: Warmstart.
09/10/2008 06:09:13 Switched Rack PDU: Communications established.
09/10/2008 06:09:04 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 06:09:04 System: Warmstart.
09/10/2008 06:03:12 System: Set Date or Time.
09/10/2008 05:49:10 Switched Rack PDU: Communications established.
09/10/2008 05:49:02 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 05:49:02 System: Warmstart.
09/10/2008 05:29:09 Switched Rack PDU: Communications established.
09/10/2008 05:29:00 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 05:29:00 System: Warmstart.
09/10/2008 05:09:07 Switched Rack PDU: Communications established.
09/10/2008 05:08:58 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 05:08:58 System: Warmstart.
09/10/2008 05:03:06 System: Set Date or Time.
09/10/2008 04:49:05 Switched Rack PDU: Communications established.
09/10/2008 04:48:57 System: Network service started. System IP is PDU IP Address from manually configured settings.
09/10/2008 04:48:57 System: Warmstart.

Message was edited by: Bosnian Child

Message was edited by: Bosnian Child

[BillP]

not sure if this is possible - but it'd be good to see if you can directly connect your computer to this PDU so that obviously the PDU and your computer are the only two items on the network. i'd want to make sure this isn't related to the network somehow..

[take a look at this article to see the type of issues i am referencing|http://nam-en.apc.com/cgi-bin/nam_en.cfg/php/enduser/std_adp.php?p_faqid=2572]

we want to rule out the network issues and see if it is one of the other issues.

interesting that it does every 20 minutes..

[Anonymous user]

I unplugged it from the network and then restarted it so it wouldn't be connecting. Since it's set to manual will it cause problems? Here is the log just after I did this. It was still restarting.

09/10/2008 11:45:17 System: Web user 'apc' logged in from 10.2.97.174.
09/10/2008 11:45:08 System: Console user 'apc' logged out from serial port.
09/10/2008 11:44:21 System: Console user 'apc' logged in from serial port.
09/10/2008 11:33:21 Switched Rack PDU: Communications established.
09/10/2008 11:33:12 System: Network service started. System IP is 10.2.97.105 from manually configured settings.
09/10/2008 11:33:12 System: Warmstart.
09/10/2008 11:13:19 Switched Rack PDU: Communications established.
09/10/2008 11:13:11 System: Network service started. System IP is 10.2.97.105 from manually configured settings.
09/10/2008 11:13:11 System: Warmstart.
09/10/2008 11:12:45 System: Restarting. Console user request.

[BillP · Accepted solution]

ahh ok so we unplugged it form the network and logged in via serial and watched it do the same exact thing.

are all 4 of the PDUs doing the same exact thing every 20 minutes?

manual TCP/IP settings won't make a difference.

when you reset this to defaults, did you reupload a config.ini for it or reconfigure it?

what i was hoping is to reset the PDU to complete defaults and then have you wait 20 minutes and see if it still does it.

another thing i thought of is maybe the PDU is trying to do something at a 20 minute interval and realizes theres a problem doing whatever it wants to do, so it restarts thinking it will correct the problem. i am still checking into that but let me know if you can possibly do what i said above.