AP9630 keeps dropping off network

[BillP]

Hi All,

Not sure if someone has ever seen this before, but for some reason the AP9630 keeps dropping off the network.

The only way i can get back into it is to either remove the data cable, or shutdown and reopen the switch port the mgmt card is connected to.

According to the card it doesn't go offline, and according to the switch end the port is still up.

Anyone have any suggestions i can look at. (SNMP is setup correctly as our monitoring centre keeps advising me when it drops out.)

Card can stay connected anywhere from 1hr to 3-4 days before dropping out again.

I have replaced both the card and the UPS and this is still happening. I have swapped ports on the switch and replaced the data cable and still happening.

Help. Anyone?????

[Terry_Kennedy_apc]

Are you running the latest firmware (6.4.0 at this time, I think)? I seem to remember an issue where high amounts of broadcast traffic would cause the card to drop off the network.

[BillP]

Hi Justin,

My thought was that SNMP is the networking protocol used by the monitoring center - how are you certain that the the AP9630 is dropping off of the network rather than just the SNMP interface/protocol having an issue? Do you try to log in to the web or telnet/ssh interfaces and very the AP9630 NMC is unreachable?

That is my first thought - verify/clarify that ALL network connectivity to AP9630 seemingly goes unresponsive versus just SNMP? Is it SNMPv1 or v3?

And are you saying that the connectivity does not recover and that is why you have to remove the data cable or bounce the switch port? How long have you left it in an "offline" state by chance to see if connectivity restores itself?

As Terry suggested, knowing the firmware version would be a good idea.

In network environments with lots of traffic, we saw an issue on older firmwares but the issue did not "fix itself" like you're describing with a bounce of the switchport or removing the data cable. It just caused high latency when the NMC had to process lots of data packets going by on a noisy network and was constant.

The other issue we had in certain environments only affected HTTP or HTTPS access over a WAN or VPN connection http://www.schneider-electric.us/en/faqs/FA235265 I don't think you have this issue..

The only other thing that comes to mind is what was in this thread http://forums.apc.com/spaces/7/ups-management-devices-powerchute-software/forums/general/10687/ap963... but that also sounds a bit different than what you're describing. Though, there very well could be something on your network that is somehow changing the default gateway of your management card? 

You may want to look into doing a network packet capture, if possible, when the problem arises to see if you can see anything.

[BillP]

Hi Terry,

I just checked, Again had to shut and reopen the switch port to be able to connect. Version says

Name:	aos
Version:	v6.0.6
Date:	Apr 5 2013

[BillP]

 

Hi Angela,

I have tried http, https, ssh, telnet.... the only way i can connect back in is to drop the link which i have just had to do again 5 mins ago. (Shutdown port on switch)

We are running SNMPv1.

Unit has been left for a month and still unable to reconnect.

Firmware is 6.0.6 (I will attempt to update firmware today and see how that goes.)

[BillP]

I have updated the firmware to 6.4.0 and will monitor the link to see when or (fingers crossed) if.

[BillP]

Hi Justin,

Ok, thanks for the update. That will be a start. Sounds very strange. Would be curious (depending on where you are located on the network) if a trace route looks any different before/after.

Also, do you have other APC Network Management Card devices that work or don't work?

[BillP]

HI Anglea,

This unit is 1 of approx 130 AP9630 management cards we monitor. The rest of them all work. This is the only one that doesn't.

Just to update the status of the card in question, it has dropped again. So firmware and DNS pointers have not made a difference. (Did DNS as there was a comment in the logs about DNS Errors) I did clear the event logs after the upgrade. I'm going to look at possiblity of high voltage causing the drop, as there is an event regarding this about the time it dropped.

I'll keep you posted.

[voidstar_apc]

If you haven't already done so, I would swap the AP9630 with one in a different UPS to see if the problem follows the card. If the problem /doesn't/ follow the card, I'd try plugging it into a different ethernet switch.

[Anonymous user]

I'm seeing this behavior as well.  Completely intermittent.  Both ICMP (Ping) and SNMP-based sensors are all failing at the same time on this one UPS w/a AP9630 in it, randomly.  I'm swapping out the cards to see if the behavior follows the card and if so, RMA the card.

Also worth noting that I have upgraded to AOS 6.4.  That did not resolve the behavior unfortunately.

[BillP · Accepted solution]

Has anyone come up with a solution for this problem. I also have this issue and it is jumping across different cards and models. The unit will sometimes come back for a time than fail again. I have spoke to APC support but that did not work out. I have the latest firmware and verified when the snmp is down the device is still accessible locally through serial or telnet. Any info would be very useful. Right now I have 6 down out of 40.