NIC Times Out After About 15 Seconds Of Inactivity

[mhaneyit]

I have a UPS that seems to timeout every 15 seconds or so. The switch will show the port go down every 15 seconds or so and then come back up after about 5 seconds. If you are logged into the UPS and are inactive for about 15 seconds and then try to navigate to a different screen it will lag or give an error and then after about 5 seconds or so it will start responding again.

I found if I ran a continuous ping to its IP the port wouldn't show it drop and it wouldn't hang up while you were logged in and inactive. Also, as long as you stayed active while logged in (changed screens every few seconds), without the ping running, it also wouldn't drop.

I tried hard setting the speed/duplex to 100 Full on the NIC and the switch port, as I saw a post where someone else said that might help, but it still drops.

The switch log is completely full of these port flapping events, since it happens every 15 seconds or so. And I am constantly getting alerts for the device going into ping failure.

Anyone seen this before or have any recommendations?

[Shaun]

Hi @mhaneyit,

Could I ask you to raise a case for this with your local support?  I'd really like to see the debug logs from the card, but that works better over a ticket than the forums, so we can keep any identifying data off the public forums.

If you have any problems at all getting traction, feel free to send me the case number on here and we can make sure the ticket's getting the attention it needs.

Thanks,

  Shaun

[prabaecce222]

Hello APC Team,

I am also having same kind of issue. Could you please help me to resolve this issue.

[dpierce4776]

Its hard to tell as there are a lot of variables. You could have a bad card, a bad port on the switch, a bad cable, or the port settings are not taking effect as they should.

First I would make sure the switch and card are set to auto for speed and duplex. Then reboot the switch and the network card. If that does not work, try replacing the network cable with a known good one. If it is still doing it try a different port on the switch. As well as plug a laptop or other device into the same port on the switch to see if it has the same issues. 

If the issue follows the card, with a new cable, then you have a bad card. If not then it could be the switch port

[fasteddye]

We probably have over a thousand APC UPS deployed and we are experiencing these same symptoms as the original post across a good number of the UPS devices.  The UPS devices are on different switches and also at different locations but all exhibiting the same symptoms as stated above with ports flapping.  I was excited to come across this thread in my last attempt searching after a lot of troubleshooting.  There haven't been recent posts but wanted to check if anyone has had any resolution to this problem?   Many Thanks!

[Teken]

As a starter if the NMC is a 9630, 9631, 9640, 9641 it should be on the latest 7.XX firmware branch. 

Any other NMC should be on whatever latest firmware branch available even if it’s EOL.

Once the NMC is on the latest / last firmware. Back up the NMC user settings and than default the NMC to a OEM state.

Input just the basic network configurations and monitor. If the flapping is gone upload the user defined settings and monitor.

If the NMC begins to flap you have problems in the network configuration. If the NMC operates fine with the user settings rinse and repeat for the rest in the corporation.

NOTE: As a break fix and to help isolate another possible root cause. Lock the port speed at the (switch) to 10/100. Do not do this at the NMC leave it in Auto. 

Report back what you see and observe along with active faults and error logs from the switch / NMC.

[fasteddye]

There are NMC 9630, 9640, and 9641 in the environment with mostly Smart-UPS 1500 and Smart-UPS SRT 1500. It looks like there may be quite a mix of firmware versions installed across these. It was mentioned that latest 7.XX firmware should be installed however when I go to APC site, v2.5.0.6 is listed as most recent for 9640 and 9641. It looks like we have mostly v2.0.0.5 installed and from what I have tracked down so far these all are the ones with switch ports flapping.

I did attempt to manually set speed to 10 or 100 and duplex fulll only on the switch port side but the problem still was present. That suggestion gave me pause anyway since that would not be best practice network configuration to have one side manually set and the other side auto negotiate. Thought I would give it a try anyways.

I have attached a snapshot of syslog from a switch showing port flapping.

It seems logical next step would be to pick a couple 9641s running v2.0.0.5 and upgrade to the latest v2.5.0.6 unless I am missing something with v7.XX being available for the 9641s. Please let me know if I am not seeing the latest firmware correctly on the APC site for the 9641.

Thanks.

[Teken]

There are two firmware branches available for the AP9640 / AP9641 NMC 3. One for the secure subscription and the other for regular none subscription.

Choose the firmware branch that applies to your environment. 👍

As it relates to the NMC 2 (AP9630 / AP9631) the latest firmware is 7.1.2.

Port Speed: It’s not best practice but helps isolate and identify root cause. There are millions of legacy 10/100 devices which do not operate correctly on the latest generation of 1GB / 10GB switches. As such locking the switch port to 10/100 insures negotiation compatibility.

Keep in mind locking the port speed (10/100) at the switch should ONLY be done on a NMC 2 AP9630 / 9631 as they only support 10/100 for testing.

Whereas the NMC 3 AP9640 / 9641 support all three port speed of 10/100/1000. 

Let us know how things turn out. 👍

[fasteddye]

How would you determine if secure subscription or regular none subscription for our environment?

Thanks.

[Teken]

Generally speaking the 2.X.X firmware branch relates to the none Secure NMC (Subscription) for the AP9640 / 9641.

The 3.X.X relates to the AP9640 / 9641 SNMC (Subscription) firmware branch. Going off memory if you try to install the wrong one the system will cry indicating that firmware is incorrect / invalid. 🤦‍♂️

Regardless, you should contact whomever is the primary lead in your organization to confirm what hardware (IS) using the SNMC Subscription! ☝️

It goes without saying don’t download a Symmettra (SY) firmware vs the Smart-UPS version. As you’ll be in a world of hurt if you can’t recover the NMC! ☹️

Lastly, it’s imperative that you understand and have on hand the NMC FAQ. As to how the recovery process works and the steps to complete should things go sideways!

Ignoring the above will quickly result in Hero to Zero . . . 

NOTE: The last firmware release for the AP9630 / 9631 NMC 2 is v7.1.2

[fasteddye]

UPDATE:

We picked about 10 Smart-UPS 1500 with NMC3 AP9641 (v2.0.0.5) that were showing the flaps to upgrade and monitor.  It appears we are SNMC (Subscription) firmware, so these were upgraded to su v3.1.1.1.  The switch port flaps immediately stopped after the upgrades and we have not seen any further flaps after 24 hours.  We are encouraged with the results.  We will continue to monitor and probably upgrade all the v2.0.0.5 in the enterprise regardless of the switch port flaps being present.

Thank you for the detailed input provided, it has been very helpful.

[Teken]

Appreciate the quick feedback and good to see a positive outcome. If the information I provided helped you solve the issue please do mark that as a solution.

As it will help the next person asking the same. 👍