UPS Network Management Card 3 (AP9640) - Switch interface to UPS mgmt card interface bouncing.

[Joho2024]

Seeing something strange here.  From switch logs, i'm seeing the switch interface connected to the UPS mgmt card (AP9640) bouncing every minute or so but only 1 interface reset is recorded on the switch interface.  Switch interface is clean, no errors, connected at Full-duplex 1000Mb.  Firmware on the mgmt cad is UPS 09.3 (ID18).  These are new AP9640 mgmt cards and new C9300 Cisco switches.  I'm seeing this same issue across several switch stacks I turned up recently.  I'm not having this issue on older APC/Schneider UPS mgmt cards.  Anyone see this issue before?

----------------------------------------------------------------------------------------------------------------------------

interface GigabitEthernet1/0/47
description Managed UPS
switchport access vlan ???
switchport mode access
spanning-tree portfast
end

Feb 23 09:04:15.297 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/47, changed state to down
Feb 23 09:04:16.298 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/47, changed state to down
Feb 23 09:04:20.252 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/47, changed state to up
Feb 23 09:04:21.252 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/47, changed state to up
Feb 23 09:04:57.930 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/47, changed state to down
Feb 23 09:04:58.931 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/47, changed state to down
Feb 23 09:05:02.885 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/47, changed state to up
Feb 23 09:05:03.886 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/47, changed state to up
Feb 23 09:05:20.752 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/47, changed state to down
Feb 23 09:05:21.752 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/47, changed state to down
Feb 23 09:05:25.708 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/47, changed state to up
Feb 23 09:05:26.707 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/47, changed state to up
Feb 23 09:06:55.120 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/47, changed state to down
Feb 23 09:06:56.119 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/47, changed state to down
Feb 23 09:07:00.073 EST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/47, changed state to up
Feb 23 09:07:01.074 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/47, changed state to up

-----------------------------------------------------------------------------------------------------------------------------------

#sh int gi1/0/47
GigabitEthernet1/0/47 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is a410.b622.fcaf (bia a410.b622.fcaf)
Description: Managed UPS
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters 1w1d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 1000 bits/sec, 1 packets/sec
72929 packets input, 8993304 bytes, 0 no buffer
Received 6968 broadcasts (5662 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 5662 multicast, 0 pause input
0 input packets with dribble condition detected
1149699 packets output, 155050470 bytes, 0 underruns
Output 44179 broadcasts (975135 multicasts)
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[Joho2024]

Anyone Schnieder available to review?

[Joho2024]

This is interesting, when I run a constant ping to the UPS mgmt card, the switch interface is stable, no longer bouncing.  I opened a ticket with Schneider on this issue.  Hopefully someone can point me in the right direction.  Thank you.

Packets: Sent = 3396, Received = 3396, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 14ms, Maximum = 42ms, Average = 16ms

[Joho2024]

Well, I tried reseating the UPS mgmt card, same issue.  I then defaulted the UPS mgmt card and reconfigured UPS, same issue.  I replaced the UPS mgmt card with a new AP9640, this cleared the issue.  All the UPS mgmt cards came from the same shipment.  The only common denominator with the UPS mgmt cards with the issue was I used one spare UPS on campus to preconfig the IP, SM, GW on these UPS mgmt cards before deployment (different subnets).   I can't see how this would cause an issue but hopefully someone from Schneider will respond.  Thanks.

[dpierce4776]

A couple of things

1. Sadly it takes a while for the admins to approve response so there could be some questions coming that have not been approved yet.

2. Maybe I over looked it but I do not see any speed setting on the switch int he logs you posted. The speed and duplex should be both set to auto. I know this is usually default but had to say something.

3. have you checked the port speed on the card?

4. have you checked the the network cable?

5. have you tried formatting the card(s) that are not working? This does more than the "reset to default" setting in the card. The format command completely erases all old and "stuck" data. The factory default setting only sets the settings it can change back to default.

Really it sounds like the cards are getting a duplex mismatch for some reason. Maybe the old switch port is set wrong and the cards are holding on to that. With the new switch port is not liking it. have you tired other ports on the same switch? To my knowledge there is nothing in the UPS itself that can change settings on the card. 

[EdRoche]

Hi All

Anyone come up with a solution for this yet?

I am having same exact issue. I have tried the backup configuration and format. I have have tried statically setting port speed and duplex on both sides. I have the following:

• Smart-UPS 1500, SMT1500C
• NCM: AP9640
• Firmware: UPS 17.2 (ID1027)
• Cisco C9200 switch

Thanks!

[Teken]

I gather what both of your trying to say is port flapping is being observed. The first thing that should be done is to insure the physical network is sound.

That would entail replacing the network cable with a certified one. Even moving the same cable to another port to see the results.

You could also use the cable test on the switch to affirm its sound. Do all the above and report what is observed! 👍

[Joho2024]

Network was verified before creating this post.  Also, if you look at the show int above, you can see the switch interface is 100% clean.   So, after seeing this exact same issue on all new AP9640 cards i received on a September shipment, I opened a ticket w/APC.  The fix was upgrading the AP9640 to "apc_hw21_su_2-5-0-6" using the NMCFirmwareUpdateUtility.exe.   This cleared the issue.