Multiple APC Smart-UPS SRT 1500 (SRT1500RMXLA) Warm Resetting NICs

[J-E-1984]

As Title describes, we have multiple UPSs deployed, and the NICs reboot every 10 or so minutes for some reason.

Following testing has been done:

UPS can ping Gateway

Time is the same on both Switch/UPS

Reseated NIC and verified/replaced Eth cable connecting UPS to switch...

We've done everything sans replacing the NIC, which we'd like to not do on multiple units at multiple sites. 

The reason this is problematic is it's causing us to get false positives that the device has reset, when it's been up (and only failed due to the NIC rebooting) - in addition, or switches are deluged with interface up/down entries with every reset.

Any thoughts?

UPS Info:

APC Smart-UPS SRT 1500 MDL:SRT1500RMXLA

Firmware Revision: UPS 07.4 (ID1025)

Battery SKU: APCRBC155

Manufacture Date: 04/16/2021

NIC Info: 

MDL: AP9641

Hardware Revision: 5

APC OS (AOS): Version v1.4.0.23

APC Boot Monitor: Version v1.1.0.2

Log (sample - IP's removed for security purposes)

09/06/2022,13:52:32,Device,"UPS: Restored the local network management interface-to-UPS communication.",0x0101
09/06/2022,13:52:29,Device,"Environment: Restored the local network management interface-to-integrated Environmental Monitor (Universal I/O at Port 1) communication.",0x0344
09/06/2022,13:52:29,System,"Network service started. IPv6 address ---.---.---.--- assigned by link-local autoconfiguration.",0x0007
09/06/2022,13:52:27,System,"Network service started. System IP is ---.---.---.--- from manually configured settings.",0x0007
09/06/2022,13:52:23,System,"Network Interface restarted.",0x0002
09/06/2022,13:40:18,Device,"UPS: Restored the local network management interface-to-UPS communication.",0x0101
09/06/2022,13:40:15,Device,"Environment: Restored the local network management interface-to-integrated Environmental Monitor (Universal I/O at Port 1) communication.",0x0344
09/06/2022,13:40:15,System,"Network service started. IPv6 address ---.---.---.---assigned by link-local autoconfiguration.",0x0007
09/06/2022,13:40:14,System,"Network service started. System IP is ---.---.---.--- from manually configured settings.",0x0007
09/06/2022,13:40:09,System,"Network Interface restarted.",0x0002
09/06/2022,13:28:04,Device,"UPS: Restored the local network management interface-to-UPS communication.",0x0101
09/06/2022,13:28:02,Device,"Environment: Restored the local network management interface-to-integrated Environmental Monitor (Universal I/O at Port 1) communication.",0x0344
09/06/2022,13:28:01,System,"Network service started. IPv6 address ---.---.---.--- assigned by link-local autoconfiguration.",0x0007
09/06/2022,13:28:00,System,"Network service started. System IP is ---.---.---.--- from manually configured settings.",0x0007
09/06/2022,13:27:55,System,"Network Interface restarted.",0x0002
09/06/2022,12:58:43,Device,"UPS: Restored the local network management interface-to-UPS communication.",0x0101
09/06/2022,12:58:41,Device,"Environment: Restored the local network management interface-to-integrated Environmental Monitor (Universal I/O at Port 1) communication.",0x0344
09/06/2022,12:58:41,System,"Network service started. IPv6 address ---.---.---.--- assigned by link-local autoconfiguration.",0x0007
09/06/2022,12:58:39,System,"Network service started. System IP is ---.---.---.--- from manually configured settings.",0x0007
09/06/2022,12:58:34,System,"Network Interface restarted.",0x0002

[BillP]

@J-E-1984 

I recommend you update to the latest NMC firmware 

https://www.apc.com/us/en/product/SFNMC3SU221/ups-network-management-card-3-firmware-v2-2-1-for-smar...

[J-E-1984]

I'll give that a try and see if it fixes it.  Sadly that's likely going to require some downtime and a scheduled maint.  

Thanks. - I'll report back if it does.

[J-E-1984]

just had a newer unit installed at another location and the same is happening on newer firmware:

Application Module

Name: Version v2.0.0.5

APC OS (AOS): Version v2.0.0.6

APC Boot Monitor: Version v1.4.0.2

[J-E-1984]

Sadly the flapping continues.

We updated the firmware on a test device and found that it stopped, briefly, but continued to flap after the fact.

[J-E-1984 · Accepted solution]

Well... after opening a ticket with APC, and attempting numerous methods of formatting, firmware updating, and the like, We have our final APC unit ready to deploy tomorrow.

By final I mean the remainder we have yet to deploy will be returned and we're merely attempting to mitigate the problems... using the older non-monitored APC devices in our deployments for now until we can find a new vendor.

Sadly, if you've experienced the same issue and are on this forum: There appears to be no solution.  Running around with APC's "Support" for a full week netted us nothing, and we tried everything from bypassing the Watchdog protocol to stepping down the NIC speed to as far as 10half just to see if, maybe, it would be stable, and I cannot tell you how many formats, firmware updates, and so on... 

Now we're looking to monitor via software (USB to local server) in an attempt to bypass the NICs altogether, as replacing the deployed units isn't an option for us as the cost of manpower alone isn't worth it.

Disappointed that APC, such a large supplier of these products, cannot seem to resolve a simple issue on their NIC.

If you have a real solution, I would love to see/hear it.  

Sadly, seems that all solutions provided are temporary or only work on one or two units.  23 out of 24 of our units have this issue.  It's impressive to say the least... Sadly soon to be 24 out of 25.  

[Hectormp]

Just wondering, why does it appear as solved if this is not solved yet? 

[JonPro]

Hi @Hectormp 

There is a possibility that the one who started the thread has marked the case as solved.

Hi @J-E-1984 

Apologies for the inconvenience this has caused.  I've handled a case that shows the same symptoms as what you've had and below are the steps that we've done.

- Upgrade FW to the latest release version
- Pull the debug file from the NMC, and format the card using either the pinhole reset button or the format command through Bootmonitor
- If port flapping does not stop, set up a continual ping to the NMC (ping <ip address> -t)

I hope you could do some of them and let us know if there will be any improvements.