Reboot Network Management Card after DHCP Request Failure

[BillP]

Hi all.  I have about 12 network management cards I changed over to DHCP Only and selected to only do 3 dhcp attempts.  Thinking it wouldn't be an issue, the actual DHCP scope or vlan was not setup at the time I set the NMC to DHCP Only.

Now the DCHP scope and vlan are all configured correctly.  However, in order for the NMC to receive an IP address, I have to console into the card and set it to DHCP again but with the 0 retries (infinite).

According to the built-in help "If a DHCP timeout/failure occurs, the Card will stop requesting network assignment until it is rebooted."

So, given some cards are at remote locations driving distance away, is it possible to reboot the NMC without having to physically be at the UPS?  So far I've tried shutdown/ no shutdown of the switch port a NMC connects to, but still no IP address is being pulled.

[BillP]

What model cards are these? FYI, for our v6.0.6 and higher of Network Management Card 2 devices, we removed DHCP retries as an option.

Anyway, two ideas we came up with here were:

• If you can access a machine locally/on the same segment as the card, set your ARP cache with the mac of the NMC, then ping with the 117 size payload as outlined in our ARP set up instructions here -> How do you make a Network Management Card communicate on a network? | FAQs | Schneider Electric US
• Try to keep the switch port off for at least 10 minutes. We have a failsafe watchdog built in that if the card does not see network traffic for 10 minutes that it will reboot in an attempt to fix it's interface. This may require valid network settings already to be there but I am not 100% sure.

[BillP · Accepted solution]

I am having this same issue with several cards at remote sites. In one location, I have been able to confirm that the Network Management Card is not sending out a DHCP request, and the IP address defaults to 0.0.0.0. That leads me to two questions:

1. Is there anyway to remotely reset a device if it does not have an IP?
2. Is there a way to change the setting so that the NMC continues to retry DHCP requests indefinitely? 

[josephcoombe]

We are also experiencing this issue. Creates a huge headache b/c a person then has to physically reset NMC card.

>Try to keep the switch port off for at least 10 minutes. We have a failsafe watchdog built in that if the card does not see network traffic for 10 minutes that it will reboot in an attempt to fix it's interface. 

Do NMC2, NMC3, and NMC4 cards all have this behavior? 

>This may require valid network settings already to be there but I am not 100% sure

Is there an official statement on this from Schneider?

[Teken]

All of the NMC you listed has a watchdog timer which you can review on page 9:

https://download.schneider-electric.com/files?p_enDocType=User+guide&p_File_Name=990-3402R-EN.pdf&p_...

DHCP: See page 49-51 as it relates to this topic and the available options.

Questions Ask . .  👍

[josephcoombe]

Thank you - that answers my question.

Followup question about
>If a DHCP server is found, but the request to that server does not work or times out, it stops requesting network settings until it is restarted."
page 51

What if a DHCP server is NOT found? Will a NMC continue to broadcast DORA discover messages indefinitely or is there a timeout?

[Teken]

The watch dog timer is set to 9.5 minutes as such would expect the NMC to try again for a period of time. 

You’ll need to use wire shark or something to monitor what behaviour is seen. There are countless scenarios and how the system is set up that can impact how the system functions and behaves.

As a starter what firmware is on the NMC. How the network infrastructure handles port flapping. 

You’re looking for TX (Transmit) packets not RX packets. As such if the the NMC is defined and monitored by a NMS (SNMP) or ping alive etc activity will be seen.

Wire Shark or similar tools will offer a more definitive answer. 

Questions Ask . . . 

[josephcoombe]

We're doing testing with NMC3 and NMC4 cards to validate reports that the cards appear to timeout and stop polling for DHCP if no DHCP server is available. This is different from other devices in network stack that poll for DHCP indefinitely until a server becomes available.

What we need to understand from Schneider is what the expected behavior should be (e.g. for latest firmware release on NMC3 / NMC4).

Specifically, if a NMC is powered on and begins to broadcast DORA discover messages and no DHCP server replies with an offer message, will the NMC timeout and stop broadcasting discover messages 

[Teken]

You should contact APC Technical Support for a more definitive answer and compare what you see in the field and report back your findings here.

Questions Ask . . . 👍