UPS: Lost the local network management interface-to-UPS communication

[rickn._apc]

Hi,

My name is Rick and I was looking through the forums and found the same issue I've been experiencing with my UPS/NMC Card. We have SMT700 and SMT1000 units with an AP9630 NMC card. There are no serial cables etc plugged into the units, just the network cable.

I had updated the NMC Firmware from V5.1.3 up to V6.2.0 on 2 of the devices.

2 days later we started to experience the same issue, the NMC card will restart for no reason and we receive the same error message.

After communicating with the APC Chat group, they said that the firmware on the UPS unit itself will need to be updated. They were at : (UPS 05.0 (ID17). Updating the UPS firmware is not an option because of the remote locations and having to remote the load from the unit.

What are the different firmware options available for the NMC card? Is V6.0.6 or V.6.1.1 available somewhere?

Thanks

[BillP]

Hi Rick,

I branched your discussion because it is likely a different problem, just same symptom. Your units are also newer and would not be affected by a connected USB or serial cable. And you do not have another accessory installed. Can you let me take a look at your logs like I asked the other post (Instructions here-> http://www.schneider-electric.com/support/index?page=content&country=US〈=en&locale=en_US&id=FA156131)? The full dump file would be preferable.

v6.2.0 is recommended and an older version likely will not help. If I review everything and it does appear to be the best option if I do come to the conclusion about the UPS firmware, I can provide it to you. Though, I have not seen any issues with the older UPS firmware causing this on new NMC firmware so I am hesitant to think that is our only option at this time.

[rickn._apc]

Hi,

I was able to get the following files: Config, Data,Debug,Dump and Event Files.

They are attached.

Thanks

Rick

[BillP]

Hi Rick,

I took a look at the logs, thank you. Yes, the Network Management Card interface is rebooting for what seems to be no good reason (shown by the Network Interface restarted events). As part of the few standard messages you receive after a restart, whether expected or not, you'll see UPS:Restored the local network management interface-to-UPS communication. I don't see any lost communication between management interface and UPS in your logs.

When I look at the dump and debug logs, for both v5 and when it was upgraded to v6, it is rebooting because of a similar issue with a particular task. Upgrading the UPS may help because the task that is reporting it is crashing when it reboots is the task that handles all of the events that come from the UPS. I also see at some point it rebooted due to the watchdog mechanism, explained here -> Why does my Network Management Card report warmstarts and coldstarts? | FAQs | Schneider Electric US - though, those seem to be when you were still on the older firmware.

If I were you, on at least one of the devices, I'd consider starting fresh with a card and formatting it (which will wipe ALL settings from it, including TCP/IP settings). If you don't have a way to reconfigure the TCP/IP settings remotely, we may want to at least start with a reset to defaults then. Alternatively, we can try downgrading one of the cards, just to try, but I am not confident in that being the solution. And because the task that reports it is crashing is the UPS event handler, I don't think it is a terrible idea to possibly plan to upgrade the UPS in the future, if not for thus, but to take advantages of any bug fixes the UPS updates provide.

I'll provide v6.1.1 firmware in an attachment to this post but please be advised it has bugs of its own, many of which are resolved in  6.2.0. But, if it does clear up with downgrading, then that could help in the investigation.

[rickn._apc]

Hi Angela,

As for the Watchdog feature, I asked our network monitoring service how often they POLL the card and its every 5 mins. Then every 1-3 mins if it does not get response.

90 % of these devices are at remote locations, so if we format the card, will it at least boot up on DHCP?

After the card is reformatted, I was thinking of a quicker way to get the card configured for our network, Radius Server, Email Servers for Warnings etc. Is it possible to download the clean Config.ini file from the reformatted card, make the changes for our setup and then upload the file and reboot?

Do you have a list of commands you can do at the FTP prompt?

Do you have the 6.1.1 firmware?

Thank you very much for the help.

Rick

[BillP]

Hi Rick,

Yes, the default for this model of card on this firmware is to boot up and do DHCP so if you have a DHCP server and can look up the IP maybe by the MAC address, you'll be fine. The MAC address is shown under About->Network in the interface.

Yes, you can do what you're asking on the config.ini file. You can also download a config.ini right before you format it as well as a backup and re-upload that or modify it upon upload.

What FTP client are you going to use - Windows Command Prompt FTP? I had a list of commands in our knowledge base but now I can't find it. Once I know the client, we can probably find a list online. You should just need to do some basic ones like get and put to download and upload respectively. There are also a few like bin to enable binary transfer mode and hash for hashmark printing before you start transferring files.

Here are some instructions on uploading/downloading a config.ini (and it also covers mass upload/download of config.ini's)-> How can I mass configure an Network Management Card (NMC) or NMC embedded product? | FAQs | Schneide...

Make sure you view this post outside of the "inbox" on this site because I did attach the firmware file to my previous post. When you're in "inbox" you won't be able to see it.

Let me know if you have any questions before proceeding.

P.S. the way you'll need to format if you're remote is to issue the format command via Telnet or SSH and then you'll need to type yes to confirm, then reboot to apply it and yes to confirm the reboot.

[BillP]

Hi Rick,

Can you get me those same log files you did last time again to see if it's any different? I am on vacation this week so I hope it'll be OK if I look at them early next week? I am especially interested in debug.txt and dump.txt with event.txt.

Thanks

[rickn._apc]

Hi Angela,

I saved off the config file, did the FORMAT command on 5 different unit and then re-applied the configuration. However, today I seen that the NMC has restarted again. Is there anything else you can think of to try? How would I down grade the firmware if needed?

Thanks

[rickn._apc]

Hi,

I've attached the logs for 4 different units that restarted.

Thanks

[BillP · Accepted solution]

Hi Rick,

You can downgrade the firmware via FTP (or a manual method) and then format it if you want to downgrade.

What I am seeing is the task that handles the INI file upload failsafe'd on 3 of them in the same way. And it looks to be like it could be memory related because they all show low memory at some point. I know you uploaded the INI file to all of them so that is a note I made. One shows that SNMP task failsafe'd but also shows low memory. I also see that they all seemed to have restarted about ~2 days after you configured them and uploaded the INI.

At this point, I don't have a magic answer. I think this is memory related. Can we monitor it for one more round now that the INI has been uploaded and see what you get? If it does not do it any more in the next few days, it may be a memory issue after the INI file upload.

[rickn._apc]

Hi,

The restart has happened again on 2 NMC devices.

I did upload the files you normally ask for.

If we have to download, how do you do it via FTP?

Rick

[BillP]

If it is easier to try downgrading for your situation (rather than updating the UPS), follow the NMC2 FTP downgrade instructions here -> How do I upgrade the firmware on an APC Network Management Card (NMC) or NMC embedded device (Rack P...

Though, because of what I see in the dump file, I do not think upgrading one of the UPSs is a terrible idea and would be a good test. UPS related processes/tasks are failsafe'ing and it could be because there is an issue with it or because something else is eating up the memory it needs.I haven't seen this specific type of scenario before.

[rickn._apc]

Hi Angela,

I've installed 6.1.1 on 3 test devices and we're still having the issue of the NMC rebooting.

Would you happen to have the 5.1.3 firmware?

That what these devices had before we did the update. The only reason we updated the firmware was to get the feature of getting the model of the UPS unit. Everything else was working fine before.

Thanks

[rickn._apc]

Hi,

The reboots are really just annoying because we receive the notification and a trouble ticket is created. Another thing is knowing that the NMC is unstable.

Our SNMP monitoring service does not pick up the reboots .

One other thing is when we try to access the NMC via a Browser, it seems to cause the reboot so then we need to wait the 2-3 mins for it to come back up.

I'll read the articles and see what we will do.

Thanks

[rickn._apc]

Hi Again,

I read the article about the concerns on doing a firmware download.

The biggest concern is losing some settings because of the database change, correct?

I have all the original 5.1.3 config files so reformatting the device and reapplying the config file is no big deal.

My biggest concern is after the downgrade we will lose connection to the device. it wont boot up at all.

We have DHCP at the locations so the UPS will at least we'll have an IP address.

What I'm thinking is

1) Format device with 6.1.1 to clear all configuration

2) Apply 5.1.7

3) Maybe format again with 5.1.7 (Not sure If I need to do this)

4) Upload original 5.1.3 config file

Do you see any problems?

Thanks

[J-E-1984]

Good Afternoon,

By any chance, did you ever reach a solution?  Our organization has the same problem: Network Cards rebooting without reason.

We replicated the issue in a lab, confirmed it is happening on all deployed units, and currently APC is investigating the issue.

Did you have anything you tried that worked?  As a note, our devices are doing this on multiple firmware revisions... So I doubt that is causing the issue.