SMT2200 with AP9631 Warmstart issue

[bkaufmann_apc]

I have an SMT2200 with an AP9631 NMC that is randomly warmstarting. I have the newest available firmware on the NMC but does'nt appear to fix the issue. The output from the UPS is'nt affected by the warmstarts, but the notifications at 3 am are a little distracting. I have attached the event and data logs along with the config. I have four more of these units to deploy so I would prefer to fix the problem before I quadruple the notifications. Any help would be appreciated.

[DOT_James_apc]

The Firmware upgrade did not stop the warmstart issue. I have reset the UPS except for the ip add/gateway and subnet mask. and the problem persists .below are all the files requested. also my Case

[BillP]

Good day James,

Have you tried upgrading to the latest firmware for the NMC2? If not, please download SFSUMX515 from this link here - [SFSUMX515 Download Link|

[BillP]

Thank you for giving us the files. While we are reviewing them, please see this APC Knowledgebase with regards to WarmStart issues and I do hope that we could at least identify which one among them is causing this issue.

[Why does my Network Management Card report warmstarts and coldstarts?|http://nam-en.apc.com/app/answers/detail/a_id/2572/kw/warmstart]

[BillP]

can you please format this NMC? what i am seeing is that there is a problem with a certain task related to network. this is not normal behavior that happens and i certainly think it is isolated. it may be related to a bad packet that the NMC is seeing on the network, causing it to reboot. is it possible to remove the network cable from the NMC for an hour or two then reconnect it to see if the error happened while it was disconnected?

alternatively, as i mentioned earlier, it may be a good idea to format the card and start over to see if it helps (it goes beyond a reset to defaults):

1.) hold down NMC pinhole reset button for 20 seconds until you see the NMC status LED go out
OR
2.) issue the format command from the telnet or SSH command line and then the reboot command for it to take effect. it will wipe out your IP settings and require a reconfiguration of your IP settings and you may also want to back up your config file too.

[BillP]

I will try the reset/format of the nmc for this ups. I have set up another ups in a lab and it is not having a problem. this could be an isolated event with this particular ups. i will watch the logs for 2 hours or until the warmstarts start to happen again. If that happens I will unplug the network conn and watch the logs again. If the warmstarts stop after the network conn is disconnected I will move the ups to the lab where i am not having this issue and test it again.

[BillP]

I think the formating of the nmc worked. I have not had a warmstart all day. I have been reconfiguring the ups one piece at a time to see if there was anyone config piece that was causing the issue. but there seems to be no problems as of now. I am going to let it be and see how it behaves over night before I start putting switches on it. But so far so good.
Thank you for all of your help and articles.

[BillP]

That is good to hear.

Please keep us posted with the developments.

[andrearp_apc]

I also have the same issue with some 1500 X series rack mount, and the AP9631 card. The card started to reboot spontaneously after a few days of service, and this happens every two/three days. The network is very "busy" (multicast vlans on a CCTV system on Cisco 2960S), but the idea of the 2bad packet" should not be an issue, as traffic is controlled and does not "flood" the network ports related to the APC cards. I have the latest firmware downloaded from the site. As the problem happens so infrequently, I think monitoring traffic is not an ooption, and after all the problem is not so "terrible", but I'd like to see it solved. Formatting is not something I would be happy to do, so I'll keep an eye here!

[BillP]

i think that any issues with random warmstarts will be addressed in AOS 5.1.6. for any Smart UPS units, it is slated to be related for August but that could change. if the actual reboot is not resolved, the new AOS portion of the firmware will at least tell us why the network management card is restarting so that we can investigate. no more unexplained warmstarts! 🙂

[BillP]

can you check in the /dbg folder to see if there is a debug.txt file? also, what is the firmware from your UPS LCD display or on the About section on the UPS tab in the NMC.

we are currently researching a similar problem in SMX units. you can disable this specific alert via email if you want. i dont think an upgrade will help at this point to be honest with you. the "problem" is more of a nuisance alert. the NMC (and only the NMC) reboots to try and fix itself, whether some memory gets overwritten or it sees a bad network packet, etc and then gives you this alert. in our next revision of firmware due out shortly, the alert is now Informational to reflect its true severity. currently is critical and a little misleading. i can give you a copy of this firmware for that if you'd like.

thus, it is a problem and is a priority here at APC but i don't think i have a fix for you at this time unless you see a debug.txt file that points out a specific problem. as soon as we get to the root cause, then a fix will be provided so that the NMC does not reboot unexpectedly. if you see the alert in the log then everything is back up and running and it already happened.

also, this knowledgebase article has a few other reasons to cause this message but it doesnt seem like thats the case for you: http://nam-en.apc.com/app/answers/detail/a_id/2572/

[bkaufmann_apc]

Thanks for the quick response.

There is no debug.txt file.

Here is the UPS firmware info:
Model: Smart-UPS 2200
Serial Number: JS0951022659
Firmware Revision: UPS 04.0 / COM 01.2 / UBL 04.0 / CBL 04.F
Manufacture Date: 12/20/2009

Apparent Power Rating: 2200 VA
Real Power Rating: 1980 W

Yes, I already looked at that knowledgebase article you referenced and I dont think any of the items apply.

Operationaly we are not having any issues other than the nuisance of the random alerts.

Do you know when the firmware will be publicly available?

[DOT_James_apc]

I also have an issue with SAU2200xl with NMC2 nic card warmstarting every 13 minutes. with a error code : 0x0002. Firmware rev: 690.19.D

[BillP]

i am HOPING by the end of this week but it has changed so many times that I don't want to make any false promises.

if you want me to follow up with you offline, i can add you to my master list, based on the data you provided to me. I need some more info to log an official troubleshooting case though.

[BillP]

I am having the same issue. I have a handful of apc 1500sumx with 9631 management cards. They are doing the same Warmstart issue, but older management cards on the same network are not doing this. Could you add me to the list to receive a notification when there is a fw fix for this please? E-mail is ojd8341@kennesaw.edu. Thank You

[BillP]

yes, i will report back on this thread when i know anything. we are continuing to pursue the problem and hope to have a resolution shortly.

EDIT - i do have access to the new firmware that changes the alert to informational if thats what you want. the firmware does not fix the unexpected/unexplained warmstarts though and can still be supressed if you want.

[BillP]

here is the file we believe will address any unexplained/requested warmstarts/network interface restarts with SMT/SMX unit. as i said, even if it doesnt, you will get a dump.txt telling you why it warmstarted anyway so we can look at that. i believe it will be a non-issue though. if you dont have an SMT/SMX, this is not guaranteed to fix your warmstarts/network interface restarts.

[andrearp_apc]

Of course, I couldn't resist immediately updating, we'll see...

Version info changed from

sumxVersion: v5.1.5
Date:Feb 25 2011
APC OS (AOS)
aosVersion:v5.1.5
Date:Oct 7 2010

to


sumxVersion:v5.1.6
Date:Aug 11 2011
aos
Version:
v5.1.6Date:May 11 2011


No changes to boot monitor.

I'll post again if something bad happens, or if it does not happen within 4 weeks.

[BillP]

great. thanks.

[wltz_apc]

I am, also,experiencing the same "Warmstart"/Network Interface Card reset issues for our Smart-UPS X 1500 devices. I recently upgraded the firmware to v5.1.5, per APC technician's request, and this did not resolve the issue. Warmstarts continue almost on a every other day basis.

Currently, we have a trouble ticket opened on this.

Please include me on your notificaton list. - wrhee@skylinenet.net

Thank you!

[BillP]

what model UPS is it? SMX1500RM2U based on what you are describing?

we are currently working on the next rev of firmware to address this but the frequency i am familiar with is more like every 3-5 days. not every other day generally..

i looked at your event logs in the incident. i also recommend that you review [this post|

[BillP · Accepted solution]

also, it may be helpful for you to post your event.txt for us to review. you can do this by following these instructions -> http://nam-en.apc.com/app/answers/detail/a_id/9321