AP9631 seems unable to boot

[Anonymous user]

Hello,

Probably I am unlucky with this network card...

Some days ago I updated the firmware to the latest version available on the APC website.

The card rebooted itself, and everything, apparently was ok.

Yesterday I had to shut down the UPS (normal shutdown, nothing strange).

When I powered up the UPS again, the card had the status led blinking rapidly orange and green for around 4 minutes, then the status led turned off.

I tried to reset the card pressing the reset button for 20 seconds but the led was turned off, not blinking as it should.

If I connect the card to a router the link led is orange and sometimes makes one blink. Anyway no IP address is assigned to the board.

When I open the console, I cannot perform any action, I don't even see anything written, except, when I reset the board. In that case I can see a line of "õõõ" being written into the console: the longer I press the reset button, the more are the "õ".

At this point I don't know if the firmware become corrupted or what else...

Is there a way to "push" the firmware on the card, like on an USB key or something like that or I have to send the card in RMA?

Thank you very much in advance!

Best regards

Daniele

[BillP]

õ is what some programs like PuTTY show when the Network Management Card is rebooting. (Hyperterminal shows ≡ instead for the same reason.)

Can you clarify what firmware you put on it? 6.1.1?

When you held down the reset button for 20+ seconds, it formats the card which is a good troubleshooting step. It erases any configuration on the system, besides firmware and can eliminate many problems. But, I can't tell if you did this properly for certain by what you described (or if the card responded correctly). When you hold it down properly to start the procedure, it should "pulse" green for the 20 second duration and then the color pulse changes to orange when it is time to let go. If the light was off while you held it, something is wrong or it was done done properly. I would also uninstall the NMC and verify the reset button is not missing or damaged. I find aligning to the top of the reset button hole with the paper clip or similar device you're using to press it works best. If you log into it quick, you could also do a few things like issue the lastrst command to see why it is rebooting (if you get that far, let me know what it says). Also, you can also try to do the format command there, type yes to confirm, then reboot and yes to confirm that to do a format via command line.

Trying USB is an option although I don't think it will be the root problem. I could be wrong. Refer to USB upgrade options here for AP9631  -> How do I upgrade the firmware on an APC Network Management Card? | FAQs | Schneider Electric US

Don't worry - I will help you through it

[Anonymous user]

Hello Angela,

thanks for the explanation about the õ

The firmware was the 6.1.1.

When holding the reset button the card was not pulsing green (was not pulsing at all) but I could imagine that the reset button is not damaged because after if I press it for 20 seconds and then I wait like 30 seconds, the card tries a reboot.

I kept the console cable inserted and the console opened during the reboot procedure and I was not able to see anything else than the "õ", nor I was able to read anything into the console itself, so I only tried to type a few characters, just to try to see if the console was stuck or not.

I'm now going to try with the USB procedure, hoping it will solve the problem.

Be right back soon with updates

Thank you so much again

Best regards

Daniele

[BillP]

Hmm, I am wondering - had you configured email notifications on the NMC?

I am wondering if it is this issue if the answer is yes -> Network Management Card 2 (NMC2) v6.X.X potential reboot loop or TCP/IP stack failure experienced wh... It just came up to me so it is fresh in my mind and sounds similar. But, the rebooting would eventually stop with this one after 6-7 unsuccessful boot attempts. If this is the answer, the rebooting would stop when you remove the network cable because I don't think it'd get far enough to try a DNS resolution perhaps. To avoid the card knowing it is connected to a network too, assuming USB firmware upgrade does nothing, you can remove the network cable, then reseat the card in the UPS slot and see if the constant rebooting stops and if you're able to access the console then. Then we know it could be this issue I mentioned or something network related.

Another option will be to potentially downgrade it via an XMODEM transfer over serial cable if the issue is the one in the kbase that is not available yet in your NMC application. I could get you the older firmware version to try that. But, if you do that, the recommendation will be to format after the downgrade too in order to ensure integrity of the system databases.

You're very welcome for the help

[Anonymous user]

Hello again Angela,

I just tried the procedure to update the firmware via USB, without success.The card keeps rapidly blinking green-amber for around 4 minutes, then nothing. I already unplugged the network cable since the port was continuously doing up/down.

May be that an USB key of 8 GB size is not recognized by the card at all?

About the email notification, the answer *should* be no, I'm not so sure about that but I suppose no (I already read that KB before writing here in the forum).

But just to be sure, I could leave the board inserted into the smart slot without any cable connected for this night and tomorrow I should eventually find it in a good mood!

Can I send the firmware via XMODEM if the console does not respond (or at least it looks like) to input?  No problem for the format, I already did that more than 10 times in the past few days.

Thank you very much for your help!

Best regards

Daniele

[BillP]

Hi Daniele,

This reference will be good -> http://www.schneider-electric.us/support/index?page=content&country=US〈=en&id=FA156047

It covers the USB and XMODEM instructions for NMC2. Is that what you referred to or did you use the user's guide? With the USB, the drive needs to have a FAT file system. When the USB upgrade is in progress, then you'd see the status LED flicker green for one file being transferred via USB then it changes to orange for the next file so you get that visual indicator saying its occurring.

Make sure that you power cycle the card after removing the cable to rule out another watchdog mechanism that has the NMC reboot if it does not see network traffic for a while. Rebooting it will start it fresh and it will never invoke the watchdog if it never sees network activity.

You need to be in what we call "Bootmonitor" mode to do an XMODEM transfer. This then allows an open window for when you transfer the file. This is what we will try if nothing else works. It is sort of like a BIOS and should be accessible by what you're telling me. It can be tricky to get to though. What tool are you using to access the local serial console - something like HyperTerminal or Putty? Putty does not support an XMODEM transfer either unfortunately. But it will allow Bootmonitor access which can at least tell us the status of the firmware files and if they are corrupted or not. Anyway, I also ask because with HyperTerminal, there is a little trick we should do in order to make sure Bootmonitor access is not prevented.

[Anonymous user]

Hello Angela,

I followed the reference you just linked to me.

Abour the USB key file system, I formatted it with FAT32 since on that guide it is written that also FAT32 should be good.

No problem to format in FAT and try again anyway.

When you say "power cycle the card" you mean short-pressing the reset button or reboot the whole UPS? Or I can also "hot" unplug-plug the card into the smartslot?

And about the program used, I was using securecrt, but I could also try directly with hyperterminal.

Thank you very much again!

Best regards

Daniele

[Anonymous user]

update:

I just uploaded via XMODEM the AOS and SUMX files, with sysinfo I get PASSED for every crc test.

but after I type reset, the behavior is the same as before, a very long orange/green blinking. Even a format has the same behavior: no blinking at all.

Perhaps I should try with older versions of the firmware

[BillP]

Hi again,

By power cycle the card, I mean just pull the NMC out of the SmartSlot for a few seconds and stick it back in. You can do this hot and it will not affect the UPS in normal situations (abnormal meaning your NMC or UPS is smoking and looks concerning). You do not need to do anything to the UPS itself.

What version did you try this most recent time - like bootmon102, AOS and sumx v6.1.1?

The command I'd type after you're done in there is boot, to tell it to go boot to the AOS.

Very strange. And you cannot format from bootmon

I'd just want to see if the rebooting stopped after removing the network cable but without seeing it, I cannot think of what else would be causing it besides some corruption some place. Have you visually verified if the reset button is broken or damaged or maybe stuck?

[Anonymous user]

Hello Angela,

Ok, the UPS and the card are not smoking

I tried the version 6.1.1 with bootmon 102, as you said.

I'm doing all those operations with the network cable unplugged.

I'm going to try right now with "boot" and with the network cable plugged in.

I'll be right back with updates.

Thank you again

Best regards

Daniele

***update***

well nothing new even with the "boot" command and with the network cable connected

Messaggio modificato da Daniele M.

[BillP]

Hmm, OK, you don't have any other UPS to try it in, right? Just to make sure the UPS is not doing something to it somehow. Doubtful, but I'll ask. Assuming you don't, I don't say it often but I think it may need to be replaced. Since I know you just got this card, but it has an older serial number, we're going to need a proof of purchase to validate the warranty and edit the warranty since it'd go by the serial number/manufacture date otherwise.

What we'll do is I'll have to see if we can track down someone in your region I can get you in touch with to process the RMA and then I will see if somehow the card could get routed back to me in the USA to take a look at, if possible. See if I can figure out what happened to it.

I'll touch base with EMEA support tomorrow morning and see if we can have someone reach out to you that will handle it properly if that's OK and you don't make any progress on your end.

[Anonymous user]

Hello again Angela,

As you thought I don't have an other UPS to try with...

Could we just try to see if it can be used an older firmware?

In case it is not working, where can I send a scan of the proof of purchase?

Thank you so much again for your help and patience!

Best regards

Daniele

[BillP]

Sure. I don't think the USB will allow a downgrade so you'll likely need to do it over the bootmonitor/XMODEM. I'll attach the older firmware here to this post so you can extract the .bin files.

I will track down someone in your region that can support a replacement and set up a case and replacement request once/if we get to that point and you'll be able to send them the proof of purchase.

[Anonymous user]

Hello Angela,

I have great news!

With the firmware version you linked me, the card is now running as expected!

Well, I will not try again to upgrade the firmware, at least for now.

Perhaps the firmware version 6.1.1 I downloaded was corrupted or broken in some way...

Anyway now it is going very well!

A question/suggestion: would it be eventually possible to place an MD5 or SHA checksum for the firmware in the download page? Just to be sure that the downloaded file is not corrupted.

Thank you so much again for your great help!

Best regards

Daniele

[BillP]

Hmm. Well, I am glad it worked . Did you format the card upon downgrading? (Which since you can get into it, can be done easily via the format command in the CLI.) You're supposed to to ensure integrity of the system on the downgrade but if you didn't, I'd be very interested in the log files (event.txt) and special debug ones in the /dbg directory on FTP called debug.txt and dump.txt that are perhaps still intact to help determine what caused that.

I don't think that if the files were corrupted they would even load. They are protected by digital signatures.

We will have another major bug release in a few months and I'd be interested to see if that worked for you if you were interested in trying it at that time. I am leaning towards a bug of some sort here is how the card got stuck. That is why I asked about the email stuff before.

[Anonymous user · Accepted solution]

Hello again Angela,

Well, I formatted the card two minutes after the "boot" command because I thought it was stuck in boot again. I understood that probably this time it went well because when holding the reset button the status led was blinking green.

I checked on the FTP but all the files you told me are fresh post-reset files. I can send them to you if needed, anyway.

Yes I know about the email bug: I don't really remember if I configured it. I suppose no, but I can't be sure about that.

I will probably try again to update with the future firmware release, hoping to not get stuck again. Anyway now I will keep the old firmware files, just to be safe.

Thank you so much again!

Best regards

Daniele