AP9635CH no connectivity

[Anonymous user]

I've lost network connectivity to my Symmetra PX 250/500 about two days ago.  No changes were made either on the network or on the UPS, yet I am no longer able to ping the UPS or access the web console. Upon physical inspection of the UPS I found that the status light on the AP9635CH NMC is constantly flashing green and amber very rapidly.  Even after disconnecting the network cable from the card the status light remained the same. I've performed the following troubleshooting steps:

• Pressed the reset button on the NMC for a few seconds and waited for it to reboot.
• Removed and reseated the NMC.
• "Hard" reset the NMC by pressing in the reset button for 20 seconds.  Confirmed the status light was flashing green while I pressed the reset button then let go once it began flashing amber.
• Reloaded the firmware on theNMC using the following files:
  • apc_hw05_bootmon_108.bin
  • apc_hw05_aos_632.bin
  • apc_hw05_sypx_632.bin

Status light remains the same constantly flashing green and amber very rapidly.  Are there any additional steps I can take to attempt to resolve this?  Or this indication of a failed NMC?  Any help/suggestions would be greatly appreciated.

Thanks.

[BillP]

Hi David,

Great troubleshooting steps so far. Have you tried connecting serially to the NMC through the console port?

When you reloaded the firmware, were you able to do that while the status LED was flashing amber/green rapidly? That would indicate the card is rebooting. But, if you formatted it like you said, this typically clears up most issues as reboot loops are often firmware issues or something hung up. It could also be that the NMC is having a problem communicating to the UPS. I've seen a reboot loop once so far be an NMC hardware problem.

Can you verify on the UPS display that the slot the NMC is in is still configured for Read/Write access? And does the display let you configure TCP/IP settings to the NMC? Can you see anything in the logs on the display?

Random question: you don't happen to have another UPS, not necessarily PX250/500, that we can install the AP9635 into to see if it exhibits the same behavior?

[Anonymous user]

Angela,

Thanks for your response.  I have not tried connecting serially to the NMC yet, at the moment I am trying to track down the cable required to do this.  Once I aquire one I will give it a shot.  To reload the firmware I used a USB flash drive.  I formatted the drive with FAT32 and loaded the necessary files.  I plugged in the USB flash drive to the top port of NMC and pressed the reset button.  The NMC rebooted and I could see the status light flash amber for several seconds and then green for about a minute.  It then appeared to do another reboot and that is where it went back to the rapid green/amber flashing. 

I checked the UPS console and confirmed the NMC is still configured for read/write access.  I am able to change tcp/ip settings.  All the logs show are entries whenever UPS loses communication to the NMC and once it restores, these are logged when I press the reset button on the card.  Unfortunately I do not have another UPS where I can test this NMC on.

Thanks.

-David

[BillP]

Hi David,

What firmware is on your UPS itself? It should be something like v4.X.X I think. I was just thinking there is a new sypx firmware out there we could try via the USB upgrade method. I don't have high confidence it will change anything but we need to make sure your UPS firmware is new enough to support the newer NMC firmware, assuming you wanted to give it a try. It is very strange it happened out of nowhere. 

I thought of one other thing too. To rule out any strange packets the NMC is having a hard time processing and is hanging it up, you could temporarily remove the network cable from the NMC and then press and release (or reseat) the NMC and see if it also behaves the same way with no network connection.

Beyond that, you may want to contact support. 3 phase support typically replaces everything () so I imagine that is the resolution they'll give you as they are less knowledgeable than us here specific to the NMC. They are experts on the UPS though. I typically review all of the NMC RMA returns from NAM so if you send it back, it'd get to me eventually. Or I can send you a special return label to get it directly to me in a shorter amount of time and I can follow up with you with what I find, assuming what I said doesn't work and 3 phase support team agrees there is nothing else they know of that would cause this.

[Anonymous user]

UPS firmware is 4.29.11.  If there is a newernmc firmware available I'd like to give it a shot, could you send me a link to the download location?

I have disconnected the cable and restarted the NMC.  However, no change in status.

Thanks.

[BillP · Accepted solution]

Hi David - sorry, was out on vacation. I will PM you a link to the newer firmware we have which I checked is okay to work with 4.29.11. Usually a field service rep is required to do these because there is often an NMC/UPS compatibility matrix but no worries for the 6.4.0 version I'll give you.