Access Problems with AP9630

[lennynpeter_apc]

Hi there,

I using an APC AP9630 network management card in a APC SUA2200RM2U UPS. Both units have been purchased refurbished and tested by the vendor prior to shipping. The UPS work well, but unfortunately due to the problems with the AP9630 I can't change its setup/behavior. Based on the UPS vendor it was tested with a AP9617 without problems.

At the first power on, after installing the AP9630 only the status LED was flashing amber. I tried all methods listed in the APC installation manual/ users manual, starting with the serial port connection via cable 940-0299, terminal program and parameters 9600 8N1 no flow control. Even after trying different PC's with native RS232 port or USB to RS232, Hyperterminal, Putty, TeraTerm and activating the AP's reset button, no connection couldn't be established. I tried the network configuration wizard, arp/ping method, http://apcXXXXXX (XXXXXX last 6 digits of the MAC address), and DHCP. Couldn't assign an IP or get access!The only change I noticed after the DHCP method was that now the Link LED is constanly on (amber) indicating a 100 Mbps, but no lease is visible in the DHCP server log.

Pressing the reset button less or more than 20s doesn't seem to make a difference, even pulling the 2032 coin battery changed something.

The only method I'm still working on is the BOOTP as it is currently indicated by the flashing status LED in amber, but unfortunately I have as well some hassle to get a TFTP/BOOTP server correctly setup in Debian Linux.

Would be very nice if someone has an additional hint or can help to resolve the problem, I would very appreciate it.

If not (and please correct me if I'm wrong) I would consider the AP9630 broken.

Thanks in advance for any help!!!!

Regards,

Peter

[BillP]

Hi Peter,

Can you take a quick video of the flashing amber you see? I suspect that your NMC may be stuck in "bootmonitor" mode which is a recovery mode when the NMC cannot boot its AOS (APC Operating System). I'll be able to confirm immediately by the flashing patteron in a video. 99.9% of the time this is due to a corrupt file and can be fixed by an xmodem file transfer over a serial cable. I analyze some of our returns on the NMC devices and many people get the flashing amber of bootmonitor mode confused with the slow flashing amber indicating BOOTP mode.

Bootmonitor is indicated by a quicker flicker of the amber status LED where bootP is a slower flash. Plus, you should still be able to get in serially @ 9600 baud rate if this was the case. If you formatted the NMC (holding down the reset button for 20+ seconds), it should've reset back to DHCP mode as the default and been alternating slowly green/orange on the status LED. Formatting wouldn't help an NMC stuck in bootmonitor mode so it all makes sense.

Let's assume we are stuck in bootmonitor mode. You need to connect @ 57600 baud rate, then 8 data bits, stop bit 1, parity none and press enter. If it says BM> then that means bootmonitor mode. You'd type ? and see all of the commands in this mode and I'd need to see the output of sysinfo command. I'd try this in PuTTy first since you have it because HyperTerminal has a weird quirk at 57600 that I would have to explain.

Let me know if you have any questions but I am confident we can fix this based on what you shared so far. I rarely see an NMC with signs of life/LEDs lit that cannot be fixed.

[lennynpeter_apc]

Hi Angela,

to be honest I'm very happy about you fast response and appreciate it a lot! I'm almost 2 weeks now in this and was on the brink to give up. We have here tenth of short or longer power outages per day and I can't use the UPS to keep the equipment alive.  Today I gave it the last shot, looking on the web and found many of your messages helping other to resolve their issues and you are the first one talking about the equipment being stucked in 'bootmonitor', so this sound very promising.

Unfortunately I'm currently at work and used just my lunch hour for digging on the web. I'm on Atlantic Time (UTC-4) and the earliest hour to send you a short video of the status LED flashing frequency will be around 6:00pm local time.

As recommended I'm will try to get the serial communication up @ 57600 baud rate, then 8 data bits, stop bit 1, parity none. I use in general TeraTerm or Putty, because hyperterminal is not available on Win 7 and completely outdated. I assume using the 940-0299 cable is the correct one to use. Right?

Please let me know how we then can take it from there! Thanks again - you really made my day!!!!

Kind Regards,

Peter

[lennynpeter_apc]

Hi Angela,

You rock! You are absolutely correct about the root cause of the problem. Really amazing - After lots and lots of conversation via the APC chat, many messages and digging for days through the web you just nailed it in an instant. Give me the email of your supervisor to get you a hugh raise, you earn it!

I had not evn crawl behind the rack to take a video of the status light, just connectedas you've said, the AP9630 responded and here is the result you asked me for:

Rhodes BootMonitor

 Error: AOS CRC Failure

BM>?
?
help
boot
reset
debug
sysinfo
mfginfo
setbaud
setdiv
XMODEM

BM>sysinfo

Model        : AP9630
MAC Address  : 00C0B7C6E634
Serial Number: JA1017015210
CPU Speed    : 200
Link Port    : 3
Warmstart: YES  NMI Count: 246

*********** FLASH MODULE INFMORMATION *****************

Boot Monitor:
  Name      : apc_hw05_bootmon_100.bin
  Version   : 100
  Cookie    : 963200
  Build Date: Oct  8 2008
  Header Crc: 5656(PASS)
  Module Crc: 9A30(PASS)


AOS:
  Name      : apc_hw05_aos_503.bin
  Version   : 503
  Cookie    : 963201
  Build Date: Aug  4 2009
  Header Crc: C048(PASS)
  Module Crc: 97DF(FAIL)


APP:
  Name      : apc_hw05_sumx_503.bin
  Version   : 503
  Cookie    : 963201
  Build Date: Aug  6 2009
  Header Crc: B673(PASS)
  Module Crc: 5218(PASS)


BM>

The UPS the AP9630 is mounted in is the following (just for the case I have to flash or update the firmware as well). THe APC support chat sent me a link for one version, but I'm sure you can give me here also the best answer:

UPS

Model: SUA2200RM2U / DLA2200RM2U

JS0636006812

IN-0T0072-18740-68T-7924

REVA00

Should you need any other information, just let me know and I'll get it! Really thanks again and have a nice weekend!!!

Kind Regards,

Peter

[BillP]

Hi Peter,

Awesome, I am glad that was it. The AOS (APC OS) somehow got corrupted is what you see there where a checksum failed. We can fix this no problem. We can do it a few different ways..

1.) We can transfer the three .bin firmware files you need over XMODEM (serially) with yes, the 940-0299 cable connected to that Console port like you had it now to get me the sysinfo output. The bootmon will take only about 2 minutes based on its file size where as the AOS and APP files you need are more like 10-12 minutes a piece @ 57600. You type XMODEM to start, you'll see CCCC which means you then send the file, making sure to specify XMODEM in TeraTerm and not something like YMODEM. After Bootmonitor transfers, type XMODEM again and browse for AOS file, and repeat for APP file. When done, you type boot to tell the NMC to boot into its AOS.

2.) You can transfer just the bootmonitor and AOS file then execute the 'boot' command at this point. Then, the NMC will be booted properly (just with no application file) and available @ 9600 baud rate so you'd have to change your baud rate in TeraTerm and you could assign an IP address through there while connected serially. You could then put the APP file on via FTP which at the end of this option, I don't know how much quicker it will be than option 1 after you disconnect/change baud rate, etc.

Grab the SFSUMX640 firmware from here (http://www.apc.com/tools/download/index.cfm) which is what you need for a Smart-UPS - SUA (or Dell branded Smart-UPS - DLA). Use the first drop down at that link to Filter by Software/Firmware and select UPS Firmware Upgrades - Network Management Card and you'll see the SFSUMX640 download part number. This download will have the .bin files you need to load up by either option. More instructions here (http://www.schneider-electric.us/en/faqs/FA156047) on upgrading firmware via this XMODEM (serially) or FTP too. 

The files need to be done in the right order too - bootmonitor, AOS, APP (sumx for you).

Let me know if you have any questions before starting! Thanks for the kind feedback as well  

[lennynpeter_apc]

Hi Angela,

I transferred all three files via XMODEM (using extraPutty) and booted the AP9630. The FTP client (FileZilla) I only used to peek into the management card (since in the telnet menu I could see a 'cd' command) and see the structure/locations of the different file if in the future I want  to use the firewall or cerificates). I did not use the FTP client for transfer. 

Below the other information in case you couldn't see any of them as attachment: 

apc>system
E000: Success
Host Name Sync: Enabled
Name: apcC6E634
Contact: XXXXXXXX
Location: XXXXXXXX
Message: SUA2200RM2U
DateTime: 04/30/2016:19:53:51
User: apc
Up Time: 0 Days 0 Hours 56 Minutes
Stat: P+ N4+ N6+ A-
Bootmon: Invalid Module Header!
AOS: Invalid Module Header!
App: Invalid Module Header!

apc>

Should I only try to do the upload of sumx.bin via FTP or start form scratch?

Thanks a lot,

Peter 

[voidstar_apc]

You only need to upload sumx.bin via FTP.

[BillP]

Yes, as voidstar mentioned, just do the sumx 640 .bin file in this scenario. Majority of the time though, you need to do all three files but because of how the NMC processes firmware upgrades and puts each file in its application space (in memory) first, this situation is an exception since sumx is the actual application file and will remain in that same space.

[lennynpeter_apc]

Hi Angela,

Finally, I was able to copy the file ‘sumx 640 .bin’ over to the Ap9630. I had to use the firmware upgrade utility, because for whatever reason it didn’t work out via FTP. Using Filezilla client it initially connects well to the AP, but as soon as you try to move the file (binary mode) the connection to the server is being lost. Trying it with the native Win 7 ftp server I got actually the same result, but it pointed a little better in the direction of the why. The native Win7 ftp asks you every time for the credentials and even if I have not changed anything in regards to the users and passwords, it refuses the default ‘apc’ credentials, while they work just fine via telnet and the web interface – strange!

Now very happy, thanks to you and ‘voidstar’, to have access via the web interface I went through all menu options as I did via telnet. Most of them are self-explaining, but is there another manual with more specific information. For example, I couldn’t change the super user password for ‘apc’.

Another question even much more important is about if the UPS has somewhere some hidden DIP switches or if there is any file in AP folders that has to be adjusted because I have still one problem.

Before you helped me even to get access the loads always receive power, if the utility power is present. As soon as the utility fails, the UPS switches as expected into battery mode, which is indicated at the status LED and clearly noticeable by the sound coming from the inverter, but as soon as the UPS is in battery mode all loads do not receive any power.

I can’t be related to the breakers, because then all outputs would be off as well if the utility power is up. The EPO doesn’t make sense either, since the apc manual defines it as a normally open contact (EPO not activated), so if there is no connection at all at the EPO connector it should work. Besides the manual states as well, that if EPO is active, battery mode is not activated.

What also makes me suspicious is the fact, that not by the status LED on the UPS front bezel nor via web interface I can see any load current.

Comparing the rear section of my SUA2200RM2U is slightly different to the ones shown in the manual. It has two pair (one breaker for each) with NEMA 5-15 and one NEMA L5-20, but all behave the same way – no utility power, no power to the loads even if the UPS is in battery operation.

The APC support once the UPS is in bypass mode, but couldn’t explain how enable/disable it. At least through the web interface I couldn’t see any option like this, nor an option to enable/disable the EPO.

So all of this is telling me there should be some other means (DIP switches, file) to set/reset this kind of options. Really hope you can help me on this as well.   Thanks a lot!

Regards,

Peter

[BillP]

Hi Peter,

Glad you got the firmware issue sorted. I do know that after each .bin file the NMC needs to reboot. In the Windows 7 FTP client within command prompt, you have to disconnect the FTP connection and make a new one or it won't work since the original session was terminated when the NMC rebooted to apply the firmware file. I am unclear if that is what you're describing but I don't think it is..

SUA models (or any Smart-UPS models I know of) do not have any hidden dipswitches or anything. Yes, I agree it is not EPO because if the EPO was activated, the UPS wouldn't power on at all (assuming its contact is held in open or closed - whichever state activates the EPO).

SUA2200RM2U does not have any bypass mode so I am pretty confused on why you were told that... only SUM or SURT/SURTD/SRT model Smart-UPS have a bypass.

This sounds like some type of hardware problem to me right off the bat. One additional clarifying question to what you've described though - when the UPS switches to battery, the load goes off but the UPS remains on? Does it do the 4 beeps every 30 seconds. Not being all THAT familiar with UPS internal workings, I wonder if this could be something like a bad output relay or stuck output relay. What is the history on this UPS as far as you know?

[lennynpeter_apc · Accepted solution]

Hi Angela,

completely agree - I never honestly thought the EPO is responsible for this situation, because with a NO dry contact it wouldn't make any sense. Good to know, that all outlets are battery covered and this confirms what I measure yesterday. That you are not allowed to share any schematics I fully understand and never expected it. I will check as soon as I come home, maybe in one of the conversations I mentioned the unit to be a DLA2200RM2U (copy/past), because both unit model numbers are many times written together, but if I'm not completely wrong, then on one of the original stickers it has the APC model number on it. Can you confirm, that my type of back plate is a Dell version?

In regards to the eBay seller - he has a very good reputation and in general even one negative rating is not what they are looking for, but for sure everything is possible.

It's sad to come so far and still be far away from the solution. I'm run a small business and can't afford to buy a new unit and since APC is an excellent brand and those UPS are being sold refurbished everywhere it was the best alternative. Maybe a wrong decision.

I will open/check the unit and see what I can find. Anyway thanks a lot for all your help and time and to resolve the issue with the AP9630 I really appreciate it! You did an excellent job!

Cheers,

Peter