Error on Rack Mounted UPS Model AP9630

[Anonymous user]

I recently replaced the battery in my AP9630 Rack mounted APC. Upon swapping the batteries i noticed that the red led next to the Battery with an X through it was blinking and beeping. I did not think anything of it as the documentation says Blinking lights are normal until a device passes its first self test only after the device holds a 75% charge or greater. However its been a full day now and the LED lights indicated charge % are all dark and the beeping has continued. all relevant data is below. The device has a network Management Card and is setup to shut down multiple server on the network and on server via USB connection. No self tests have passed 

Hardware Factory

Model Number: AP9630
Serial Number: XXXXXXXXXXXXX
Hardware Revision: 05
Manufacture Date: 12/03/2014
MAC Address: 00 C0 B7 8A F3 F2
Management Uptime: 0 Days 0 Hours 40 Minutes

Application Module
name: sumx
Version: v6.1.1
Date: Mar 7 2014
Time: 12:05:01

APC OS (AOS)
Name: aos
Version: v6.1.1
Date: Mar 7 2014
Time: 11:52:49

APC Boot Monitor
Name: bootmon
Version: v1.0.2
Date: Jan 21 2010
Time: 13:35:57

Critical Alarm Present

02/18/2016 12:14:35 UPS: The local network management interface-to-UPS communication not established

02/18/2016 12:55:57 System NTP update failure.

Emails im receiving from the network card: 

Name : apc8AF3F2

Location : Unknown

Contact  : Unknown

http://apc8AF3F2.XXX.com

http://10.1.1.92

http://FE80::2C0:B7FF:FE8A:F3F2

Serial Number : XXXXXXXX

Device Serial Number : NA

Date : 02/18/2016

Time : 12:14:35

Code : 0x0102

 Critical - UPS: The local network management interface-to-UPS communication not established.

[BillP]

Hi Jordan,

The information you gave to the forum is actually the information regarding the network card, an AP9630, that is installed in your particular UPS and not the UPS itself.

What is the UPS model itself?  There should be a tab on the web interface called "About" and then beneath that a tab called UPS.  In lieu of that, the information can be taken off a label on the back of the UPS itself.

Waiting to hear from you with the UPS information.

[BillP]

Hi Joe, Jordon,

It seems the AP9630 UPS Network Management Card (NMC) is reporting it lost internal communication to the UPS so I am guessing About->UPS will be missing or not having information listed. We'll need it from the sticker on the UPS like you suggested.

We'll want to troubleshoot why the AP9630 NMC cannot talk to the UPS in order to get them talking and then understand the condition of the UPS that it can report to us. Knowing the UPS model will be important too along with that. Based on what you described, it sounds like an older unit with no LCD screen and something like an SUA, SURT, SU, SUM type model.

Depending on UPS type, we'll want to make sure it does not have any serial or USB cables connected to the rear of it..only the Ethernet cable to the NMC. Serial or USB cables connected with an NMC installed can cause this problem. Once we know the UPS model too, we can double check if there is anything else along these lines to look at.

The full NMC logs would also be helpful to see the unit's history and how long this issue has been reported. If you can go to the web interface and get to About->Support and can generate logs and download logs, we'd like to see the .tar file. Detailed instructions on getting the logs through this method and others are here -> http://www.schneider-electric.us/en/faqs/FA156131 

[Anonymous user]

If there is anything I can do about this issue?

[BillP]

Hi Jordon,

Thanks for all the info.

To start, the architecture and design of the now older SUA model units do not allow for simultaneous communication between USB and serial communication busses. Newer UPS models can handle it though we don't typically recommend it. It'd still work though. The problem is that the UPS SmartSlot (where the AP9630 is installed) is part of the serial communication bus so connecting the USB cable stops the serial bus from working properly. Typically you have to "braindead" the UPS to get it to work again. This means turning the equipment off from the UPS, turning the UPS off itself, and unplugging the UPS from the wall. Then, holding down the off button, which looks like this (o) and the UPS will click and flash its LEDs indicating it has been bread deaded.

What you can try first, since you removed the USB cable now, is rebooting the management interface of the UPS to see if that works. To do that, go to Control Menu->Network->Reset/Reboot and the radio button defaults to the correct option of "Reboot Management Interface" so Apply that, confirm, and let the management interface of the NMC reboot to see if it will start talking to the UPS after it reboots. If it doesn't, you'll have to do the "brain dead option" when you can afford to turn off the server. And, the USB will have to stay disconnected or else you'll get back to where we started again. 

Unfortunately with this UPS, you cannot use the USB and AP9630 at the same time because you'll constantly have this issue and the NMC becomes useless because it cannot operate on the serial bus with the UPS connected. You'll have to either just use AP9630 or just use USB. Alternatively, if the UPS was upgraded to a model with SMT or SMX prefix in their SKU, you could get away with the USB and NMC at the same time as the communication busses on the new units are all isolated and won't interfere.

Let me know if you have any questions.

P.S. the AP9630 firmware is out of date so once you sort out what you want to do, I'd suggest we upgrade this to AOS 6.4.0/sumx app 6.4.0. It has a lot of fixes since what you have (6.1.1) including security related items.

[Anonymous user]

Sad day. the Net card reboot did not work. it might be time to purchase a smarter smartups.

[Anonymous user]

Quick question. while i wait for an opportune time to brain dead this APC and bring it back should I be concerned about it functioning correctly? This specific device supports all important Switches and could be disastrous if it went down.

[BillP]

Hi Jordon,

It's hard to say since the NMC cannot talk to the UPS, we don't know if there are any faults present on the system to be concerned about. In general, if there are also no red LEDs on the front panel of the UPS, there will likely not be an issue. I of course can't be 100% sure but it'd be my educated guess that if the UPS is not acting strange with beeping and red lights, it should be fine to turn it back off and back on again. The only other thing would be to know when the battery was last changed and also, if you know if the UPS self tests (quick battery load tests) have been running in order to catch any battery issues. Sometimes you can run into an issue where a dead/old battery would cause the UPS not to turn on again. That is really worst case scenario though. I don't want to NOT mention anything and then you find a problem.

[Anonymous user]

well in my case the unit is beeping. and there are no light light up on the load side. So I probably should be concerned then?

[BillP]

Oh, yes, you did mention that. I apologize. So the load side is the left bar graph. What other LEDs do you have illuminated and what is their color/flashing pattern? And what is the beeping pattern - chirping, constant tone, 4 beeps every thirty seconds, etc? I also don't know if you have the capability to take a video that we could look at if that is easier?

Here is the UPS front panel LED reference if you need it. Yours should look the same.

[BillP]

Hi Jordon,

OK, so the load graph is the one on the left. The one with no lights on the right is the battery charge LEDs. And the flashing red LED you have is the replace battery LED. When it is flashing, it means the UPS thinks the battery is disconnected. This means it is physically disconnected which you could verify or that it is "bad" and it is so bad the UPS can't even "see" it anymore so it reports disconnected.

Long story short, you have a battery issue. If the battery is more than 2 years old, it likely actually needs a replacement. If it is relatively new, then it could be physically disconncted or faulty.

I'd start by removing the front bezel and check the blue (I think its blue on this one) connector and make sure it is plugged in. Also check for any swelling or leaking too, just in case, You may want to put the back of your hand on the battery area and see if it is heating up. When you have a battery that is bad like this (assuming it is not unplugged), it could continue charging and cause a problem.

You probably need to plan on a replacement battery for this to swap in or if you were interested, look into an entire UPS upgrade. We can tell how old the UPS is if you find the first six characters of its serial number on the white sticker where you saw the model number. And then with that, understand too if battery is original or was ever replaced since the serial number won't help if the battery was replaced since it was manufactured.

I don't want to tell you to turn the UPS off with this issue because I can't be sure in this state that it will turn back on 100%. I do urge you to investigate this battery issue ASAP as I suggested so we can get it fixed and also get your original issue fixed.

Let me know if you have any questions.

[Anonymous user]

This Battery is brand new out of the box. so it "shouldn't" be faulty. Since the device is not reading any battery at all would it be safe to say i could unplug the Blue connector and plug it back in while still being live? in case it just didn't seat correctly when i replaced it last week? I feel like this could be possible.

[BillP]

Yep, that should be fine since it is a new battery. I would've suggested caution if it was older and you noticed leaking, swelling, etc. 

[Anonymous user]

So this model is hotswapable then? 

[mitsmith_apc]

Hi Jordan,

Yes the battery is hotswapable on this unit 

[Anonymous user]

FIXED! the issue was that the cable connecting the new battery and the device was not seated properly. It literally took a hammer to correct the issue.

[BillP]

How long did you let the battery charge after you got it re-connected it properly? I'd make sure you let it charge at least 8 hours before running the self test in general and even now before you try again. Keep in mind, the battery was sitting for a bit not charging while it was disconnected so it is possible it could've caused the UPS self test to fail since it sat there not charging for days (weeks). And, if it wasn't at 100% when it was purchased, that could be an issue too.

So, give it a least 8 hours and let's try another self test. Hopefully it will be fine afterwards 

Otherwise, if it still is a problem and we're confident it was fully charged, it could be replaced (assuming it was a genuine APC branded battery). If you got a different brand battery from somewhere else, they may be able to do the same for you.

[Anonymous user]

It is a genuine APC battery. but I will wait another day and run it again. fingers crossed.

[Anonymous user]

Looks like it passed! took about 12 hours to recharge.

[BillP · Accepted solution]

Glad to hear it!