AP9613 require manual reset each time batteries are drained

[Anonymous user]

Hi,

I have a SYH6K6RMT UPS which include an AP9631 network management card and a AP9613 dry contact I/O card.  Both cards are in their respective slots, respecting their priority level .  The UPS is not supported by a generator, thus it occurs once in a while that we completely drain out the batteries.  Each time this situation occurs, it is required that we reset manually the AP9613 card, otherwise its status LED is flashing red and slow, and it reports failure on all its dry contact outputs.

Is this behaviour normal?  I would expect this system to be able to be autonomous and restart itself fully operationnal without any manual intervention.

Thanks for your help.

[Anonymous user]

The first thing I would check is to make sure your AP9631 and the 9613 as well has the latest up to date firmware installed.  For the 9631 This can be determined either by using the web interface go to the "Administration " tab and click "General" , then select the "About" from the side menu ( for 5.1.7 or earlier firmware) or log into the card serially via hyperterminal and use the console command to get that info by using the "about" command at the apc> prompt. I believe the latest version is 6.0.6 ...if you go to the APC support web page for firmware and there you should look for the firmware for the single phase symmetra. I haven't used the 9613 personally but if you go the same firmware support there will hopefully be the latest firmware available.

[Anonymous user]

Yes, I'm already at 6.0.6. and also at the latest for the AP9613.

By the way, I'm also using their new Web interface which is now completely diferrent from the old one.  Just finding the firmware revision is tricky, since they replace the tabs by unfolding menus.

Thanks!

[Anonymous user]

I've actually left my cards at 5.1.7 for now as I found the new interface tricky as well. I have another suggestion for you which is to configure your 9631 card to command the ups to do a graceful shut down when the battery level (during outage) reaches a low level rather than to just let it drain/die completely. If you preserve some battery life hopefully your 9631 and 963 won't be starved of power somehow and this is also way healthier for your batteries - the deeper the discharge the more life is stolen from the battery cells.

[Anonymous user]

Maybe APC can correct me if i'm wrong, but if you set your "Basic Low Battery Duration" to something like  8 minutes (instead of the default 120 seconds), and then set the "Low Battery Duration" to something like 2 minutes, the UPS will shut itself down with  6 minutes of battery life left.

( "Basic Low Battery Duration" minus "Low Battery Duration" )

[Anonymous user]

That's a good observation.  I'll configure the UPS to shut down a bit earlier in order to try to keep a bit of power in the batteries.  I hope this will be helpful for the AP9613 card, as it could also be related to the simple fact of rebooting the UPS that create this bug.

[Anonymous user]

Hi Angela,

Here are more details.  We noticed both the AP9631 and the AP9613 were no longer communicating with the UPS after the outage (Nov. 20th).  Today, I simply resetted the AP9613 card using the reset hole on its back and then both cards started back to communicate with the UPS.  I have attached the event log from Nov. 20th up to today.

Thanks for your help!

[BillP]

I am awaiting some input from someone I asked. I feel like I am missing something. I'll report back in the next day or two.

[BillP]

We have this article outlining details about what the status LED means -> AP9613 Status LED Behavior Explanation | FAQs | Schneider Electric US

It is possible that when the UPS shuts down, it is losing communication with the UPS intelligence since it is off. I'd also expect that if the UPS goes completely off and is powered cycled (so then the AP9613 is), that it should reboot and be fine assuming communication has been regained, especially if the management card gained it. Depending on my questions below and what we find, perhaps I need to find more about how and when the AP9613 would attempt to regain communication if it does not appear to be doing it on its own like I would think it should.

I'd need to know specifically what state your UPS is in to replicate this. Let me know if you want to proceed further. There are several states on a Symmetra beyond just on/off (load disconnect, etc).

Something that might help me is if you can provide to me the logs from your AP9631 from the time you see this issue so I can see what is going on with the UPS at the time the AP9613 shows this behavior.

[BillP]

Thanks.

Are we looking at this timeframe as one example?

2013-11-20 08:37:07 System Network Interface coldstarted. 0x0001 [You are pressing the reset button or this is reported when you turn the UPS on? - coldstart means NMC was power cycled]

2013-11-20 07:13:02 Device UPS: The output power is turned off.The output power is turned off due to a low battery. 0x0114 [UPS turns off?]

So correct me if I am wrong, the UPS runs and turns off. Events above are recorded in the NMC log. How long after the UPS reports the output power does AC power return in these scenarios? I ask because the UPS will go braindead after 10 minutes with no AC input - wondering if it's related. If it's less than 10 minutes, the UPS would just stay off sleeping but there should still be NMC and AP9613 communication to the UPS I think.. Data.txt might help me with some of this if you have it from this period.

When you walk over to the UPS because you know you cannot access your NMC (over the network?) or AP9613 is indicating a fault on something connected to an output (I assume), what LEDs do you see on the NMC? I need to know what the right and left LEDs are showing if you recall. And the AP9613 you said is flashing slowly red.

And I know you said you installed these in the right order - so I believe for this UPS, it would be NMC on top slot, AP9613 on the bottom?

[Anonymous user]

Hi Angela,

2013-11-20 08:37:07 System Network Interface coldstarted. 0x0001 The UPS retrieve its A/C input power so it started back.  The NMC started by itself, no one push the reset button until Dec. 11th at 10h50.

2013-11-20 07:13:02 Device UPS: The output power is turned off.The output power is turned off due to a low battery. 0x0114 That's due to the electrical outage we got, it automatically went down after battery depletion.

Power returned at 8h37 on Nov. 20th.  Giving an outage of about 2h16 (08h37 - 06h21) with 2 brief returns of power at 07h01 and 07h12.

I went to the UPS because we got alarm on the PLC interface (someone told me this only today, but alarms were present since Nov. 20th), since the AP9613 relays were all not enegized.  NMC was reachable via its web interface, but it was unable to get inverter status.  I have seen the red LED on the AP9613 first and decided to reset it first.  I haven't seen any LED alarm on the NMC, execpt the normal RJ45 led showing normal activity of the network.

You are correct about the order.  NMC being installed at the factory, we inserted the AP9613 on the available slot on the bottom.

I have attached the Data Log for the same period.

Regards,

[BillP]

Haven't forgotten about you! Still trying to track down an answer on how to explain what is occurring.

[BillP]

I'll need to do some testing to see where the issue is, perhaps with the AP9613. Can you confirm to me what firmware version your actual Symmetra RM has? I'd be looking for the MIM firmware (part of "intelligence system). Mine has firmware 157, slave 226 (this is the latest).

I am going to try to replicate the scenario.

[Anonymous user]

MIM has FW 157 and slave at 226

RIM has FW 226 and slave at 226

[BillP]

Ok - we did our testing today. Our Symmetra RM frame is a bit older but the IMs were the same as yours per my earlier note.

We did the first test with the NMC in the bottom and AP9613 on the top. We ran the UPS down and let it "braindead" itself after 10 minutes of sitting with no AC power. Then, we reapplied AC power and I saw what you were talking about partially. The AP9631 NMC came on but we had no issues with NMC to UPS communications from the get go. At this time, we noticed our UPS had a battery fault (it only had one battery in it too). I did watch the AP9613 and it did not gain communication all that quickly. It was flashing green for a bit then went to flashing red but then, while I was saying "oh we are seeing the same issue," I noticed i changed to solid red on the AP9613. Solid Red is used to indicate a fault on the UPS itself - which to me was the battery fault we had. I removed the battery from the system (or had some muscle do it for me ) and then the AP9613 LED went to solid green. So, I believe it did re-establish comm for me when it went to solid red (since it saw the UPS fault and changed from flashing to solid red) and then went back to normal state when the battery was removed.

Then I decided to reverse the smartslot card order (which is technically wrong) and saw what happened. The mentality testing that would be that the UPS slots are somehow wired backwards on your unit and this is really what you have right now - who knows but I figured I'd test it. In doing that, the UPS shut down and the NMC came back right away but did indicate lost communication. The AP9613 was flashing red too. Upon removing it, as we suspected, and reseating it, both regained communication. This makes sense to me if they are in the wrong order and you remove the AP9613 so then its not disrupting the communication bus.

So - up to you but I think I will have to log an issue about this but if you feel like it, it may be worth a try to reverse the order from what you have it as and see what happens for testing if you're able to. I say this since if my first test, my AP9613 did appear to regain communication...

[Anonymous user]

Thank you very much for your support.

This would make sens, but is it realy physically possible that the slot could be wired backward?  Is there a non-disruptive way I can validate this assumption? Would the hotswap removal of the AP9613 should affect the NMC if the slot order were not respected, and therefore be used to simulate?

[BillP · Accepted solution]

While it is maybe possible, I don't honestly think it is likely myself. You could try what you suggested - that should be fine.