AP9630 sending "UPS: On battery power in response to an input power problem" email every 2 minutes

[Anonymous user]

Hi, we had a power outage last month. It seems that the power went off and back on almost immediatly after. Shortly after that , I can see another "UPS: On battery power in response to an input power problem" message in the logs and after this we kept receiving emails every 2 minutes saying that it was on battery power even though it was not on battery. Powerchute interpreted this as real and initiated a shutdown 2 hours later. The only thing that could stop the network management card from sending emails was to putting the ups in bypass and then back online. Here's a copy of the logs :

2015-07-05	18:15:48	System	Web user 'admin' logged out from 192.168.150.23.
2015-07-05	18:12:22	admin	Web user 'admin' logged in from 192.168.150.23.
2015-07-05	18:09:35	System	Web user 'admin' logged out from 192.168.150.23.
2015-07-05	18:06:02	admin	Web user 'admin' logged in from 192.168.150.23.
2015-07-05	15:29:44	Device	UPS: On battery power in response to an input power problem.
2015-07-05	15:29:24	Device	UPS: No longer on battery power.
2015-07-05	15:29:19	Device	UPS: On battery power in response to an input power problem.
2015-07-05	14:45:14	System	NTP update successful.

Now I don't really know if I can rely on the unit since it's messages and alarms may be different on front panel, web interface and emails. Any way I can solve this?

[BillP]

Hello,

You are certain the unit was no longer on battery during these messages?  This was confirmed by physically checking the front of the unit?  I just want to make sure it is not an assumption being made because "the lights came back on" or other pieces of equipment were turning back on.  If power came back on, but the voltage or frequency was not of good enough quality as judged by the unit, it could continue on battery power without turning back to online operation and accepting utility power.

If that has been confirmed, and there is a difference in reporting between the unit and Nmc, I would try rebooting the NMC (not the unit) and test a power outtage again during downtime to make sure everything is working fine.

Model and serial numbers of the card and unit would also be helpful in further diagnosing an issue.  What firmware do you currently have on the NMC?  As always, full data and event logs also go a long way.

[Anonymous user]

We've received these emails for 2 days, way past the battery runtime and there was no alarms on the front panel. NMC has been reboot but it continued to send emails after reboot. Bypass stopped the emails.

Unit:

Model:	Symmetra 16000
Serial Number:	JD0829024019
Firmware Revision:	260.143.I
Manufacture Date:	07/19/08

NMC:

Model Number:	AP9630
Serial Number:	ZA1336005373
Hardware Revision:	05
Manufacture Date:	09/05/2013

Version:

v6.2.1

For the log, do you need a specific time frame?

[BillP]

I'd recommend you attach the .tar file available from the AP9630 web interface under About->Support menu so we can review it in full to identify the problem. When you go to that web page, click generate log files and then download and you can put the .tar here.

I think it'd also be helpful to know what the UPS reports for available battery runtime under Status->UPS.

[Anonymous user]

On 8/10/2015 10:52 AM, Angela said:

I think it'd also be helpful to know what the UPS reports for available battery runtime under Status->UPS.

Runtime remaining: 2hr7min

[BillP]

Could you share what you see in the PowerChute Network Shutdown client's (192.168.100.83) event log? That may help or shed some additional light.

I see no reason why the emails would've still been coming through and have never really seen that issue before in my 8 years dealing with this product. Had you reviewed the PowerView display log after the event to see what it reported? Sometimes you can have a faulty battery that discharges current or power module that remains on battery by itself and can use issues like this (making the entire UPS think its on battery) but I would've expected the On Battery LED on the PowerView to remain on and typically that issue slowly discharges the battery and would not cause a shutdown.

I agree that since that outage a while back, not too much has happened either in the log. I did note after the second UPS: On battery power in response to an input power problem. there was no clearing event (UPS: No longer on battery power.)

So the emails have stopped after you put the UPS in and out of bypass? And everything appears normal now?

This will be a tough one to diagnose with 100% certainty because the problem is not actively happening. I'd like to see the PowerChute client log though if at all possible from that time period and see if it reported anything strange to see if it helps us understand better.

[Anonymous user]

Here's the log for PowerChute Network Shutdown. The PowerView display log doesn't seem to show it stayed in a UPS: On battery power in response to an input power problem state and there's no errors related to power modules or intelligence modules, everything seems normal. I can see though the shutdowns initiated by PowerChute Network Shutdown. We disabled PowerChute Network Shutdown for now to prevent it to issue any graceful shutdown commands. We had another episode like this a couple of months back but we didn't have PowerChute Network Shutdown implement at that time so there was no shutdown. Only the same UPS: On battery power in response to an input power problem emails every 2 minutes that also stopped once the unit was but in and out of bypass. It is very puzzling, could it be a faulty NMC?

[BillP]

Thanks for sharing the PowerChute log. I am going to ask Bill P to check this from the PowerChute side when he returns from vacation.

I do not think a new NMC will do anything. Let's say it was a "faulty" NMC - it'd likely be a firmware issue and a new piece of hardware wouldn't help. I haven't seen what you've reported though yet and I am very intimately involved with this product every day.

I do tend to think whatever happened, was an NMC problem unless the first time the UPS went to battery and switched back, it got caught in some weird state that the NMC did not properly pick up on or reported incorrectly, maybe because it was so fast.

This is a Symmetr LX, correct? The Intelligence Module has its own log that can be obtained from the serial port directly on the SYMIM5 module to the right of the powerview. The MIM is the bottom module. This would be another piece of diagnostics we could explore if you have a  940-1524 or 940-0024 cable handy. This could at least show us what the UPS itself reported and hopefully it'd go back a month.

Let me know if you're interested in grabbing that and I can tell you how but essentially you connect at 19,200 to this port on the SYMIM5 and press some letters while capturing the text output to a file for review.

[Anonymous user]

No it's not an LX, I think it's older than that. The color of the unit is beige instead of black instead of black and grey for LX. Would retrieving the data from the intelligence module still possible?

[BillP]

Ahh, OK. I thought the 2008 serial number would make it an LX but I guess not. It sounds like the older Power Array. In that case, no unfortunately it is not possible on this older model to retrieve anything directly from the intelligence module.

[BillP]

Hi,

The clocks seem to be slightly off between the NMC and PCNS but on 5 July the NMC recorded an on battery event at 15:29:44

07/05/2015    18:06:02    admin    Web user 'admin' logged in from 192.168.150.23.    0x0015
07/05/2015    15:29:44    Device    UPS: On battery power in response to an input power problem.    0x0109
07/05/2015    15:29:24    Device    UPS: No longer on battery power.    0x010A
07/05/2015    15:29:19    Device    UPS: On battery power in response to an input power problem.    0x0109

PCNS log recoreded

07/05/2015         15:28:29         UPS has switched to battery power.
07/05/2015         17:08:29         UPS critical event: On Battery.
07/05/2015         17:08:30         UPS critical event: On Battery triggered a shutdown on Hosts srv-esx6.delafontaine.com, srv-esx3.delafontaine.com, srv-esx4.delafontaine.com, srv-esx5.delafontaine.com, srv-esx7.delafontaine.com.

It looks like there is a 6000 second delay set in PCNS for an on battery event and then shut down is started. At 15:29:44 The NMC reported on battery and the next entry is at 18:06:02 web users login. The NMC needed to report UPS: no longer on battery.

[Anonymous user · Accepted solution]

Any idea why the NMC didn't report that the ups was not on battery any longer?