Smart-UPS 1500 logs - UPS: The output power is turned off.

[Anonymous user]

Hi,

We had an incident recently where some servers that are attached to a Smart-UPS 1500 similtaneously shut down (not gracefully). Checking the logs via the Network Management card we have installed there are entries that read "UPS: The output power is turned off." followed by "UPS: The output power is turned on." at the exact time of the servers going off.

As I understand it these messages mean that the UPS shut off power to it's outlets briefly which obviously meant any devices connected would lose power. Surely this is not expected behaviour as we've never seen anything like this before? We've since performed a shutdown test using powerchute and some scripts and everything performed as expected but we are concerned about the health of the UPS.

Should we be thinking of replacing this UPS? It is quite old now even though the battery is new, we are concerned that the unit is faulty. Are there any diagnostics or tests we can perform to check the health of the UPS?

Thanks

Trevor Dale

[Anonymous user]

Hi,

Just to elaborate the power event was a momentary outage (max 2 seconds) in which time the UPS had logged that it had switched off it's outputs even though the expected runtime is currently showing 30+ minutes.

Is there something wrong with this UPS?

Thanks

[BillP]

Hi,

Can you provide the details on the UPS model? We have several Smart-UPS 1500 so the model number on the physical unit would be helpful. It'd be on a white sticker on the back of it. Otherwise, whatever you see in the management card would help for model/firmware on the UPS itself (versus the management card). I might be able to piece it together.

The management card logs would be helpful to see if anything jumps out at us. (Instructions here on gathering those logs -> How can I download Event, Data, Configuration, and Debug files from my Network Management Card? | FA...)

This could be due to a faulty battery (that has not failed its self test yet) and if you had a power blip, the battery did not support the load. But, when you say you tested it, can you elaborate more? Did you test with tripping an upstream breaker or pull the plug from the wall?

The reported runtime is a really good estimate. Thus, when the UPS goes to battery, it is supposed to constantly recalculate and update it. With a faulty UPS or battery, sometimes it can still look correct. More often than not, when a battery is failing or aging, the runtime will gradually drop based on real time "mini" calibrations when you have blips.

[Anonymous user]

Hi,

Thanks for the response.

The UPS in question is a Smart-UPS 1500 (SUA1500I) and for information the network management card is an AP9617.

We had replaced the battery shortly before the incident due to the previous battery being faulty. After the incident the test we performed was simply switching off the power at the mains socket to check that the UPS switched to the battery okay and it worked just fine.

Please see snippet from event.txt below from the time of the event. I've also attached the files.

13-Nov-14     12:45:23    UPS: No longer on battery power.    0x010A

13-Nov-14     12:45:23    System: Email: Could not resolve 'mailserver'.    0x001A

13-Nov-14     12:45:09    UPS: On battery power in response to an input power problem.    0x0109

13-Nov-14     12:45:19    System: Email: Could not resolve 'mailserver'.    0x001A

13-Nov-14     12:28:28    UPS: The output power is now turned on.    0x0113

13-Nov-14     12:29:44    System: Email: Could not resolve 'mailserver'.    0x001A

13-Nov-14     12:28:24    UPS: The output power is turned off.    0x0114

13-Nov-14     12:29:04    System: Email: Could not resolve 'mailserver'.    0x001A

06-Nov-14     17:34:48    System: Web user 'Admin' logged out from xx.xx.xx.xx [].    0x001F

06-Nov-14     17:29:11    UPS: Passed a self-test.    0x0105

06-Nov-14     17:29:10    UPS: The battery is now installed.    0x0130

06-Nov-14     17:28:51    UPS: The battery is not installed properly.    0x012F

06-Nov-14     17:27:29    UPS: Passed a self-test.    0x0105

06-Nov-14     17:27:13    UPS: Started a self-test.    0x0137

06-Nov-14     17:27:13    UPS: A faulty battery no longer exists.    0x011A

Many thanks for your help.

Trev

[BillP]

Hi Trev,

Thanks - one question.

13-Nov-14     12:45:23    UPS: No longer on battery power.    0x010A

13-Nov-14     12:45:23    System: Email: Could not resolve 'mailserver'.    0x001A

13-Nov-14     12:45:09    UPS: On battery power in response to an input power problem.    0x0109

13-Nov-14     12:45:19    System: Email: Could not resolve 'mailserver'.    0x001A

13-Nov-14     12:28:28    UPS: The output power is now turned on.    0x0113

13-Nov-14     12:29:44    System: Email: Could not resolve 'mailserver'.    0x001A

13-Nov-14     12:28:24    UPS: The output power is turned off.    0x0114

For those events above I've highlighted, was that a legit power outage (after the unexpected turn off) and the UPS acted normally on its own? Or was that your testing you said you did? I was assuming it was the testing because the event log does not have any other shutdown related events from any other testing but it also does not indicate a full graceful shutdown happened.

[Anonymous user]

Hi Angela,

These events are related to the test we performed i.e. turning of the mains power. We only turned off the power for a short while to check the battery would kick in as expected so we did not let the shutdown procedures commence (gracefull shutdowns).

Thanks

Trev

[BillP]

Hi Trev,

Ok, thanks for confirming. The UPS just turned back it's output by itself, correct (4 seconds later)? There was no manual intervention required? I assume so since it was only a 4 second delay.

I don't see anything obvious in the log and your subsequent test passed so I am sort of stuck on what could've caused it to turn off and back (going by what you've provided). You could contact EMEA support (your local support region) to see if they have any ideas or I can ask them for feedback if you want.

Edit - I did forget to mention, this could be something like an Automatic Voltage Regulation (AVR) fault which uses an internal transformer and a relay to activate it. Usually, this is recorded in the log though and you'd likely see the center 6 LEDs flashing on the UPS. It also usually requires manual intervention though to reset it.

[Anonymous user]

Hi Angela,

Thanks for all you input thus far. It's really reasurring to see such an active response on these forums.

I think based on the event that we have seen and the fact that we have never seen anything like it previously on any of our UPS devices we will be contemplating purchasing a new unit. The unit itself is some 11 years old I believe so it may well be time for a new unit.

One thing I haven't mentioned is that when we replaced the battery it was by way of a hot swap. As far as I know the unit was not recalibrated but it did perform a self test after the battery was replaced which passed no problem. Not sure if this could have caused the issue as we have replace batteries in this manner previously with no problems but thought I should mention it.

Another question if I may, could you tell me if the Network Management Card we have installed (AP9617) is compatible with the newer UPS models? We will most likely be looking at a Smart-UPS 1500 (SMT1500I) unless you would you recommend another model. Our current UPS generally runs at around 30% load and reports around 30-40 minutes runtime, overkill probably but we like plenty of time to allow our gracefull shutdowns to initiate and complete.

Thanks again for your input.

Trev

[BillP]

Hi Trev,

You replaced the battery properly. Hot swapping it and doing a self test is all it should take.

AP9617 will not work with SMT/SMX units. It is replaced by AP9630 as far as features and functionality. AP9631 can also be used if you'd like to add environmental monitoring and USB firmware upgrade (which is what the USB currently supports on AP9631).

I don't blame you on your decision. It makes sense to me. Depending on how you're going to do it, you could consider the APC Trade-UPS program to trade in your old UPS to get a discount on the new one.

[Anonymous user]

Hi Angela,

Thanks again for all of your help.

What are the main advantages of the AP9631 is it just the environmental monitoring and USB ports?

Trev

[BillP · Accepted solution]

Hi Trev,

No problem! Glad to help. Yes, Environmental Monitoring (temp probe included, humidity and dry contacts optional, future accessories possible) and USB ports.