Smart-UPS X 1500 powered off

[BillP]

One of our clients had an issue this morning where 2 of their Smart UPS 1500's were off.  He says the load was reading 0% and the battery was reading 100%, and he had to power them back on manually.  Looks like the issue started around 3am.  Any ideas what could have caused this?  They have 4 total backups.  The 2 that failed are both on the same circuit breaker, but nothing tripped.  Here's the snippet of the log around that time.  Just trying to come up with a cause for my client.  Thanks.

01/21/2016 07:03:46 UPS: The output power is now turned on.
01/21/2016 07:03:46 UPS: The power for switched outlet group 2, Dell Switch2, is now turned on.
01/21/2016 07:03:45 UPS: Switched outlet group 3, VMSERV1 PS1 RVPE, has been commanded to turn on, sequenced.
01/21/2016 07:03:45 UPS: Switched outlet group 2, Dell Switch2, has been commanded to turn on, sequenced.
01/21/2016 07:03:45 UPS: Switched outlet group 1, EQ PS1, has been commanded to turn on, sequenced.

01/21/2016 04:47:24 UPS: The output power is turned off.
01/21/2016 04:47:24 UPS: The power for switched outlet group 3, VMSERV1 PS1 RVPE, is now turned off.
01/21/2016 04:47:24 UPS: The power for switched outlet group 2, Dell Switch2, is now turned off.
01/21/2016 04:47:24 UPS: The power for switched outlet group 1, EQ PS1, is now turned off.
01/21/2016 04:47:23 UPS: Restored the local network management interface-to-UPS communication.
01/21/2016 04:47:14 System: Network service started. IPv6 address FE80::2C0:B7FF:FEC3:E094 assigned by link-local autoconfiguration.
01/21/2016 04:47:12 System: Network service started. System IP is 172.16.164.74 from manually configured settings.
01/21/2016 04:46:56 System: Network Interface restarted.

[BillP]

Hi Rob,

Could you please provide the full log files? We would want config.ini, event.txt, and data.txt at a minimum from the management card to see if there are any clues from before the actual event as to what happened. You can attach them to your post and sanitize them if you need to.

Instructions here on different methods of gathering log files, depending on your firmware version or access to the management card -> http://www.apc.com/support/index?page=content&country=ITB〈=en&locale=en_US&id=FA156131 

[BillP]

Hi Rob,

Thanks, my only confusion is in the eventLog.txt which is from PowerChute Network Shutdown. The date/time is off and I can't match that up with the NMC log well. Can you confirm to me what date there matches up with today's date (1/21) in the NMC logs?

[BillP]

Where do I see the NMC log?  I'm not sure why the time would be off.  I know the event happened around 3:07am EST this morning 1/21

[BillP]

Hi Rob,

The logs you attached from the NMC in the UPS are event.txt, data.txt, and config.ini. I see the correct date and time here and see the event on 1/21/2016.

You also attached EventLog.txt which appears to be from PowerChute Network Shutdown. In that PowerChute Network Shutdown log (which is nice to have), there are no dates from year 2016 in there.. so to me, I didn't know if the date/time was wrong or is it the fact PowerChute logged nothing since 2015... which I am doubting that is the case. I can't find 1/21/2016 data in the PowerChute log.

[BillP]

Hi Angela.  I'm attaching 2 more logs from the actual host servers which is where the APC software is configured. I think the other one I sent you was from the wrong server.

[BillP]

Thanks, just to confirm, the Smart-UPS units, they both have NMCs but the logs are identical as well as firmware? I know there are two UPS units involved and we probably should have both sets of NMC logs. I'll ask some other colleagues to check these since I don't see a glaring answer yet and want to make sure I got all the data possible and confirm the UPS logs are identical.

[BillP]

I'll see what I can get for you. I guess I'm more concerned with why they didn't power back on even though the settings are set to to so.  I can explain to the customer that there was some electrical event that triggered the unit to shut down, but the main concern is why didn't they power back on.

[BillP]

Yeah, I understand. What I see so far is just a turn off event. No on battery event or power related event(s).

When the UPS shuts down due to a low battery after a power outage or after PowerChute clients turn off and tell the UPS to turn off, the UPS is set to come back on by default when AC power returns. I see that setting set correctly for the one UPS in the config.ini. The problem is, I don't see evidence of a low battery shutdown or a shutdown PowerChute did so I suspect that is why this happened - since the UPS was not turned off due to low battery or PowerChute command. I just need someone to review these logs to see if we can determine if there was a UPS fault or if it is possible some other type event caused the UPS to turn off quickly rather than go to battery, for example.

If there were any LEDs or characters on the screen solid, scrolling, or flashing upon having to manually turn the UPSs on, we would definitely want to know too.

My other thought too was if someone was local and could check the UPS LCD menu to see if it logged any faults on it that don't show in the NMC logs we have.

[BillP]

Hi Angela. I checked with my client and this is what he told me about when he found the APC's off.

The screens on the two that were "off" were blank. I hit one of the arrow keys to see if they were fully off and it popped up the typical indicator telling me there was 0% load and 100% battery. I hit the power button at the top on each and it gave me a message with a timer, I believe it was a startup message of some sort but I don't 100% remember what it said. It didn't seem to be an error or anything like that.

[BillP]

Hi Rob,

Thanks for the feedback. I asked a few of my colleagues and they did not see anything obvious and neither did I unfortunately. Considering two UPSs on the same circuit did this, I would guess something caused the UPSs to turn off somehow or some type of event happened that they could not react to. We did not see any evidence of what though from either of the NMC logs or the PowerChute logs. It is almost like someone forced the UPSs to turn off from the front panels intentionally, but at least in newer NMC firmwares, there is a message in the log after that event. I am not sure if there would be on this older NMC/UPS firmware combination.

The only suggestions I received were to upgrade the outdated firmware/software involved here so that we are in a better spot if worst case, something similar happens again. I feel terrible saying that but I just can't see any answers in the log as to what occurred. But, I don't want to tell you to upgrade to "fix" what happened since we can't tell what it is unfortunately.

The NMC, PowerChute, and UPS versions are fairly out of date so I don't think it is a bad idea so that we can get best possible logging on everything for any further issues. 

If you would like some help on how to get things up to date, if the client is allowed to and wants to, let me know.

At a high level, for an NMC upgrade, it is pretty straight forward and can be done remotely with no affect on the UPS. Though, the latest version of NMC firmware has a different web interface and features so I'd say it is a "major" upgrade. The UPS would need to be powered down for itself to be upgraded over a serial connection. And PowerChute 4.1 is the latest version.. you have to uninstall 3.0.0 and upgrade to 4.1 I learned since it is older.

[BillP]

Thanks for your time. I will relay the information to our client.

[BillP · Accepted solution]

You're welcome. Sorry I don't have better news or information!!