Unexpected "Turned off for a defined period of time" on Symmetra RM w/ AP9631

[Terry_Kennedy_apc]

I was awakened due to an unexpected power outage in my server room. When I got to the server room, all indications on the UPS were normal and systems were beginning to turn back on as the UPS had decided to turn its output back on.

The event log contains "UPS: Turned off for a defined period of time in response to a software command, or off while waiting for input power to return to normal. 0x0115" Nobody did anything to the UPS - we were all (including the cat!) asleep at the time.

I am attaching a censored event.log to this message, which shows that the NMC2 has been up for 22 days and that there isn't anything odd in the logs. I removed a bunch of stuff between the last self-test and 24 hours before today's fault. The ftp login/logout is an automated config polling script which I've been running on hundreds of APC gizmos for over a decade, and which was last modified over 3 years ago. So that shouldn't have anything to do with it.

If any of the APC folks here want the complete debug file, just add me as a friend so I can PM you, and I'll send a link.

[BillP]

Terry, 

Do you have the information from the "Logging" menu on the power view to see how the RM itself reported this? 

[Terry_Kennedy_apc]

Quite a lot of stuff:

03:39:21 On Batt
03:39:31 On Line
03:39:43 UPS Turned Off
03:39:43 UPS Rebooting
03:39:49 UPS Into Bypass
03:39:49 UPS Reboot Complete
03:39:51 UPS Turned On
03:39:54 UPS Out of Bypass
03:39:54 On Line

The self-test passed on the 6th is the entry before that. Apparently my Powerview is in a different time zone . None of the 7 other APC units here (including 2 Matrix, so looking at the whole 208V) reported any power issues that night, so I'm not sure why the RM logged an "On Batt" event.

[BillP]

Pretty strange. Not a whole lot to go on. Do you have anything set up to do graceful shutdowns of the system? The power view states it went through a reboot process as you can see. So it seems to have received a command from somewhere to do this (SNMP, serial, etc). 

[Terry_Kennedy_apc · Accepted solution]

Nope. Nothing was changed on it recently (the last change was about 3 weeks earlier, upgrading to 6.4.0 on the NMC). The unit is polled every 5 minutes via SNMP (the only community is read-only) and hourly via FTP (which only does a single GET). There is a serial cable connecting it to a Cisco router's terminal server card, but that has been connected for several years (current configuration) and more than 5 years before that on a different Cisco router.

Unfortunately, I expect that the Symmetra RM6K is a mature product and there aren't going to be any changes to functionality (particularly as the firmware is in the microcontrollers and not field-upgradable). For future products, it would be great to have the reporting enhanced to say "Power off by NMC", "by serial command", "by simple signalling", etc. - with something like that, it should be possible to determine what caused it.

I believe there is an undocumented serial command to get debugging info out of the SYMIM3. If that info might have a clue as to what happened, just let me know what commands I should send to the UPS and I'll report back with the output. The SYMIM3 has pretty recent firmware - 226/143.