Unknown reason for failover

[dg6]

Dear All,

We have a couple of AP7723 in our DC and the ATS is switching sources.

We would like to know the exact reason for switch, but we can't find anything in the logs.

Is there a way to know the reason for the ATS to decide to switch source?

Regards.

[Teken]

Please upload the logs as it relates to the input / output voltage, frequency. Provide a screen capture of the ATS settings / programming. 👍

[dg6]

Do you mean this log:

16.07.2024 08:35:49 System: Configuration change. Data log web display time selection.
16.07.2024 08:35:25 System: Configuration change. Event log web display time selection.
16.07.2024 08:34:41 System: Web user 'admin' logged in from 10.1.1.56.
15.07.2024 18:16:55 System: Web user 'admin' logged out from 10.1.1.56.
15.07.2024 18:06:45 System: Web user 'admin' logged in from 10.1.1.56.
15.07.2024 18:02:05 Automatic Transfer Switch: ATS has switched source. Source B is current source.
15.07.2024 18:01:33 Automatic Transfer Switch: ATS has switched source. Source A is current source.
15.07.2024 16:28:52 System: Web user 'admin' logged out from 10.1.1.56.
15.07.2024 16:26:23 System: Configuration change. Event log web display time selection.
15.07.2024 16:22:52 Automatic Transfer Switch: Event counts cleared.
15.07.2024 16:21:44 System: Web user 'admin' logged in from 10.1.1.56.

Here it is:

I did reset the counters before the last failover:

Please let me know if you need more information.

Thanks.

[dg6]

Here is the log:

15.07.2024 18:16:55 System: Web user 'admin' logged out from 10.1.1.56.
15.07.2024 18:06:45 System: Web user 'admin' logged in from 10.1.1.56.
15.07.2024 18:02:05 Automatic Transfer Switch: ATS has switched source. Source B is current source.
15.07.2024 18:01:33 Automatic Transfer Switch: ATS has switched source. Source A is current source.
15.07.2024 16:28:52 System: Web user 'admin' logged out from 10.1.1.56.
15.07.2024 16:26:23 System: Configuration change. Event log web display time selection.
15.07.2024 16:22:52 Automatic Transfer Switch: Event counts cleared.
15.07.2024 16:21:44 System: Web user 'admin' logged in from 10.1.1.56.

And I don't know which settings you wants but here is the frequency/voltage settings:

And here is the event count that I had reset before the failover.

Please let me know if you need more information

[dg6]

Here is the log:

15.07.2024 18:16:55 System: Web user 'admin' logged out from 10.1.1.56.
15.07.2024 18:06:45 System: Web user 'admin' logged in from 10.1.1.56.
15.07.2024 18:02:05 Automatic Transfer Switch: ATS has switched source. Source B is current source.
15.07.2024 18:01:33 Automatic Transfer Switch: ATS has switched source. Source A is current source.
15.07.2024 16:28:52 System: Web user 'admin' logged out from 10.1.1.56.
15.07.2024 16:26:23 System: Configuration change. Event log web display time selection.
15.07.2024 16:22:52 Automatic Transfer Switch: Event counts cleared.
15.07.2024 16:21:44 System: Web user 'admin' logged in from 10.1.1.56.

And I don't know which settings you wants but here is the frequency/voltage settings(attached)

And here is the event count that I had reset before the failover (attached)

Please let me know if you need more information

[Teken]

Appreciate you taking the time to screen capture some of the information. Based only on what you provided here the (Event Log) indicates two surge / sag events.

It would be very helpful to know how the ATS is connected to the infrastructure. Typically the primary is connected to a UPS. The secondary is connected to a back up generator once the UPS has been exhausted.

There should be a tab indicating the input / output voltage and frequency seen at the ATS. 

Please just provide a screen capture of all the tabs in the ATS for review. Regardless, the ATS will transfer over from primary to secondary based on the user settings.

Once the defined threshold has been met the ATS will automatically transfer over to the secondary input. 

Normally all of the network devices would be communicating to a Syslog server / SNMP NMS. 

Both would offer greater insight as to the reason for the transfer. If there is a UPS connected to the ATS please do share the event & data logs from this UPS.

As I am confident it will show the input / output voltage & frequency before, during, and after this event. Keep in mind if your facility conducts a monthly (Lights Out) test to validate all critical systems are fully operational this may very well be the reason as to when. 

The (Why) could very be the UPS battery is near end of life and wasn’t able to supply the required operational runtime before the ATS transferred over to the secondary backup power.

Whether that be another UPS / Generator.

[dg6]

Regarding the connections:
The ATS is in a tier 4 DataCenter and is connected to two different UPS.
The provider tells us that no event have been detected during the failover of the ATS.

Regarding the screenshot, see attached file.

I am surprised that the ATS inform of a failover without mentioning the reason.
And since there is no event on the UPS that explain the failover, it is a very unconfortable situation.

There is no way to know why the ATS is switching source (on the ATS)?

[Teken]

All the settings and data logs appear just fine on this ATS. Are you monitoring this specific equipment with a Syslog / SNMP NMS?

As this would offer much better insight as to why. Lastly, if it’s possible to obtain the event / data logs from the two UPS that would offer some insight and validation.

As the only thing called out is the surge / sag event from the ATS. I would not expect to see either impact the ATS given more than likely these UPS are double conversion pure sine wave. So a surge / sag should not have come through unless the UPS is also having issues.

Lastly, if this ATS is more than eight years old I would be tracking & monitoring the three power supply rails via SNMP.

The screen capture you provided show they are fine but that is just a static view - now. 

As this is one of several culprits of an ATS showing odd behaviour and soon after failing abruptly. 

Let me know if you have more questions or can provide more data from the other supporting UPS units. 👍

[dg6]

Yes we monitor with both Syslog and Snmp.

Syslog has the same information as on the ATS log, and with SNMP we just check if both power source are ok.
If Snmp can help more, please let me know what to monitor.

We have asked for the UPS logs to our provider. I'll give it when received.

> Lastly, if this ATS is more than eight years old I would be tracking & monitoring the three power supply rails via SNMP.
I don't understand this, sorry. Can you explain or give me the OIDs?

Thanks again.

[Teken]

The ability to track and monitor the specific OID's relies on what ever NMS you're using. This is from one of my AP4450 ATS. As you can see not all the same information can be obtained from the different NMS.

[dg6]

For SNMP, we use personal script so we can monitor any OID.

You mentioned "I would be tracking & monitoring the three power supply rails via SNMP."

Since I don't know what is "the three power supply rails", I was asking for the OIDs.

[Teken]

You’ll need to reach out to APC Technical Support requesting the MIBS / OID’s for this specific ATS. As it relates to (IF) the three power rails are available for polling.

Let us know what they share so others can benefit. Along with how everything turns out relating to why the ATS transferred over. 👍