ATS dropped power to all protected devices

[phild_apc]

This was originally posted on APC forums on 8/11/2016

---

I have an AP7721 ATS.  Both inputs are fed from separate APC UPSes (SMX3000RML).

Yesterday there was a power surge that tripped the breaker for the UPSes.  Both UPSes reported that they were "on battery power in response to distorted input".

When the breaker was flipped back on, power to all devices (network switches) behind the ATS was dropped.

There's nothing quite like the feeling of running to the server room and seeing your core and access switches rebooting...

Servers connected to the UPSes (not through the ATS) did NOT lose power (they are fed through a different outlet group from the ATS).

How is this even possible?  Equipment backed by redundant UPSes and fed through an ATS should not loose power unless BOTH UPSes are faulty (in which case the servers should have gone down too).

[Terry_Kennedy_apc]

This was originally posted on APC forums on 8/11/2016

---

The ATS relies on a very short time period between deselecting one input and selecting the other. Anything like old relays or other aged components can lead to a "make before break" transfer where both the A and B inputs are connected to the output. This can cause various problems including dropping the load, damaging the ATS relays, or even tripping one or both upstream breakers.

My (locally-developed) best practices are:

• Make sure the ATS reports the inputs as "in sync". If not, move one source to a different phase until the ATS reports "in sync".
• Make sure the ATS has the latest firmware. The firmware bundle includes a separate firmware file for the actual ATS controller. Observe the warnings in the knowlegebase article about disconnecting one source before updating.
• Configure the ATS as non-reverting (preferred source none). This will eliminate non-essential transfers (transfers just for the sake of getting back to the preferred source after a switchover).
• Configure the ATS with sensitivity low if possible to eliminate unnecessary transfers.
• If connecting the ATS with one or both sources connected to the output of a UPS, configure the UPS with a delay when returning to AC power (going off battery). If using multiple UPSs, configure a reasonable difference (30 seconds or more) between the UPSs, so they don't both transfer at the same time.

[phild_apc]

This was originally posted on APC forums on 8/11/2016

---

Yes, I'm aware that the ATS complies to power switching that is within the tolerance for most device PSUs (1/60th of a second).

The UPSes are less than 3 years old.  Everything on the ATS is brand new (purchased this year) network switches.  Cisco and HP.

The ATS was configured with a non-preferred source (it may not be right now, but it definitely was before).

In initial testing only the ISP supplied network device has issues when switching between sources (reboots no matter what if the input source is changed).  This continues to be the case when testing after the outage.

Since the outage I have changed sensitivity to low, and increased the allowable voltage range (now at medium - 12V).

I will look into changing the UPSes to resume AC power with different delays.  Thanks for that tip - I had not considered that before.

[Terry_Kennedy_apc · Accepted solution]

This was originally posted on APC forums on 8/12/2016

---

There are some devices that you just can't please, no matter what. I have a piece of equipment (Meridian Audio) that will reboot from the UPS just changing taps (for voltage boost).

You might try asking your ISP for either a different piece of equipment, or adding the redundant power supply if the unit is dual-supply or RPS capable. If it is a piece of Cisco equipment, would you mind sharing the model number? I use a fair amount of Cisco gear and might be able to shed some light on issues with specific models, like bad caps in some Catalyst 3750's.