AP7753 unexpectedly dropped load

Terry_Kennedy_apc · ‎2021-07-09

This was originally posted on APC forums on 8/28/2017

I had a very unpleasant experience the other night while attempting to update an AP7753 ATS. It was showing "Source B selected, Switchover Possible" so I unplugged the Source A power cable (the firmware update has a huge warning about not updating the ATS with both sources connected, because if there is new ATS microcontroller firmware there is a possibility of shorting the 2 sources together). Immediately after unplugging the Source A cable (remember, Source B was supposedly selected and providing power) the ATS dropped all loads. I checked and the cable was clearly labeled Source A, and even if it was mislabeled, the ATS was reporting "Switchover Possible" so the worst case is (supposedly) a source switch.

Immediately after I heard all the equipment power down, I plugged the source cable back in, but due to power sequencing in the downline AP7900 PDUs it took some time for the connected equipment to power back on, boot, etc. Meanwhile I'm looking at several dozen dead 10GbE links and hundreds of GigE links as all of the network switching gear lost power, along with 4 core routers.

The ATS apparently doesn't think it did anything wrong:

08/26/2017      21:19:29        Automatic Transfer Switch: Fault cleared on Source A.   0x0C13
08/26/2017      21:19:28        Automatic Transfer Switch: Redundancy has been restored. Source A is available. 0x0C03
08/26/2017      21:19:22        Automatic Transfer Switch: Fault exists on Source A.    0x0C12
08/26/2017      21:19:05        Automatic Transfer Switch: ATS has lost redundancy. Source A is not available.  0x0C02
08/26/2017      21:18:26        System: Console user 'root' logged in from xxx.xxx.xxx.xxx.        0x0014

The 23 seconds was my reaction time to get the cable plugged back in. Nowhere in the log does it show the load dropping. The AOS/G2ATS firmware is 374/371 and the ATS microcontroller firmware is 2.0.3.

We have been following the suggested best practices, including having both sources on the same AC phase (though from different circuits on different datacenter PDUs), low sensitivity, preferred source none (non-reverting), etc.

This is very disappointing. I need to replace at least the 4 AP7753 at this site, and probably the units at the other sites as well. The question is, go with APC again and hope things are better / more reliable, or with some other brand. If a different brand, which?

Terry_Kennedy_apc · ‎2021-07-09

This was originally posted on APC forums on 9/5/2017

This was the first load drop in well over 4 years (and also the first firmware upgrade performed since they were installed). The first unit went fine, this was the second unit, and we abandoned plans to upgrade the 3rd and 4th units that night.

If it comes out, something else needs to replace it - either the new APC unit or something from another vendor. We'd probably schedule a time period to do all four at this site.

See Answer In Context

B-DoubleE_apc · ‎2021-07-09

This was originally posted on APC forums on 8/29/2017

Hi Terry,

I'm sorry for this issue with your ATS. Do you happen to remember what LED's were lit up on the front panel (just to confirm if it showed the same information as the web interface)? Additionally, would you be able to send me:

Serial number of the ATS
Copies of the event/data/config files (instructions to download here: http://www.schneider-electric.com/en/faqs/index?page=content&id=FA156131)
A screenshot of the "Event Counts" page in the Web UI, under the "Unit" tab

So you're aware, we have a new generation of ATS launching in the coming weeks (manual here: http://www.apc.com/salestools/HLAE-AJCRQQ/HLAE-AJCRQQ_R1_EN.pdf) that has many improvements over the previous generation, such as improved transfer time and an embedded NMC2.

Terry_Kennedy_apc · ‎2021-07-09

This was originally posted on APC forums on 8/29/2017

The front panel showed A and B LEDs green, the lower curved arrow lit, and the load LED green.

I have collected all of the data you requested, but I can't send you a private message with the link due to the way the forum is set up. Can you PM me so I can reply?

I'll look at the detailed specs of the new unit (AP4453) once it is announced.

B-DoubleE_apc · ‎2021-07-09

This was originally posted on APC forums on 8/30/2017

Hi Terry,

PM sent with my email address

B-DoubleE_apc · ‎2021-07-09

This was originally posted on APC forums on 9/5/2017

Hi Terry,

I've reviewed the files, but there aren't any smoking guns in the logs. Is this ATS still in deployment, or are you able to bench test it? Have you tried to force source switches since the load drop to see how the ATS behaves?

Terry_Kennedy_apc · ‎2021-07-09

This was originally posted on APC forums on 9/5/2017

This was the first load drop in well over 4 years (and also the first firmware upgrade performed since they were installed). The first unit went fine, this was the second unit, and we abandoned plans to upgrade the 3rd and 4th units that night.

If it comes out, something else needs to replace it - either the new APC unit or something from another vendor. We'd probably schedule a time period to do all four at this site.