ATS output cut when Input B is connecting back to ATS

[Anonymous user]

I have my ATS firmware upgrade issue fixed in the the thread below:

Referenced content removed

Now, I have a bigger problem.

During the firmware upgrade, it is required to take out one Input. In my case, Input B is un-plugged.

Once the firmware is up and running normally, in connect Input B again.

Then I got my network disconnected.

Afterwards, network resumed and I find the following:

1. Input A changed to Not Available
2. ATS is taking power from Intput B
3. On the same PDU take power from ATS, some of the devices are reloaded due to lost power.
4. Input B, the MCB switch is now OFF

Why was that?

[BillP]

i agree with you. this procedure looks good.

[BillP]

thanks. i just need data.txt now to review along with your event log 🙂 can you attach that one too?

[Anonymous user]

Here it is.

Please look around 15:20 for the insert of Input B.

Thanks a lot!

[BillP]

ok, so let me make sure i understand the sequence of events.

you fixed your firmware issue that we worked on. in your case, you had A removed since it was your secondary source and B was feeding the ATS and the load. you when to reconnect B and it seems that it tripped an upstream breaker upon reconnecting B to power? so the ATS switched back to A and then the load rebooted?

could you please provide that event.txt file i asked you for that wasnt there before as well as data.txt and config.ini? they should all be in the same location.

[Anonymous user]

This is the sequence of events:

0. Both Input are normal. Input B is preferred. Input B is active.
1. Unplug Input A to prepare for firmware upgrade
2. Firmware upgrade failed to "cannot transfer application"
3. Web console app cannot be accessed
4. Put Input A back to ATS
5. Get help from forum
6. As check from front panel, Input A is active.
7. Remove Input B, to prepare of ftp upload of BIN file
8. BIN file uploaded (3.7.0). Web console accessible
9. Use EXE updater to update to 3.7.1. Success without issues
10. Connect Input B back to ATS
11. Network connection losts
12. Network access to ATS resumed
13. Input A is OFF (later confirmed MCB switch jumped to OFF)
14. Input B is active
15. Within the loadings of the ATS, Router reloaded. 1 of the 7 switches reloaded. (my 8th switch is dual-powered)
16. Input A is still OFF, as I afraid turning it on again will reboot my loading again.

Please find my event.txt as attached.

Thanks a lot!

[Anonymous user]

This is the log from my Switched PDU AP8959EU3, which takes power from my ATS:


10/12/2012 16:00:42 1.59 1.60 534.3 8.0 8.0
10/12/2012 15:50:42 1.57 1.60 534.1 7.9 8.0
10/12/2012 15:40:42 1.59 1.60 533.8 8.0 8.0
10/12/2012 15:30:42 1.59 1.59 533.5 8.0 8.0
10/12/2012 15:20:42 1.47 1.67 533.3 7.5 8.4
10/12/2012 15:10:42 1.60 1.61 533.0 7.9 8.0
10/12/2012 15:00:42 1.59 1.61 532.8 8.0 8.0

Message was edited by: sampson_fung@executivecentre.c

[BillP]

hi again,

i have a few questions below for you but ultimately, i think you should try reapplying A input to the ATS. if there is a problem, it should trip again. if it does not, then it might be one of the attached devices of load you have.

other questions I had:
• what are A and B - A UPS, commercial power, etc?
• do you know if those sources are in phase?
• i know you did not reapply power to A again on the ATS but did you reset the breaker upstream and just leave the ATS unplugged, or is the A MCB still off?

[Anonymous user]

As show in the logs, my loadings to my ATS is 7.8 to 8.1A
A is Utility Power, 16A circuit breaker
B is the 16A output of 5KVA UPS, which in turn connects to a 32A circuit
A&B are out of Phase
MCB of A is still off.

My main concern is:
1. After firmware update failed, (B active, A is off), re-insert A, everything is fine.
2. After firmware fixed (A is active, B is off), re-insert B, ATS loading rebooted (not all), A's circuit breakers trips, and Source changed to B
3. As in the manual of ATS, while connecting One to Utility Power and the Second to UPS, those are usually our of phase
4. In my Switched PDU, the direct downstream to ATS, there is no record of Voltage drop, but loading did drop lightly. This reflects certain PDU sockets rebooted.
5. At the very begining, my Input A is from a 3KVA UPS. But when I test of re-insert of input, the upstream circuit breaker of the 3KVA tripped (But loading never rebooted). Then I found the FAQ from APC as below. It recommended to get the latest firmware
http://www.apc.com/site/support/index.cfm/faq/index.cfm

6. But now, with that firmware, I got loading rebooted.


Common Rack Automatic Transfer Switch Scenarios and Questions

This article pertains to the following Next Gen Rack ATS models specifically: AP7721, AP7723, AP7724, AP7730, AP7732, AP7752, AP7753.

With these models, please ensure that after analyzing the situation, the latest available ATS firmware has been applied and instructions followed - INFORMATION BULLETIN - Upgrading Rack ATS Firmware (AP77XX) - potential to permanently damage ATS during download.

When both sources are available and stable, the ATS will switch back to its preferred source after a 30 second wait period. Attempts to continually switch the unit prior to the completion of the 30 second period could result in loss of load or damage to the ATS. Anytime an anomaly is detected within this sampling period, the 30 second timer will be reset.

The following is a list of potential causes of ATS to source switching when the upstream UPS goes to battery or when power appears to be present at the ATS input:



Switching of the ATS when the UPS goes to or from "on battery" mode

The switching of the ATS occurs as a result of an internal source synchronization when the UPS transfers from mains power to battery power. The UPS creates a frequency shift to speed up or slow down the frequency in order to align the inverter in phase with the sync source chosen. This frequency shift actually creates a deviation in the way the ATS measures the voltage coming from the UPS. This voltage deviation can be large enough for the ATS to interpret this deviation as a voltage drop resulting in the switching from the preferred source to the secondary source.

Dropping the load when the ATS switches sources while both appear to have power

The tripping of the circuit breakers during the switching process has been determined to be the result of switching between two sources that are out of >>phase with each other producing a voltage potential that exceeds the rating of the internal relays responsible for switching from one power source to the >>other. The latest ATS firmware resolves this issue and will prevent the ATS from tripping the upstream circuit breakers.

Here are some recommended actions that can be put into place to reduce the probability of the ATS switching when the UPS transfers to battery mode:

Revise the voltage transfer range from medium to wide on the ATS:


This will increase the amount of voltage deviation allowed prior to switching, reducing the probability of switching during this event.


Set the sensitivity preference from high to low:


This will allow more time for the frequency synchronization to occur before the ATS takes action and switches.


Set the source preference from source A to none on the ATS:


If neither source is preferred, this is the ideal setting. This will avoid switching back to Source A when the two UPSs are most likely to be out of sync with each other.

[Anonymous user]

Just one more point.

During the last day, my UPS did not recording any input problems.

Thank you very much!

[BillP]

are you going to reapply A? also, how have you narrowed down that there is not an issue with one of your specific devices? i.e. shorted power supply

[Anonymous user · Accepted solution]

As the ATS loading is being affected, the situation is out of my control now.

I need to submit a action plan for approval before any further actions can be taken.

This problem is also stopping my next project to include 4 units of the same ATS.

As on the possible problem with my loading, my line of thinking is like this:

1. My equipment has been on line for more than 1 month.
- Initially, they are connected to passive PDU and to Utility Power direct.
- Once my UPS is online, the PDU is changed to take power from UPS
- Once my switched PDU is ready, they are changed from the Passive PDU to Switch PDU
- Once my ATS is ready, the Switch PDU is changed from UPS to UPS + Utility Power.

2. During the firmware upgrade, we do not physically plug-unplug at the ATS side. Instead, we tripped the circuit breaker for the input links.

Hence I consider the chance of equipment shorting to very rare.

Thanks a lot!