AP7724 and Phase Sync:Out of sync

[hachete_apc]

Hello everyone,

We have two different AP7724 ATSs which show the "Phase Sync: Out of sync" message.

This is the setup:

ATS feed 1: Smart-UPS RT 8000 RM XL  with this data:

Date	Time	Vmin	Vmax	Vout	Iout	%Wout	Freq	%Cap	Vbat	TupsC	%VAout	TIambC	%HumI

		Smart-UPS RT 8000 RM XL										Integrated EMU
Date	Time	Vmin	Vmax	Vout	Iout	%Wout	Freq	%Cap	Vbat	TupsC	%VAout	TIambC	%HumI
23.05.2016	11:53:12	221.7	224.6	231.6	10.17	32.0	50.01	100.0	216.0	32.4	29.0
23.05.2016	11:43:12	221.7	224.6	231.6	10.28	32.0	49.98	100.0	216.0	31.9	29.0
23.05.2016	11:33:12	221.7	226.0	231.6	10.07	33.0	50.00	100.0	214.9	32.4	29.0
23.05.2016	11:23:12	223.2	224.6	231.6	10.07	32.0	50.00	100.0	216.0	31.9	29.0
23.05.2016	11:13:12	221.7	226.0	231.6	10.07	33.0	49.99	100.0	216.0	32.4	29.0
23.05.2016	11:03:12	221.7	226.0	231.6	10.17	32.0	49.97	100.0	216.0	32.4	29.0
23.05.2016	10:53:12	223.2	226.0	231.6	10.17	32.0	50.00	100.0	216.0	32.4	29.0
23.05.2016	10:43:12	223.2	226.0	231.6	10.28	32.0	50.00	100.0	216.0	32.4	29.0
23.05.2016	10:33:12	223.2	226.0	231.6	10.07	32.0	50.01	100.0	216.0

ATS feed 2: Smart-UPS RT 10000 RM XL  with this data: 

Date	Time	Vmin	Vmax	Vout	Iout	%Wout	Freq	%Cap	Vbat	TupsC	%VAout	TIambC	%HumI

		Smart-UPS RT 10000 RM XL										Integrated EMU
Date	Time	Vmin	Vmax	Vout	Iout	%Wout	Freq	%Cap	Vbat	TupsC	%VAout	TIambC	%HumI
23.05.2016	11:01:11	226.0	228.9	230.4	0.00	0.0	49.98	100.0	213.8	33.0	0.0
23.05.2016	10:51:11	227.5	228.9	230.4	0.00	0.0	50.00	100.0	216.0	33.0	0.0
23.05.2016	10:41:11	226.0	228.9	230.4	0.00	0.0	49.98	100.0	216.0	33.0	0.0
23.05.2016	10:31:11	227.5	228.9	230.4	0.00	0.0	49.99	100.0	216.0	33.0	0.0
23.05.2016	10:21:11	227.5	228.9	230.4	0.00	0.0	49.99	100.0	216.0	33.0	0.0

 I've spent all morning reading about this and I can`t seem to find the problem because both feeds seem to me to be in sync.

Anyone around to give a hand?

Thanks

Daniel

[BillP]

Hi Daniel,

The Rack ATS products, such as AP7724, report phase out of sync when your input power sources are as little as around 10-11º out of phase. The ATS can switch sources that are as much as 180º out of phase (with firmware v3.5.7 or higher).

I would say it is extremely rare to have two input sources that are exactly 0º out of phase and thus many users see this message. You'll notice that this a status item on the ATS but not an alarm. The information you showed from the UPS data log really cannot show anything related to this message. You just need to make sure your UPSs are fed from different sources themselves and not use two UPS's that are on the same input source to avoid any problems with the ATS operation. This phase sync item won't be able to detect or report that to you though - I just meant it as a side note.

Beyond that, there is no problem here if your UPSs are fed from different input power feeds themselves as you won't be able to sync two power sources to exactly 0º majority of the time.

I hope that helps. Let me know if I was not clear.

[hachete_apc]

Hi Angela,

We indeed feed from two different sources, but we're now running into this problem when swithching sources although we're following all the recommendations : http://www.apc.com/us/en/faqs/FA156201/

2. Dropping the load when the ATS switches sources while both appear to have power

The tripping of the circuit breakers during the switching process has been determined to be the result of switching between two sources that are out of phase with each other producing a voltage potential that exceeds the rating of the internal relays responsible for switching from one power source to the other. ATS firmware v3.5.7 and higher aims to resolve this issue and will help prevent the ATS from tripping the upstream circuit breakers

Any idea of what the solution could be to this problem?

Thank you,

Daniel

[Terry_Kennedy_apc · Accepted solution]

I think that FAQ was written in response to a discussion I had with APC...

It can still be an issue in 3.9.2, so we now insist on our datacenter A and B power being on the same phase (but from different sources) to avoid it.

There isn't really an ideal solution for this other than the above. Theoretically, you'd want to switch over as fast as you can. But the units use mechanical relays, so there is unavoidably some time delay during the cutover. As an example, if the ATS waited 1 second between opening (disconnecting) one source relay and closing (connecting) the other, the problem would never happen. But your equipment will get power cycled because most things can't tolerate a 1 second interval without power. So the delay gets shortened to something most equipment will tolerate. The other configuration options work to prevent reverting back to the original source (preferred source none), not switching for minor disturbances (sensitivity low), etc.

BTW, the reason the release notes warn you to disconnect one source when upgrading firmware is that there are 2 pieces of firmware - one for the (slow but complicated) management card, and one for the (fast but simple) ATS microcontroller. If there is a new version of microcontroller firmware, when it is written to the micrcontroller it will reset when done (like a reboot after Windows Update). While the microcontroller is rebooting it has no control over the source relays and they can be in an indeterminate state. If they happen to end up with source A and B on different phases, that is the same as trying to do a short circuit between the phases. That causes the breaker(s) to trip and damages the ATS relays.

It might be better for the hardware to use DPDT relays, with one side being source A and the other being B, with the common being the load. I'm not sure why the units don't do that - perhaps it does not meet isolation requirements.