3 AP7723 Issues I would like help on.

[Anonymous user]

I have been given the task of managing our Rack mounted ATS and PDU at my datacenter. For the last 3 years they just plugged them in and did no form of configuring. Well now I come in and configure them so they can actually work (before all of them had alarms thus could not switch). Upon configuring I have come across 3 issues. none are causing downtime but has stopped the ability to switch over. The issues are:

1- One of the ATS are alerting that the input voltages on both sources have went down to 110 volts in comparison to the 208 they should be. I also noticed that the -12v power supplies in the measuring tool within the ATS shows it at -22
.

2- Another is showing that its frequency is constantly changing from the normal 60hz all the way to 120 Hz and anywhere in between. This is only for source A

3  -Everything is showing out of phase.

Things that you should know are that I have taken a multimeter and tested everything that it could. All firmware is up to date. I am not allowed to unplug anything due to everything being in production. The topology of our power situation is as follows. Each Rack has 2 legs of power fed to it. Lets call them Leg A and B. each ATS has 1 source from each leg of power. those legs are rackmount PDU's that connect to a much larger leibert PDU that is ran off a leibert UPS. We have 2 independent UPS. Any help would be appreciative.

[Anonymous user]

I have been given the task of managing our Rack mounted ATS and PDU at my datacenter. For the last 3 years they just plugged them in and did no form of configuring. Well now I come in and configure them so they can actually work (before all of them had alarms thus could not switch). Upon configuring I have come across 3 issues. none are causing downtime but has stopped the ability to switch over. The issues are:

1- One of the ATS are alerting that the input voltages on both sources have went down to 110 volts in comparison to the 208 they should be. I also noticed that the -12v power supplies in the measuring tool within the ATS shows it at -22
.

2- Another is showing that its frequency is constantly changing from the normal 60hz all the way to 120 Hz and anywhere in between. This is only for source A

3  -Everything is showing out of phase.

Things that you should know are that I have taken a multimeter and tested everything that it could. All firmware is up to date. I am not allowed to unplug anything due to everything being in production. The topology of our power situation is as follows. Each Rack has 2 legs of power fed to it. Lets call them Leg A and B. each ATS has 1 source from each leg of power. those legs are rackmount PDU's that connect to a much larger leibert PDU that is ran off a leibert UPS. We have 2 independent UPS. Any help would be appreciative.

[BillP]

Hello,

Can you clarify - are items #1 and #2 happening on the same ATS?

Too, can you verify the firmware version, just to be sure? Have to ask - people say it's the latest all the time and sometimes its not so forgive me

• Anyway, on item #1, the -12V power supply is used for voltage and current measurement. So, considering this power supply is faulting, it makes sense to see the weird voltage reading. If under warranty or not, the device should be replaced.

• Item #2 - Is this the same unit as item #1 - you say "another" so I am thinking no but I wanted to be sure. I'd want to see the logs on this unit if possible and verify whether or not there is an active power supply fault here too. (Instructions on obtaining event.txt, data.txt, and config.ini are here -> How can I download Event, Data, Configuration, and Debug files from my Network Management Card? | FA...

• Item #3 - It is rare that two input power sources are in phase completely (meaning 0º phase angle). Our Rack ATSs can support out of phase switching up to 180º degrees, depending on firmware, so this is likely a non issue unless the firmware is really out of date (more technical info here -> http://www.apcmedia.com/salestools/THON-7GTPY6/THON-7GTPY6_R5_EN.pdf). Ideally, those two legs of power you mentioned are from each one of the independent UPSs you mentioned...?

[Anonymous user]

Yes these are 2 different ATS but same model. the above is the one with frequency issues and the below is the one with voltage issues. Both with same version.(its the version the apc guy told me was most recent.). All my ATS and PDU that are rackmounted shows out of sync. I attached the logs from about 2 days ago but this issue honestly probably been happening for months or years before i came along. The most awesomely drawn pic is the power arrangement. so hopefully it explains things more.

[BillP]

Ok, thank you. And the picture was nice

yes, the firmware is the latest.

On the first one, I am surprised there is no other fault here (power supply, etc) along with this problem. Ultimately though, this should be replaced. If you can, the only thing you could try would be to reset the ATS controller under the Unit->Controller menu but if you do, please read this first -> What does the "Reset ATS Controller" command accomplish on a Next Gen Rack Automatic Transfer Switch... because depending on the attached load, it could reboot/turn off briefly. It could reset the reading though, especially since there is no other fault.

On the second one, it seems like this is all related to the power supply fault. Does the unit actively report a power supply fault within the 5 active alarms it has? (or can you give me the event.txt for that one?)

And for the out of sync stuff, this is normal. It is not likely the two sources will be in sync because they are least several degrees apart. But, you won't see this as an alarm like I said because the ATS can switch sources if they are out of source.

So, it is likely these two units need to be replaced because of the frequency and voltage readings if the resetting of the ATS controller does not help the first one.

[Anonymous user]

Without the customers approval I cant unplug anything or do anything that would result in power loss. given the frequency one has nothing plugged in it and the voltage has only 1 thing but still. Also here are the event logs.

[Anonymous user]

Update. Soooo.... This last Saturday we had our electricians replace our UPS batteries as per another issue. I checked the ATS logs for the one with frequency issues and it cleared about 5 minutes of when the electricians finished. So that appeared to be an UPS issue. The weird thing is how Source A has the issue the UPS that was worked on was source B. So they shouldnt be connected but are. I checked the one with the voltage issue and it is still alerting the same as before. However when I telneted to it it said that the power supply that showed the -12v ps that shows -22v is ok. I am not sure if that said it before but it makes me think its a ups or main pdu issue. On that matter I know something is plugged up and running on the ATS as there is 1 server it shows 0 total power output. I then plugged up my laptop which is basically a 200 watt PS 2 monitors and a test server up to it and it still showed as 0 Watt output. All my devices are powered on and each of the outlets are showing correct voltage when I use a multi-meter.

[BillP]

Hmm, OK.

Well the logs don't tell me anything new unfortunately. I still think either they need to be reset or likely replaced to resolve this each of these issues.

[BillP]

Well, I learned my lesson then, lol. Does your UPS have logs where it was reporting high frequency on the output? I wonder if your UPS has a frequency slew rate setting to tighten but I can't imagine if it did, that it should still somehow be outputting 120Hz. I have never seen that before but now I have!

I did check with engineering on the -12V power supply reading -22V. It should not have an affect on the ability on the frequency measurement. But, that one is also reading low input voltage..

We are interested in swapping that unit though for evaluation..

[Anonymous user]

well the thing is that the one with voltage issue is still alarming and it never had the frequency issue. The one that had the frequency issue did not have a voltage issue and is  now fixed. Another weird thing is it was source A that threw the alarm. The ups that was worked on was for Source B. I dont know but it is now fixed. Someone said to me that I may have a bugged upgrade on the voltage one. He said I should try down grading to previous firmware then reinstalling the newest firmware. Because I believe all the measurement tools are bunked. I mean all the outlets are measuring correctly with the multimeter and was able to power several devices. And no these leibert ups units does not give us any logs. They might give leibert logs but not us. I recently discovered a schnieder electrics building while driving on i75. I wouldnt mind dropping off the voltage unit if it is totally bunked. I know that they are out of warranty most likely.

[BillP]

I don't agree with downgrading on this. If anything, resetting the ATS controller under the control menu is the next step. A botched firmware upgrade would not have this symptom.

If anything too, the unit is somehow mis-calibrated and that cannot typically be fixed by the user - you'd only be able to reset the controller in an attempt to do that.

Can I ask what state you're located in? Your serial numbers did not look out of warranty but I did not verify 100%.

[Anonymous user]

Cincinnati Ohio. If I reset the controller will that cut power to the devices on the ATS?

[BillP · Accepted solution]

Yes, it will according to What does the "Reset ATS Controller" command accomplish on a Next Gen Rack Automatic Transfer Switch... - for 330-340ms it will cut the output unfortunately. I imagine most equipment will have a problem with that.

In Ohio, we don't have any APC/IT Business Unit facilities so it must be one of the other Schneider Electric BU's facilities. I am in Rhode Island and our facility that does most of the design and evaluation stuff on returned ATS units is in St. Louis, MO.

The one serial number I see for I believe the unit with the frequency issue is mid-late 2012 manufacture date and these have a two year warranty.

I think when you get some time as to not put a risk to the load, we should try to reset the ATS controller and if that does not work, ideally we could replace it for you and take a look at it because I am pretty sure it will still be in warranty at least until the end of 2014. Now on that other one that magically fixed itself when the electrician worked on the other source's UPS, I am honestly clueless unless there is a mistake on what is connected to what and he really did touch the UPS that is feeding that input.