Silcon 60-80KW UPS/Autostart

[Anonymous user]

A site on our radio system experienced a commercial power failure. The UPS kicked in without a problem, but the backup generator failed to start. After 45 minutes, the batteries were apparently depleted and the Silcon UPS shut itself down ("**System OFF**") taking down the site. When power was restored, the UPS failed to come back online automatically. I ended up calling APC technical support who talked me through manually bringing the UPS back online. A day or so later when I checked the parameters on the UPS, I checked the "Autostart" parameter expecting to find it set to "No" but instead found it set to "Yes". So, why did the UPS did not come back up automatically as soon as it saw power restored? One hypothesis I had was that a technican who had been in a few weeks earlier checking the batteries had set it Autostar to "Off" and neglected to turn it back on. However, this would require some sort of default setting of "Yes" in order for the UPS to return the parameter to "Yes" after the shutdown and restart cycle. I don't see anything in the documentation that suggests that a configurable default value exists. Any ideas? We're meeting next week to review the failure and this is the only issue in the whole series of events that I still do not understand.

[TheNotoriousKMP_apc]

Eric,

If you wouldn't mind, please report your findings here. That way I can have the 3-Phase team revisit it if need be.

[TheNotoriousKMP_apc]

Eric,

Do you have a Network Management Card installed in the Silcon? Is it possible at all that there are configured shutdown and return parameters that have been setup to work with PowerChute Network Shutdown for graceful, unattended shutdown of your server equipment.

The most common thing seen is that customers have setup a return battery capacity which is set at a reasonable return percentage, but after a deep discharge of the batteries, the UPS output power remains off until the battery capacity has reached that percentage.

[Anonymous user]

Thanks,
Someone else (my boss, actually) had also suggested that there might be a battery charge threshold issue but if that were the case I would expect to see something besides "** SYSTEM OFF **" on the display.
The site contains radio equipment. I'll double-check but I am not aware of any feedback mechanism between the radio equipment and the UPS to coordinate a graceful shutdown.
I'm going to visit the site tomorrow and, among other things, examine the event stack to see if I can get the play-by-play summary of what the UPS saw during the power failure.
This whole episode is turning into a learning experience for a number of people, most of whom didn't work here when the system was installed.
Thanks again,E

[Anonymous user]

Here's the play-by-play:

12:50:04 "Battery Operation", (Power Lost to Site)
13:12 "Battery Weak", Battery Capacity Below 75%
13:41 "Low DC Warning", Battery Nearly Discharged
13:42 "System Off", UPS Has Been Off (Full Outage Begins)
13:42 "Low DC Shutdown", Battery Discharged to Minimal Permissible Level
14:17 "MPU is Reset", UPS Has Been Completely Shut Off
14:17 "Inverter Current Limiter Active", Inverter Overload/Reduce UPS Output Load
14:40 "Q003 On", External Bypass is “ON” (Full Outage Ends)
15:38 "Q003 On", External Bypass is “ON” (Partial Outage Begins. Lasts ~ 10 minutes)
16:18 "Q002 Off", UPS Output “OFF”. (On Phone with APC Technical Support/restarting UPS.)
16:18 "Inverter Voltage Error", Inverter Voltage Outside Limits (Normal during Stop/Start)
16:19 "Normal Operation",
16:19 "System Turned On",
16:19 "Bypass Operation", UPS Has Been in Bypass Operation
16:20 "Normal Operation"

I added the definition of each event for my own reference.

There were a large number of "Synchronization Error" and "Mains Outside Tolerance" events that I did not list.

I am most curious about what the events at 14:17 indicate. There was no one in the room at that time but it may correspond to when our backup to the backup generator was engaged. 14:40 is where I turned on Q003 on my own initiative. 15:38 is where I turned off Q003 after commercial power was restored, thinking the UPS would come back online on its own, then turned it back on immediately when it did not.

Thanks for all your help.

Eric

[TheNotoriousKMP_apc]

Eric,

Brought the post to 3-PH escalation and their Senior Technician (their equivalent of what I am for single-phase), the most knowledgeable and tenured person, had the following to say:

The default setting for autostart is off. If he had it set to on, his idea that the technician working on the batteries turned it off during the service makes sense, otherwise the UPS would keep restarting after it saw 60s of good input power, despite no one pushing the on button. Perhaps they he did not set it back to on after the service. But since he is saying that he found it set to Yes, then I doubt this is what occured, since this does not change by itself.

The other possibility, is the customer was standing in front of it, and when power was restored and he saw the UPS stay at system off and then closed Q3. The UPS waits 60s to see good AC input before auto starting. If he closed Q3 before this 60s time frame, the UPS would see Q2 AND Q3 on at the same time, and it will not start under those conditions.

Since I don't do 3-Ph and have MINIMAL (and I stress that word HEAVILY) experience with anything 3-Ph, that's pretty much foreign language to me.

[Anonymous user]

Thanks, we're getting closer.

The backup to the backup generator is a portable generator on a fire truck. I'm wondering if the power coming from this generator was sufficiently clean to allow the UPS to kick back on? I have determined that the UPS has a Out-Of-Band management card on it so I am planning on returning to the site tomorrow to obtain the entire event log. One of those "Synchronization Errors" may have been relevant. It'll be a big deal (for us) if the portable generator power has issues.

The comment on the Q003 switch is helpful. Once commercial power was restored, I did switch Q003 off of bypass. When the UPS did not come back online immediately, I switched it back, probably within 5 seconds. If I had waited the 60 seconds it might have started.

[TheNotoriousKMP_apc]

Thanks again for the Update.

I would definitely look at the fire-engine Generator. It's possible that the harmonic distortion caused by the generator would be outside of spec for the UPS to operate sufficiently.

[Anonymous user]

Here's the complete event log of this incident in reverse chronological order. I don't have the exact time written down of exactly when the fire truck generator was attached, but I speculate that it was at 14:17:08. I'd be interested in the opinion of one of your 3-phase folks, especially if they think there might be an issue with the fire truck generator. Thanks!



Logged mode:
Normal operation 08.04.28 16.20,55

Logged mode:
Bypass operation 08.04.28 16.19,55

System
Turned On 08.04.28 16.19,00

Logged mode:
Normal operation 08.04.28 16.19,00

Output is out
of tolerance 08.04.28 16.18,59

Inverter voltage
error 08.04.28 16.18,59

** Q002 off **
08.04.28 16.18,13

** Q003 on **
08.04.28 15.38,35

Synchronization
error 08.04.28 15.06,47

Bypass sync.
error 08.04.28 15.06,47

** Q003 on **
08.04.28 14.40,37

Synchronization
error 08.04.28 14.29,53

Bypass sync.
error 08.04.28 14.29,53

Synchronization
error 08.04.28 14.29,50

Bypass sync.
error 08.04.28 14.29,50

Synchronization
error 08.04.28 14.24,50

Synchronization
error 08.04.28 14.20,12

Bypass sync.
error 08.04.28 14.20,11

Synchronization
error 08.04.28 14.17,43

Bypass sync.
error 08.04.28 14.17,43

Synchronization
error 08.04.28 14.17,40

Bypass sync.
error 08.04.28 14.17,39

Logged mode:
System off 08.04.28 14.17,18

Mains is moment.
out of tolerance 08.04.28 14.17,08

Inverter current
limiter active 08.04.28 14.17,08

Synchronization
error 08.04.28 14.17,08

Bypass sync.
error 08.04.28 14.17,08

* BATTERY *
* WEAK * 08.04.28 14.17,08

Bypass is out
of tolerance 08.04.28 14.17,08

Bypass is moment
out of tolerance 08.04.28 14.17,08

Mains is out
of tolerance 08.04.28 14.17,08

Logged mode:
MPU is reset 08.04.28 14.17,08

Low dc shutdown
08.04.28 13.42,18

Logged mode:
System off 08.04.28 13.42,18

Low dc warning
08.04.28 13.41,25

* BATTERY *
* WEAK * 08.04.28 13.12,22

Mains is out
of tolerance 08.04.28 12.50,54

Bypass is out
of tolerance 08.04.28 12.50,54

Bypass is moment
out of tolerance 08.04.28 12.50,54

Mains is moment.
out of tolerance 08.04.28 12.50,54

Mains is out
of tolerance 08.04.28 12.50,05

Bypass is out
of tolerance 08.04.28 12.50,05

Bypass sync.
error 08.04.28 12.50,04

Bypass is moment
out of tolerance 08.04.28 12.50,04

Mains is moment.
out of tolerance 08.04.28 12.50,04

Synchronization
error 08.04.28 12.50,04

Logged mode:
Batt. operation 08.04.28 12.50,04

[TheNotoriousKMP_apc]

Eric,

First and foremost, if you find this gruesome to go back and forth via email, feel free to call 3-Phase and ask them to just read the thread. Most of them sit right next to me. Here's the feedback I received:

*"There are quite a few sync errors, which is indicative of a sycnronization to generator issue. There are settings on the UPS that can be modifed to make the UPS more generator friendly. I am also not sure what size Silcon or size of the genset, but the rule of thumb for APC 3phase UPS's is the genset must be at least 1.3 times the size of the output KW rating of the UPS. (We only need 1.3 since we have unity input power factor and very high efficiency).*

*The settings for generator compatibility are located in the 28 and 29 stacks on the display of the Silcon. I would suggest he max out the delta softstart in the 28 stack to 40s, as well as the syncronization to 4hz/sec. In the 29 stack, he can ensure the freq tolerance is set to 8%."*

[Anonymous user · Accepted solution]

I have a long (long!) history of providing technical support so my inclination is to keep things public so that if someone else down the road has the same issue, the answer is out there. I don't mind the bulletin board approach, and I think we're done anyway.

My conclusion is that the UPS failed to restart automatically because either the fire trucker generator was out of tolerance, using the parameters in the UPS to define "tolerance", or the parameters in the UPS were (and still are) too restrictive for the generator. The equipment ran on the backup generator when I bypassed the UPS so I'm inclined to think the parameters are too restrictive. Thanks for your help, and extend my thanks to the 3-phase folks as well.