Smart-UPS 3000RM Problems While On 14kw Generator Backup Power

[Anonymous user]

We have four Dell 2850 Servers that are connected to a 2006 APC Smart-UPS 3000RM battery backup unit. When our power goes out (as it did today), our 14kw Generac natural gas generator automatically turns on after 60 seconds. This means that the UPS unit only needs to provide one minute of backup power. The load on the UPS unit is only 36% with the four Dell servers running, and the PowerChute Business Edition says that the UPS unit will provide 11 minutes of backup power at this load. When the natural gas generator is running, it provides a consistent 118-120 volts of backup power according to the PowerChute console.

The problem that we encounter happens after about four or five minutes of the system running on the backup generator. The UPS unit will make a really loud humming sound, followed by what looks like a brownout in the office that is apparently being caused by the UPS unit trying to draw too much power from the generator all of a sudden. This causes the UPS unit to shut down and turn right back on, in turn causing the Dell servers to reboot. This process loops several times and then eventually the UPS unit just turns off completely and has to be manually powered back on - not to mention rebooting the servers again.

Because of this problem (which happened every time the power went out), we put this UPS 3000RM back in its box shortly after we purchased it in late 2006. We left it there for about a full year or so, and used a UPS XS1500 unit on each server to replace it. Then about six months ago I put the UPS 3000RM unit back in the rack after reading that I could switch the voltage parameter from high to low using the PowerChute console. I thought I had this problem solved, because after I made the voltage parameter change we had a power outage and the UPS unit didn't crash. However, the power went out today and the UPS unit crashed again. I came into the server room about 15 minutes into the power outage, and although the generator was running - the UPS unit and all four Dell servers were turned off. When I booted the UPS unit back on and the servers came back on, about five minutes later the UPS unit started making the loud humming noise followed by the lights dimming, and then the UPS unit and the servers rebooted. This happened about three times, and then the UPS unit finally turned itself off, along with the servers.

After the power came back on, I checked the PowerChute event logs, and it didn't even log when the unit crashed. All of the voltage logs during the five minutes that the unit was on backup generator power showed that the voltage levels were well within the acceptable range. I have no idea what to do about this, and am afraid that I will have to go back to using a UPS XS1500 on each Dell server and just scrap the 3000RM unit that we paid about $1,500.00 for. Right now I am using four XS1500 units on my Dell 600SC tower servers, and they work just fine on our backup generator. I have no idea what is causing the UPS 3000RM unit to drag the generator down to its knees and then crash.

If anyone has any tips or advice on this issue, I would greatly appreciate it. The good thing is that I can turn off the power in the building, which will cause the generator to turn on and allow me to test any new configuration changes.

Thanks in advance!

Message was edited by: Johnny

[TheNotoriousKMP_apc]

Hi Johnny,

Sorry to hear about your experience. It definitely sounds like something abnormal is occurring. For starters, do you happen to know if the generator output is being distorted at all? The harmonic distortion that could be coming from the output side of the generator into the UPS may explain the humming. However, as far as the UPS rebooting, does the software event log report anything for a UPS event? It's also possible that the UPS humming and rebooting are directly correlated, in that an AVR component of the UPS malfunctioned back when originally being used, and simply, is just causing any time AVR is in use, to crash everything. It would also work that way if there was a EMI/RFI filtering failure as well. Again it's tough to tell since you can't see what each individual part is doing in reaction to the input power from the generator. You also stated you let the UPS sit for over a year, was the battery connected? If so, what's the battery charge at now? Have you at all ruled out a possible bad battery in the system? I know it's probably more unlikely that it's that as opposed to the previously mentioned diagnosis because the UPS would run online for 5 minutes, but if the AVR or filtering went outside of the UPS's online-state threshold, and kicked to battery with a bad battery, that may cause unexpected downtime.

[Anonymous user]

Thanks for the very informative reply. I believe the battery is still in good condition. If I pull the plug on it, it runs all four servers for at least 10 minutes. Right now, the battery is 100% charged. I cannot remember if the battery was disconnected while in storage or not, but I do remember allowing it to completely discharge by using the calibration feature in the PowerChute console.

Here are the current stats from the battery while on normal AC power:

!http://www.myebayfiles.com/stuff/apc-1.jpg!

!http://www.myebayfiles.com/stuff/apc-2.jpg!

Message was edited by: Johnny

[Anonymous user]

Also, here is the event log file from yesterday's events. I'm not sure why it says "Low Runtime Remaining" because the generator was running at that time.

!http://www.myebayfiles.com/stuff/apc-3.jpg!

Thanks again!

Message was edited by: Johnny

[Anonymous user]

Also, I thought I would mention that back in 2006 when I first bought the unit, we went on vacation and while we were gone the power went out. Since the UPS unit was on high sensitivity (by default), when it sensed the slightest voltage fluctuation from the generator power, it put the unit back on battery power. Then when the battery would die, it would reboot the system, charge for a few minutes, go back on battery power when it sensed fluctuation again, which would reboot the servers each time. This went on for about seven hours while I drove back from Wisconsin in a rental car (I left my family in the Dells). I wonder if this caused what you have described here:

It's also possible that the UPS humming and rebooting are directly correlated, in that an AVR component of the UPS malfunctioned back when originally being used, and simply, is just causing any time AVR is in use, to crash everything.

Is there a way I can test the unit for any damage that may have occurred to it during the vacation outage outlined above?

Message was edited by: Johnny

[TheNotoriousKMP_apc]

Johnny,


Thanks for all the useful information. There's a good chance that this rapid changing back and forth between online and on battery due to the input to the UPS could've destroyed a component or two (transfer switch, rectifier, inverter, battery charger, filterer), there's no surefire way for you to isolate the problem to a specific component unless you possibly speak with KVar outside of these forums who can assist you in isolating where the humming is coming from on the board, and the component (as he doesn't work for APC, and we won't allow proprietary information to be publically posted here) that failed.

[Anonymous user]

I wonder why the voltage parameter issue is not something that new users are warned about? Obviously this is something that every new APC user who is a natural gas generator owner needs to know about, or they will face the same set of problems that I have been facing. I guess the voltage parameter is set to high by default, but this will not work with a generator and needs to be changed to medium or low by the user. I wonder how many new customers find this out the hard way like we did?

Message was edited by: Johnny

[TheNotoriousKMP_apc]

Johnny,

I totally agree with you. I'm going to see if we can get a couple of the other moderators together and get some new K-Bases written to post this type of information. Probably a Generator Use with Smart-UPS since there's only one for a Back-UPS, but it doesn't make much reference to the sizing of the generator vs. the output of the generator to the input of the UPS.

To answer your question on how many customers have to deal with this? I'm not sure. I've been an APC employee for a little under 3 years. In that time, I've come across less than 10 instances where a natural gas powered generator is being used. The majority of them were diesel. (Should probably take into account that at one point on our Smart-UPS Tech Support team, I was handling on average 40+ calls a day in 8 hours for 2 months straight. So whatever that number is, that should speak for itself).

But again, I'll see if I can get with JoeMomma and IPickedAWinna and see what we can come up with.

[Anonymous user]

Would you suggest buying a new APC 3000 to replace the unit? Will the same thing happen again?

I need to find a solution. We had a storm and the power went out again today. When the power goes out, the APC unit fails and reboots. However, if I just pull the plug on the APC unit - it doesn't reboot, and the servers stay online.

What gives?

[Anonymous user]

I just ran across a customer review of a Guardian natural gas generator that is similar to mine. Could it be that the waveform type will not work with the APC Smart UPS unit?

+When on automatic, there is a delay of 15-20 seconds before the generator supplies power after utility power is lost. This means that you still need a UPS of some sort if you're trying to protect computers. I did have some trouble with my APC Smart-UPS, which didn't like the power supplied by this generator. Voltage was ok (in the 124v range), so it was either frequency or waveform that the UPS didn't like. I haven't yet narrowed that down. However, many UPS' have an adjustable sensitivity to these things. In my case, I was able to tell the UPS to be less picky and all is well now.+

Message was edited by: Johnny

[Anonymous user]

I believe I figured out what caused the problem: My generator is not big enough. It only pushes 13kw on natural gas (15kw with LP). This would be enough to accommodate the server room and its mini-split air conditioner, but not for the rest of small building.

Now I need to find out if my APC unit is damaged. I'm not sure it would be worth paying to have it evaluated for repair. By the time I pay shipping both ways to somewhere, I could have had a good down payment on a new unit.

[Anonymous user · Accepted solution]

We have four Dell 2850 Servers that are connected to a 2006 APC Smart-UPS 3000RM battery backup unit. When our power goes out (as it did today), our 14kw Generac natural gas generator automatically turns on after 60 seconds. This means that the UPS unit only needs to provide one minute of backup power. The load on the UPS unit is only 36% with the four Dell servers running, and the PowerChute Business Edition says that the UPS unit will provide 11 minutes of backup power at this load. When the natural gas generator is running, it provides a consistent 118-120 volts of backup power according to the PowerChute console.

The problem that we encounter happens after about four or five minutes of the system running on the backup generator. The UPS unit will make a really loud humming sound, followed by what looks like a brownout in the office that is apparently being caused by the UPS unit trying to draw too much power from the generator all of a sudden. This causes the UPS unit to shut down and turn right back on, in turn causing the Dell servers to reboot. This process loops several times and then eventually the UPS unit just turns off completely and has to be manually powered back on - not to mention rebooting the servers again.

Because of this problem (which happened every time the power went out), we put this UPS 3000RM back in its box shortly after we purchased it in late 2006. We left it there for about a full year or so, and used a UPS XS1500 unit on each server to replace it. Then about six months ago I put the UPS 3000RM unit back in the rack after reading that I could switch the voltage parameter from high to low using the PowerChute console. I thought I had this problem solved, because after I made the voltage parameter change we had a power outage and the UPS unit didn't crash. However, the power went out today and the UPS unit crashed again. I came into the server room about 15 minutes into the power outage, and although the generator was running - the UPS unit and all four Dell servers were turned off. When I booted the UPS unit back on and the servers came back on, about five minutes later the UPS unit started making the loud humming noise followed by the lights dimming, and then the UPS unit and the servers rebooted. This happened about three times, and then the UPS unit finally turned itself off, along with the servers.

After the power came back on, I checked the PowerChute event logs, and it didn't even log when the unit crashed. All of the voltage logs during the five minutes that the unit was on backup generator power showed that the voltage levels were well within the acceptable range. I have no idea what to do about this, and am afraid that I will have to go back to using a UPS XS1500 on each Dell server and just scrap the 3000RM unit that we paid about $1,500.00 for. Right now I am using four XS1500 units on my Dell 600SC tower servers, and they work just fine on our backup generator. I have no idea what is causing the UPS 3000RM unit to drag the generator down to its knees and then crash.

If anyone has any tips or advice on this issue, I would greatly appreciate it. The good thing is that I can turn off the power in the building, which will cause the generator to turn on and allow me to test any new configuration changes.

Thanks in advance!

Message was edited by: Johnny