Current UPSs won't stop server reboots. What equipment is reccommended?

Anonymous user · ‎2021-06-29

We're currently running four racks of servers and phone equipment in two of our buildings. Each rack has two rackmount SmartUPS 1400XLs (and a random Minuteman UPS) to keep everything up in the case of short power outages or brownouts, which are quite frequent here. Equipment is spread out across the multiple UPSs, all have good batteries, and none of them have loads showing more than 2-of-5 bars.

Problem is, our UPSs aren't switching over fast enough to stop the servers from blinking off and rebooting, and we then have to baby about half the servers back online, deal with corrupted databases, hardware failures, etc. We've hobbled through regular outages for a couple of years now, stalling till we were ready for a major equipment update. Most of the servers are being virtualized which will consolidate everything down to just one rack with three blades, some network storage equipment, and the phone system.

We're done with servers being knocked offline due to power. What should we be looking for as far as a replacement here, because all we really know is the APCs we have now are not what we need.

What should we be looking for here to get the best performance when we start ordering for the server room upgrade? Is there a best-value option for us? Are the new 1500s that much improved over the older 1400s we've got that just upgrading to the new model will take care of us? Or are we looking at retiring the SmartUPS for one of the Symmetra solutions?

Thanks.

Anonymous user · ‎2021-06-29

I have some audio equipment from Meridian (not the same as Meridian phones), with extremely picky power supplies. Any sort of utility blip, regardless of duration and whether the UPS goes to battery or not, will cause the Meridian processor to shut down. This is on a Matrix 5000. There's a mechanical relay in the UPS (you can hear it when the UPS adjust taps or goes to battery) which produces a very brief dropout of less than one AC cycle (1/60th of a second).

Any decently designed power supply should be able to handle far more than a single AC cycle dropout. Meridian supplies are less than decent that way (there have been a number of "we suggest you visit your dealer to have your power supply upgraded - at your cost, of course" notices from Meridian).

In some cases this brief dropout is absolutely necessary - for example, on the APC Rackmount Automatic Transfer Switch, it HAS to be break-before-make when switching sources, as the 2 sources may not be phase-synchronized.

30 years or so ago, the "Continuous On-line UPS" was all the rage as it had 0 transfer delay (it always ran the load from the batteries - the AC input just charged the batteries). There were a lot of disadvantages to that design, and since then there have been many advances in UPS design which have been adopted by APC and others.

I'd check with Angela here to see which APC models have 0 transfer delay - I'm not up-to-date on APC's current product offerings (as you can tell by the fact that I'm running a Matrix 5000).

See Answer In Context

BillP · ‎2021-06-29

reading this my first question was "how do you know the actual problem is that the UPSs are not transferring fast enough?" are the servers doing a hard shutdown or rebooting during the transfer to battery and then shutting down?

are these configured for graceful shutdown with powerchute software? - this scenario could occur if you do not have any software or graceful shutdown software configured properly.

have you done a pull plug test or similar (pulling the plug on the UPS to see if it supports the load)?

any information in any log files? i wouldnt want to suggest something else until I knew the definite root cause to make sure we didnt get you something you didnt need or that would not fix the problem.

Anonymous user · ‎2021-06-29

The supply not taking over fast enough is just a guess of ours. ('Cause that's what UPSs are supposed to do, right? ;D)

From watching them go down when we're present during a blip they appear to losing power, not just a reboot. They're shutting down hard then coming back up as soon as power is restored a moment later, no different than just flipping the switch on the PSU then back on again. (The logs don't have anything useful aside from a note that there was a shutdown due to power outage.) They do this on a "plug test" as well.

If the outage is more than a blip everything starts to come back online and it will run the servers till the batteries are drained, but at that point it doesn't matter. I thought it could be load-at-startup being too high but even the dedicated UPS running just the mail server and nothing else doesn't stop that moment of silence when every fan in the closet stops spinning at once. =/

Only a few if any of the servers would be running PowerChute, though. (I think the phone system is on a dedicated UPS with shutdown configured, since that one never comes back properly on its own. Can't verify from here at the moment, though.)

The software can make that much of a difference? That might be a place to start. We were under the impression that only mattered for configuring the sensitivity and audible warnings.

Message was edited by: Larva

BillP · ‎2021-06-29

ok based on some of your comments, your guess sounds a bit more likely - the fact that they reboot during a long power outage but run on battery after they come back up (after the damage is done).

does this happen to every single UPS/server combination? Are the servers all the same model? does the UPS reboot at any point also (based on what you said I am guessing the answer is no)? one other thing i also might try is using a voltmeter on the UPS outlets (with maybe no load attached) and see if you see a disruption in output voltage during the switch to battery or if it is indeed just specific to your load (rather than an issue with the UPS hardware).

you mentioned a phone system. a lot of those do indeed require an "online" UPS which would be the Smart UPS RT line. if it is indeed the transfer time, you'd look at something like Smart UPS RT, Symmetra LX or Symmetra RM depending on requirements.

Anonymous user · ‎2021-06-29

I have some audio equipment from Meridian (not the same as Meridian phones), with extremely picky power supplies. Any sort of utility blip, regardless of duration and whether the UPS goes to battery or not, will cause the Meridian processor to shut down. This is on a Matrix 5000. There's a mechanical relay in the UPS (you can hear it when the UPS adjust taps or goes to battery) which produces a very brief dropout of less than one AC cycle (1/60th of a second).

Any decently designed power supply should be able to handle far more than a single AC cycle dropout. Meridian supplies are less than decent that way (there have been a number of "we suggest you visit your dealer to have your power supply upgraded - at your cost, of course" notices from Meridian).

In some cases this brief dropout is absolutely necessary - for example, on the APC Rackmount Automatic Transfer Switch, it HAS to be break-before-make when switching sources, as the 2 sources may not be phase-synchronized.

30 years or so ago, the "Continuous On-line UPS" was all the rage as it had 0 transfer delay (it always ran the load from the batteries - the AC input just charged the batteries). There were a lot of disadvantages to that design, and since then there have been many advances in UPS design which have been adopted by APC and others.

I'd check with Angela here to see which APC models have 0 transfer delay - I'm not up-to-date on APC's current product offerings (as you can tell by the fact that I'm running a Matrix 5000).