OS Shutdown Delay - Not enough time to properly shut down.

[Anonymous user]

For years I have configured APC UPS devices - mostly in small business environments.

I have almost exclusively used the bundled Power Chute Basic editions and took for granted their seemingly easy setup and configuration.

That is until we started experiencing quite a few power events which were leading to server corruption, even though the servers should have been properly shut down.

I believe I have traced the problem to the "OS Shutdown Delay" settings in the Shutdown Sequence settings of Power Chute.

Now, part of the problem is I can't seem to find a proper description anywhere of what the Shutdown Delay is supposed to do. However, from my testing, what I have been told by APC techs (after working on the phone for longer than we should have needed), and from a post here , I believe the following description is accurate:

OS Shutdown Delay:
+The maximum time required for OS shutdown after the configured command file has completed. After this delay, the UPS will turn off its output, turning off power to the protected system.+

The problem is that the maximum time that I am allowed to set for this value (either by manually dragging the bar, or trying to put in the numbers) is 10:30 minutes. I have tried this on several servers running PowerChute versions 7.0.5 and prior connected to several different models of Smart-UPS.

In my particular example I have a Windows server connected to a SmartUPS 1500 taking up a negligible load. I have almost 80 minutes of runtime on a full charge. I have configured my server to begin the shutdown sequence 2 minutes after power is lost. This means I should have well over 70 minutes of runtime when a shutdown event is initiated.

Why am I only allowed to give my server a maximum of 10:30 minutes to shutdown when my server often takes longer than this to gracefully shutdown (+read: Exchange+). What is the purpose of purchasing an UPS which provides high runtimes if the software can't support more than 10:30 minutes?

I have opened a ticket with APC and their techs have been less than stellar. 3 different techs give 3 different answers and then don't respond when clearly presented with tests that show this to be the case.

I have attached a file which shows the maximum of 10:30 minutes that I am allowed to set (the screenshot shows me using a command file, but this happens whether I use a command file or not).

I would be interested to know what people using other versions (like the Deluxe) have in terms of maximum OS Shutdown Duration limits. This seems to be a bug/limitation in the software. If that is the case, then is seems APC is saying that only users who purchase a deluxe version can support servers with shutdown durations longer than 10:30 minutes. If that is the case, it would make sense to bundle the deluxe version with devices which support more than 10:30 minutes of runtime!

Please help me at the very least confirm this issue, because the folks I have spoken with so far at APC support doesn't even seem to want to recognize it.

If indeed I am doing something wrong, then I will eat my hat and be thankful that someone out there is smarter than the people I have worked with on the phone so far and can show me how to set this up!

thanks

[voidstar_apc]

That's a limitation on many current generation Smart-UPS. PowerChute tells the UPS ahead of time to shutdown. The hardware on your UPSes physically don't support waiting longer between the command to shutdown and actually shutting down.

I don't have a good, simple solution with Powerchute Personal Edition. Perhaps someone with more knowledge of the product lines could recommend an APC UPS that supports longer delays, if any.

Another option is to use a Network Management Card in conjunction with the free [PowerChute Network Shutdown|http://www.apc.com/products/family/index.cfm?id=127]. According to the feature list "In extended outages, UPS proactively coordinates and confirms safe, complete shutdown of all systems before UPS turns off. "

[Anonymous user]

Thanks for that info Void...and not that I don't trust you but I always like to see something "official" that states what is the case.

I understand (and agree) that PowerChute tells the UPS ahead of time when to shutdown, but if indeed the UPS hardware doesn't support waiting longer than 10:30 minutes that is a major, major issue.

SmartUPS are used extensively in the small business world, as is SBS. SBS runs SQL and Exchange, two applications which definitely can make a server take a while to shutdown. It is not uncommon for a server to take more than 10:30 minutes, and by having the UPS cut power when there is still 60-70 minutes of runtime left on the box seems like an absolutely ridiculous configuration.

I know that I could write a command file to shut down the Exchange services prior to the OS shutdown. The problem there is that is can be difficult to estimate the needed time, and things get even more complicated if you bring in more than one server on the UPS which now affects the runtime.

I believe that it should be possible to set these devices up in a configuration where, at the very least, the UPS will never go into sleep mode until it has fully discharged. This would give all the runtime available (and paid for) to shut systems down.

I hope that there are some more people out there with additional insight that can add to this.

Thank You.

[voidstar_apc]

I've edited my response to include the possibility of using a network management card. Unfortunately I didn't edit it quick enough 🙂 Perhaps that would work for you.

[Anonymous user]

Yes, I have been reviewing that as a solution and will want to confirm that works before purchasing.

However, it is really not a good solution, at least not the way APC sells its current line of SmartUPS.

All indicators would point to the fact that if I buy a single UPS to supply power to a single server (note: I will just use a single server here, because I do realize the caveat about PCBE supporting only one server) and that UPS has a high enough rating to support the necessary runtime, then I should expect a safe shutdown of my system.

Not to beat it dead, but if I pay for a UPS with 80 minutes of runtime when connected to my server, I shouldn't expect it to pull the plug after 10:30 when my system may take 30 minutes to shutdown.

I will also note that multiple APC techs never mentioned anything about this limitation you mention (not that I trust them over you). And, in fact, when I described my situation was assured multiple times that the UPS would not ever cut power as long as the OS was still up and runtime still available. This has been proven blatantly false.

Oh one other thing which I wanted to comment from your edited post about the Network card solution:

The term +"In extended outages"+

The fact is, extended outage or not, once a shutdown event occurs there is no way to stop it. So, assuming we want to be safe and begin shutdown only 2 minutes after a power loss, or wait 30 minutes (or more with a larger unit) we still are given only 10:30 minutes to complete the full shutdown.

Thanks for your advice so far.

[voidstar_apc]

>
Oh one other thing which I wanted to comment from your edited post about the Network card solution:

The term "In extended outages"

The fact is, extended outage or not, once a shutdown event occurs there is no way to stop it. So, assuming we want to be safe and begin shutdown only 2 minutes after a power loss, or wait 30 minutes (or more with a larger unit) we still are given only 10:30 minutes to complete the full shutdown.
>

I believe extended outage here means an outage long enough that you will begin a shutdown. It's only in that case that the Network Management Card will coordinate a safe shutdown of all systems. If the outage is not long enough, then your computers will ride through it none the wiser.

[Anonymous user]

So is my original problem of not having more than 10:30 minutes to shut down a server confirmed by any other sources?

Additionally, is there any confirmation that a network card or a deluxe version will solve this issue?

I don't want to invest in additional hardware/software only to find the same issue.

Thank You.

[voidstar_apc]

I asked one of the UPS engineers about it. Ultimately, they started with a 10 minute, 30 second limit and upped it incrementally on newer generations of UPSes. They were unaware of software like Exchange needing a half hour to shutdown. One of the latest UPSes I've come across could shutdown in 30 minutes. Hopefully now that they know about this issue, future UPSes will be able to wait for hours.

The network management card on UPSLink (but not Microlink at the moment) UPSes does it's own timing which should avoid this problem. You'd basically set up a command file to shutdown Exchange and set the command file execution time to 1900 seconds or so. I agree that this setup should be tested before buying these cards.

[Anonymous user]

Thanks for the follow up.

That just seems very strange to me... Does anyone in an SMB environment buy something like a Smart1500 to actually be able to have 60+ minutes of runtime if the power goes out? Unless you are in a 24/7 uptime environment, no one wants their UPS to actually provide runtime power...they just want to ensure their systems get shutdown gracefully. And I don't think someone in a critical environment is using SmartUPS devices!

This really seems like a glaring oversight, especially since you seem to indicate that it is a problem with the UPS itself and not the software. I also can't believe they were unaware of Exchange needing long shutdown times, especially because they provide the shutdown DLLs.... it is unfathomable that a UPS company doesn't know that a server might take more than 10 and a half minutes to shut down. You also mention that they upped it on newer models...well these UPS devices are not even a year old and have this issue, so I don't think they have resolved anything.

I was just in at another client who had an older SmartUPS 3000 with an AP9606 card. After upgrading the firmware I was able to setup the Power Chute Network Shutdown software like you described.... I let the Exchange shutdown script run for 15 minutes. The UPS didn't seem to kill power until after that server was done shutting down, even though another server had shut down 10 minutes earlier... so it does appear that way does work. I was still confused why the UPS took over 5 minutes to go into sleep mode though after the server shutdown, since I thought I had that delay set to the lowest value of 20 seconds.

This issue is really going to make me look at other UPS vendors for small customers. To pay the additional money for a network management card on top of the high prices of APC just to get more than 10 minutes for an SBS server shutdown seems a little ridiculous. Not to mention that no one from APC has responded either over the forum or gotten back to me on my ticket after numerous e-mails and calls.

I appreciate your assistance void.

[voidstar_apc]

I certainly understand that. We're always rolling out improvements so be sure to keep an eye out.

You have a good question about why other SMB environments don't seem to be vocal about longer timeouts before shutdown. I did a quick Google search for "[Exchange shutdown slow|http://www.google.com/search?q=exchange+shutdown+slow]" and there seems to be a lot of information on speeding it up. Specifically, on windows domain controllers running exchange, such as SBS systems you'd likely find in SMB environments, shutdown times can be cut from 10 minutes down to 2 minutes by shutting down the exchange service first via a command script or [applying Microsoft's fix|http://support.microsoft.com/default.aspx?scid=kb;en-us;887539]. I'm certainly not a windows server expert, and you may have already come across this, but if not it's worth a shot.

I hope that helps.

[BillP]

Is there anything new to this issue?

We have several Smart-UPS for our small customers. Even the newer ones will not allow me to set a higher value for "OS shutdown delay". So the UPS will pull the plug during the shutdown. Also the description in the "shutdown sequence configuration" is confusing: It says "Time Required for Shutdown Sequence". If you click "configure", it says "Runtime Remaining Threshold". The description on the bottom mentions only the delay for the script and the "OS shutdown delay". Even the support could not clearly communicate what value means what exactly.

If the maximimum time for "OS shutdown delay" is hardcoded in the device and i can not prevent the unit from pulling the plug after 10min:30sec, the unit is useless. This would, at least in germany, considered a hidden defect that would allow me to return the unit to where i brought it.

Is there any setting i could use to disable the power off after X minutes? I don't need it, and i think most of the other "non enterprise customers" don't need it too. When power fails, i wan't the server to shutdown and the UPS to provide battery power for as long as possible. No need for shutting the UPS down. It will shut down by itself when the battery is empty.

I have i ticket with support and will tell you if there is anything new.

[voidstar_apc]

>
Is there any setting i could use to disable the power off after X minutes? I don't need it, and i think most of the other "non enterprise customers" don't need it too. When power fails, i wan't the server to shutdown and the UPS to provide battery power for as long as possible. No need for shutting the UPS down. It will shut down by itself when the battery is empty.
>

The good news is that what you want is possible. The bad news is that doing so would mean your computers may not turn back on after a power outage.

I'm not a PowerChute Business Edition (PCBE) expert (perhaps someone else here has more information on this aspect) however I know you can achieve that using an NMC card and PowerChute Network Shutdown (PCNS). You configure PCNS to shutdown the computer when the UPS has been on battery for X minutes, and then uncheck the box to turn off the UPS after shutdown finishes.

apcupsd should also be able to do this. They call turning off the UPS after OS shutdown "killpower" and it is optional.

The reason why we typically want to turn off the UPS after the OS shuts down is to handle the case where we've shut down the OS however the power returns before the UPS' battery is depleted. The attached computer will stay off unless the UPS cycles the power off and then back on. So ideally you'd want to get the OS shutdown under ten minutes and then have the UPS turn off after the OS shuts down.

edit I should also mention two other reasons: it conserves the battery for the next power failure and it lengthens the lifespan of the battery.

Sometimes you want to turn off a non-essential device like a power-hogging server while keeping essential equipment running. Our answer to that is going to be putting them on separate outlet groups.

>
Is there anything new to this issue?
>

Engineering made changes to the UPS firmware to perform longer off delays, however I expect it'll be a while before those change makes it into manufacturing. This only helps if you can't get the OS to shutdown fast enough for the existing 10m30s delay, and isn't the most desirable option because a momentary power surge could leave you waiting a while for the server to shut down and turn back on.

[BillP]

voidstar wrote:
The reason why we typically want to turn off the UPS after the OS shuts down is to handle the case where we've shut down the OS however the power returns before the UPS' battery is depleted. The attached computer will stay off unless the UPS cycles the power off and then back on. So ideally you'd want to get the OS shutdown under ten minutes and then have the UPS turn off after the OS shuts down.

edit I should also mention two other reasons: it conserves the battery for the next power failure and it lengthens the lifespan of the battery.

I am aware that there are some advantages of not draining the battery and auto-restart the server. However, the main concern for us is a save shutdown in the event of power loss. It's ok for the customer to wait until I switch it back on for them.

I talked to the APC support and was very disappointed. They don't understand the issue and they seem to not know exactly what the entries under "Shutdown Management" mean. 😞 This is sad.

I will test a workarround: Running a command script with "shutdown -f -s -t 60" an not ticking the "Shutdown" box at all. This should work arround the issue. I will post here if it is working.

Message was edited by: djar

[BillP · Accepted solution]

I tested the "shutdown -s -f -t 60" command file workarround at the SBS 2008 server from a nice customer that let me reboot the server several times during the day. I even got coffee. 🙂

The good thing:
It work's out of the box. There is a sample command file named default.cmd in the APC software folder. Adding "shutdown -s -f -t 60" and setting the command file to run will shutdown the server even if no user is logged on. There will be no "power cut-off", the UPS will not be contacted to "shutdown and OFF" simply because the APC service will be down seconds after the command file runs.

APC support suggested two other things:

- Buy network card for UPS
- Use APC UPS daemon, an open source tool

The bad thing:
Buying a bigger UPS will not help. Support confirmed that this issue is not only affecting the small ones. They told me that this is a software limitation and not hardware related, so switching the unit will not help. I'am not sure if this is correct. Maybe someone using the APC UPS daemon can clarify if it is possible to set a higher value there.

What bothers me more is that it took me several calls to APC until they admit that there might be an issue. They made me change several timing settings that had nothing to do with this problem. I even got the dangerous tip to set the "OS shutdown delay" to the lowest possible setting and increase another value. This almost crashed the server. I would consider the shutdown settings somewhat basic and would expect the support to know what they mean.

Anyway, if there is something new to this, i would like to hear it. Maybe i'am missing somthing.