Shutdown configuration problem on 5000RM & Powerchute

[Anonymous user]

Setup: 5000RM with NMC, connected to 8 servers (2k3 and esx).

All clients has powerchute installed with default configuration (no changes). Default setting for "Turn off the UPS after the shutdown finishes" is ON, and we left that settings on all clients. No events are configured (all are aeft to "Log event" by default)

UPS settings:
- Low battery duration - 5 min
- Shutdown delay - 000 sec
- On battery shutdown behaviour - Restart when power is restored

Other settings as default....

When input power is off, UPS waits for "runtime remaining" becomes equal "low battery duration" or so, and start graceful shutdown of all clients. After our servers are turned off, UPS DOES NOT SHUTDOWN ITSELF. Of course, remaining time is now significantly increased, since there is almost no load, and if power breakage lasts for long enaugh to exhaust remaining time, everything is OK, because UPS will then turn off itself, and turn back again, once the power is back.

Problem is that duration of our power outages is mainly 20-30 minutes, enaugh to start shutdown sequence, but not enaugh to spent all "runtime remaining" when all servers are off. After power is back, UPS is not more on battery, but it will not restart our servers.

What is the simplest settings which will asure that UPS will shutdown itself once all servers are completed shutdown, regardless how many batteries are left.


Thanks.

[BillP]

Good day Vladimir,

What I usually recommend to our customer is to mark the "+Turn off the UPS after the shutdown finishes+" on the last computer that would turn off, or the one that takes the longest time to turn off. By doing so this would prevent the UPS for immaturely shutting off the outlets. Correct me if I'm wrong, the way I understood it is the PowerChute Network Shutdown waits until all of your UPS sends a "+Low Battery Signal+".

Can you send us the config.ini, events.txt and data.txt of your card and if possible send us a screenshot of the Event logs of your PowerChute Network Shutdown.

[BillP]

Vladimir,

Here is the guide that would help you download the files from your network management card --> [Click here for the link.|http://nam-en.apc.com/cgi-bin/nam_en.cfg/php/enduser/std_adp.php?p_faqid=9321].

Based from the screenshots that you've attached, I see that you have a return delay of 300 seconds (5 minutes) once the power is restored. Have you ever saw your UPS to enter what we call as sleep mode? You'll know that your UPS has entered sleep mode if you see the 6 center LEDs scrolling up and down.

Have you also tried it wherein only one of the PCNS clients have a mark on "+Turn Off UPS when shutdown finishes+".

[Anonymous user]

Hi,

Actually with current settings everything works fine, and all our servers have graceful shutdown, when remaining runtime decreased to 5 minutes or so. Problem is that UPS itself does not shutdown, after all servers goes off, and if power is back in that stage (which is most often the case), UPS is still on, and didn't turn off power to servers. All servers are shutdown, but has the power all the time, and when power is back, UPS just switch from battery to normal, but servers will of course not reboot, since they had power all the time. As far as I understand, UPS should turned off after some time and shut power off to all servers, after shutdown servers has been started, regardless if power comes back meanwhile. Is it default behaviour ?

Unfortunatelly I couldn't download files you mentioned from UPS FTP. Is there some way to extract files from the UPS web interface ? I am sending couple of screenshots from both UPS and clients.

Thanks, Vladimir.

[Anonymous user]

Hi again,

It seems now everything works just as it should.

I left only one client with the setting "Turn off UPS after the shutdown finishes" (one with the longest shutdown process"), and clear that setting for others. Also I decrease "Return delay" to 60 seconds.

I simulated power outage two times. First time it takes abt. 15 mins to reboot, after graceful shutdown process is initiated, second time abt. 10 mins. In both cases I turned on power back as soon as shutdown process was initiated.

First time I set "Low battery duration" to 14 min (to speed up process), and after it reboots, "runtime remaining time" was 15 min. Second time I put "Low battery duration" to 8 minutes, and after reboot "runtime remaining time" was exactly 8 min, so it seems that it will wait until it has enaugh battery to cover low battery duration before it reboots, although "Minimum battery capacity" in the "End of shutdown" section was set in both cases to 000.

I'm just curious what is logic behind various times it takes to start reboot process after shutdown is initiated. I'm sending files as requested.

Thanks you, Vladimir.

[BillP]

Vladimir,

I apologize for the delay responding to your post. I did check the files that you've sent and from what I see on the logs, your UPS functions without any problem. Let me provide you with an outline on how our UPS should work during power problems. Once the UPS detects power interruption the NMC will send a signal to all the PowerChute Network Shutdown clients, depending on the clients configuration it'll wait for a shutdown command from the NMC. PCNS clients can be configured to shutdown on low battery or shutdown when the UPS has ran on battery for a certain amount of time.

When the shutdown process on the clients starts, there is no way for us to stop it. If power is restored back to the UPS during this cycle the UPS will return online but it will restart since the PCNS shutdown cycle is still not yet done with the process. Typically, once a shutdown was initiated it adds 2 more minutes on your low battery settings for any unforseen circumstances plus the shutdown delay. It's like Low Battery Duration + 2 Minutes + Shutdown Delay, so if you set the LBD to 8 minutes the total time that your UPS needs to wait to turn off is around 10 _+ Shutdown Delay.

Also, looking at the files that you've sent it appears that your network management card is using an old firmware. I advise that you download and upgrade to the latest firmware that we have for your card. I've made an outline on how to upgrade the firmware of a network management card 1 below.

Please go to our APC download page --> [Download Page|http://www.apc.com/tools/download/index.cfm] and then on the first drop down box please choose Firmware Upgrades - Network Management Card. If done correctly you will be given a list of all the firmwares available for both the Network Management Card 1 and 2. I would like you to click on the SFSUMX358(Network Management Card v3.5.8 firmware executable for Smart UPS and Matrix w/ AP9617/8/9) download button, you might be asked to login first before you can download it so please do so. This is a self-extracting file and I suggest that you create a folder/director in which the files can be extracted.

To help you out, here is an APC Knowledge Base Article regarding firmware upgrade --> [please click here|http://nam-en.apc.com/cgi-bin/nam_en.cfg/php/enduser/std_adp.php?p_faqid=768].

Goodluck and keep us posted.

[Anonymous user]

Hi Jon,

Thank you for clear explanation, I'll try to repeat that on the other way, correct me where I'm wrong:

1. UPS is on power, battery at 100%, all clients are running, Low Battery Duration set to 8 min, and the rest (shutdown delay, min battery capacity, return delay) set to 000. Runtime remaining is at full load 19 minutes.
2. There is power outage, UPS goes to battery, and UPS note all clients about that. Since there is no event (except logging) "assigned" to that event, nothing will happen on clients at this stage.
3. When runtime remaining decrease to about Low battery duration (8 min in our case, total 11 min (19-8) after power outage), it will start process which can not be canceled from that moment on - shutdown
4. Shutdown of clients starts, and at the same time countdown for UPS reboot begins.
5. After Low battery duration (8 min) + 2 min margin (total 10 min after shutdown begins) UPS should restart itself and temporary put all clients out of power before it returns power back, which will restart all clients
6. At this stage if power outage comes again shortly, UPS will immediately turn power off to all clients, since there is not enaugh battery capacity to cover 8 min. This can be overcome by setting "Minimum battery capacity" in "End of shutdown" section ?

If this works more or less like that, then everything is clear to me, although it is not completely clear why UPS waits for so long before it rebooted in two simulations I've performed, as explained in previous post.

Also two other questions:
1. What will happen if no clients is configured with switch "Turn off the UPS after the shutdown finishes". On the other way: if UPS will reboot itself exactly after 8+2 minutes after shutdown begins, then why is needed that clients reports that their shutdown is completed ?
2. Similar question: If setting "Turn off the UPS..." is checked on server with the shortest shutdown time, does that make any change in the time UPS will reboot itself.


By the way, I have upgraded firmware, as you suggested. Do I need to update PCNS clients as well ?


Best regards, Vladimir.

[BillP · Accepted solution]

Vlad,

Once the UPS goes on battery it sends a signal to the PCNS clients which will then check how it is configured. It can be configured to shutdown on Low Battery or when the UPS has been on battery for a certain amount of time. Based from what you have explained your setup waits until the UPS sends a Low Battery State, depending on the Low Battery Duration.

On this kind of setup once the UPS is on Low Battery State it sets the Shutdown Timer equal to the Low Battery Duration and then sends a shutdown signal to all the connected clients. Once the Shutdown Timer or the Remaining Runtime is equal to 0 it then checks the network management card for the On-Battery Shutdown Behavior settings. At the end of the process, the UPS will turn off when the time specified as "Shutdown Delay" expires.

Here are my replies for your questions:
1. It really depends on how is the PCNS and NMC configured to shutdown the clients. This will actually send a signal to the UPS that it's time for it to start the shutdown timer. If this is not marked, the UPS will just exhaust all of it's available runtime. This is normally marked if you opted to choose to shutdown after the UPS has been on battery for a certain amount of time.

2. If you put mark on this option on the computer that has the shortest shutdown time, it can cause your setup to drops its load or experience an ungraceful shutdown. Just like what I've said above, the PC will notify the UPS to start the shutdown process. During that process what if another computer is in the middle of the process and then UPS reached the timer then it'll turn off the outlets causing the other computer to turn off unexpectedly.

I've also attached a screenshot that will explain the shutdown timeline, this is also available via the Help Menu of PCNS.