Help with PCNS and NMC scheduling on Outlet Group

[Anonymous user]

Ive tried really hard to understand PCNS/NMC shutdown / restart scheduling from the manuals; Real world testing is difficult as all our systems are 'in production', so trial and error is not very possible. Id welcome some advice on configuring PCNS / NMC.

Our Environment

• Smart-UPS SRT5000 with NMC and 2 Outlet Groups with Bypass Switch and switched PDUs, All load on outlet group 1, Runtime reported under full load = 60 minutes.
• Powering Physical Servers (ILO), ESXi VSAN Cluster (IDRAC), Switches
• PCNSInstalled on 5 physical servers, with 1 physical server 'VISAYAN' nominated as 'master' to execute shutdown scripts (send tweets, alerts users, main ESXi shutdown script) and startup scripts.

Design Goals

Maintain uptime in the event of power failure, time to alert ICT and Users to save and shutdown, Protect Hosts VSAN cluster through controlled shutdown when needed.

Restart UPS/Power only when there is enough power to ensure safe shutdown, should power fail again.  On restart run service resumptions scripts (Eg Tweets, ESXi Startup)

Notifications to ICT - Email/Twitter/Yammer after 60 seconds of power loss and power resumption

Notifications to Users - Email, Twitter/Yammer, Citrix of impending service loss and service resumption.

All the scripts  are working...it is just the scheduling and UPS and outlet groups power on /off times that im not confident about..

Configured so far

VISAYAN - event 'UPS on battery' - Command File 'delay 10s' 'script to notify ICT'

Intention - Inform ICT when power has been lost for more than instant.

VISAYAN - event 'Runtime remaining below threshold' - Command File 'Threshold 30 minutes' 'script to notify users'

Intention - Inform Users that service is going down, (shutdown needs to start now, so everything has enough time to shutdown)

VISAYAN - event 'Runtime remaining below threshold' - Shutdown 'Threshold 40 Minutes'

Intention - ?The setting which tells PCNS to start what is configured in shutdown settings?

VISAYAN - Shutdown Settings - Turn off Ups Yes

VISAYAN - Shutdown Settings - Run Command 'script to shutdown esxi' 'Duration 10 minutes'

Intention - ESXi is shutdown (Script takes <=10 Minutes), Then Visayan is shutdown, then UPS Outlet group and UPS is shutdown

NMC - Low battery duration - 30 Minutes

Intention - when there is 30 minutes of power left, shutdown needs to start....

?Complete unknowns.?

What event can i use to distinguish between when

• Power is resumed before a shutdown has had to start (then just inform IT, no need to do anything else.)
• Power has resumed after a shutdown and a power up (then also run service resumption scripts, eg, restart ESXi / VSAN).

How do i configure the UPS and OUtlet group to only restart when there is '40 minutes of battery charged', and not before?

Im really hoping someone can help clear the fog on this...the manual makes no sense to me.

thank you

Nick

[BillP]

Hi,

PowerChute network Shutdown does not send e-mail alerts however you can script it to do so. Another option is with PCNS version 4.2 you can have it send SNMP alerts. See the User's Guide starting on page 18 for SNMP assistance.

The Network Card will send e-mails based on events and the card supports SNMP. 

On 1/27/2017 4:51 AM, nick said:

Power is resumed before a shutdown has had to start (then just inform IT, no need to do anything else.)

You could have configure PCNS to send SNMP notification for Input Power Restored or again have the NMC provide the information. 

On 1/27/2017 4:51 AM, nick said:

Power has resumed after a shutdown and a power up (then also run service resumption scripts, eg, restart ESXi / VSAN).

For power has resumed after a shutdown that would have to come from the NMC. Once PCNS starts the shutdown process it should be completed. If you configure PCNS properly it will send a power down signal to the NMC which will power down the UPS. If the UPS is configured properly it will power down wait what ever delay you have set and then power back up. If the server/s BIOS have been configured properly the server will see that AC has been restored and will start up.

On 1/27/2017 4:51 AM, nick said:

How do i configure the UPS and OUtlet group to only restart when there is '40 minutes of battery charged', and not before?

 That is done via the NMC under the outlet group settings. You can configure a shutdown delay, and power on delay. 

[Anonymous user]

Thanks Bill

On 1/27/2017 9:08 PM, Bill said:

PowerChute network Shutdown does not send e-mail alerts however you can script it to do so

To clarify, my scripts to eg. send email alerts are working well...but i am requesting help on the configuration of PCNS and NMC to action those scripts at the right time to make a functional shutdown and startup solution.

On 1/27/2017 9:08 PM, Bill said:

For power has resumed after a shutdown that would have to come from the NMC. Once PCNS starts the shutdown process it should be completed. If you configure PCNS properly it will send a power down signal to the NMC which will power down the UPS. If the UPS is configured properly it will power down wait what ever delay you have set and then power back up. If the server/s BIOS have been configured properly the server will see that AC has been restored and will start up.

I have the power recovery settings to ON not 'LAST': reason after graceful shutdown the last state is OFF...so I think this should work.

So as per my other post, i will configure VISYAN to run a startup script to run after a successfull UPS shutdown to 'unwind' all the shutdown commands and put ESXi/VSAN back into service.

On 1/27/2017 9:08 PM, Bill said:

 That is done via the NMC under the outlet group settings. You can configure a shutdown delay, and power on delay. 

I tried to configure the Outlet group power off delay to ZERO, but it auto reset to 720 seconds...I think the Outlet Group power off delay takes into account the PCNS shutdown command file duration + some padding? which is fine i just need to understand that.

I set the outlet group 'Min Return Runtime' to 40 minutes...Intention - dont startup the servers until there is 40 minutes of runtime (enough to gracefully shutdown in the event of a further power loss)..I do wonder whether NMC is clever enough to remember what the load was (40 minutes of current runtime with 0 load is alot different to 40 minutes runtime once all the servers are running) ..because of this doubt I have also set Power on Delay to X seconds...to ensure that the UPS has charged enough to provide a real 40 minutes of runtime under full load.

i set Shutdown Threshold on PCNS on VISAYAN to 20 Minutes and the (shutdown) command file duration to 10 minutes..am i right to expect/understand?

• With 20 minutes of runtime remaining..my shutdown script will execute/start..and the power off delay timer of outlet group will start.
• My shutdown script will finish with roughly 10minutes of runtime remaining, VISAYAN will shutdown,
• When the outlet group power off delay timer reaches 720 seconds the outlet group and UPS will switch off.
• When power returns to UPS, the power on delay timer of outlet group will start (no power delivered to hosts connected to outlet group yet)
• Subject to 'Min Runtime remaining/Power on Delay' timer on outletgroup, power will return to hosts.

I have set my low battery duration to 1 minute. my understanding is i dont ever want to hit this point..everything would have shutdown long before this..so it would never come into effect?

thanks

Nick

[BillP]

Hi,

On 1/29/2017 9:53 AM, nick said:

I tried to configure the Outlet group power off delay to ZERO, but it auto reset to 720 seconds...I think the Outlet Group power off delay takes into account the PCNS shutdown command file duration + some padding? which is fine i just need to understand that.

The NMC does take into account the delays set in PCNS and add that time to the outlet group shut down delay. 

On 1/29/2017 9:53 AM, nick said:

I set the outlet group 'Min Return Runtime' to 40 minutes...Intention - dont startup the servers until there is 40 minutes of runtime (enough to gracefully shutdown in the event of a further power loss)..I do wonder whether NMC is clever enough to remember what the load was (40 minutes of current runtime with 0 load is alot different to 40 minutes runtime once all the servers are running) ..because of this doubt I have also set Power on Delay to X seconds...to ensure that the UPS has charged enough to provide a real 40 minutes of runtime under full load.

The NMC records the load when the shut down command is given and will base the needed battery capacity based on that recorded number. 

On 1/29/2017 9:53 AM, nick said:

With 20 minutes of runtime remaining..my shutdown script will execute/start..and the power off delay timer of outlet group will start.

Yes. Here is a diagram of the process assuming PCNS has not been install to communicate with VMware and shutdown the VMs/vApps.

Here is a diagram of the process when PCNS is communicating with VMware and shutting down VMs/vApps.

On 1/29/2017 9:53 AM, nick said:

• When the outlet group power off delay timer reaches 720 seconds the outlet group and UPS will switch off.
• When power returns to UPS, the power on delay timer of outlet group will start (no power delivered to hosts connected to outlet group yet)
• Subject to 'Min Runtime remaining/Power on Delay' timer on outletgroup, power will return to hosts.

Correct on all 3 points. 

[Anonymous user]

Thanks Bill, its starting to make more sense.

with no confidence/being new to the subject matter, the manual (images) are easy to dismiss as 'not really understood ' (partly because its not obvious to the leyman why the solution has to be designed and documented in such a complex fashion) ...once confidence grows I see the manual (images) as 'of course, thats what they are saying all along'.

but before i get too confident...are you able to elaborate / help understanding on the purpose of 'low battery duration' in a PCNS/NMC shutdown design..does it sound right to you that i am 'tucking it out of the way' at 1 minute.

On 1/29/2017 2:53 PM, nick said:

I have set my low battery duration to 1 minute. my understanding is i dont ever want to hit this point..everything would have shutdown long before this..so it would never come into effect?

Ive been led to be believe that LBD is some sort of 'backstop', that initiates a shutdown of PCNS Hosts and Outlet groups regardless of any PCNS event configuration or NMC configuration. what would be the result of setting LBD to 1 minute?

[BillP]

Hi,

The purpose of the low battery duration is to alarm when the UPS has reached a runtime level that is the minimum needed to sustain the load that will allow the load to power down safely. The allowable low battery duration settings are 1 - 30 minutes. When the UPS reaches low battery level the NMC sends a signal to all PCNS clients command them to start OS shutdown. When that shutdown is started PCNS will not honor any delays such as run this script first and wait 8 minutes before start OS shutdown. The OS shutdown will begin as soon as the command is received. 

The low battery duration should be 2 minutes greater that amount of time it takes for all systems to power down safely. As an example there are 5 servers attached to UPS, they all require 10 minutes to power down. 1 of the systems has PCNS configured to run a script that requires an 8 minute delay. The UPS has 40 minutes of available runtime and PCNS has been configured to start a shutdown after the UPS has been on battery for 10 minutes. the The low battery duration should be set to 20 minutes. Once the UPS has been on battery for 10 minutes all 5 servers will begin the shutdown process. The server that needs to run the script will kick off the script while the others begin to power down. Once the 8 minute delay has completed the final server will begin the shutdown process allowing the system to power down safely. 

NOTE: In the scenario above I would set the low battery duration to 22 minutes because my preference is to increase the needed shutdown time to be 4 minutes greater not 2 but it is not necessary. That is just my preference. 

[Anonymous user]

thanks bill,

I assume the logic in your previous post can be transposed into my design. I am using 'Runtime Remianing Threshold' rather than 'Time on Battery Duration'

the bit that is confusing me is that in your example sequence of events....

the actual timings of when things shutdown and scripts execute are ALL driven by PCNS configuration.....(time on battery in your sequence (or runtime remaining in my case), command file duration)...it seems that the low battery duration  is not having a tangible effect in your sequence of events ....

 or to put the question another way...

On 1/31/2017 1:33 PM, Bill said:

When the UPS reaches low battery level the NMC sends a signal to all PCNS clients command them to start OS shutdown

My understanding is that this 'signal' is entirely separate from the timers created by PCNS event configuration "Runtime remaining below threshold" and "ups on battery delay"..my understanding is that these timers are not based on Low battery Duration timer...rather 'time on battery' and 'runtime remaining'. Maybe this is where the nub of my confusion is...

if LBD is indeed a completely separate timer...and LBD initiates host shutdown without honoring any scripts configured in PCNS (if i have understood correctly)....then i definitely would never want LBD to come into effect...I need my scripts to run to ensure graceful shutdown...

 i think im almost there in my understanding and this is the last piece of the puzzle 🙂

thanks again for your help

Nick

[BillP]

Hi,

On 1/31/2017 9:38 AM, nick said:

the actual timings of when things shutdown and scripts execute are ALL driven by PCNS configuration.....(time on battery in your sequence (or runtime remaining in my case), command file duration)...it seems that the low battery duration  is not having a tangible effect in your sequence of events ....

In my example the effect is the NMC reported the UPS is on battery and it was not cleared in 10 minutes so PCNS starts the shutdown process. In your setting the NMC reports the UPS is on battery and the runtime has fallen below X so PCNS starts the shutdown process. Low battery does not affect either of these scenarios. 

On 1/31/2017 9:38 AM, nick said:

if LBD is indeed a completely separate timer...and LBD initiates host shutdown without honoring any scripts configured in PCNS (if i have understood correctly)....then i definitely would never want LBD to come into effect...I need my scripts to run to ensure graceful shutdown...

Correct. If the UPS hits the low battery threshold and a shutdown command has not already been executed by PCNS then PCNS will execute the command but it will skip any delays and begin the OS shut down immediately. That is why I said to be sure Low Battery Threshold is a minimum of 2 minutes greater that the time needed to complete the shut down process. If the script needs 8 minutes and the OS needs 10 minutes low battery should be set to 20 (I prefer 22). If using Runtime Remaining Below Threshold it should be set to 30 minutes. That will give the script 8 minutes to run and the OS shutdown to have began 2 minutes prior to Low Battery. 

[Anonymous user]

Apologies i am still not getting it..:-(

I know the answer,

the LBD should be set to longest of any host (os shutdown + any shutdown script + 2/4 minutes) ,

but i still dont know LBD relevance and how it comes into play in the shutdown sequence of events...

My current understanding (which i know must be flawed)

• example LBD is 22 minutes, OS shutdown takes 10 minutes and Shutdown Script take 8 minutes
• Low battery duration (LBD) represents a period of time leading up to UPS switch off and a power off event..the last 22 minutes..
• LBD represents the minimum shutdown period the slowest shutdown in the environment requires (not sure sure why/the purpose).
• LBD is an entirely separate timer to 'On Battery Delay' and 'Runtime Remaining Below Threshold' (Events on PCNS that are driven by the NMC communicating ' time on battery' and 'runtime remaining' ; PCNS timers are 100% decoupled from and not driven by 'low battery duration timer'.
• When LBD is hit (22 minutes before power off event/22 minutes runtime remaining), all PCNS clients start os shutdown and execute PCNS shutdown scripts (command file event scripts are ignored by an LBD event), but OS shutdown does not wait for shutdown script timer to complete before OS shutdown. If the script has not finished prior to os shutdown 'too bad'.
• The admin can solely rely on PCNS event timers to shutdown the environment properly, LBD timer / event is not required..
• LBD event would only come into effect if the admin configured the LBD to be a large enough value to come before/interfere with event timers configured in PCNS (which do properly respect script duration timers). (But this is exactly what a value of 22 minutes would do)
• Indeed if the goal is to provide as much runtime as possible, then the admin might well configure LBD to be 1 minute (its minimum value) to give PCNS events the chance to complete properly without interference from an LBD event.*

* this last statement can not be correct; it is at odds with your recommendation  but it is the conclusion of my understanding...so something is still not understood.

sorry for my stupidity

Nick

[BillP]

Hi,

On 2/1/2017 4:30 AM, nick said:

the LBD should be set to longest of any host (os shutdown + any shutdown script + 2/4 minutes) ,

but i still dont know LBD relevance and how it comes into play in the shutdown sequence of events...

The low battery duration has no affect on ant of the events that are configured in PCNS. Low battery is an un-configurable event in PCNS. When the NMC reports low battery any PCNS clients will begin the shutdown process. As an example no shutdown events have been configured in PCNS. The UPS switches to battery and the NMC reports the event. PCNS does nothing since no shut down event have been configured however when the NMC reports that the UPS has reached low battery PCNS begins a shut down. 

This is a log of a shutdown test I ran this morning. 

02/03/2017 09:05:59 UPS has switched to battery power. .3.5.1.5.4.1
02/03/2017 09:18:43 Low-battery condition has occurred. .3.5.1.5.3.3
02/03/2017 09:18:43 UPS critical event: Low Battery. .3.4.9.9
02/03/2017 09:18:43 Shutdown process started. Windows will shut down soon. .3.5.1.4.3
02/03/2017 09:18:43 PowerChute is turning off UPS 10.0.0.8 .3.4.9.9
02/03/2017 09:18:43 Running Shutdown Command File. .3.5.1.4.16
02/03/2017 09:18:49 UPS Outlet Group: Switched Outlet turn off has been initiated. .3.5.1.4.8
02/03/2017 09:18:49 UPS critical event: Outlet Group Turn off initiated. .3.4.9.9
02/03/2017 09:23:43 Shutdown Command File has completed. .3.5.1.4.14
02/03/2017 09:23:43 Shutting down physical machine that PowerChute is running on. .3.4.9.9

I had configured PCNS to run a command file with a 5 minute delay. The low battery setting was configured for 30 minutes so once that threshold had been breached PCNS started the shutdown. I had also configured PCNS to send a shutdown signal to the UPS. 

A correction I need to make is I had written earlier that once low battery was reached and PCNS began the shutdown process the command file delay would not be honored. I was mistaken as you can see in the log the delay was honored and the system was able to power down safely because I set the low battery threshold to a number that was greater than the time needed to run the command file and power down the system. When I wrote that I was thinking about multiple critical alerts when PCNS is configured with redundant UPS with is not the proper scenario. 

Sorry for the confusion and I hope this explains LBD. 

[Anonymous user]

hi Bill thanks for the feedback and perseverance,

Im still not clear about the part LBD plays in my PCNS driven shutdown workflow

Im hearing two conflicting messages

Message 1 - LBD has nothing necessarily to do with a PCNS event driven shutdown (and so not required).

On 2/3/2017 3:11 PM, Bill said:

low battery duration has no affect on ant of the events that are configured in PCNS

Message 2 - LBD should be set at a high enough value (os shutdown + any shutdown script + 2/4 minutes) (where it will interfere with a PCNS event driven shutdown).

 eg.

OS Shutdown takes 10 minutes

Shutdown Script duration is 8 minutes

PCNS Shutdown Event 'Runtime remaining below 20 minutes' enabled

Low battery Duration = 10+8+4=22minutes

My Intention

PCNS intention= start shutdown with 20 minutes power remaining.

Actual Result

LBD 'interferes' and starts shutdown with 22 minutes power remaining.

The 2 minutes difference isnt the problem...its understanding why I need LBD in my workflow..if its going to interfere with my PCNS driven shutdown ....why would i willingly have this point of confusion in my shutdown workflow?

(I am half expecting for you to correct my understanding around message 1 and tell me that LBD is in fact CRUCIAL to a PCNS driven shutdown workflow..and i just havent understood how yet)

thanks

Nick

[voidstar_apc]

Hi Nick,

The way I think about it is there are multiple ways to trigger a UPS shutdown: PCNS conditions, NMC low battery, NMC load shedding (UPSes with switched outlet groups), NMC controlled early shutdown (UPSes without switched outlet groups), and simple signaling.

All that matters is which one triggers first. So if you set up PCNS to shut down when runtime remaining is less than X and low battery setting is Y and X>Y, then only the PCNS setting really matters.

Then why have low battery at all? At times it's useful, for example:

- If the PCNS machines are already off, such as for maintenance. In that case PCNS will not turn off the UPS -- that will be the job of low battery.

- If PCNS is set to turn off the UPS after X seconds on battery. Many people prefer to trigger off time on battery, but this can be unsafe when the battery ages and and eventually cannot support the load long enough for safe shutdown + X seconds. Low battery can trigger the shutdown before X seconds.

- Some people prefer to do their configuration on the NMC and leave PCNS unconfigured. Low battery can be used in place of "runtime remaining below threshold" on PCNS. Of course in this case I think using load shedding settings makes more sense (be sure to disable skip outlet delays).

[Anonymous user]

That helps clear a bit more of the fog..

On 2/3/2017 11:28 PM, voidstar said:

All that matters is which one triggers first. So if you set up PCNS to shut down when runtime remaining is less than X and low battery setting is Y and X>Y, then only the PCNS setting really matters.

So PCNS is clever enough to not let LBD break the shutdown workflow...Whichever  event  initiates shutdown sequence first the other diminishes and does not interfere...this make sense.

On 2/3/2017 11:28 PM, voidstar said:

- If the PCNS machines are already off, such as for maintenance. In that case PCNS will not turn off the UPS -- that will be the job of low battery.

makes sense to me

I will make a shutdown down workflow using these lessons learnt, thank you Bill and Voidstar for your explanations, im much more confident that i can design the event workflow properly now and theres a good chance my first test will be a successful test conforming to a predicted outcome.

[Anonymous user]

"Skip Outlet Off Delay" was checked by default on NMC Outlet Group...

Help file says....If you use PowerChute Network Shutdown (PCNS) on this outlet, be sure "Skip outlet off delay" is not checked so that PCNS clients can safely shutdown. Also, each PCNS client has its own set of conditions that may be configured to start a shutdown...

 I have tried to uncheck "Skip Outlet Off Delay"..but it goes back on...

any ideas please?

[voidstar_apc]

Hi Nick,

"Skip outlet off delay" in the load shed section only applies to a shutdown triggered by load shedding. My guess is that none of the load shed triggers are enabled on the outlet group, therefore the "skip outlet off delay" setting is irrelevant and has no effect. For whatever reason (and I've personally reported this behavior as a bug), in the case where it has no effect, the UPS always resets that setting to the default value of "true".

[Anonymous user · Accepted solution]

Thanks Voidstar,

If not a bug, its a good example of how accessibility of the design is quite challenging to a new comer..

theres even a bigger gap between the previous option and this one in the interface, implying the option is not related to the load shedding section (imo)