Interaction between Smart-UPS 3000 NMC2 & PCNS for Hyper-V Servers

[Anonymous user]

Hi everyone,

Firstly, grab a coffee - this is a long post!

I've been reading posts in this forum here, as well as  blog posts, official documentation etc. and I am somewhat perplexed.

References I have used: 

• http://www.apc.com/us/en/faqs/FA156160/
• https://www.stevejenkins.com/blog/2013/07/howto-configure-low-battery-duration-and-pcns-to-shut-down...
• APC, Spiceworks forums, etc.

I'll state our requirements here, and then offer screenshots  of settings.

We have a APC Smart UPS 3000 (SUA3000I) w/ Integrated NMC AP9630. I upgraded the firmware for this in Feb 2017.

We would like the simple approach for the UPS to shutdown both physical hosts, gracefully shutting down HyperV-VMs  in the respective servers. thereafter, the UPS is to remain powered-off and wait for manual intervention to turn it back on.

I understand the requirements for Hyper-V (integrated services installed: "Operating System shutdown' ticked, etc.)

Where my problem lies is in the operation of the UPS and how it communicated with the PCNS running on both servers (2012 R2 Datacenter), and understanding  the interaction between both.

We performed a calibration on the UPS last night, and there appears to be ~27mins of runtime remaining with a current load of just under 30% (we calibrated with a load of 30.5%)

What we would like: after approx. 15mins of a power outage, i.e. the UPS is supplying battery power for approx. 15mins, that we initiate a shutdown of the two physical servers, both hosting Hyper-V VMs, and have the UPS turn off completely and wait for a manual turn-on.

The main Hyper-V  server would have the longest shutdown time requirement, and I would like to let it have approx. 8-10mins  to shut down - for the VMs  it's hosting and the host OS itself.

The other server, a VEEAM server running Hyper-V  also, and this should should take between 3-5mins.

So here are the UPS settings at the moment:

As you can see, I have chosen the options to keep the UPS powered-off when it is indeed off, but left the other fields at default values.

So I am making an educated guess here in saying that I should make the 'Low Battery Duration' 14mins, meaning that it would have used approx. 13mins of run time  before hitting the threshold + 90 sec shutdown delay, and giving us approx. 14mins remaining before the battery is exhausted and during this time the shutdowns of both physical servers and their respective VMs  must occur.

I am confused as to what happens at these points:

• When mains power is lost, does the UPS issue an "on battery" alert or "shutdown" command immediately to each of the servers running the PCNS  agents?
• Or, when the "Low Battery Duration' of 14mins  is met, i.e. the UPS is on battery power until the runtime remaining is 14mins, does the UPS issue an "on battery" alert or "shutdown" command to each of the servers running the PCNS  agents?
• (Just need to get this right in my head!)

Later, I will need some help understanding the "PowerChute  Shutdown Parameters:" Maximum Required Delay - 8mins / Force Negotiation" setting, but firstly, I'd like to review PowerChute Network Shutdown settings:

So, on our main Hyper-V  server, I would like to give it the aforementioned 8-10mins  to both shutdown the VMs  and then shut down the host OS. It may not take this long (we need to test), but this is an educated guess from my interactions  with the server.

In the PCNS  web console on the main Hyper-V  server, I have the following settings:

Configure Events:

Event Name: "UPS On Battery"

Enable Shutdown: unticked -

• Do I need to tick this for both physical servers?
• If I tick this and leave it at 120 second default, does the 120 seconds start the moment the UPS loses mains power, or at the point where the 'Low Battery Duration' gets to 14mins  remaining on the UPS?
• Is this delay a delay before issuing a "shutdown" to the physical server/VMs/both?
• What happens at the end of the 120 seconds? Should this be increased to more than the value of seconds in the VM Shutdown Settings (see below)?

I am just a little confused as to when the UPS signals the PCNS  to initiate the OS shutdown.

Shutdown Settings:

I am not allowing PCNS  to shutdown the UPS:

Virtualisation Settings:

Setting for 8mins, but  also preventing PCNS  from starting them up:

There is no mention of the individual VMs here, so am I right in assuming that PCNS  will issue a shutdown command to all the VMs  on the server firstly, thereafter issue a shutdown command to the physical host when all of the VMs  have been shut down?

Or is it that the command is issued to the physical host, and the VMs  are shutdown because of the OS being shut down, and this setting is only a delay to allow for all the VMs  to shut down before the host OS is instructed to shut down?

For both the Hyper-V and Veeam servers, I have the following settings (8mins and 5mins  respectively):

Hopefully, my timings make sense here!

Getting back to:

PowerChute Shutdown Parameters

Maximum Required Delay

8 minutes

Force negotiation

... do I need to perform a force negotiation as per documentation here: http://www.apc.com/us/en/faqs/FA156160/ ?

When I get all of these thoughts in my head cleared up, I'd like to draw out a diagram similar to the one offered in the "What does the Force Negotiation setting achieve in the Network Management Card?" documentation, but have it with my servers' timings:

This diagram confuses me somewhat, as the Server Shutdown: "On Battery" event seems to start before the "UPS Turn Off Starts" - I thought that the UPS would immediately switch to battery power when mains power is lost, then  after the Low Battery Duration is met, that it signals PCNS with an "On Battery Event".

I just need to understand what triggers an "On Battery Event" in PCNS's interaction with the UPS, and what triggers the "UPS Turn Off Starts", and why there is a time difference between both.

Another thing that confuses me in this diagram is the section at the start of the in the top bar, it mentions "Event Shutdown Delay" - is this the same as the PCNS setting:

•  Event Name: "UPS On Battery", Enable Shutdown: ticked, Delay = no. of seconds?

I've seen mention of this "Event Shutdown Delay" in Steve Jenkin's  post, but not in the PCNS  consoles for the two servers:

"Your answer to that question determines whether you prefer a “time since” shutdown method (which is driven by the Event Shutdown Delay setting in the PCNS client software), or a “time until” shutdown method (which is driven by the Low Battery Duration setting in the UPS’s NMC software)." 

Maybe I have everything mixed up here, but having it drawn out will help me (and my manager!) immensely.

(Wouldn't it be terrific if the settings could be determined by interacting with a dynamic version of the graphic above? I live in hope...)

[BillP]

Hi,

On 4/5/2017 2:31 PM, Stephen said:

• When mains power is lost, does the UPS issue an "on battery" alert or "shutdown" command immediately to each of the servers running the PCNS  agents?
• Or, when the "Low Battery Duration' of 14mins  is met, i.e. the UPS is on battery power until the runtime remaining is 14mins, does the UPS issue an "on battery" alert or "shutdown" command to each of the servers running the PCNS  agents?

When power is lost the NMC/UPS issues an on battery event. That event notification will be picked up by all PCNS clients configured to listen to that NMC. The shutdown would only start immediately if you configured PCNS to start a shutdown when the UPS was on battery and set the delay to zero.

When low battery is met the NMC/UPS issue a power down command to all PCNS clients that are configured to listen to that NMC/UPS.

On 4/5/2017 2:31 PM, Stephen said:

Event Name: "UPS On Battery"

Enable Shutdown: unticked -

• Do I need to tick this for both physical servers?
• If I tick this and leave it at 120 second default, does the 120 seconds start the moment the UPS loses mains power, or at the point where the 'Low Battery Duration' gets to 14mins  remaining on the UPS?
• Is this delay a delay before issuing a "shutdown" to the physical server/VMs/both?
• What happens at the end of the 120 seconds? Should this be increased to more than the value of seconds in the VM Shutdown Settings (see below)?

When you select UPS on Battery and set it to enabled you are asked to enter a delay. The default delay is 120 seconds. What will happen is when the UPS switches to battery the NMC/UPS send an on battery event notification. PCNS will start a count down and when it reaches 120 seconds PCNS will begin the shutdown process. If during that 120 seconds a cleared notifiacation is received that the count down stops. There is a screenshot of the 120 second delay in effect attached to this post http://forums.apc.com/spaces/7/ups-management-devices-powerchute-software/forums/general/14073/pcns-...

On 4/5/2017 2:31 PM, Stephen said:

There is no mention of the individual VMs here, so am I right in assuming that PCNS  will issue a shutdown command to all the VMs  on the server firstly, thereafter issue a shutdown command to the physical host when all of the VMs  have been shut down?

Or is it that the command is issued to the physical host, and the VMs  are shutdown because of the OS being shut down, and this setting is only a delay to allow for all the VMs  to shut down before the host OS is instructed to shut down?

When the shutdown process is started the VMs will be commanded down. PCNS will wait the delay you set and then command the host down.

On 4/5/2017 2:31 PM, Stephen said:

This diagram confuses me somewhat, as the Server Shutdown: "On Battery" event seems to start before the "UPS Turn Off Starts" - I thought that the UPS would immediately switch to battery power when mains power is lost, then  after the Low Battery Duration is met, that it signals PCNS with an "On Battery Event".

I just need to understand what triggers an "On Battery Event" in PCNS's interaction with the UPS, and what triggers the "UPS Turn Off Starts", and why there is a time difference between both.

Another thing that confuses me in this diagram is the section at the start of the in the top bar, it mentions "Event Shutdown Delay" - is this the same as the PCNS setting:

•  Event Name: "UPS On Battery", Enable Shutdown: ticked, Delay = no. of seconds?

I've seen mention of this "Event Shutdown Delay" in Steve Jenkin's  post, but not in the PCNS  consoles for the two servers:

As an example you configured the PCNS On Battery Event to start a shutdown after the UPS has been on battery for 120 seconds. The diagram show that the On Battery event starts a point 1. PCNS waits the 120 second delay and moves to point 2 at the same time it moves to point 3 (point 2 and 3 are simultaneous) If you have configured PCNS to run a command file / script the command file duration / delay will be honored and then PCNS will move to point 4 (OS shutdown) There is a 70 second delay and then the OS will power down.

As for points 5, 6, 7, When the UPS receives the power down command from PCNS it will wait the greater of either Low Battery Druation or Maximum Required Delay. Then wait an additional 2 minutes then the UPS shutdown delay and then cut power to the outlets. So you screenshot shows that low battery and maximum required delay are both 8 minutes and the shutdown delay is 90 seconds. The UPS when it receives the power down command from PCNS will wait a total of 11 1/2 minutes and then cut power to the outlets.

The maximum required delay is based on the amount of time PCNS requires to power down. So in you case PCNS Hyperv1 needs 8 minutes to power down the VMs and 2 minutes to power down the OS. The maximum required delay would be set to 10 minutes when you run force negotiation or it will be changed when PCNS is restarted or when you run the PCNS setup wizard.

[Anonymous user · Accepted solution]

Hi Bill,

Sorry for the long wait to reply - I was out of the country on annual leave for the past two weeks.

Following up on your response:

I have drawn-out a diagram similar to the screen shot, but for my case on one of the servers, and more clearly defined interactions between the UPS and PCNS (note: not drawn to time-scale):

What I'd like to establish is the following:

1. As in the diagram above, that when the Host OS is instructed to shut down on the Hyper-V server, that the VMs are automatically instructed to shut down and that 480 seconds (a custom value that I have inserted) will elapse along with the additional 70 seconds before the host OS is shutdown.
2. "The maximum required delay is based on the amount of time PCNS requires to power down. So in your case, PCNS Hyperv1 needs 8 minutes to power down the VMs and 2 minutes to power down the OS." - where are the 2 minutes you mention coming from, or how do I assign that value for the host OS shutdown delay (I may be missing something obvious here)?
3. As per diagram above, I assume that the MRD of 10 mins = PCNS 'Event Shutdown Delay' 120 secs + 'VM Shutdown Duration' 480 secs, but that it ignores the 70 secs additional delay?
4. If none of the PCNS agents are instructed to turn off the UPS, when does the UPS actually turn off? Is it when all PCNS agents have reported that they have turned off the OS's that they are installed on?
   • Or based on my earlier screenshot, will it turn off if/when the battery actually depletes?
   • Should I have the longest-running PCNS agent power off the UPS?
   • Is there any harm in not instructing the PCNS agents to turn off the UPS, allowing the battery to deplete (as switches and other devices will eventually suck the battery dry even with the servers powered-off) and leaving the UPS off until someone intervenes and manually turns it back on again (IT Manager here would prefer this method)?
5. The UPS has about 30 mins runtime. If I wanted to start the shutdown process after 15 mins of being 'On Battery', should I set the 'Event Shutdown Delay' on all host servers to 900 secs? Is this good practice?

Again, I really appreciate your help.

Regards,

Stephen