New enivronment: Best practices re: PCNS, VMWare vCenter

[Anonymous user]

I'm finally ready to engage in a long-overdue task getting our APC/PCNS house in-order. This is a sprawling environment and it's not quite something I've setup with PCNS before. What's more, we have a lot of flexibility to come up with other hardware if we need to.

If the best way to help plan this is to call into support, please let me know.

Hence, my question to everyone here.

Here's the environment:

2 sites:
  Site 1: 2 ESXi 5.5 setup with failover/vMotion
  Site 2: 1 ESXi 5.5 setup
  Sites are connected via IPSec VPN. Tunnel is "always" up.*

vCenter is running _as a VM_ on one of the ESXi hosts in Site 1. vCenter manages both sites.

SAN:
  Dell/Equalogic. Holds the VMs at Site 1. The Site 2 storage is local.

QNAP (4x):
  SAN for security camera DVR hardware. Attached to ESXi via iSCSI. DVRs are VMs at Site 1 and write data to the QNAP.

AppAssure BDR:
  Dell box running AppAssure software. AppAssure backs up all VMs via Agents installed directly on VMs. Data is replicated offsite to eFolder.

Goals:

• Have PCNS manage shut down of both sites.
• Have PCNS gracefully shut down the QNAP, the SAN, and the BDR.
• Map all possible breakdowns of process and construct resolutions. 

* While the tunnel is connected via a leased line, the traffic still goes over the Internet at some point and it's possible that in, say, a weather related emergency that produces an extended power outage, that the tunnel will go down. This would render PCNS ineffective if it's running from the other side of the tunnel.

We have some flexibility here. For instance, I've had some issues when PCNS and vCenter are both VMs. We could, for example, setup a new physical box to host vCenter and PCNS if necessary.

I'm doing various reading surrounding all this, of course, and I have some other experience. I'd like to see what others have done and thing are some good approaches to this project. The sky could be the limit.

If anyone needs specific models of the APC equipment we have on-site, I can provide that as well.

Cheers,

Mike...

[BillP]

Mike,

Application Note 180 and the PCNS - VMware User's Guide should answer most of your questions. A similar configuration to what you described is listed on page 11 of the Application Note. 

On 12/14/2015 9:53 AM, Mike said:

Have PCNS gracefully shut down the QNAP, the SAN, and the BDR.

To manage the shutdown of systems PCNS cannot be installed on you will write a script or scripts that PCNS will kick off that will command the other systems down. So to power down a SAN you would write a script that will tell the SAN to go down. 

[Anonymous user]

Thanks, Bill. Always great to hear from you.

I will read those documents. I have in the past, but not when I was planning something like this in mind. I'll re-read them with this new project in mind and come back with specific questions if I have any.

Cheers,

m

[Anonymous user]

I am finally winding my way through the documentation. I've started a sketch of what I hope to accomplish.

I called into support today to get a few clarifying questions answered did okay. There are two questions in which the person I spoke to sounded a little uncertain and I'd like to double-check here.

Question 1: A Dell Equalogic SAN publishes the Datastores for our ESX hosts. Does this immediately stop our ability from running PCNS as a VM? I believe so.

Question 2: The most immediate physical candidate to run PCNS is a box we use for backups. However, PCNS will need to run a script to gracefully shut down some services that run on the box. Is it possible to run PCNS on a physical box to manage a vCenter Cluster and, on that very same physical box, send scripts to shut down services on the box on which PCNS runs? The tech said I'd have to "install [PCNS] twice" to the same box and... I don't understand that answer.

Cheers,

m

[BillP]

Mike,

On 1/6/2016 1:08 PM, Mike said:

Question 1: A Dell Equalogic SAN publishes the Datastores for our ESX hosts. Does this immediately stop our ability from running PCNS as a VM? I believe so.

Since PCNS needs to run a command file to stop the SAN and that SAN will be hosting PCNS if running as a VM you should follow the configuartion on page 3 of Application Note 180 and have PCNS running on a standalone PC. 

On 1/6/2016 1:08 PM, Mike said:

Question 2: The most immediate physical candidate to run PCNS is a box we use for backups. However, PCNS will need to run a script to gracefully shut down some services that run on the box. Is it possible to run PCNS on a physical box to manage a vCenter Cluster and, on that very same physical box, send scripts to shut down services on the box on which PCNS runs? The tech said I'd have to "install [PCNS] twice" to the same box and... I don't understand that answer.

I apologize for the confusion but the rep you spoke with is mistaken. With the latest versions of PCNS you would install a single instance of PCNS and script it to shutdown the SAN and any other apps you need powered down prior to OS shutdown. With older versions of PCNS you may have installed multiple instances but not any longer. If the standalone PC is powered by the same UPS as the ESXi hosts and SAN then the configuration on page 3 of  Application Note 180 will be correct. If the standalone PC is powered by a separate UPS than you  will have a similar configuration to the one discussed on page 6 - 10 of Application Note 180.  

[Anonymous user]

Thanks again for the answer. I'm speccing a couple of micro systems for the sole purpose of running PCNS and controlling shutdowns.

I mentioned in my OP that I have a Dell Equalogic SAN and a QNAP NAS that I hope to shut down with PCNS. The SAN is very straightforward. I have a document from Dell that explains how to do it.

I've been puzzling over the QNAP though. I see that it's supposed to be able to receiving SNMP traps that tell it to shut down, however I see all kinds of wishy washy answers on QNAPs forums about what really works. PCNS is mentioned as sometimes working as does apcupsd and something else called NUT.

Do you have any knowledge of trying to gracefully shut down a QNAP from PCNS? Can PCNS send SNMP traps?

I apologize if these answers are in front of me. I am really trying to gather information from a lot of sources and am getting lost in the details.

[BillP]

Hi,

On 1/7/2016 6:34 PM, Mike said:

Do you have any knowledge of trying to gracefully shut down a QNAP from PCNS? Can PCNS send SNMP traps?

I do not have any experience with QNAP and this version of PCNS does not offer SNMP. The next version will and I expect it will be release sometime in September of this year. The NMCs offer SNMP so you could configure the NMC to sent a trap to the QNAP when it is communicating with PCNS. 

[Anonymous user]

Thanks again. I'll see what I can turn up regarding the NMC and SMNP. The QNAP side doesn't have much in the way of support. The good news is that I can send the trap which would kick off a timer...

[Anonymous user]

Bill,

I've set up one of our UPS to send SNMP traps to a QNAP. However, I can't tell on the QNAP side whether it has received a trap or not.

What does the "test" in the UPS confirm? Send? Receive? Both? Neither?

Cheers,

m

[BillP]

Hi,

Below is how it is explained in the NMC help file. 

Last Test Result

The result of the mostrecentSNMP trap test. AsuccessfulSNMP trap test verifies only that a trap was sent; it does not verify that the trap was received by the selected trap receiver. A trap test succeeds if all of the following are true:

• The SNMP version (SNMPv1 or SNMPv3) configured for the selected trap receiver is enabled on this device.
• The trap receiver is enabled.
• If a host name is selected for the "To" address, that host name can be mapped to a valid IP address.

To

Select the IP address or host name to which a test SNMP trap will be sent. If no trap receiver was ever configured, a link to the "Trap Receiver" configuration page is displayed.

[Anonymous user]

Hi Bill,

Again I'm back. This is all going slowly, but that's probably for the best. The major change that has occurred is that I've purchased two small Windows 7 boxes that will run PCNS for two individual sites PCNS will manage.

This morning, I installed PCNS on one of those boxes and set it up for VMware vCenter management. I ticked the option to indicate the vCenter is running as a VM. I also setup prioritization for the various VMs (High, Med, Low).

In PCNS, when I look under Virtualization Settings, Virtual Machine Startup/Shutdown, it indicates 2220 minutes required for shutdown. This number is a total of the numbers I input in VM Prioritization, Set VM Shutdown Duration (300 for High, 600 for Medium, 1200 for Low, and 120 for Un-prioritized). I am assuming 120 seconds per VM which, admittedly, is a guess. Since we have over three hours runtime I don't necessarily have to wait until the UPS is running on fumes before I send shutdown signals.

In the Event logs of PCNS, I noted an "Available Runtime Exceeded." I've dealt with that before the in the past and had a general idea that I needed to adjust the "Low Battery Duration" setting inside the NMC. Using the 2220 figure from above, I attempted to set that field to 37 minutes. However, I got an error that the number was out of range. Reading up on it a bit more, I tried to apply the "Force Negotiation" under Maximum Required Delay. After doing this, I get a Maximum Required Delay of 45 minutes which the drops to 30 minutes after a time. I still couldn't set any number above 30 minutes in Low Battery Duration.

I called into support and they tried to help me. I realize that I never know who I'm going to get there. The tech wasn't sure if the 2220 was correct or there was something else wrong. As a general type of fix, he suggested a firmware update on the NMC which I've done. The only difference is that now I see a range of values I can put into that Low Battery Duration Field "minutes [0-30]".

I guess I main concern here is that I know there will be more devices I need to shutdown. After the VMs and the hosts are shutdown, I will also have to shutdown a SAN and two NAS via a command file. This will, of course, increase the amount of run-time needed to shutdown everything. If I can't go past 30 minutes, I don't think I'm going to have enough time.

No doubt I'm missing something but I don't know what. The tech I spoke to promised to escalate to a product manager but I don't know how far that's going to get me. ("Product Manager" sounds like someone who has a lot more to do than talk to people for support. 😉

I've read the Application Note for this setup, but I need to read it again. No doubt I've forgot more than I've retained.

Cheers,

m

[BillP]

Hi Mike,

How do you plan to trigger the shutdown process? On battery for x minutes, when run-time has fallen below x minutes, or when UPS hits low battery? Also, what is the model UPS and what is the model and firmware of the NMC? I assume the NMC is AP9617, 9618, or 9619 but I would like to be sure. 

You can ignore the available run-time has been exceeded error as long as you never let the UPS reach low battery prior to VM-Host shutdown. 

BTW: The reason force negotiation changed to 45 minutes is because the time calculated includes the VM and host shutdown delays, any and any start up delays. I have posted the shutdown description below. 

1. PowerChute reports that the UPS is on battery.

2. Shutdown delay for the On Battery event elapses. PowerChute starts a maintenance mode task on each Host. At the same time it sends a command to turn off the UPS or Outlet Group.

3. PowerChute starts VM shutdown followed by vApp shutdown.

4. VM/vApp shutdown durations elapse and PowerChute gracefully shuts down the vCenter Server VM.

5. vCenter VM shutdown duration elapses.

6. PowerChute starts executing the shutdown command file. - VM & vApp Shutdown enabled with a 120 second delay (i.e. 120 seconds allocated for each action to complete). - The option to turn off the UPS or Outlet Group is enabled. - A shutdown command file has been configured with a 120 second duration. When a critical UPS event, such as UPS on Battery occurs, the following sequence is triggered:

7. Shutdown command file duration elapses. PowerChute shuts down the VMware hosts using the order on the VMware Host Protection page. (The host running the vCenter VM will be shut down last in this scenario).

8. OS shutdown sequence starts on the PowerChute physical machine.

9. After a 70 second delay the OS starts to shut down.

10. UPS waits for greater of Low Battery Duration/Maximum Required Delay (NonOutlet Aware UPS‟s) or the Outlet Group Power Off Delay (initiated during step 2).

11. UPS turns off after the user-configurable Shutdown Delay time has elapsed or the Outlet Group turns off after the power off Delay elapses.

[Anonymous user]

On 2/9/2016 11:04 AM, Bill said:

How do you plan to trigger the shutdown process? On battery for x minutes, when run-time has fallen below x minutes, or when UPS hits low battery? Also, what is the model UPS and what is the model and firmware of the NMC? I assume the NMC is AP9617, 9618, or 9619 but I would like to be sure. 

Well, I suppose the best answer re: the shutdown trigger is that I want the devices to remain up as long as possible. Seeing as we have  3 hours of run-time at average load, it doesn't help us to start shutting things down 2 hours in if it takes 37 minutes to shut it all down. Does that make sense? However, if it's simpler to put a hard time on it such as in the "on battery for x minutes" scenario, I am fine with that too. I don't recall where I would adjust the setting.

The NMC is an AP9631. The firmware is 6.4.0. The UPS is a SmartUPS 5000RT with 3 total batteries. Runtime remaining is: 3h 10m 11s.

On 2/9/2016 11:04 AM, Bill said:

BTW: The reason force negotiation changed to 45 minutes is because the time calculated includes the VM and host shutdown delays, any and any start up delays. I have posted the shutdown description below. 

Well, to be sure, it jumps to 45 then eventually goes back to 30. I don't understand why it jumps up then back down.

In other situations you've encountered, do you find that people will manually shutdown everything and time it all in order to get a real sense of how long things will take? I am assuming 120 seconds per VM, for instance, but the time may be less (or more!).

[BillP]

Hi,

You should set Runtime remaining below threshold to 47 minutes (I added an extra 10 minutes for safety) so the shutdown will start when the UPS report it has fallen below 47 minutes of available runtime. Since the system will not allow for anything lower than 30 minutes for low battery leave it at 30 minutes. 

On 2/9/2016 12:32 PM, Mike said:

In other situations you've encountered, do you find that people will manually shutdown everything and time it all in order to get a real sense of how long things will take? I am assuming 120 seconds per VM, for instance, but the time may be less (or more!).

Yes. People usually test / time the shut down to be sure they have enough time entered to complete the task. So if you find you need more time than 37 minutes you should increase the shut down delay and increase the Runtime remaining below threshold. 

1 thing you should not do is configure a shut down for Runtime exceeded. Since the needed time to shut down the system is greater than the Low Battery setting you could run into an issue. 

[Anonymous user]

Hi again,

You're talking about this area, right?

What we plan to do is actually run two tests. The first will be a manual shutdown in the order that PCNS will shut everything down. This will give us accurate shutdown times and will also (hopefully) expose problems with our shutdown ordering.

The second test will be an automated shutdown triggered by an extremely low _on battery_ time limit (5 minutes) so we can test the actual procedure and expose any problems therein.

[Anonymous user]

Bill!!! Don't leave me hangin!

[BillP · Accepted solution]

Hi,

Sorry I missed your post. Yes. That is the setting I referred to.