PCNS 3.1 with VMWare

[Anonymous user]

I have a 3 host vmware cluster with 2 x APC UPS with NMC providing redundant power to each of the hosts and to the SAN.

I have set up PCNS 3.1 as a virtual appliance on 1 of the hosts (a), Vcenterserver is a VM on another host (b). Other VMs are on host C.

I have completed the wizard to set up communication with the NMC and onto the process of fine tuning the shutdown process.

After logging into the Powerchute VM console page I have set the following:

On the Configure Events page I have set the "UPS On Battery" to shutdown after a 300 second delay.

On the Shutdown Settings I have enabled VM shutdown with a 240 second delay.

On the VCenter Server VM shutdown I have set it to 480 seconds.

Within VMware host protection, the servers are automatically listed with C at the top, then A then B.

My question is, what is the sequence that will follow a power cut? As I understand, 300 seconds after a power cut the shutdown process will start.  However will the VM shutdown occur on all hosts at this time? Or will it work through each host leaving a 240 second gap between each one? ie after 300 seconds the VM shutdown starts on host C, then 240 seconds later on host A and then 240 later on host B?  Should I adjust the order of the hosts?  Should the powerchute vm be on the same host as the vcenterserver VM?  At what stage does the vcenterserver shutdown occur?

Thanks

[Anonymous user]

Well I managed to test it yesterday.  Out of my 3 hosts, only the first 1 shutdown, the other 2 hosts, with vcenterserver on 1 and PCNS on the other stayed on and didn't turn off.  is this correct?

Thanks

[Anonymous user]

Thanks, how do I access those files with the virtual appliance?

[BillP]

Hi

SSH access is enabled on the Virtual Appliance. You can transfer the files using WinSCP.

I think part of the problem is covered in FA228172 - there is an issue in PCNS 3.1 when both PCNS and vCenter Server are running on VMs the ESXi host running vCenter Server does not get shut down. When the vCenter Server goes down PCNS attempts to connect to each ESXi host directly using an Active Directory account. In addition to Configuring an AD account via VMware SSO, each ESXi host must be joined to the domain and the AD account needs to be added on each ESXi host - ESXi hosts are not integrated with VMware SSO.

To resolve the issue you would need to install PCNS on a separate Windows machine and check that each ESXi host has been joined to the AD domain and that the AD user account has been added on each ESXi host (connect directly to each Host using vSphere Client).

[Anonymous user]

Thanks, so the virtual appliance is useless then which is a disappointment. 

I don't want to add my hosts to the domain as the hosts contain VM which are domain controllers.  So if the host have been off, I can't login and start the VMs as the DC is off.

[BillP]

In the case where vCenter Server is running on a VM, yes the Virtual Appliance cannot be used. This will be addressed in the next release.

Investigating an alternative solution using vMA and PCNS 3.1. Will provide the details shortly.

[Anonymous user]

Thanks

[BillP]

I think the following solution will work in your environment.

1. Deploy vSphere Management Assistant (vMA) on one of the ESXi hosts.

2. Install PCNS 3.1 using ESXi build and configure in the same manner as you did for the Virtual Appliance.

3. Add the ESXi hosts as targets for vi-fastpass using the command vifp addserver

4. Create a script e.g. shutdownhost.sh with the content below on the vMA - only the lines in bold need to be changed for your environment.

#!/bin/sh

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/vmware/vma/lib64:/opt/vmware/vma/lib

export LD_LIBRARY_PATH

export PERL_LWP_SSL_VERIFY_HOSTNAME=0

SAVEIFS=$IFS

IFS=$(echo -en "\n\b")

hosts=(10.216.254.167 10.216.254.168)

ups_vm="vSphere Management Assistant \(vMA\)"

for host in ${hosts[@]}; do

echo $host

source /opt/vmware/vma/bin/vifptarget -s $host

for i in `vmware-cmd -l --username xxxxx --password xxxxx`; do

  if [ `echo $i | egrep -c $ups_vm` -eq 1 ]; then

  echo "Found vMA Host $host"

  vma_host=$host

  fi

done

done

source /opt/vmware/vma/bin/vifptarget -c

for host in ${hosts[@]}; do

source /opt/vmware/vma/bin/vifptarget -s $host

if [ $host = $vma_host ]; then

  echo "Skip vMA host shutdown"

else

  vicfg-hostops -o shutdown --force

fi

done

echo "Shutting down ESXi Host running vMA"

source /opt/vmware/vma/bin/vifptarget -s $vma_host

vicfg-hostops -o shutdown --force

source /opt/vmware/vma/bin/vifptarget -c

6. Save the file and grant it execute permissions (chmod +x shutdownhost.sh)

7. In the PCNS UI go to Shutdown Settings->Run Command and enter the full path to shutdownhosts.sh. Enter a duration value - this should be long enough to allow the host shutdown commands to be sent. Apply changes.

When the UPS goes on battery the following sequence occurs:

1. After the shutdown delay for the on Battery event has elapsed PCNS will start shutting down VMs on each Host (it does this in parallel threads). VM shutdown delay is the time allowed to shutdown VMs on all Hosts.

2. When the VM shutdown delay has elapsed PCNS will shut down vApps if this option is enabled.

3. After the vApp shutdown delay has elapsed PCNS will shutdown the vCenter Server VM (vCenter Server VM Shutdown Duration).

4. After the vCenter Server VM Shutdown Duration has elapsed PCNS will run the shutdown command file (shutdownhosts.sh).

5. The shutdownhosts.sh script uses vi-fastpass to connect to the ESXi hosts (no need for Active Directory authentication) and shut them down. It detects which host is running the vMA with PCNS installed and leaves this until last.

Output from the script:

Caveat: Since PCNS is not shutting down the Host running vCenter Server it does not track that it has shut it down and will not re-start it automatically when the hosts come back online.

If you have any trouble creating/running the script please let me know.

[BillP]

Just to point out - the Virtual Appliance will work correctly if it is deployed on the same Host as the one running the vCenter Server VM. The issue with the hosts not shutting down occurs when the Virtual Appliance and vCenter Server are on different hosts. DRS affinity rules can be used to keep the Virtual Appliance and vCenter Server (VCSA) on the same host.

However since you cannot join your ESXi hosts to the Active Directory domain the only solution I can see is the one outlined above.

[Anonymous user]

Thanks, I will give it a try.  I used the vMA solution with 3.01 but it never worked correctly as the vcenterserver vm was shutdownat the same time as the other VMs which meant it couldn't keep track of their status so when power was back on it was a mess in vsphere.

[BillP]

This shouldn't be an issue with PCNS 3.1  on vMA since it deliberately shuts down the vCenter Server VM after the other VMs have been shut down.

Let me know how it goes.

[Anonymous user]

Hi,

Sorry picking this up again.

I am assuming that now a local user has been added to windows server that has vcenterserver installed that it should be that user on the  "vCenter Server Details" in PCNS rather than the domain account I used before?

Also, would it be advisable to have the PCNS virtual appliance on the same host as my vcenter server virtual machine?

Thanks

Ed

[BillP]

Hi,

Yes you should use the local user account that has been added to the Windows VCenter Server VM in PCNS. In addition to this you need to log into each ESXi host directly and add a user with the same username and password (Local Users and groups) with the administrator role. This is so that PCNS can log in to each ESXi host directly to complete the shutdown after the vCenter Server VM is shut down.

VMware tools needs to be installed on the vCenter Server VM.

There is no strict requirement to have PCNS virtual appliance on the same host as the vCenter server virtual machine.

[BillP]

No problem.

The new local user on the vCenter Server machine can just be an ordinary user - it does not need admin privileges on the windows machine itself, it just needs to be assigned the Administrator role in ESXi/vSphere permissions.

[Anonymous user]

Thanks for the quick reply.

Now I have my new local admin set up in windows of vcenterserver. Do I need to enter the username as vcenterservername\username or just username in PCNS?

Ditto on each of the host when I add them?

Thanks

Ed

[BillP]

Hi,

Just username in PCNS and for each ESXi host as the ESXi hosts will not allow you to enter vcenterservername\username.

[Anonymous user]

Thanks, hopefully last questions:

On point 2 you say: Log in to vCenter Server using the vSphere Client and  click on the Permissions tab at the root inventory level. Do you mean datacenter or cluster level?

Sorry for all the questions, I want to get this right as I have seen the effects of hosts and VMs not gracefully shutting down!

Edit: having thought about it, if propagate is select it shouldn't matter. Is that correct?

[BillP]

Hi,

No problem. The permissions should be added at the root inventory level as shown in the attached screenshot.

The PowerChute vSphere plug-in is created at the root level so if you're using that you'd need to define the permissions at that level. In theory, if you don't use the plug-in and only want to allow PCNS to access certain datacenters or clusters you could add the permissions at the datacenter/cluster level to restrict access.

[Anonymous user]

Thanks, all set up.

I assumed that the new local user on the vcenterserver machine should be a local admin.  Or can it just be a normal user?  I have made it an admin on each host.

Thanks again.  I hope to test later.

[Anonymous user]

Well the test didn't go too well.  More or less the same thing happened.

The hosts went into maintenance mode, VMs shutdown correctly but on the next stage of shutting down vcenterserver it went wrong, 1 host turned off at this point, which was the same host as PCNS and the vcenterserver machine and vsphere moved them onto another host.  After that nothing else happened apart from the UPS starting a count down on the LCD screen to turn off the outlet group.  1 UPS depleted very quickly which doesn't help (which I have another thread about). Any ideas?

[Anonymous user]

Looking at the emails sent out at the time I got this message

Alarm Definition:

([Event alarm expression: event.UPS Critical Event occurred; Status = Red] OR [Event alarm expression: event.UPS Critical Event initiated; Status = Yellow] OR [Event alarm expression: event.UPS Critical Event cleared; Status = Green])

Event details:

  event.UPS Critical Event occurred.fullFormat (UPS Critical Event occurred)

Would that cause the host to be shutdown immediately?

[BillP]

Hi,

So it sounds like vCenter VM Shutdown is being skipped and PCNS is moving directly to ESXi host shutdown. You have vCenter Server running on a Windows Virtual Machine? Can you check if VMware tools are installed on that VM? PCNS uses VMware tools to find the vCenter Server VM (FindByDNS, FindByIP API functions) - if VMware tools is not installed/running then PCNS cannot locate the vCenter  Server VM and it will skip the vCenter Server VM shutdown duration (default 240 seconds). If VMware tools are running then another possible cause of this issue is if PCNS cannot resolve the vCenter Server VM hostname e.g. old/duplicate DNS records etc. In the Shutdown settings page of PCNS do you see an option to Configure vCenter Server VM shutdown duration

Could you attach the following files from PCNS: pcnsconfig.ini, EventLog.txt, error.log, they are located in the /opt/APC/PowerChute/group1 folder on the Virtual Appliance (SSH is enabled so you can access the files using WinSCP for example). I'll have a look at the files and try to confirm what the problem is.

[Anonymous user]

Thanks for doing further research.

When you say "Log in to vCenter Server machine and add a user via Computer Management -> Local Users and Groups for Windows" Do you mean add a local user to the windows OS?

I can't do the Ad route as I don't want to add my hosts to the domain due to the virtual domain controllers.

Thanks

[BillP]

Hi,

If your vCenter Server is running on a Windows VM, yes you add a local user via Computer Management -> Local Users and Groups to the Windows OS.

If you are running vCenter Server Appliance you need to login locally to the console and use the terminal commands "useradd" to create the user and "passwd" to set the password for that user. Having done this the new user will appear under "(server)" in Select Users and Groups dialog when assigning permissions in vCenter Server.

On each of the ESXi hosts being managed you need to log in directly to the ESXi hosts and go to the Local Users and Groups tab - you then need to add a new user with the same username and password as the one created for vCenter Server.

That way PCNS can connect to vCenter Server and the ESXi hosts if the virtual domain controllers are shut down and PCNS can still connect directly to the ESXi hosts to shut them down after it tells the vCenter Server VM to shut down.

[BillP]

Hi,

Would you please send the pcnsconfig.ini, EventLog.txt, and error.log The files will be in /opt/APC/PowerChute/group1

[Anonymous user]

I have a 3 host vmware cluster with 2 x APC UPS with NMC providing redundant power to each of the hosts and to the SAN.

I have set up PCNS 3.1 as a virtual appliance on 1 of the hosts (a), Vcenterserver is a VM on another host (b). Other VMs are on host C.

I have completed the wizard to set up communication with the NMC and onto the process of fine tuning the shutdown process.

After logging into the Powerchute VM console page I have set the following:

On the Configure Events page I have set the "UPS On Battery" to shutdown after a 300 second delay.

On the Shutdown Settings I have enabled VM shutdown with a 240 second delay.

On the VCenter Server VM shutdown I have set it to 480 seconds.

Within VMware host protection, the servers are automatically listed with C at the top, then A then B.

My question is, what is the sequence that will follow a power cut? As I understand, 300 seconds after a power cut the shutdown process will start.  However will the VM shutdown occur on all hosts at this time? Or will it work through each host leaving a 240 second gap between each one? ie after 300 seconds the VM shutdown starts on host C, then 240 seconds later on host A and then 240 later on host B?  Should I adjust the order of the hosts?  Should the powerchute vm be on the same host as the vcenterserver VM?  At what stage does the vcenterserver shutdown occur?

Thanks

[BillP]

Hello,

The following application note 180 explains the different PowerChute scenarios:

http://www.apcmedia.com/salestools/SJHN-92FLU8/SJHN-92FLU8_R0_EN.pdf

The host that PowerChute is installed on, always shutdowns last regardless of the way you configure it on the VMWare Host Protection page. The host running the vCenter VM, shuts down second last before the Host that shuts down PowerChute. The sequence in which the shutdown process is as follows --> VM's, vApps, Host C, Host B and then Host A. VM shutdown delay is the total amount of time it takes for all VM's to shutdown.

Regards,

B

[Anonymous user]

Thanks for the reply.  I had found that note already doesn't actually mention the timings.  Does it start shutting down the VMs on the first host (in my case c) and after 240 start on the VMs on the next host? and then the next? IE all the VMs are shutdown after 720 seconds (3 x 240)

What does the value for "vCenter Server VM Shutdown Duration" represent?

Thanks

[BillP]

The 240 seconds refers to the time it takes for all the VM's on all the hosts to shutdown. I would recommend you have enough time set here for all the VM' to shut down in your environment.

Since your vCenter is running on a VM, I would advise you configure an AD or LDAP account with an Admin role for vCenter server,so that PowerChute will still be able to connect directly to the VMWare hosts using this AD or LDAP account.

[Anonymous user]

Thanks, so in theory all VMs should be shut down after 540 second. At which point PCNS will shutdown the hosts?

I created a group in vsphere with the following permissions http://www2.schneider-electric.com/resources/sites/SCHNEIDER_ELECTRIC/content/live/FAQS/177000/FA177...

I then added an AD user to that group. This was the user account I used when connecting to the vcenterserver during the config wizard.

Going back to one of my previous questions, What does the value for "vCenter Server VM Shutdown Duration" represent?

My set up is more or less the same as scenario A on http://www.apcmedia.com/salestools/SJHN-92FLU8/SJHN-92FLU8_R0_EN.pdf

I am assuming it shuts down the vcenter server last ie after 240 seconds and then after the vCenter Server VM Shutdown Duration the hosts are then shutdown?  I expect this is so that vcenter server knows the state of the VMs before being shutdown itself which is the problem I have with PCNS 3.01, the vcenter server was shutdown with all the other VMs and there was a big mess when the cluster was turned back on.  Is there a need to use a command file?

Thanks again.

[BillP]

Yes, the VM's will all be shutdown after 540 seconds providing you have VMWare Tools installed and also the right timings set in the duration field for all the VM's to shutdown. PowerChute will wait for this duration to be elapsed before moving on to the next step of the sequence which is shutting down vApps (if you have any), and then the hosts.

vCenter Server VM Shutdown Duration represents the time before the vCenter VM will be shutdown. It is shutdown once all the other VM;s have been shutdown as described in the VMWare documentation.

It shuts down the host with PowerChute installed on last ALWAYS. The host that hosts vCenter VM will be shutdown second last in the sequence.

[Anonymous user]

Many thanks for your help.  I suppose all there is for me to do is pull the power to test!

[Anonymous user]

Hi Guys,

I am facing similar issues with PCNS 3.1, on VMware 5.5 with a virtualised vCenter server (on Windows server not appliance). I have created local user accounts for vcenter and each of my two vmare hosts as described above, and updated credentials in PCNS configuration.

Our environment has:

UPS1 - 10.61.213.51

UPS2  -10.61.213.52

VMware Host1 - 10.61.213.111

VMware Host2 - 10.61.213.112

My findings are that PCNS is behaving as follows:

-successfully shuts down guest VMs, on the host not running vcenter guest

-successfully shuts down the Host not running vcenter guest

-successfully shuts down guests on host running vcenter guest (and PCNS)

-successfully shuts down vcenter guest

-unable to shutdown host running PCNS, due to a failure to communicate with vcenter (expected as vcenter was powered down in the previous step, but though communication should pass to the host itself in this instance?)

The strange thing is, it looks like host2 (running PCNS) is being instructed at some point to exit maintenance mode? I can only assume this is due to some recovery condition being signalled from the UPS?

I would expect the final step to be PCNS instructs the host on which it is homed, to shutdown (by direct communication with the host), and go down with the ship. - This is not what we are seeing. The final host remains online, along with PCNS appliance online.

Am i missing something?

I have attached our diagnostic log from testing and pcnsconfig

Many thanks

Jon

[BillP · Accepted solution]

Jon,

Please provide copies of error.log and VirtualizationFileStore.properties. They are located in the /opt/APC/PowerChute/group1 folder. Also, is PCNS running on vMA or did you install the Appliance/PCNS_3.1_OVF10.ova?