PCNS 4.2 for VMware - ESXi Host Shutdown Problem

[cBOB_apc]

Hello APC Support Team,

first some details about our environment/configuration:
- 1 vSphere 5.5 HA cluster (with full automated DRS enabled) for management purpose, hosting the vCenter and vMA (with PCNS 4.2 for VMware) VMs.
- 1 vSphere 5.5 HA cluster (with full automated DRS enabled) with 5 ESXi Hosts (on 2x Smart-UPS VT 15 kVA (AOS v6.4.6) - redundant setup)
- VM prioritization ON (all VM's are grouped), UPS shutdown OFF, trigger: 'UPS on battery'

In our first test-scenario with only one ESXi host involved, everything works fine:
- event 'on battery' occurs
- shutdown delay elapses
- shutdown sequence starts
- task to 'set host in maintenance' is triggered
- DRS starts to migrate the VM's (from the specified host only)
- host successfully entered maintenance mode
- VM showdown phase elapses (no VM shutdown required)
- ESXi host shuts down


In the real scenario when all hosts of the cluster are involved, the ESXi hosts shutdown fails.
The sequence is almost the same, beside the fact that the migration (cause there is a lot of it) and when the hosts successfully entered maintenance mode takes more time, but i think that's normal.
Even the 'VM shutdown phase' itself works as desired, and finally the DC's (@ HIGH group) shut down at last.
All VM's are off, all hosts in maintenance mode but no ESXi host shut down is triggered. For me it looks like that PCNS cant communicate with the cluster anymore at this point.

The Error Log only shows 'Error checking maintenance mode for Host x'!

Strange behavior is, when i manually take the hosts out of maintenance after a certain time, and restart all the VM's to recover the cluster, almost exactly 1h (in both tests i've done!) after the last host successfully went to maintenance mode, suddenly all the ESXi hosts shut down!?

Any ideas what causes this, or what i'm doing wrong?

If i forgot to mention some facts, let me know please

Thanks in advance

Robert

[BillP]

Hi,

Sounds like a communication issue between PCNS and the hosts. Are you using Active Directory or did you create a local user account that PCNS uses to communicate with VCSA and the hosts?

You should also review the troubleshooting section of the PCNS VMware User's Guide and page 16 for information concerning the use of Active Directory and local user account.

If you continue to have issues put PCNS into debug mode run the shutdown test and then review the error log. FAQ FA230514 discusses how to put PCNS into debug mode. 

[cBOB_apc]

Hi Bill,

thanks for your quick reply.

First, to get this clear, i read many things on forums about configure vifastpass on the vMA in order to get PCNS running. Is this still an issue, cause it is nowhere mentioned in the manual for PCNS?

Another question, does PCNS connect directly to the ESXi hosts to shut them down or via VCSA?

Actually PCNS connects via AD account with the VCSA, but local ones are created already on each host/vcsa. So yeah, i think with all the VM's off within the Cluster the ESXi hosts can't authenticate the PCNS user who tries to shutdown the host himself.

I will try 2 things next time:

- use local accounts to avoid the missing AD

- configure the ESXi hosts to use other/still available DC's (if possible)

Thank you again

Robert

[BillP]

Robert,

On 3/30/2017 12:24 PM, Robert said:

First, to get this clear, i read many things on forums about configure vifastpass on the vMA in order to get PCNS running. Is this still an issue, cause it is nowhere mentioned in the manual for PCNS?

Beginning with PCNS version 4.0 we no longer use vifastpass. 

On 3/30/2017 12:24 PM, Robert said:

Another question, does PCNS connect directly to the ESXi hosts to shut them down or via VCSA?

 Yes. The shutdown order is VMs, VApps, then VCSA, and finally the hosts. Application Note 180 describes most shutdown scenarios.  

[cBOB_apc · Accepted solution]

Hi Bill,

Thanks for the 'vifp' info, good to know not to worry about.

So i decided to go for a local account setup in the next test.

I removed the AD account from all ESXi hosts and the VCSA, so only the local ones exists, and I verified the login for each ESXi host / VCSA successfully.

During the "PowerChute Setup" via PCNS web-interface at the host-setup screen I received an error (see attached file).

But I think its caused by the fact that the VCSA and vMA/PCNS are not within this protected cluster and I ignore that at this point. The setup procedure finishes successfully too.

Clicking at 'host-protection' afterwards, I see the cluster hosting VCSA and vMA and the one which is protected by PCNS, exactly as permissions are granted. So everything looks fine in my opinion. Otherwise I will let you know... 😉

Have a nice weekend

Robert