One host fails to shutdown properly

[BillP]

This question was originally posted by Lee on APC forums on 6/12/2018

---

Greetings!

I have two ESXi hosts in a cluster managed by vCenter and have gone over the configuration steps many times and made sure everything is set up correctly but cannot figure out why one of my hosts fails to shutdown. In the error.log I see the following repeatedly:

11.components.Shutdowner.Hosts.ESXManagedHost - Failed to take host out of maintenance mode: 192.168.139.6

12-06-18 10:35:03,067 ERROR pool-185-thread-2 com.apcc.m11.components.Shutdowner.Hosts.ESXManagedHost - Failed to take host out of maintenance mode: 192.168.139.4

The host that fails enter maintenance mode but that is as far as it goes. I believe the error above sheds a light on the issue but don’t know what steps to take to correct it. 

If it helps, the host 192.168.139.4 is the one running vCenter and the PowerChute appliance. 

[BillP]

This reply was originally posted by Bill on APC forums on 6/12/2018

---

Hi,

I would check the permissions. Once PCNS powers vCenter off it will then take the host out of maintenance mode to allow the host to start up and start PCNS when AC is restored. When vCenter is off line if permissions are not correct PCNS will not be able to communicate with the host and therefore not take it out of maintenance mode. 

[BillP]

This reply was originally posted by Lee on APC forums on 6/13/2018

---

I made sure permissions were set correctly by creating the accounts on each host and configuring PCNS to use that account in case vCenter is not available. Is it possible that the host running PCNS and vCenter is shutting down before the one that fails? 

[BillP]

This reply was originally posted by Lee on APC forums on 6/13/2018

---

Bill Here's a screenshot of vCenter showing the PowerChuteAdmin user I created. This user also exists on the VM running vCenter (Windows) and also on each individual ESXi host.

[BillP]

This reply was originally posted by Bill on APC forums on 6/13/2018

---

Hi,

The shutdown process is

1 - put hosts into maintenance mode

2 - shutdown VMs and vApps

3 - shutdown vCenter Server

4 - Shutdown hosts starting with the host not running PCNS

Host 139.6 should remain in maintenance mode until restarted. Host 136.4 should be taken out of maintenance mode to allow it to restart when AC is restored which in turn will allow PCNS to restart. 

Have you put PCNS into debug mode and reviewed the error.log to see what errors are posted. If you need assistance putting PCNS into debug mode see Schneider Electric FAQ FA230514

[BillP]

This reply was originally posted by Lee on APC forums on 6/16/2018

---

Hi  Bill ,

I have enabled debug mode per your instructions attached the log file and would appreciate if you could take a look. For your reference, the debug log starts at 16-06-18 08:11:24,042.

Again, I have followed the instructions over and over to make sure I didn’t miss a step but for some reason that one host fails to shutdown every time. It will however enter maintenance mode but that’s as far as it goes. The only thing I haven’t tried is placing the PC appliance on the host that fails in which by the way, runs ESXi 6.7 and the one that shuts down correctly runs 6.5 if that matters. 

Thanks for your help. 

[BillP · Accepted solution]

This reply was originally posted by Bill on APC forums on 6/18/2018

---

Hi,

The error.log file shows a permissions issue

16-06-18 08:18:23,976 ERROR Thread-34 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - Unable to log directly into host (NoPermission or InvalidLogin): 192.168.139.4

16-06-18 08:18:23,976 DEBUG Thread-34 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - com.vmware.vim25.InvalidLogin

16-06-18 08:18:23,976 DEBUG Thread-34 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - getESXiHostConnection end

16-06-18 08:18:23,976 ERROR Thread-34 com.apcc.m11.components.Shutdowner.Hosts.ESXManagedHost - Unable to establish a connection to shut down the host.

16-06-18 08:18:23,976 DEBUG Thread-34 com.apcc.m11.components.Shutdowner.Formatters.VirtualizationFeaturesFormatter - format: MsgID=96

16-06-18 08:18:23,976 DEBUG Thread-34 com.apcc.m11.components.Shutdowner.Formatters.VirtualizationFeaturesFormatter - formatted:TargetHostShutdownFail.Log to:Shutdown Host failed for Host 192.168.139.4.

16-06-18 08:18:23,976 DEBUG Thread-34 com.apcc.pcns.errorservice.ApplicationErrorHandlerImpl - Adding error to application error handler. String: Shutdown Host failed for Host 192.168.139.4.

What happens when you log into 139.4 using PowerChuteAdmin and the password? 

I don't see any errors for 139.6

Also, would you please sent the eventlog. I would like to review that as well.

[BillP]

This reply was originally posted by Lee on APC forums on 6/18/2018

---

I can log in to 139.4 just fine with the PowerChuteAdmin user. 

Here's the event log:

[BillP]

This reply was originally posted by Lee on APC forums on 6/18/2018

---

So after you said it was a permission issue, I kept digging further and discovered that this part

PowerChute Security
"The Username and Authentication Phrase are used to authenticate communications between PowerChute and the NMC. Therefore, you must set these values to be the same in both PowerChute and the NMC."

was misconfigured.

So it appears that the user in the NMC must be the same as the user in PowerChute appliance. If I stuck to the default configuration of the user apc I would probably not have ran into this issue but I insisted on creating my own user and forgot to change it in the PowerChute Security section. 

Thank you very much for your help  Bill .