"Find By IP" messages under recent tasks in ESXi host

[BillP]

After properly configuring PCNS, I am now seeing these messages under my recent tasks in both of my ESXi hosts. These tasks are started by the shared user I created for PowerChute (user pcns).

Attached are a couple screenshots of the PowerChute event log and ESXi. It happens about every two minutes. 

Any ideas?

[BillP]

Hi,

What version of PCNS are you running? Also, look in the PowerChute install folder / directory. There will be a file named VirtualivationFileStore.properties. Is that file empty or are there entries in it? 

[BillP]

I am running the latest version (4.2.0 Build 6755) and the file VirtualizationFileStore.properties has all of my VMs listed but on the second host, one of the VMs repeat. Here's what it looks like:

#Mon Jun 18 17:42:17 EDT 2018

192.168.139.4_VMs=VM1,VM2,VM3,VM4,VM5,VM6,VM7

192.168.139.7_VMs=VM2,VM8

I've noticed that VM2 shows on host x.139.4 AND on x.139.6 even though it only exists on host x.139.4. 

Is this the issue?

[BillP]

Update:

I deleted the contents of the VirtualizationFileStore.properties and restarted PCNS appliance and ran another test and everything shutdown correctly. Now the contents of the file have changed to this:

#Tue Jun 19 19:56:57 EDT 2018

LOCK_192.168.139.4=true

192.168.139.4_VMs=MyVM_1,MYVM_2,MYVM_3,MYVM_4,MYVM_5,MYVM_6,MYVM_7

192.168.139.7_VMs=MYVM_8

VCSA_VM=MYVM_1

VCSA_HOST=192.168.139.4

LOCK_192.168.139.7=true

Unfortunately the messages remain.

[BillP]

Bill  Anything else you suggest I try?

[BillP]

Hi,

Stop the PowerChute service and clear the file. Then check to see if the messaging stops.

The file should be cleared automatically when PowerChute restarts after an outage. Once you know the massaging has stop you should look at the event log and error logs to see if anything is being logged in PowerChute to indicate why the file was not cleared. In most cases it is caused by a host or hosts not powering down or restarting properly. 

[BillP]

I have followed your instructions but the message still appear every 2 minutes but now only on the host running vCenter and PCNS. The file did NOT clear properly again. I still see 

LOCK_192.168.139.4=true

192.168.139.4_VMs=MyVM_1,MYVM_2,MYVM_3,MYVM_4,MYVM_5,MYVM_6,MYVM_7

192.168.139.7_VMs=MYVM_8

VCSA_VM=MYVM_1

VCSA_HOST=192.168.139.4

LOCK_192.168.139.7=true

Attached you will find the event.log and error.log of my last test. 

[BillP]

Hi,

After the last restart the error log shows 

21-06-18 21:11:12,173 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - java.net.UnknownHostException: DELLR710.HOME: unknown error
21-06-18 21:11:12,180 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - getESXiHostConnection, Host DELLR710.HOME - (RemoteException) VI SDK invoke exception:java.net.UnknownHostException: DELLR710.HOME

Prior to that the log shows 

21-06-18 18:57:55,392 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VirtualizationUtils - java.net.UnknownHostException: DELLR710.HOME: unknown error
21-06-18 18:57:55,399 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - getESXiHostConnection, Host DELLR710.HOME - (RemoteException) VI SDK invoke exception:java.net.UnknownHostException: DELLR710.HOME
21-06-18 19:54:03,250 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - validateESXiConnection Host 192.168.139.6 - RemoteExcepion occurred, attempting reconnection
21-06-18 19:54:09,263 ERROR pool-3-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - getESXiHostConnection, Host 192.168.139.6 - (RemoteException) VI SDK invoke exception:java.net.NoRouteToHostException: No route to host

When the hosts were added to PCNS were they added using domain name or IP? Would you attached a copy of the pcnsconfig.ini file?

[BillP]

Bill I added the hosts using their IPs since I don't have a domain setup in my environment. 

Attached you will find my configuration file.

[BillP]

Hi,

To verify both PowerChute 139.121 and VCSA 139.96 are running on host 139.6? Also, when you look at the name assigned to the server 139.6 is it DELLR710.HOME? I am trying to determine where that name is coming from.

What appears to be happening is when PowerChute starts it is attempting to communicate with 139.6 and cannot establish communications. PowerChute using the VMware API find by ip. 

To troubleshoot please open the PowerChute web interface and deselect All host on prior to start up under the Virtualization Settings . Then stop the PowerChute service, clean out the VirtualivationFileStore.properties file and put PowerChute into debug mode (see Schneider Electric FAQ FA230514), then restart. Run the shutdown test and if the issue persist attach pcnsconfig, error.log, EventLog.txt, VirtualivationFileStore.properties.

NOTE: Be sure to take PowerChute out of debug mode once the test has completed or the error.log will get very large. 

[BillP]

No, PowerChute (139.121) and VCSA (139.96) are running on host 139.4 (DELLR710.HOME).

[BillP]

Ok, so I think we are getting close and I just may have a clue as to what could be happening. Just so we are clear, this how everything is setup:

139.4 ESXi Host = PowerChute Appliance (139.121) and vCenter Server inside a Windows Server VM (139.96) along with a few other VMs. 

139.6 ESXi Host = A few VMs. 

139.116 - APC Network Management Card

One thing I did notice after a test is once power is out, PowerChute shuts down all VMs and places all hosts in maintenance mode and them shuts them down but once power is restored and I start the hosts once again, only the host 139.4 starts up OUT of maintenance mode but the host 139.6 stays in maintenance mode. I believe that this is what triggers the "find by ip" message. 

What I don't understand is how can host 139.4 power on already out of maintenance mode if the PowerChute Appliance is not even running yet? Is the host taken out of maintenance mode right before shutdown? If so, PowerChute is failing to do the same with host 139.6 and I have no idea why.

Attached you will find the ini. file, error.log and Virtualizationfilestore. I did take a quick glance at the error.log and found:

ERROR Thread-35 com.apcc.m11.components.Shutdowner.vmShutdownRunnable - No Shutdown required.  Host 192.168.139.6 may be powered off or in maintenance mode.  Waiting duration anyway

P.S. I hate to complicate things further but unfortunately I gave the Windows VM running vCenter server a static ip so when you see the log, 139.96 is now 139.7.

Thanks for your help so far.

[BillP]

Hi,

In the error log I see

25-06-18 19:51:18,722 DEBUG Thread-34 com.apcc.m11.components.Shutdowner.Hosts.ESXManagedHost - Attempting to take host out of maintenance mode: 192.168.139.4

25-06-18 19:51:31,127 ERROR pool-6-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - getESXiHostConnection, Host 192.168.139.4 - (NoPermission or InvalidLogin) [no details available]

25-06-18 19:51:31,128 DEBUG pool-6-thread-1 com.apcc.m11.components.WebServer.util.virtualization.VMWareConnection - com.vmware.vim25.NoPermission

25-06-18 19:51:31,135 ERROR pool-6-thread-1 com.apcc.m11.components.Shutdowner.Hosts.ESXManagedHost - Failed to take host out of maintenance mode: 192.168.139.4

The standard process is to put all hosts into maintenance mode. Shutdown VMs and any vApps. Shutdown the vCenter VM and then take the host PCNS is running on out of maintenance mode to allow it to start up and start PCNS. Shutdown down all host finishing with the PCNS VM and its host. The log is showing that the host is not exiting maintenance mode and I suspect it is not powering down properly but cannot tell without the Event log. It also indicates there is a permissions issue happening between PCNS and host 139.4 once vCenter is powered down.

[BillP]

The log section you posted is from 06-18-18 and those errors have already been corrected. I did what you suggested two days ago and posted the relevant log but you still quoted 06-18-18. 

Is there any way I can contact you directly through chat? Unfortunately we are not getting anywhere this way specially since I have to wait a day or so for a reply. 

[BillP]

Hi,

I am sorry you are not satisfied with my assistance. The logs you provided and what I posted clearly show the errors and the date stamp is 25 of June 18. The log clearly shows there is a permissions or login issue and the host is not being taken out of maintenance mode which is the most likely cause of the find by ip calls. 

Does the event log show the host being taken out of maintenance mode and does it show the host being shutdown? 

[BillP]

The even log shows both hosts being shutdown but only 139.4 is taken out of maintenance mode. 

[BillP]

Hi,

That is strange since the error log show an issue with communication and that it could not take the host out of maintenance mode. Both of which would cause the VirtualFileStoreProperties file note to be cleared. Causing PowerChute once back on line to search for the IP address and the API call find by ip. 

Would you please attach the EventLog.txt file.

[BillP]

I've been troubleshooting this further for the past few days and made a few changes to my setup in order to simplify things. I am now running vCenter Appliance instead of vCenter inside of a Windows Server VM. Before I run another test and post the event log, I wanted to make sure of a few things.

If I understand correctly, a user with administrative privileges need to exist on both of my hosts and as well as on the vCenter appliance. So I created a user pnc and gave administrative privileges to both on my ESXi hosts. Since I am now using VCSA, I don't see an option anywhere in vCenter to create a user. When I was running it inside a Windows Server VM I could just follow the guide for setting up a Windows user. 

What do you advise? I believe this setup to be much simpler and will certainly help eliminate any permission errors. 

Edit - I believe the user needs to be created by using "useradd" command via ssh. Unfortunately that doesn't work in VCSA 6.7.

[BillP]

Hi,

On 7/5/2018 9:52 PM, Lee said:

Edit - I believe the user needs to be created by using "useradd" command via ssh. Unfortunately that doesn't work in VCSA 6.7.

I searched the web and found this https://github.com/vmware/photon-controller/wiki/Authenticating-Multitenant-Users-and-Groups

I also looked in the VMWare Admin guide. I suggest you contact VMware support since their doc does not list any such command. https://github.com/vmware/photon-controller/wiki/Command-Line-Cheat-Sheet

[BillP]

Hi Bill,

I was able to add the user to the VCSA appliance. Now all VMs and Hosts shut down gracefully. One small issue is once power is restored and hosts are restarted, the PowerChute appliance would not auto start so I configured it to do so when the host is back online (Using the ESXi autostart setting). After another test I found that the PowerChute appliance is started and all hosts are taken out of maintenance mode and all VMs started BUT the VCSA. I don't know if it was just a fluke since I have yet to run one more test but do you think I need to also set the VCSA VM to autostart? 

Again, this is my setup to help you visualize it:

ESXi Host 1 - 139.4

VCSA, PCNS, Virtual Machine 1

ESXi Host 2 - 139.7

7 Virtual Machines

The log shows that once the UPS signals the hosts to shutdown, all hosts enter maintenance mode and all VMs start shutting down one by one. In the end, the host 139.4 exits maintenance mode so it can shutdown the VCSA. Later a message warns that VCSA is down and commands cannot be issued. PCNS then shuts down hosts directly. Once hosts are restarted, 139.4 is out of maintenance mode and PCNS is auto started by the ESXi host. The 139.7 host never exits maintenance mode until told to do so by PCNS. It is then told to exit maintenance mode and all VMs are started by PCNS. For some odd the reason the VCSA was never started. 

I plan on running one more test but first wanted to see what you think.

The messages "find by ip" are gone. 

Thank you for your help so far.

p.s Don't mind the IPs, they were changed again to help simplify my setup. 

[BillP]

hi,

On 7/10/2018 8:31 PM, Lee said:

The 139.7 host never exits maintenance mode until told to do so by PCNS. It is then told to exit maintenance mode and all VMs are started by PCNS. For some odd the reason the VCSA was never started. 

That is odd since PCNS should start VCSA first and then start the VMs. See page 5 of Application Note 180 for the start up process. You should look in the error.log file to see if there is an entry that explains why the VCSA was not started.

[BillP · Accepted solution]

The publication says HA is responsible for starting the PowerChute appliance? If so this may be the reason since I don’t have HA enabled. 

I thought that since the PowerChute appliance is powered off instead of shutdown that it would automatically start when the host is restarted.

Just a bit confused, I apologize for so many questions.