PCNS 4.3, vcenter 6.7 u2b, - PCNA does not shutdown a thing, EVER

[DaleMurray_apc]

PCNS is installed on hardware Windows 10 computer dedicated to managing Pivot3 (hyperconverged system) and APC.

I have done the following.

• Each ESXi host is domain joined
• The following key (Config.HostAgent.plugins.hostsvc.esxAdminsGroup) on each ESXi host is configured to utilize the domain security group assigned to vm management
• I have verified a specific user within that security group - vmanager - can log in to each ESXi host and vcenter the following ways:
  • domain\username
  • username@domain.com
    All other users within that security group can also login without issue
• I can access the APC plugin from vcenter server when running in Flash mode
• It appears PCNS is communicating with vcenter without issue as it lists VM's running in my stack
• The user ID used in PCNS to communicate with vCenter is the vManager user mentioned above.
• PCNS has been uninstalled/reinstalled multiple times.
• SMX3000LVNC UPS has been factory reset and network configurations re-entered

What simply does not work, and frankly the only reason to bother with PCNS, it get this software to shutdown ANY VM when power is lost.
I have unplugged power and waited more than 10 minutes and nothing EVER gets shutdown, EVER.
I have attempted to have PCNS shutdown my vmware stack with vm prioritization checked, and unchecked. Did not work.
I have tried every variation I can think of under Virtualization Settings. Did not work.

I have been trying to get this to function for more than a week and the only thing that works so far, is a batch file I have run after 10 minutes to shutdown Pivot3 vm – which must be the very last vm running prior to maintenance mode.

[DaleMurray_apc]

So what does one have to do to get support from APC?

I've opened a ticket - no response.
Created this post - no response.
Contacted support via chat twice yesterday - both sessions were closed by tech prior to answering ANYTHING.

[BillP]

Hi,

Sorry for the inconvenience. What is the ticket number you have open? I would like to review to see what has happened with that.

In the mean time, when you tested what was posted in the PowerChute event log. If you do not mind please zip these files and attach to this post.

error.log, EventLog.txt, VirtualizationFileStore.properties, pcnsconfig.ini, javaVersionFile.log, debug.log

If installed to the default path the files are in 
C:\Program files\APC\PowerChute\group1

[DaleMurray_apc]

My configuration is similar to "Scenario A" on page 16 of this document:  https://www.apc.com/salestools/SJHN-92FLU8/SJHN-92FLU8_R7_EN.pdf

• Physical computer running PCNS connected to a UPS, however, the UPS servicing the PCNS computer does not have NMC.
• All vmware hosts connected to a single UPS with NMC.

We are not running vSan to create the hyperconverged environment, we are using Pivot3.

My ESXi hosts and vCenter are all domain joined and tested with domain user logins.

I am sure I missed something but not sure what. I am also quite certain if somebody were to create a remote session they could review my setup and resolve the issue quickly.

Tickets:
59638274 - Arvin did reply a couple times but seems to have fallen off the radar.
59806305

[BillP]

Hi,

The pcnsconfig.ini show that shutdown will start when the UPS reaches low battery. If you would like the shutdown to start earlier, you should go to configure events in the PowerChute web interface and select UPS on battery or runtime remaining below threshold. Click on shutdown, then select enable shutdown, and enter the delay. When selecting UPS on battery the delay will be counted down and then the shutdown process will begin. When selecting rumtime remaining below threshold the UPS will wait until the runtime falls below the set delay and then the shutdown process will begin. 

Something else I noticed is PowerChute is losing communication with vCenter Server. This may cause issue with shutdown. It would be helpful if you would first correct the comm issue with vCenter. Then stop the PowerChute service, move debug.log, error.log, EventLog.txt from the group1 folder to a new folder. Then edit log4j2.xml so the line reads  Once the change has been save to log4j2.xml restart the PowerChute service and run a shutdown test. Once the test has completed if you are still experiencing an issue zip these files error.log, EventLog.txt, VirtualizationFileStore.properties, pcnsconfig.ini, javaVersionFile.log, debug.log, and attach to this post.

After zipping the files stop the PowerChute service, edit log4j2.xml so that it reads  and then restart the service. You should not leave the application in debug mode.

When I reviewed the files attached earlier along with the lost comm with vCenter I also see available runtime has been exceeded. Since the shutdown process is beginning at low battery the low battery threshold would need to be 2 minutes greater than the time needed to power down all the VMs and the hosts. The low battery threshold is configured via the NMC web interface under Configuration, Shutdown, Low battery duration. The default is 2 minutes. 

Do you have vApp configured? if not you should remove the 1140 second delay for vApps to power down and the 120 second delay for them to start up. I would then set the low battery duration on the NMC to 30 minutes to allow for the other delays that are set. 

Side note: ESXi 6.7 update 2 is not an officially supported version however it should work. 

[DaleMurray_apc]

This morning I pulled the plug on the UPS and prayed.

The UPS has about 70 minutes runtime and from what I gathered my shutdown timing should have started about 50 minutes into running on battery. When the UPS showed it had less than five minutes of available run time (about 7am) I chose to shutdown vCenter and another server to protect them from corruption.

All of the "Cannot connect to vCenter" is during the time when I shutdown vcenter followed by the battery dying on the UPS. The PCNS computer was up and running and could not connect because the whole vmware stack had to completely boot from power down.

What would be nice is if somebody over in support actually contacted me, setup a support session and reviewed my setup. Instead I get a bunch of leg work and links to PDF's.

[BillP]

Hi,

One more thing you should do before running another shutdown test is to edit C:\Windows\system32\drivers\etc\hosts and add the hosts IP addresses and domain names. The error log is showing java.net.ConnectException: Connection refused:  for the hosts and vCenter. This could be caused PowerChute not being able to contact the DNS server. By adding the domain name and IPs to C:\Windows\system32\drivers\etc\hosts the issue will be resolved. 

[DaleMurray_apc · Accepted solution]

The debug log I provided begins on 6/23/19
The first instance of java.net.ConnectException: Connection refused occurs at 6:59am on 6/25/19 which is just about the same time I powered down vCenter.

The event log I provided begins on 6/19/19
The first instance of Cannot connect to vCenter Server. occurs are 6:59am on 6/25/19 which is just about the time I powered down vCenter.

All of those connection issues came AFTER I manually shutdown vCenter as it was clear PCNS was not going to. The reason it continued for an hour is that is how long it took to bring things back online.

Available runtime has been exceeded. errors.
Those were a regular occurrence until 6/21 when I ran a calibration cycle. Once calibration cycle completed I no longer see the error.
The error shows up again 6/25/19 after I had depleted the battery from that mornings test. Reviewing current log I see if has happened again this morning - I assume I would need to run a calibration again to clear the error?

Group 1 Power Off Delay is 1260 seconds (21 minutes); significantly less than the nearly 75 minutes of available runtime.

So far all this shooting back and forth on a forum has been a total waste of time.

Yesterday I was assured by my CDW rep I would receive a call from APC support before 3pm - never happened.
I did receive a call at 4:44pm yesterday, which is nearly an hour after my normal exit and nearly 2 hours after the time I was supposed to receive a call. Arvin did leave a message telling me support would contact me today - phone has yet to ring.