Need help troubleshooting ESXi shutdown failure from powerchute vma client

[BillP]

I've been trying to get this working for a few days and it might be time to get some extra assistance.

I have a pair of new Dell servers running ESXi 6.5u1.

I downloaded the latest powerchute agent, pcbeagent-9.5.0-301.x86_64.tar and installed it onto a newly deployed vma on one of the ESXi hosts.

The vma is connected via serial cable to brand new APC SMT1000RM2UC.  The vma is communicating with the UPS.

I have both ESXi hosts as fastpass targets and showing successfully in the vma;

esxi1.honeycomb.local ESXi 6.5.0 6765664

esxi2.honeycomb.local ESXi 6.5.0 7388607

So, here's the problem.

I run a manual shutdown from the powerchute web interface.  When I do this, host # 1 shuts down gracefully and powers off, but host # 2 does not.  Host #1 has about 6 VMs installed including the VMA.  Host #2 only has a single VM and that VM is not running.  Both hosts have the same stop/start settings in ESXi.

How can I further debug this problem?  I don't see log entries for the powerchute agent in the VMA so that I can see if there was any bad interaction with the ESXi host.  Is there a way to execute the shutdown on the VMA via command line so that I can test the behavior on the ESXi host that isn't powering down, by monitoring its logs at the time I issue the shutdown command?

Now here's the extra odd wrinkle.  The very first time I executed the shutdown, both servers shutdown and powered off successfully.  Every other time, the 2nd server does not power down and if I access the console the server is still up.  No changes that I am aware of were made.

Thanks.

[BillP]

Tough time getting support via this forum it seems... can someone just confirm for me that shutdown of multiple hosts is supported by installation of a single vMA on a single host?

The documentation indicates it is supported;

Fastpass Target

In order to shut down an ESXi host using vMA, you must add the ESXi server as a fastpass target for that vMA.

1. Use the command:

         vifp addserver 
   

2. When prompted, enter the root password for the ESXi host.

3. Confirm that the ESXi host has been added as a target server with this command:

         vifp listservers
   

   NOTE: You are prompted to add the ESXi server during the installation. If you skipped this step, you must manually add the ESXi server as a target host after the installation completes.

   You can add additional ESXi hosts for shutdown if required by using the sudo vifp addserver command. 

[BillP]

Hi,

Shutting down 2 ESXi host using a single install of PowerChute Business Edition is not supported. You have a few options. The one I prefer is to add a network management card (part number AP9630) to the UPS and install the PowerChute Network Shutdown VM on each host or you could install VMware vMA on each host and install PowerChute Network Shutdown the vMAs. Another option is to add a serial port expansion card to the UPS (part number AP9624) and install VMware vMA on each host and then install PowerChute Business Edition on each vMA. 

I prefer adding the network card because is allow greater flexibility managing the UPS and allow the UPS / NMC to power down multiple hosts. The serial port expander does not offer management of the UPS and allows the connection of only 2 more hosts that will communicate with the UPS in simple signaling mode. 

Simple signaling is the recognition of whether certain pins of the serial port are high or low, and allows for 3 states only; on battery, low battery and UPS turnoff. PowerChute Business Edition will recognize these states and act on them. 

The built in serial port of the UPS is Smart signaling.  In smart signaling mode, the UPS and PowerChute communicate allowing the UPS to provide a large array of information to the software. you would need to connect one of the host to the Smart signaling port and 1 to the expander card if you chose this method. 

[BillP]

Thanks Bill for responding and your reply is really helpful in clarifying this.

I guess I am most confused by the documentation I used to set this up claiming that multiple hosts are supported, they just need to be added via sudo vifp addserver.

Since in fact it's not possible to shut additional hosts down from a "master" vMA perhaps that section of the documentation should be revised, would have saved me a lot of time.  It's still a mystery to me why the shutdown appeared to work a single time.

Thanks again for your time, looks like I need to order some additional hardware.

[BillP]

Hi,

I apologize. You are correct the manual does list adding multiple hosts via fastpass for shut down. I still prefer PowerChute Network Shutdown but PowerChute Business Edition should be able to command 2 host down from a single vMA. Have you tried to remove the second host from the fastpass list and re-adding it?  

[BillP]

Yes, that was the very first thing I tried (great minds think alike! ).

At this point, I can think of two possibilities.

#1, vMA hosted APC Business is sending the shutdown but the host is not shutting down properly.

#2. vMA hosted APC Business is doing something wrong and not sending the shutdown correctly.

If there was a log to review of what it was doing that would be a good place to start.  Another option is to review what the ESXi host is doing when the shutdown is executed but I've found the ESXi host logs difficult to look at.  To do that I would need to be able to run a command from the vMA that executes the shutdown to just the one host that I'm having problems with.

[BillP]

Hi,

The PCBE.log file shows the shutdown started but does not list the host that are commanded down since they are save by the vMA. You can check vmksummary.log to see if the host is getting the signal. 

2018-04-05T12:30:24Z bootstop: Host is powering off
2018-04-05T12:36:01Z bootstop: Host has booted

You can access vmksummary.log via web browser at https://host ip address/host

To test shutdown you can run the shutdown with this command from /opt/APC/PowerChuteBusinessEdition/Agent

sudo ./bin/shutdown 

to test just the one host you could remove the other from the fastpass list temporarily. vifp removeserver  

[BillP]

Bill,

This support thread (http://forums.apc.com/spaces/7/ups-management-devices-powerchute-software/forums/general/9982/pcns-3...)

includes a script that appears to execute the shutdown from the command line, this might be the key line;

 vicfg-hostops -o shutdown --force

If this vicfg is the actual executable that runs the shutdown I could potentially get it to run against my problem host and see if it shuts down properly when the shutdown is executed from the command line, vs. when APC software is trying to do it.

[BillP]

On 4/5/2018 8:43 AM, Bill said:

Hi,

The PCBE.log file shows the shutdown started but does not list the host that are commanded down since they are save by the vMA. You can check vmksummary.log to see if the host is getting the signal. 

2018-04-05T12:30:24Z bootstop: Host is powering off
2018-04-05T12:36:01Z bootstop: Host has booted

You can access vmksummary.log via web browser at https://host ip address/host

To test shutdown you can run the shutdown with this command from /opt/APC/PowerChuteBusinessEdition/Agent

sudo ./bin/shutdown 

to test just the one host you could remove the other from the fastpass list temporarily. vifp removeserver  

Thanks Bill, our posts crossed, I hadn't seen this reply when I posted.

This looks like exactly the information I need to get to the bottom of this, thank you.

[BillP]

On 4/5/2018 8:43 AM, Bill said:

Hi,

The PCBE.log file shows the shutdown started but does not list the host that are commanded down since they are save by the vMA. You can check vmksummary.log to see if the host is getting the signal. 

2018-04-05T12:30:24Z bootstop: Host is powering off
2018-04-05T12:36:01Z bootstop: Host has booted

You can access vmksummary.log via web browser at https://host ip address/host

To test shutdown you can run the shutdown with this command from /opt/APC/PowerChuteBusinessEdition/Agent

sudo ./bin/shutdown 

to test just the one host you could remove the other from the fastpass list temporarily. vifp removeserver  

Hi Bill.

I've made some real headway here and have a better understanding of the issue.

When I issue the shutdown command from the shell I can see that the shutdown happens so fast that the vMA is shutdown from ESXi before it has time to execute the shutdown to the 2nd host.

If I re-order the vifp list so that the primary host (the one that has the serial connection to the UPS) is 2nd in the list then the 2nd host gets the shutdown command and then after that command is completed the primary host gets the shutdown command, disconnects the vMA and almost immediately is shut down.

So, this is good!  However it leaves me with the question of what the "180 second OS shutdown timer" in the PowerChute software is doing.  Wouldn't it normally be good to do a non forced shutdown in order to give host VMs time to get gracefully shutdown by ESXi before the ESXi hastily shut down and powered off?

Thanks again for all of your help.

[BillP]

Hi,

On 4/6/2018 4:42 PM, Jason said:

So, this is good!  However it leaves me with the question of what the "180 second OS shutdown timer" in the PowerChute software is doing.

The 180 seconds is the delay that the UPS will wait before powering down the outlet group/s. From the web interface you will see the delay under Shutdown - Outlet Sequence If you selected the switched outlet group in PCBE as the outlet group the servers area connected to you will see the switch outlet group will wait 180 seconds and power down.  The main outlet group will wait 180 seconds for the switched outlet group to power down and then wait an additional 180 seconds to power down. 

[BillP · Accepted solution]

On 4/9/2018 8:25 AM, Bill said:

Hi,

On 4/6/2018 4:42 PM, Jason said:

So, this is good!  However it leaves me with the question of what the "180 second OS shutdown timer" in the PowerChute software is doing.

The 180 seconds is the delay that the UPS will wait before powering down the outlet group/s. From the web interface you will see the delay under Shutdown - Outlet Sequence If you selected the switched outlet group in PCBE as the outlet group the servers area connected to you will see the switch outlet group will wait 180 seconds and power down.  The main outlet group will wait 180 seconds for the switched outlet group to power down and then wait an additional 180 seconds to power down. 

Ah, that makes sense.  So I guess what I will need to do is see if there's a way to configure the shutdown action on the ESXi host to be more graceful or maybe I can modify the shutdown command that the script is sending so it gives the VM guests more time to shut down.

I haven't observed any problem with the "forced" shutdown that is issued to the ESXi box, but I'm concerned that in a production environment this could cause things to not be shut down gracefully, resulting in data loss.

I do have one small recommendation for the documentation against this, which is that it would probably be good to have some note when it indicates multiple fastpass targets can be configured that the ESXi host the vMA is running on needs to be the last server in the list or the script won't complete properly.