PCNS Install on vSphere 5?

[stanj]

{color:#000080}I{color}{color:#000080} am setting up the APC to use PCNS to shut down VMs in vCenter.

I am running vSphere 5.0.

I installed the vMA app in vCenter on the ESXi Server.

Do I install the PCNS v3.0.1 onto the vMA, then unzip and run ./install.sh?

Any particular steps required to have the VMs shut down with PCNS?

thanks{color}

[BillP]

you'll need to install PCNS on the vMA as far as I know.. check FA159780 @ http://www.apc.com/site/support/index.cfm/faq/


here is information on shutting down your VMs - refer to FA159688 @ http://www.apc.com/site/support/index.cfm/faq/

[stanj]

{color:#000080}I received an alert that the question was updated, but I do not see the response in the forum,,

In ant case, {color}
{color:#000080}I installed the PCNS on the vMA.{color}



{color:#000080}From what I can tell, it is up and running.{color}


{color:#000080}I was able to log into the PCNS GUI by entering the IP of the vMA and port number 6547
However, I was not sure of the selection for outlet group during the configuration.
I selected outlet group 1.{color}



{color:#000080}What is the difference between UPS Outlets and Outlet Group 1?{color}



{color:#000080}From what I recall of the hardware we set up, a PDU was plugged into one of the UPS outlets, but I do not remember which one.{color}



{color:#000080}The esxi server running the VMs was then plugged into the PDU.{color}


{color:#000080}Is this ok?{color}




{color:#000080}Is there a way to test if the VMs will shutdown in case of low battery power?

thanks
{color}

[stanj]

i just saw the response now,,stange that it appeared after I reposted

[BillP]

there is no test feature really. i'd recommend doing a pull plug on the UPS if you can at all possible to simulate an actual outage. did you do any configuration on the PCNS client you installed on the vMA and accessed via port 6547?

[stanj]

{color:#000080}thanks,

yes,

i installed PCNS onto the vMA and followed the attached doc.
i can access PCVS via port 6547.

i can pull the apc plug that connects to the 30 amp wall plug..

also,
what software would I install on a Dell PC runing Windows 7 (windows 7 home premium) to have it shut down wehn the UPS battery is drained?{color}

[BillP]

so is this Windows 7 PC connected to the same UPS? if so, we have powerchute network shutdown for windows which would be what you'd want to install.

if you go to http://www.apc.com/tools/download/index.cfm and filter by software/firmware (the top drop down), you can find software upgrades - powerchute network shutdown and then download the windows version.

[stanj]

{color:#000080}ok,

thanks,,

i have a unique problem now..

the set up is,,
i have the APC NMC connected to our dell 5524 switch along with other hadware.
ESXi Servers, PC,, etc.

When I attempt to make a change in the PCNS via the web interface, and save, i get

{color:#ff0000}*Explorer cannot display the webpage ERROR: Cannot connect to the NMC. Please verify that the NMC is accessible* and that the server may no longer be protected

{color}When I go to the PCNS logs and click on the IP for the NMC, I get directed to a vSphere ESXi download page,,as if there is a redirection to our ESXi Server.

I tried moving the NMC connection in port 5 of the switch to port 24 and placing it a diffrent VLAN, but it did not help.

It is also seems to be intermittent becasue I can get to the APC NMC via IP once in a while.

Are there any requirements for connecting the APC UPS to a switch?

thanks{color}

[BillP]

hmm no. when you installed PCNS on your dell workstation, you need to input the IP address of the network management card in your UPS, not the IP of your ESXi server unless there is a conflict there. so when you go to PCNS, it gives you a hyperlink to your management card on the left (instead of your ESXi server like it sounds like it did..weird).

the requirements are just IP, subnet mask, gateway. beyond that, http, telnet, ftp, and SNMPv1 are on by default..

what does the PCNS log show on your dell workstation? can you paste it here?

[stanj]

The PCNS is installed on the vSPhere VMA appliance VM, not a Windows Workstation.
Below is the log. When I click on the IP, it takes me to the vSphere ESXi client down load page.
I am going to connect directly to the NMC today to verify the IP.

-----------
Date Time Event
11/01/2012 16:11:27 ERROR: Cannot connect to the NMC. Please verify that the NMC is accessible using the URL: http://xxx.xxx.xx.xx:80
11/01/2012 16:11:27 PowerChute cannot communicate with the Network Management Card
11/01/2012 13:56:13 ERROR: Cannot connect to the NMC. Please verify that the NMC is accessible using the URL: http://xxx.xxx.xx.xx:80
11/01/2012 13:24:29 ERROR: Cannot connect to the NMC. Please verify that the NMC is accessible using the URL: http://xxx.xxx.xx.xx:80
11/01/2012 13:07:47 ERROR: Cannot connect to the NMC. Please verify that the NMC is accessible using the URL: http://xxx.xxx.xx.xx:80
11/01/2012 10:54:37 ERROR: Cannot connect to the NMC. Please verify that the NMC is accessible using the URL: http://xxx.xxx.xx.xx:80
10/31/2012 19:44:22 ERROR: Cannot connect to the NMC. Please verify that the NMC is accessible using the URL: http://xxx.xxx.xx.xx:80
10/31/2012 18:02:59 ERROR: Cannot connect to the NMC. Please verify that the NMC is accessible using the URL: http://xxx.xxx.xx.xx:80
10/31/2012 17:40:12 ERROR: Cannot connect to the NMC. Please verify that the NMC is accessible using the URL: http://xxx.xxx.xx.xx:80
10/31/2012 17:38:42 ERROR: Cannot connect to the NMC. Please verify that the NMC is accessible using the URL: http://xxx.xxx.xx.xx:80
10/31/2012 17:38:38 ERROR: Cannot connect to the NMC. Please verify that the NMC is accessible using the URL: http://xxx.xxx.xx.xx:80
10/31/2012 17:37:26 ERROR: Cannot connect to the NMC. Please verify that the NMC is accessible using the URL: http://xxx.xxx.xx.xx:80
10/29/2012 19:50:20 Input power has been restored: UPS has switched to On Line operation
10/29/2012 19:50:19 UPS has switched to battery power
10/28/2012 23:43:05 Communication has been established
10/28/2012 22:07:27 PowerChute successfully opened TCP port 6547.
10/28/2012 22:07:26 PowerChute successfully opened TCP port 3052.
10/28/2012 22:07:25 PowerChute Network Shutdown version 3.0.1 started
10/28/2012 22:07:25 PowerChute successfully opened UDP port 3052.

[BillP]

ok, maybe i got confused.

you asked for PCNS for Windows 7 so I thought it was for a separate computer besides your ESXi host/guest VMs.

generally this error has to do with blocked ports for one reason or another, specifically I think UDP 3052 is what maintains communication.

[stanj]

I checked with our IT firewall staff,,
they said the ports are open.

I rebooted the APC and it now works. I can connect to PCNS running on the vSphere VMS and the UPS.

I do need PCNS for a Windows PC that is connected to the APC UPS.

[BillP]

ok, so you can go to the link i mentioned before and download PCNS 3.0.1 for Windows and install it the same way.

[stanj]

yep,

i downloaded 3.0.1 and installed it on a PC running Windows 7 and configured it.

how exactly will the PC shutdown once the UPS is at a low battery level?

is there a way to send out email to users?

[BillP]

any email sending option gets sent directly for the UPS Network Management Card. PCNS cannot send email - you can configure shutdown or configure a script to run to possibly do this type of thing. we support using Messenger Service on Windows OSs if the service is available and enabled as well.

if you check the PCNS help file, there is a good description of how this whole thing happens. In a short explanation, the UPS sends a "hey i am at low battery" signal and PCNS will see that message via the network and begin and OS shutdown. there are are settings you can change and configure to alter that basic timeline though and it is noted in the help file within PCNS.

[stanj]

ok,

how do you use Messenger Service on Windows OSs to notify a user?
Windows 7 has no messenger service

[BillP]

well if its not there, or there and just disabled on the newer Windows Operating systems, then it won't be an option. you'd need to try and do it a different way with some type of scripting with another utility or just rely on the network management card email options.

[stanj]

ok,

the only option I have seen is to install an instant messaging program or use Windows Live Messenger

will Windows Live Messenger work?

[BillP]

PCNS does not have any direct integration with Windows messenger. the only option might be to issue a command file to script some type of message via Windows messenger if that is even possible.

[stanj]

i was testing the PCNS shutdown.

I set the VMs startup and shutdown according to the guide in the vSphere client.
I logged into the PC that is connected to the UPS.
I also was monitoring the VMs from vSphere client.


I launched the PCNS gui interface on PORT 6547.
I launched the APC gui interface connected to the NMC.

I pulled the plug and monitored the power.
The PCNS gui reported a log message – switched to battery power.
The APC gui showed the minutes left and % battery.

When the APC gui showed 4 minutes left of battery power, a critical message was shown in th APC gui and
the PC that was connected to the UPS received a PCNS shutdown command and the OS shutdown in a normal fashion.

the VMs were not notified…..

When I tried to check the PCNS gui interface on 6547, I get an IE message Internet Explorer cannot display the webpage
Like as though the PCNS process on the vMA was stopped/killed when the critical 4 minute message was sent?

The APC gui showed 0% battery ...so I reconencetd the APC UPS to the power outlet and the battery charged.


Any idas?

[stanj]

additional info ,,

PCNS is installed on the vMA running on the ESXi Server,,

wondering if PCNS is compatible with vMA on ESXi 5?

[BillP]

yes, its compatible.

http://swhelp.apcc.com/pcns/help/troubleshootingguide_EN/

at that link, is a link to the compatibility chart and other helpful documents.

it'd be helpful if you can provide the NMC event log and event log from the PCNS client. did you verify the set up on getting the VMs to shutdown per our knowledge base artlce I sent? that is not part of what our software does so i don't personally know much about it.

[stanj]

ok,

where are the NMC event log and event log from the PCNS located?

[BillP]

for the NMC, you can download the logs per these instructions please -> http://www.apc.com/site/support/index.cfm/faq/ -> FA156131

for PCNS, log in via the PCNS web interface (https://:6547, and you'll see a log available in there (it is the default start page on mine - "View event log") that you'll need to copy and paste. i don't think there is an easier way to export it.

[BillP]

ok, let me clarify after double checking on my end. PCNS installs on the vMA and uses remote command line (RCLI) to tell the host the shut down. The host is configured to shut the VMs down. Then you install the vmware tools on all the VMs per the kbase I mentioned and you followed. Also, if the host is not licensed (kbase FA159705) or if you have not installed the PCNS patch (mentioned in kbase FA159783), the host will not shut down hence the VMs will not shut down.

[stanj]

I have the log info,,

The VMs did not shut down. I was monitoring the battery % via the NMC gui. Even when it was at 0%, the VMs did not shut down
As I stated earlier, I was connected to the PCNS gui and monitoring the log while the battery was being drained and then lost contact with the PCNS software.

The PCNS software stopped running as if when it received a command to shut down the VMs but the PCNS software exited.
PCNS software did not start until I rebooted the vMA.


this is strange,,,The VMs don't seem as if they were contacted by PCNS

[BillP]

PCNS does not contact the VMs, that is why. You need to configure ESX itself to initiate the VMs shutdown as mentioned in an earlier knowledge base (FA159688 @ http://www.apc.com/site/support/index.cfm/faq/). The PCNS software only shuts down the host ESX operating system.

[stanj]

I have that set (see atatched).

I thought vMA was used to shut down the VMs on ESXi Servers with PCNS installed on the vMA?

[stanj]

ok,

the host is licensed

i did not see the "PCNS patch" listed in the installation guide for PCNS Network Shutdown Installation for ESXi.

Was it omitted?

I will try it out this week..

thanks

[BillP]

no problem. i don't think the patch is mentioned in the install guide (which maybe it should be) but i think it was just a reaction to when VMWare changed something which caused this issue.

[stanj]

ok,,

question,

I will be logged into the APC and PCNS gui's (attached).

Once I pull the plug on the UPS and the battery begins to take over, based on the attached screen of the Battery Capacity, when should the ESXi and VMs begin to shut down at what %?

[BillP]

ok, i think we are seeing issues with ESXi itself.

it appears PCNS is executing the RCLI command to do the shutdown and ESXi is not accepting it for some reason.

when the NMC reaches a low battery, that means any client left running will be forced to shutdown. that is what seems to have occurred here which is normal. UPS is running low on battery and says to everything listening that it needs to shutdown. so then the vMA is trying to issue the RCLI command to ESXi and ESXi is not accepting it (apparently due to permissions).

i suggest that you maybe try issuing a RCLI command from the vMA outside of PCNS to verify it works. but there is going to be no shutdown until we understand why the RCLI is not being accepted on the ESXi box from the vMA.

another thing you can try is Vicfg-nics –l and if nothing is there, then you need a license.

beyond that, i dont know anymore regarding ESXi itself and you may need to contact VMWare or research that side of it more.

[BillP]

well, i can't say 100% by looking. also, it reports 1 hr 38 minutes there. when the UPS goes to battery, it provides a more accurate runtime report. while it is online, that is a very good working estimate.

a few questions i need to confirm before i can make a guess on that - have you configured anything to shut down the client under Configure Events (i.e. UPS on battery)? If so, what is the amount of time there. let's say it is 35 minutes. after 35 minutes of on battery time (assuming you actually get 1 hr 38 minutes exactly) the UPS should be around ~64 because it will have about 63 minutes left to go. i think I did that correctly 🙂

if you have NOT configured anything there, the UPS will force clients to shutdown when it hits its low battery duration (a default of 2 minutes) which would probably a very low percent of battery..

[stanj]

There are no Events configured, all events have the “log event” checked.

Back in this chain of postings, I described what I saw when I pulled the power.
I was logged into the APC and PCNS guis’. When I pulled the plug of the UPS and it switched to battery power, I monitored the time and it reported 45 minutes of battery power remaining in the APC gui. PCNS gui reported - switched to battery power at 17:24 on Nov 9 (see attached screen).

At about 4 minutes of battery power remaining, the PCNS software must have shutdown on the vMA because I lost connection to the GUI and could not reconnect on port 6547. When the APC gui reported 0 min of batter power and the ESXi Server and VMs were still running, I reconnected the APC to the power. There is a message (see attached screen) at 17:58 on Nov 9 stating “Shutdown Process Started” but the only system to shutdown was a physical PC that was running with PCNS 3.0.1

[stanj]

in addition to the last post,,


During the APC PCNS shutdown of the ESXi Server, will the VMs below be shutting down one after another or are they queued based on the delay time?
With PCNS running on the vSphere vMA VM (in attached screen), I set the delay to 300, will this cause issues when PCNS is notified by the APC UPS?

Will the order shown in the attached allow the VMs to shut down?

[BillP]

how long does your ESXi host take to shutdown normally when we aren't dealing with PCNS or the graceful shutdown?

you need to make sure that you either raise the low battery duration on the NMC or add a delay on the PCNS client side to assure your server has enough time to perform a shutdown and tell the VMs to shutdown.

i am not sure on your other questions because i don't know ESXi really well. what i can tell you is that PCNS running on the vMA sends a remote CLI command to the ESXi server to shutdown..so however that works and what happens with the delays set on ESXi is how it'd go.

I think that you need to first confirm how long the ESXi server takes to shutdown with all of the VMs you have outside of PCNS. then knowing that, you can fix your configuration on PCNS - for example, maybe set the PCNS client to initiate a shutdown after 30 minutes on battery so that there is 15 minutes remaining for the shutdown of the ESXi server and VMs to complete.

[stanj]

ok,,
i enabled the PCNS log and tried again,,
i get the infow below in the log.

It may have to do with vSPhere ESXi access.
I have the ESXi console disabled but I do not think that is the issue.
I can SSH to the ESXi Server and login..
Are there other ESXi settings that need to be checked,, like:

I/O Redirector (Active Directory Service)
Network Login Server (Active Directory Service)
lbtd
vpxa
ESXi Shell
Local Security Authentication Server (Active Directory Service)
NTP Daemon
SSH
Direct Console UI
CIM Server

Also firewall ports?

-----
vi-admin@redd_vma:/tmp> more *PCNSHostShutdown.log
*--30.11.2012, 13:11:07--debug: Connecting to server localhost
--30.11.2012, 13:11:07--debug: Login by vi-fastpass to: our esxi server
--30.11.2012, 13:11:10--debug: Error: Unable to login to host: "our esxi server"! Host may not be accessible over the network or is not managed by VMA
--30.11.2012, 13:11:10--debug: Login by vi-fastpass to: our esxi server
--30.11.2012, 13:11:16--debug: Error: Unable to login to host: "our esxi server"! Host may not be accessible over the network or is not managed by VMA
Connecting to server localhost ...
Operation :shutdown

[stanj]

esxi 5 takes about 5-10 minutes to shutdown if the VMs are off.
the problem is, they are on and there may be warnings.
typically, you place the esxi server in maintenance mode and then shutdown VMs

where (what menus) allows me to
raise the low battery duration on the NMC
and
add a delay on the PCNS client side

thanks

[stanj]

ok,

i found the LOW low battery duration fiield on the GUI for the NMC and raised it to 10 min

there is also a Basic Low Battery Duration filed ,,but I did not set it.

i pulled the plug on the UPS and now have 1hr 18 min to wait,,,

i have all of the VMs set to shutdwon with system and are in manual

[stanj]

no go, I had to plug in the UPS.

at 8 minutes or so, the NMC GUI reported
,,,The battery power is too low to support the load; if power fails, the UPS will be shut down immediately.

I was monitoring the ESXi Server events via the vsphere client and got this error:

Cannot login user @xxx.xx.xx.x: no permission
(where xxx.xx.xx.x is the IP of the vMA where PCNS is installed.

Now, I cannot access PCNS via xxx.xx.xx.x:6547.
i get back,,,, Internet Explorer cannot dispaly the webpage.

same issue I saw last time,
I will need to restart the PCNS or reboot the VMA to get PCNS running again.

not sure what permisson or what process is trying to login ? PCNS?

[stanj]

please define what RCLI command i can execute from the vMA

also,
from the vMA I issuesd the following:


vi-admin@redd_vma:~> vicfg-nics -l
Enter username: vi-admin
Enter password:
Error connecting to server at 'https://localhost/sdk/webService': Connection refused
vi-admin@redd_vma:~>

[BillP]

the command to run the shutdown would be - cd /opt/APC/PowerChute/ then type sudo bin/shutdown - i am not sure what the actual linux command is though unfortunately since its embedded in a perl script..

from what i can tell based on the response from vicfg-nics -l, you need to find out why your vMA and ESXi are not communicating properly. so you've verified that you're licensed and you've applied the patched we discussed a bit ago? the server is added to the server list?? (checked by issuing vifp listservers if it is not there, then adding the server by vifp addserver )

another thing to try is enabling debug mode in this file: /opt/APC/PowerChute/group1/shutdown

if you go into the vi editor and edit this file, there is a line in this file that allows you to enable debug mode while executing the shutdown. you can comment out the normal line that is active now and then enable the line that allows for debugging if you wanted to.

[stanj]

ok,

as a precaution, I reinstalled vMA and PCNS and copied the patch over,,

I did not find shutdwon in /opt/APC/PowerChute

I do have the one that is patched in /opt/APC/PowerChute/group1/bin


what will issuing the shutdwon do?

thanks

[stanj]

update:

found some info on VMware forum, but not sure it makes sense,

If I issue the vicfg-nics –l, I get

Error connecting to server at 'https://localhost/sdk/webService': Connection refused

Forum states,,

sudo vifp addserver
Then run,

vifptarget -s

Only then will the 'vicfg-nics -l' command will work without additional options required.

I tried this and vicfg-nics –l works,
but when I reboot vMA and try the vicfg-nics –l, I get the same

Error connecting to server at 'https://localhost/sdk/webService': Connection refused

[BillP]

sorry, yes the one in the bin folder. this will basically execute the shutdown that PCNS does without you having to do it by pulling the plug on the UPS. so maybe it won't be all that helpful.

as far as your second post here, it sounds like a licensing issue but i am not sure. it does not sound like something with the PCNS software..

when vicfg-nics –l worked, what did it list there?

[stanj]

issuing,,,
vicfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 01:00.0 bnx2 Up 1000Mbps Full 78:2b:cb:47:bf:36 1500 Broadcom Corporation Broadcom NetXtreme II BCM5716 1000Base-T
vmnic1 01:00.1 bnx2 Up 1000Mbps Full 78:2b:cb:47:bf:37 1500 Broadcom Corporation Broadcom NetXtreme II BCM5716 1000Base-T
vmnic2 07:00.0 bnx2 Down 0Mbps Half 00:10:18:f3:c3:c0 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T
vmnic3 07:00.1 bnx2 Down 0Mbps Half 00:10:18:f3:c3:c2 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T
vmnic4 08:00.0 bnx2 Down 0Mbps Half 00:10:18:f3:c3:c4 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T
vmnic5 08:00.1 bnx2 Down 0Mbps Half 00:10:18:f3:c3:c6 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-T
vi-admin@redd_vma:~[esxi-castle]>

but, if I rebbot the vMA and log back into the vMA and try again, I get

Enter username: vi-admin
Enter password:
Error connecting to server at 'https://localhost/sdk/webService': Connection refused


the only way i works again is if I enter vifptarget -s

[BillP]

from our instructions, you should only need to do the vifptarget -s once and not have to worry about re-doing it after each time if you reboot.

i bet that after you do that and then do vicfg-nics -l and everything looks good (as it does in your post there with all of the NICs listed) PCNS will work properly.

at this point, i am not sure what to do next - it appears to be an issue with ESXi as to why when you reboot, your server is being removed from the server list (and thus not communicating properly to be able to issue the shutdown).

[stanj]

i did not see in the instructions where the vifptarget -s is issues,,I did see visp addserver
I am going to remove the server and add it back..


i am then going to try a test in about 30 minutes,,

either issue the shutdown cmd as you pointed out or
in the PCNS gui, checking off the shutdown feature on the "loss of communication", and then unplugging the Network Management Card ,,,E

[stanj]

i ran the shutdown command and the VMs began to shutdown.
the ESXi Server then shutdwon but I am not sure the VMs shutdown before the ESXi Server shutdown.

I am rebooting now

i may try PCNS gui,
check off the shutdown feature on the "loss of communication", and then unplugging the Network Management Card

[stanj]

looks like it is working,,
i tried a second test by pulling the UPS power cable from the wall,,

waited 50 minutes and the PCNS GUI log reported
--Low-battery condition has occurred
--Shutdwon process started. Linux will shut down soon

the ESXi and VMs were shut dwon.

A strange thing ,,,
the PC running windows 7 with pcns 3.0.1 for windows I have connected to the UPS, appears to have shut down also, but I do not see a long entry in the PCNS GUI Log.
the windows event log under system show event 13 (shutdown) at the indicated time,,,

Any idea why no entry was made?