UPS Communication Lost on ESXi

[mrteo_apc]

This was originally posted on APC forums on 3/13/2013

---

Hi,

I'm having this problem with my SMT1500RMI2U.

It is connected to my ESXi host via the provided Rj45->serial cable.

3 weeks ago I installed powerchute business agent on vMA following this guide: http://www.apc.com/support/index?page=content&id=FA165747&actp=search&viewlocale=en_US&searchid=1363...

Everything was working fine. I logged in to :6547 and set it up.

But, a couple of weeks ago I started having some problems.

When I manually shut down the host, the UPS seems to freeze.

-The load percentage does not go down to 0%, even disconnecting host's power cables.

-The lcd menu doesn't let me make any changes (can't shut down ups, can't start a test, can't reset to factory defaults).

The only way to reset it is to disconnect the power cable and the battery.

Also, on the PBEAgent web page I just get the "UPS Communication Lost" message.

There is no way to access it.

I don't know what to do!

I tried to reset the UPS to factory defaults, I tried to reinstall the PBEAgent, I even removed and reinstalled the whole vMA Appliance.

But nothing changed.

I cannot even understand why this is happening!

Everything was working fine.

The only "strange" thing I can think of is that a couple of weeks ago I decided to try if the shut down process was working.

So I started it from the PBEagent web page but it did not work, it just powered off the outlets.

Can someone please help me?

This is driving me nut!

Thank you!

[mrteo_apc]

This was originally posted on APC forums on 3/13/2013

---

Hi,

I'm having this problem with my SMT1500RMI2U.

It is connected to my ESXi host via the provided Rj45->serial cable.

3 weeks ago I installed powerchute business agent on vMA following this guide: http://www.apc.com/support/index?page=content&id=FA165747&actp=search&viewlocale=en_US&searchid=1363...

Everything was working fine. I logged in to :6547 and set it up.

But, a couple of weeks ago I started having some problems.

When I manually shut down the host, the UPS seems to freeze.

-The load percentage does not go down to 0%, even disconnecting host's power cables.

-The lcd menu doesn't let me make any changes (can't shut down ups, can't start a test, can't reset to factory defaults).

The only way to reset it is to disconnect the power cable and the battery.

Also, on the PBEAgent web page I just get the "UPS Communication Lost" message.

There is no way to access it.

I don't know what to do!

I tried to reset the UPS to factory defaults, I tried to reinstall the PBEAgent, I even removed and reinstalled the whole vMA Appliance.

But nothing changed.

I cannot even understand why this is happening!

Everything was working fine.

The only "strange" thing I can think of is that a couple of weeks ago I decided to try if the shut down process was working.

So I started it from the PBEagent web page but it did not work, it just powered off the outlets.

Can someone please help me?

This is driving me nut!

Thank you!

[BillP]

This reply was originally posted by Jonathan on APC forums on 3/13/2013

---

Hi,

Using the PCBE Agent interface or the LCD menu of your SMT unit, can you please grab the firmware of your UPS.  Also, what version of ESXi and vMA are you using?

[BillP]

This reply was originally posted by Bill on APC forums on 3/13/2013

---

Hi,

To troubleshoot you should first uninstall PCBE, disable the serial port on the vMA, remove the serial cable, and reboot the vMA. Then test the manual shutdown to see if the UPS still hangs. If it does not hang then reattach the serial cable, re-enable the serial port on the vMA, and the run the manual shutdown test again. If the UPS hangs that would indicate a comm issue between the serial comm port and the UPS. If it does not hang then reinstall PCBE on the vMA, be sure that all outlet group are configured to shutdown via the outlet sequence page. Sent the Shutdown Setting - When power fails - After the UPS has been on battery for 60 second; this will start the OS shutdown in 60 seconds. Cut the power to the UPS / pull the plug. The vMA should shutdown in 60 seconds along with the host and outlet groups will begin powering down depending on there delay settings (see outlet sequence for settings). Once all the outlet groups have powered down the UPS will go into sleep mode for approximately 9 minutes then power down completely to preserve battery life / power. If power is restored after the UPS goes into sleep mode or powers down the outlet groups will be powered back up depending on the settings applied to each group. If this test works then you should test the manual shutdown again.

If you still have an issue we will need to know how the manual shutdown is performed e.g. pushing the power button, via RCLI along with the info Jon P requested.

[mrteo_apc]

This was originally posted on APC forums on 3/13/2013

---

Hi,

I'm using latest versions of both ESXi and vMA

Firmware:

-UPS 06.5

-MCU 11.0

-UBL 08.2

-MBL 11.0

@bill

I'm sorry but I don't get it...

What do you mean for "manual shutdown"? Are you talking about ESXi host or the UPS?

If you are talking about the first, well, it works, as I'm shutting down the machine almost every day, because I'm still testing it. But I shut it down from vsphere client, not through the PCBE agent.

If you are talking about the UPS, well, it doesn't work. It hangs every time I power off the ESXi host. Even now, I just powered off the machine, but the UPS's lcd still tells me I have a 27% load. And the only thing I can do is to unplug the battery and the power cord.

Thanks for your support!

Matteo

[BillP]

This reply was originally posted by Bill on APC forums on 3/14/2013

---

Matteo,

You wrote "When I manually shut down the host, the UPS seems to freeze." You should follow those same steps with PCBE uninstalled from the vMA, with the serial cable connected and the serial port configured on the vMA. If the UPS hangs with this configuration that would indicate a comm issue between the serial comm port and the UPS.

[mrteo_apc]

This was originally posted on APC forums on 3/17/2013

---

You're right Bill. I uninstalled PCBE and tried first with the cable plugged in.

The UPS hangs.

I then "reset" the UPS and tried with the cable unplugged and everything worked fine.

So, looks like you identified the problem, but...what is causing this? And, what am I supposed to do now? How can I solve this?

Many thanks for the help

Matteo

[BillP]

This reply was originally posted by Bill on APC forums on 3/18/2013

---

Matteo,

Try this, before shutting the host down power down the vMA and then power down the host.

[mrteo_apc]

This was originally posted on APC forums on 3/18/2013

---

Thanks Bill

Just tried. The UPS is still hanging

[mrteo_apc]

This was originally posted on APC forums on 3/25/2013

---

Update:

I tried to change ESXi License switching from evaluation mode to a free license.

And WOW, everything was working. I was able to "see" the UPS and configure it.

So, I decided to try again the manual shutdown from PCBE Agent WebUI (Shutdown->Shutdown now)

Well, it did not worked. It just cut off power to the outlets. And after rebooting, communication with UPS was lost, the UPS was frozen again, etc...

So I tried again to switch licenses and reboot, but nothing. I'm back again to the same problems.

This is really driving me nut!

[BillP]

This reply was originally posted by Bill on APC forums on 3/26/2013

---

Matteo,

This is very strange! What is the hardware the the ESXi OS is running on and are there other VMs running? I just ran a test with ESXi 5.0 with vMA 5.1 and told set PCBE to start a shutdown in 30 seconds with a manual restart. First the vMA shutdown, then the host shutdown and finally all 3 outlet groups powered down.

[mrteo_apc]

This was originally posted on APC forums on 3/26/2013

---

Hi Bill,

I know this is weird! I can't figure out where the problem is hidden.

What happened to me is:

-Set shutdown to start in 5 secs

-Set shutdown time to 120 secs

-Set reboot delay to 30 secs

-Shutdown procedure started

-After 120 secs no VM was powered off nor was shutting down and the outlets powered down

-After 30 secs the UPS and the host machine started and then UPS communication was down again

I also had to reconfigure the license because (i think due to the bad shut down) it came back to the evaluation license

My ESXi 5.1 is running on a Dell PE2950:

Processor: 2 x 1.86GHz QUAD CORE INTEL XEON L5320

Memory : 16GB (8x2GB)ECC DDR2 FB-DIMM 667Mhz

Raid Controller : PERC 5i Raid controller

Graphics : embedded ATI ES1000 WITH 16MB OF SRAM

Networking : embedded Dual Gigabit Ethernet

There are 3 VMs configured to auto power on:

-vMA

-vCMA

-Debian

There are also 2 other VMs on stand by.

[BillP]

This reply was originally posted by Bill on APC forums on 3/27/2013

---

Matteo,

You have multiple issue happening. First, PCBE will not be able to shut the host down properly unless the host is licensed. VMware ESXi 5.x does not support remote command line (rcli) with the free versions and PCBE uses rcli to communicate shutdown commands to the host. The second issue is the UPS hanging when the host is powered down even without PowerChute installed. That indicates there is an issue with the serial port. My guess is when the host starts to shutdown it is sensing something on the serial port and attempting to communicate with it causing the UPS to hang.

You need to get the ESXi licensed and then PCBE should be able to shutdown the host. The host with then command the VMs to shutdown. We have a Kbase on the subjects FA159705 and FA159688.

I will ask our engineering team if anyone has seen or heard of an issue with Dell systems causing a UPS to hang when attached via serial.

On a side note, you could purchase a network card for the SMT and use PowerChute Network Shutdown (PCNS) instead of PCBE. This would alleviate the serial comm issue. You will still need the license from VMware for PCNS to work since it relies on rcli also.

[mrteo_apc · Accepted solution]

This was originally posted on APC forums on 3/29/2013

---

Bill,

Before registering it with a free license, ESXi was in evaluation mode. As far as I know, in evaluation mode you have all the features enabled.

BTW, the biggest problem is that I cannot even manage the UPS because of the "communication lost" issue...

I also just got my UPS replaced by italian tech guys, but the same problems are showing up: no communication, UPS hanging...

I know about the possibility to purchase a network card, but I'm almost out of budget.

-----EDIT----

I tried again with this:

-unistalled pcbeagent

-switched back to evaluation mode on ESXi

-turned off all VMs except vMA

-rebooted the host

-connected the UPS to serial port and installed pcbeagent

At this point I was able to access the webui of pcbe.

Configured everything and tried the shutdown procedure.

I unplugged the UPS from power and the procedure started.

This time everything worked fine, vMA shutted down and then the host shutted down gracefully.

But, when i powered it up, there was again that "Comm lost" message. And the UPS was still frozen!

Well, I really don't know what to do anymore...

I'm attaching the log file found in /opt/APC/PowerChuteBusinessEdition/Agent/log/

Hope this helps you identify the issue...

------EDIT 2-------

The issue seems to come up in the startup phase.

I tried to connect the serial cable after everything was powered on and it works.

Attaching another log (pcbe2.log) to let you see what happens.