PowerChute cannot communicate with the Management Card - only on 1 server

[Anonymous user]

Greetings,

I am new here, but have been using PowerChute Network Shutdown for years. Today, I have encountered my first real stumper.

I work for a school district, and we have APC UPS's at nearly every school in our district (37) with Network Management Cards. Each site runs a VMware ESX 4.0 server with 3 or 4 virtual servers. In most cases, the VMs are one Windows 2008 server, one NetWare 6.5 server, and one SuSE Linux Enterprise Server (SLES) 10 server.

At one specific site, the NetWare and Windows servers have no problem communicating with the management card, but the SLES server keeps giving me "PowerChute cannot communicate with the Management Card" in the Events. When I installed PCNS, it registered with the NMC without any trouble, and the IP address of the VM showed up under "Clients" in the NMC's web management interface. I have uninstalled and reinstalled several times, each time making absolutely sure that my username, password, authentication phrase, IP address, and subnet mask are correct. The firewall is completely disabled on the SLES VM. I have upgraded to the latest firmware on the NMC (3.7.2; it had been at 3.5.5). I have tried deleting and manually re-adding the client IP through the NMC's management interface. I have not encountered any problems with PCNS under SLES at any other site; just this one.

I am not sure what else to check. Any suggestions?

Thanks! With much appreciation,

Bryce Newall
Systems Administrator
Poway Unified School District

[Anonymous user]

I am having the same issue.

After the 2.2.4 powerchute installs, it registers the IP with the NMC, but it still say cannon communicate with the NMC when you check the powerchute page.

I have used firebind, and port 3052 is open

[BillP]

yes, i agree. strange.i personally dont have any more ideas :-(. havent used SLES in some time and even so, this is beyond me.

[BillP]

when it cannot maintain communication but registers successfully, it makes me think that this a port issue.

here are the ports:

NMC ->PCNS for shutdown command, any other communication: TCP 3052
NMC and PCNS maintaining communication: UDP 3052
PCNS->NMC: TCP/UDP 80 (or whatever port the HTTP (not HTTPS) port is set to on the NMC)

it sounds like something may be blocking UDP 3052 so that they cannot maintain communication after registering.

is there anyway you can verify/deny that this is the case? maybe temporarily disable the firewall on SLES and if that works, we can configure it to allow the aforementioned port? all ports above should be enabled for bi-directional communication. i find that temporarily disabling any firewall will save us sometime rather than trying to allow it first, so we can verify its definitely the issue.

[Anonymous user]

Hi, thanks for your response.

I don't think it's a firewall issue. I mentioned in my original message that the SLES firewall is completely disabled. I should have said that it is permanently disabled. I didn't just disable it to try to troubleshoot this issue; I disabled it during installation, and verified through YaST that it is still disabled. (For what we use the server for, we don't need it.)

That being said, is there a way to test connectivity to UDP ports from a SLES machine? Testing TCP is easy, but I don't know how to test UDP.

Thanks!

[voidstar_apc]

>
I don't think it's a firewall issue.
>

You're probably right. I believe SLES 10 runs a 2.6 kernel with iptables built-in, so we can check for any firewall rules by issuing:

sudo iptables -L

If there are no rules, we should see something like:

voidstar@seaquest2:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

That being said, is there a way to test connectivity to UDP ports from a SLES machine? Testing TCP is easy, but I don't know how to test UDP.
>

Yeah, it's a little tricky since UDP isn't connection oriented. I would use a network sniffer like wireshark on the SLES box to check for traffic on UDP port 3052. You should see the NMC periodically sending UDP packets (port 3052 I believe) to either your machine or to a broadcast address.

Winna's suggestion to use netstat allows you to see if PCNS is listening.

sudo netstat -an|grep "3052"

You can also see which process is listening on that port (PCNS or something else)

sudo lsof -n|grep "3052"

[BillP]

hi - sorry about that. i did miss that part so i apologize.

only thing that comes to mind is a port scanner. is there an online port scanner you can try to check with? i guess it'd have to be one that allows you to try and probe a specific port.

we see these issues commonly and it generally ends up being a port issue so that is why i suggested it. is this VM bridged directly to the same network as the NMC? i assume so. i would say its something on the network possibly blocking the traffic but you do have other devices working so that is not likely.

what do you think about a port scanner? for open ports we usually try to check with the netstat command - does SLES have that? it shows open ports but i thought it'd be the active ports.

[Anonymous user]

Hi, sorry for the delay; I got swept up in a couple of other projects and am just now getting back to this.

I haven't figured out how to quote a message to be able to reply to specific points, but I'll answer voidstar's suggestions here:

I ran "iptables -L" and the output was exactly as you showed, i.e. there are no rules.

netstat -an|grep "3052" produced the following output:

tcp 0 0 0.0.0.0:3052 0.0.0.0:* LISTEN
udp 0 0 0.0.0.0:3052 0.0.0.0:*

lsof -n|grep "3052" produced the following output:

java 8970 root 5u IPv4 312152 UDP *:apc-3052
java 8970 root 7u IPv4 312564 TCP *:apc-3052 (LISTEN)

It looks to me like only PCNS is using port 3052. I'll have to talk with someone else here about using Wireshark (we do have it), since I'm not familiar with how to use it.

Thanks!

[Anonymous user]

Hello again,

Well, I figured out how to work Wireshark! (Go me...) I'm having it do a packet capture filtering for UDP port 3052. About once every 25-26 seconds, I'm seeing a packet originating from the NMC's IP address, and the destination IP is the broadcast address. That's ALL I'm seeing; no UDP traffic originating from the servers. I also tried expanding the filter to include both UDP and TCP traffic searching for Port 3052, and all I get is that same packet every 25-26 seconds.

Any ideas where to go from here?

Thanks!

-- Bryce

[BillP]

well i can tell you that packet is normal. that is the packet PCNS looks for to maintain communication with the network management card.so if you're seeing that, the NMC is doing what it is supposed to..

[Anonymous user]

i picKed a winna wrote:
well i can tell you that packet is normal. that is the packet PCNS looks for to maintain communication with the network management card.so if you're seeing that, the NMC is doing what it is supposed to..

Indeed. I don't have any doubts about the NMC. It's this one server that is totally perplexing me. I'm actually now having the same issue at a second site with the same setup. In both cases, the server that CAN'T communicate with the management card is a virtual SLES 10 machine running under VMware ESX 4.0. Both of these SLES servers were recently set up. I'm not having this problem at another site with an equally-recent SLES VM, though. The other VMs, one ea. of Windows 2008 and NetWare 6.5, have no problem. It's totally baffling me.

[Anonymous user]

i picKed a winna wrote:
yes, i agree. strange.i personally dont have any more ideas :-(. havent used SLES in some time and even so, this is beyond me.

I think what may help me is to have a better understanding of how the NMC and the PowerChute clients communicate. I set up my Wireshark capture to listen to anything going to or from the IP address of the NMC. I'm seeing the periodic broadcasts FROM the NMC, but nothing going TO the NMC, even from the machines that are working properly, even when I first start the service. If I ping the NMC, either from my own workstation or from one of the virtual machines, I can see that traffic register in Wireshark. This makes it sound to me like the NMC is the one doing all of the work, i.e. the PowerChute clients just sit there and listen for the periodic broadcasts, and if they don't get one after about 2 minutes, then they log that "PowerChute cannot communicate with the Management Card" error. Does that sound about right? If it is, it's ultimately not helpful, because it doesn't help me to determine where the communication breakdown might be. It seems that the only work done by the PowerChute client is the initial registration with the NMC during installation, which was successful. Perhaps something is blocking the broadcast from the NMC, but I think it's going to be difficult to determine what it is.

[voidstar_apc]

>
I think what may help me is to have a better understanding of how the NMC and the PowerChute clients communicate. \[...\] I'm seeing the periodic broadcasts FROM the NMC, but nothing going TO the NMC, even from the machines that are working properly, even when I first start the service. \[...\] This makes it sound to me like the NMC is the one doing all of the work, i.e. the PowerChute clients just sit there and listen for the periodic broadcasts, and if they don't get one after about 2 minutes, then they log that "PowerChute cannot communicate with the Management Card" error. Does that sound about right? \[...\] It seems that the only work done by the PowerChute client is the initial registration with the NMC during installation, which was successful.
>

That's correct. PowerChute initiates a connection to the NMC to register with it and to start a UPS shutdown. During normal operation, it monitors the NMC's periodic status updates.

So we know that the VM delivers packets to the SLES machine based on the wireshark results. We know that there's no firewall rules on the SLES machine. We could have a little more confidence that a userspace program can see the packets by running

nc -l -u -p 3052

and seeing if the packets show up, but I'll assume this works. We know that powerchute is running and is listening on the correct port.

All that seems to leave is the admin user phrase. Sometimes the problem is they don't match between the NMC and PCNS, but in your case you made sure the authentication phrase is correct. And if it wasn't correct, PCNS wouldn't work on your other servers.

That just leaves me with the possibility that Java isn't quite as run-anywhere as I might wish it were. I wish I could be more help.

[Anonymous user]

voidstar wrote:

nc -l -u -p 3052

and seeing if the packets show up, but I'll assume this works. We know that powerchute is running and is listening on the correct port.

I couldn't find "nc" at first, but after some YaST searching, I found it was actually spelled out "netcat" in this SuSE distribution. So, this is interesting. I stopped the PCNS service and ran netcat using the parameters that voidstar gave. I was still running the Wireshark capture. As soon as the NMC sent out the first packet (first after starting Wireshark and netcat), Wireshark saw it, and netcat saw it at the same time. However, netcat did NOT register any subsequent packets, even though Wireshark saw them. An even more interesting development is that if I pressed ENTER in the terminal window while netcat was running (which shouldn't do anything), Wireshark detected a malformed packet originating FROM the SLES machine, with a destination address of the NMC. Weird! I could hit ENTER 10 times, and each time, Wireshark would capture the same packet. I don't know why Wireshark would think the packet was destined for the NMC. This is getting a bit beyond my expertise, so I'll take any advice I can get! 🙂

Message was edited by: bnewall1

[voidstar_apc]

>
However, netcat did NOT register any subsequent packets, even though Wireshark saw them. An even more interesting development is that if I pressed ENTER in the terminal window while netcat was running (which shouldn't do anything), Wireshark detected a malformed packet originating FROM the SLES machine, with a destination address of the NMC. Weird
>

That's just how netcat works... it receives the first packet and sends anything you type back.

[Anonymous user]

voidstar wrote:
>
However, netcat did NOT register any subsequent packets, even though Wireshark saw them. An even more interesting development is that if I pressed ENTER in the terminal window while netcat was running (which shouldn't do anything), Wireshark detected a malformed packet originating FROM the SLES machine, with a destination address of the NMC. Weird
>

That's just how netcat works... it receives the first packet and sends anything you type back.

Ahh, I see. I wasn't familiar with netcat so I didn't know that's how it works. So if it's sending back what I type, and Wireshark is seeing that, that sounds like the server is seeing the packets from the NMC. So... anything else anyone can think of that I can check? I haven't tried calling APC, but I didn't know if they would offer phone support on a free product.

[Anonymous user]

"is there a way to test connectivity to UDP ports from a SLES machine? Testing TCP is easy, but I don't know how to test UDP."

If you're trying to see if a port is being blocked by an intervening firewall, you can use firebind.com. There is a java client that allows you to choose TCP or UDP, and any port (or range) from 1-65535. It then sends packets back and forth from your machine to the firebind server on the internet on the chosen ports, thereby verifying if there is an intervening firewall block or not. They have a javascript client too but it's TCP only and it's subject to some port restrictions due to browser limitations.

It's not a port scanner (since a port scanner would scan open ports on a target machine.) Instead it's a path (to the Internet) scanner.

http://www.firebind.com

- ProtocolGeek

[BillP]

you are receiving top tier APC support forum the forum to be honest with you. phone support will most likely escalate it to the folks that already post on this board.

[Anonymous user]

Greetings,

This topic got back-burnered for a while, but I'm finally getting back to it.

Strangely, one of the two sites that I was having problems with mysteriously started working. I have no idea what was done; nothing that I did! Anyway, my original site still isn't working.

I tried running Firebind, as suggested by another user here. I'm not sure I know exactly what I'm doing, but here's what I did:

Chose Port 3052 as the outbound port.
Chose UDP as the protocol.
Clicked "Start".

The results are as follows:
================================
Firebind Port Test Results

Ports: 3052
Start time: Wed Apr 13 16:47:37 PDT 2011
End time:Wed Apr 13 16:47:38 PDT 2011
Duration: 0 seconds

Port Detail:
Passed: 3052

For more information visit http://www.firebind.com
================================

Anything else I can check?

Thanks!

[Anonymous user · Accepted solution]

I had a similar issue when using the PCNS VM under VMware.
I was able to ping both ways between the PCNS VM and the NMC.
Connection would establish, but at the registration phase I'd get the message "PowerChute is not receiving data from the Network Management Card".
I found what the issue was in my case: due to some network changes I needed to change the IP address of the PCNS VM. I used nmtui to do this. The issue is that you must enter the IP address along with the subnet mask. ie. I enter x.x.x.x instead of x.x.x.x/24 . However, I only entered x.x.x.x . Hence it was my error, but the interface should have been a bit more idiot proof. It should have either not accepted the entry without the /xx , or it should have had a separate subnet mask field.
This probably won't help the OP, but may help others.