[PCNS] One server can't communicate with NMC

[mwhalen_apc]

This was originally posted on APC forums on 8/27/2010

---

HOLY CRAP I FOUND THE PROBLEM. It's the DNS Server service running on BKSERVER. I realize that it's grabbing 3052 because that port hasn't been reserved. There was a security update published for SBS and other server products that addresses DNS spoofing. [http://support.microsoft.com/kb/953230/] This update increases the amount of ports the DNS server will reserve for itself. DNS basically just grabs everything it can.

The downside is that this has a tendency to interfere with other services that need their own ports. Sometime in the last year, we updated the server with this update among others and I recall that it broke Blackberry Enterprise and the ability to login remotely. We were aware that we had just applied the port and adjusted the Reserved Ports registry key in the MSKB article that addresses this: [http://support.microsoft.com/kb/956189/]

But this was a long time ago. I am fairly certain we had put PCNS on this computer before we updated the server with that patch. Apparently that's when the PCNS connection broke and hasn't been able to work since.

So, what I did was set up the reservation in the registry. (This will become permanent on next reboot.) I then shut down the DNS Server and started up PCNS' Service. It connected to the NMC! I then restarted the DNS Server. (Since PCNS already has that port, DNS Server can't grab it. And when we reboot, DNS Server won't be able to grab that port because it is reserved.

Of course, what clued me in was the java output that the port was already in-use (see the last update below). What sealed it for me was using TCPView from SysInternals. When I turned on the "unconnected endpoints" option, I saw DNS.exe grabbing "*" for 0.0.0.0. That reminded me of the DNS security update.

---------

Hello,

Oy. I am so fried and frustrated.

I installed an AP9631 in a SmartUPS-2200SUA. I configured it successfully. I installed PCNS on two servers and configured them for shutdown when the environment temperature went above maximum. PCNS is running on two Windows servers. One is SBS 2003. The other is Server Standard 2003.

Two months later a colleague went to test the scenario. (It just took at that long to get out to the site to test.) He turned a hair dryer onto the temperature probe, artificially forcing the temp above the maximum. As you'd guess, the alarm went critical, indicating a temperature well above maximum.

But... the servers didn't shut down.

Since I had setup the original configuration, I was tasked with troubleshooting what happened. When I first logged into PCNS within each server, there were no Environment events listed under "Configure Events."

It was at this point I reviewed the installation material but, having not found much of anything, I called APC. I spoke to three people. The last one was the most knowledgeable, but he said that he had never heard of this problem.

So, with that statement in mind, we did the following:

1. Uninstalled PCNS on both servers -- BKSERVER and 2003SERVER.
2. Reinstalled the PCNS software on both servers.

(Note: During the uninstall and reinstall on one of the servers -- the server that's still not working -- I noticed a Java update. I applied it, which seemed to break Java entirely as the "verify java" pages on Sun's website did not work. I then uninstalled Java, rebooted that server, and installed the latest Java from the website. Java seems to be functioning now.)

On 2003SERVER, the Authentication Phrase I documented during the initial install didn't work. So, during the process of reinstalling PCNS on 2003SERVER, I re-set the Authentication Phrase to something new, documented it, then applied it during the install wizard. I was able to successfully register 2003SERVER and its installation of PCNS can see the Environment events.

I performed the same install on BKSERVER. It still, however, does not show those Environment event options. The installer accepted the parameters, the IP is registered in the NMC, but after about two minutes, the PCNS logs an unable to communicate event.

Everything is on the same subnet, 192.168.1.x. When I point telnet or a web-browser from 2003SERVER to the various ports that are supposed to be open and accessible on BKSERVER, the ports respond appropriately. 3052 shows a response from Jetty which doesn't respond if the PCNS service on BKSERVER is stopped. I've confirmed via netstat that java.exe is listening on 3052 under TCP and UDP. The HTTPS port 6547 is available from any machine within the subnet. I've also taken the extra step to shutdown all IIS websites listening on port 80 to be sure that PCNS wasn't listening on that port. (It was not.)

I have also reset the Management Card back to defaults, reconfigured it clean, and tried again to no avail.

I do not have a software firewall on BKSERVER. There is an edge firewall in the form of a Cisco ASA, but considering the other server can communicate over all the ports fine, I am unclear what detrimental effect the Cisco would have.

And this is where I sit. Of course, messages to the tech at APC I spoke with yesterday have gone unanswered. I am about to throw a packet capture on the server to see if I can see anything going across 3052.

UPDATE: Wireshark sees a broadcast to 192.168.1.255 from the NMC. I don't see any communications on either server going back to the NMC, so I am unsure if what I'm supposed to be seeing...

-------------

UPDATEx2: I used netcat to confirm that 3052 via UDP is open on BKSERVER. I connected to the server via UDP port 3052 and sent data. Wireshark sees the data coming in. If I telnet to port 3052 and set "GET /" I get an error from Jetty. This all proves that both TCP and UDP 3052 are open and listening on BKSERVER.

-------------

UPDATEx3: I spoke to someone at APC today and they gave me the default authentication phrase. I decided to plug that into the various places that need the auth code to see what happens. First, I plugged in "admin user pass" into the auth. phrase field within PCNS of the server that can communicate properly, SERVER2003. After two minutes, the an event was logged that the server could not communicate with the NMC. I then changed the authentication phrase within the NMC's UI to "admin user pass" and, after about a minute or so, SERVER2003 indicated it could speak to the NMC again.

I then logged into the PCNS UI on BKSERVER and changed the auth.phrase to the new "admin user pass" and no new events were logged. I expected at least one other "cannot communicate" message. I got nuthin' until I cycled the service.

I wonder if one of the .jar files is failing to load...?

-------------

UPDATEx4: OOOKay. Maybe I am getting somewhere.

This is logged by java when I run the entire java command line that the PCNS service runs. Note: This is AFTER I've stopped the PCNS service:

2010-09-01 17:46:27.656::INFO: Started SocketConnector @ 0.0.0.0:3052
java.net.BindException: Address already in use: Cannot bind
at java.net.PlainDatagramSocketImpl.bind0(Native Method)
at java.net.PlainDatagramSocketImpl.bind(Unknown Source)
at java.net.DatagramSocket.bind(Unknown Source)
at java.net.DatagramSocket.(Unknown Source)
at java.net.DatagramSocket.(Unknown Source)
at java.net.DatagramSocket.(Unknown Source)
at com.apcc.m11.components.StdPowerSource.BroadcastReceiver.run(Unknown Source)

Here's the full output:

E:\Program Files\APC\PowerChute\group1>java.exe -Xrs -cp .\lib\jetty-6.0.0.jar;.\lib\jetty-util-6.0.0.jar;.\lib\servlet-api-2.5-
6.0.0.jar;.\lib\collections.jar;.\lib\jsdk.jar;.\lib\m11.jar;.\comp\ds.jar;.\comp\AAOL.jar;.\comp\CommandFileRunner.jar;.\
comp\EventLogger.jar;.\comp\Notifier.jar;.\comp\Omaha.jar;.\comp\PacketRepeater.jar;.\comp\PowerSourceAggregator
.jar;.\comp\PSAggregator.jar;.\comp\RunTimeVerifier.jar;.\comp\Shutdowner.jar;.\comp\StdPowerSource.jar;.\comp
\WebServer.jar;.\comp\shutdownerlets\OSShutdownerlet.jar com.apcc.m11.arch.application.Application
2010-09-01 17:46:26.938::INFO: Logging to STDERR via org.mortbay.log.StdErrLog
2010-09-01 17:46:27.031::INFO: jetty-6.0.x
2010-09-01 17:46:27.109::INFO: NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
2010-09-01 17:46:27.563::INFO: Started SslSocketConnector @ 0.0.0.0:6547
2010-09-01 17:46:27.563::INFO: jetty-6.0.x
2010-09-01 17:46:27.563::INFO: NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
2010-09-01 17:46:27.656::INFO: Started SocketConnector @ 0.0.0.0:3052
java.net.BindException: Address already in use: Cannot bind
at java.net.PlainDatagramSocketImpl.bind0(Native Method)
at java.net.PlainDatagramSocketImpl.bind(Unknown Source)
at java.net.DatagramSocket.bind(Unknown Source)
at java.net.DatagramSocket.(Unknown Source)
at java.net.DatagramSocket.(Unknown Source)
at java.net.DatagramSocket.(Unknown Source)
at com.apcc.m11.components.StdPowerSource.BroadcastReceiver.run(Unknown Source)

[mwhalen_apc]

This was originally posted on APC forums on 9/1/2010

---

I accidentally marked it answered.

[BillP · Accepted solution]

This reply was originally posted by Alan on APC forums on 3/9/2011

---

Your post proved to be incredibly helpful. Thanks for the extensive documentation. I spent several hours working on this and don't think I would have figured out DNS was the culprit without your post. Like you said, nothing specific appeared to have 3052 open using netstat or TCPView. APC technical support did not appear to be aware of your solution. They should add it to their list of things to check!

I could not see that the security update you reference was installed on my system, but it may have been rolled into another update.

Thanks again.