DCE Client keeps dropping connection

[DCIM_Support]

I'm new to DCE, so this might be a very simple answer.

One of our DCE clients keeps dropping connection with the DCE server. I log in and maybe 40 minutes to an hour later get an error that says "The client has lost communication with the server." This particular computer is connected to a couple of other monitoring softwares so I don't think it's the network. Also, my own client install has remained connected since I logged in this morning so I don't think it's the server.

Where should I start when trying to troubleshoot? Are there any logs that might give me a clue why the connection keeps dropping? We're on DCE 7.2.2.

Your assistance is much appreciated.

(CID:89555492)

[DCIM_Support]

Hi Dominic,

So this seems to be isolated to one DCE client? Is that correct? (the one that is monitoring other software solutions)? If this is so, what other differences are between this client and the client installed on your laptop? Are they on different network subnets? What OS are installed on both clients? Are you using the same connection to log into both clients? (using SSL or not using SSL)? Are you using a VPN connection? Is there any security policy in place on the problematic DCE client that cuts connection after a certain length of time? Is there anything else running on that OS that may interfere with client/server connection? (backups, etc)? When was the first time you noticed this issue? Was it ever working correctly? Was it always timing out? Has there been any system/network changes since you noticed this issue? 

How many devices are you monitoring? How many surveillance cameras are you monitoring? Have you tried to re-install the client? 

There is a client log that may help us determine why the client is dropping connection. This is located on the local machine under the C:\users\username\.isxc\7.2.2\.metadata. 

For the DCE client that is losing communication I would suggest (if possible), that we could stop the other monitoring software solutions to see if this could possibly be impacting the client/server connection (network traffic caused by the monitoring systems, causing a dropped connection from the client-server). By any chance are the other monitoring software solutions monitoring the same devices as DCE? 

Is this a hardware or a virtual DCE? If its a hardware server, I would suggest you connect directly to the private LAN and try and use the DCE client there. If the issue does not happen there or happen as frequently it might suggest that it is more and likely a network issue. 

If it is happening a couple of times a day or every 40 mins, a packet capture might be useful if the above suggestions don't work. 

Just for your review there is a KBase that outlines the ports that need to be open for client/server communication. FA158328, The link to the KBase is below:

http://www.apc.com/site/support/index.cfm/faq/

Regards,

B

(CID:89555498)

[DCIM_Support]

I have noticed this as well. I was thinking that it could be an internal timeout on a firewall that disconnects the port after a time of inactivity. I have seen this on both dce and dco, it would be nice if the dce could just reconnect like the dco client does without requiring a full client restart. Dave

(CID:89555499)

[DCIM_Support]

Thanks Breda and Dave for your responses. Yes, the problem is isolated to the one client. At first I thought it was a load issue, as the computer was at a pretty constant 90%+ load due to everything else running on there, but I reduced the load to ~30% and it's still disconnecting. The client connects over port 443, so after reading Dave's suggestion I found an https:// website that refreshes every minute in order to maintain traffic over that port, but it didn't help. I had a look at the log file, it shows a "read time out" every time the client disconnects. Does that mean anything significant to you?

(CID:89556679)

[DCIM_Support]

Hi Dominic,

Have you tried to connect using HTTP rather than SSL to the DCE server? Does the same issue happen? What is your timeout settings when connecting via SSL? Have you tried to increase this? Have you tried to stop the other monitoring solutions on that machine to see if that is impacting server/client connectivity? Have you tried to connect directly to the server (if a hardware server) to the private LAN and see if the issue persists? It would also be helpful to run a packet capture to see what is going on. 

Regards,

B

(CID:89556714)

[DCIM_Support]

Thanks for your assistance. It's much appreciated. I don't have the option to connect over HTTP, it's set up to only connect over SSL. I've tried reinstalling the client but it didn't help. Friday evening it ran for 18 hours without a timeout but since then has disconnected within a few hours at most. I also tried a continuous ping to the server, but I had no dropped packets and the client still got disconnected. Timeout settings are 30 seconds, but I tried 60 and 90 seconds and it still dropped the connection. The other monitoring solutions are just web pages that refresh every minute or few minutes. I'm not sure what you mean by "connect directly to the server" - I have my own desktop client connected for days at a time without an issue so I don't think it's the server. Please explain how I would run a packet capture.

(CID:90898738)

[DCIM_Support]

Hi Dominic,

You can run an application like Wire-Shark to carry out a packet capture. If the communication drop is happening intermittently it will be very hard to filter the list on the capture to see where the issue lies. (If it was happening every couple of minutes it would be easier to see what is happening, opposed to every couple of hours). What OS is installed on your machine? If this doesn't happen to your colleagues if would indicate an issue locally to your laptop. (I'm guessing your colleagues and yourself are on the same network/subnet?) Are you able to run an misinfo on your machine and your colleagues machine to check the different application/programs installed? Are you able as a test to stop the monitoring software and anti-virus on your machine to rule out any software conflicts that maybe would cause the connection to drop? If you are seeing an improvement, then I would introduce each application individually. (I understand this might be difficult in your environment) I have seen in the past that a client can be disconnected if there is backup software running. If the client was running for 18 hours on the Friday, is there anything that you can think of that is not running that day? i.e backup software/virus scans?  

What I mean by "directly connecting to the server" is that if you have a physical machine there is a GB2 port in the back. If you connect using a network cable to the back of the server on the private LAN to see if the issue persists?(to rule out a network issue). I know this would be difficult to do since the server would be likely installed in a Data Center. 

I hope this helps. 

Regards,

B

(CID:90898758)

[DCIM_Support]

Hi Dominic, One of my colleagues also pointed out if by any chance you are behind a NAT? (a NAT between the laptop and DCE) Sometimes with SSH connections they sometimes drop if no traffic goes through for a few minutes? Maybe if you could set your TCP keepalives to 60 seconds?

(CID:90898772)

[DCIM_Support]

Thanks Breda. I've tried running a webpage that auto-refreshes an HTTPS connection - wouldn't that do the same? It didn't help. My colleague suggested it might be the NIC being set to auto-negotiate, but I manually set it to 100Mbit Full Duplex and it didn't make any difference. I'm still getting disconnects. I have noticed that if I have the client running and connected but then change the Windows clock time, I immediately disconnect. Same if I have the clock update from an internet time server. Any idea why a time change would cause a disconnect?

(CID:90899044)

[DCIM_Support]

Hello,

I would like to attach myself to this topic as my organisation suffers from this issue as well. On the initial install DCE client did not drop the connection, however since version 7.2.6 and 7.2.7 we experience this issue on several machines in different network locations. One of the affected machines is connected directly to the DCIM network hence, ports problem is out of the discussion. The issue appears randomly during the day but usually no more than twice. Thanks to Breda response I have find out about the place where DCE Client stores log files, unfortunately I am unable to find there one specific cause for the connection problems therefore I would like to ask you for help in error analysis of this log and point me where to look for solution to this permanent problem. I have uploaded the client log file .log.

Best regards,

Krzysztof

(CID:102924678)

[DCIM_Support]

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.