DCE running very slow and Logs keep filling

[jake.l]

Hello, DCE is running very slow and the logs keep filling with this specific error message. The client has been restarted, but the problems persist. Any suggestions? DCE is currently running version 7.8.1.

<log4j:event logger="com.netbotz.server.services.sensor.controllers.SensorsController" timestamp="1788346246973" level="WARN" thread="TP-Processor28">

<log4j:message><![CDATA[Handler execution resulted in exception]]></log4j:message>

<log4j:throwable><![CDATA[com.thoughtworks.xstream.io.StreamException:  : null

at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:77)

at com.thoughtworks.xstream.core.util.QuickWriter.write(QuickWriter.java:49)

at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.writeText(PrettyPrintWriter.java:279)

at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.writeText(PrettyPrintWriter.java:231)

at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.setValue(PrettyPrintWriter.java:214)

at com.thoughtworks.xstream.io.WriterWrapper.setValue(WriterWrapper.java:45)

at com.thoughtworks.xstream.io.StatefulWriter.setValue(StatefulWriter.java:124)

at com.thoughtworks.xstream.io.WriterWrapper.setValue(WriterWrapper.java:45)

at com.netbotz.server.xfer.nbVarValueServerXStreamConverter.marshal(nbVarValueServerXStreamConverter.java:132)

at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)

at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)

at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)

at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)

at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)

at com.thoughtworks.xstream.converters.collections.MapConverter.marshal(MapConverter.java:79)

at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)

at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)

at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)

at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshallField(AbstractReflectionConverter.java:250)

at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.writeField(AbstractReflectionConverter.java:226)

at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.<init>(AbstractReflectionConverter.java:189)

at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doMarshal(AbstractReflectionConverter.java:135)

at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:83)

at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)

at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)

at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)

at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)

at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)

at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)

at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)

at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)

at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)

at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)

at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)

at com.thoughtworks.xstream.XStream.marshal(XStream.java:1022)

at com.netbotz.server.xfer.XStreamServerProtocol$LocalXStream$1.writeToStream(XStreamServerProtocol.java:77)

at com.thoughtworks.xstream.core.util.CustomObjectOutputStream.writeObjectOverride(CustomObjectOutputStream.java:84)

at java.io.ObjectOutputStream.writeObject(Unknown Source)

at com.netbotz.server.services.sensor.controllers.SensorsController.handleXmlRequest(SensorsController.java:146)

at com.netbotz.server.spring.servlet.AbstractXmlController.handleRequestInternal(AbstractXmlController.java:60)

at com.netbotz.server.spring.servlet.AbstractController.handleRequest(AbstractController.java:34)

at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)

at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:585)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)

at com.netbotz.server.filters.CompressionFilter.doFilter(CompressionFilter.java:143)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)

at org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)

at com.netbotz.server.services.security.WebAppAwareFilterSecurityInterceptor.doFilter(WebAppAwareFilterSecurityInterceptor.java:45)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)

at org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101)

at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)

at org.springframework.security.wrapper.SecurityContextHolderAwareRequestFilter.doFilterHttp(SecurityContextHolderAwareRequestFilter.java:91)

at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)

at org.springframework.security.providers.anonymous.AnonymousProcessingFilter.doFilterHttp(AnonymousProcessingFilter.java:105)

at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)

at org.springframework.security.ui.basicauth.BasicProcessingFilter.doFilterHttp(BasicProcessingFilter.java:174)

at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)

at org.springframework.security.context.HttpSessionContextIntegrationFilter.doFilterHttp(HttpSessionContextIntegrationFilter.java:235)

at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)

at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:175)

at org.springframework.security.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:99)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)

at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)

at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)

at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)

at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)

at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)

at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)

at java.lang.Thread.run(Unknown Source)

Caused by: ClientAbortException:  java.net.SocketException: Broken pipe (Write failed)

at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:366)

at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433)

at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:348)

at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:392)

at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:381)

at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:88)

at java.util.zip.DeflaterOutputStream.deflate(Unknown Source)

at java.util.zip.DeflaterOutputStream.write(Unknown Source)

at java.util.zip.GZIPOutputStream.write(Unknown Source)

at com.netbotz.server.filters.CompressionResponseStream.writeToGZip(CompressionResponseStream.java:192)

at com.netbotz.server.filters.CompressionResponseStream.write(CompressionResponseStream.java:168)

at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)

at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source)

at sun.nio.cs.StreamEncoder.implFlush(Unknown Source)

at sun.nio.cs.StreamEncoder.flush(Unknown Source)

at java.io.OutputStreamWriter.flush(Unknown Source)

at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:75)

... 88 more

Caused by: java.net.SocketException: Broken pipe (Write failed)

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(Unknown Source)

at java.net.SocketOutputStream.write(Unknown Source)

at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518)

at org.apache.jk.common.JkInputStream.doWrite(JkInputStream.java:147)

at org.apache.coyote.Response.doWrite(Response.java:559)

at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:361)

... 104 more

]]></log4j:throwable>

</log4j:event>

[ATrabert]

While there could be number of reasons this could be happening, it would be best to open a tech support case and provide them with capture logs output from DCE https://www.se.com/us/en/faqs/FA158539/ 

What's your DCE server uptime?  In DCE 8.2 and below, there was a bug where DCE would eventually run out of HTTP connections and users would be unable to connect through the desktop/web client.  It would also manifest itself as slowly degrading performance.  Rebooting the DCE server would temporarily clear the issue.  How long was variable, could be week, months, or year +.  The capturelogs output I recommended gathering would be able to pinpoint if it was that issue or not.

As you may already know, DCE 781 is pretty dated at this point, so it's recommended that you spin up a new 790 VM, migrate to that, then sequentially upgrade to a newer release.  Or you could start fresh with 8.3 and reconfigure from scratch.   You could also go to straight to the latest DCE 9.2 and reconfigure, but that requires working with tech support to convert your existing perpetual licensing over to subscription licenses based on remaining support contract dates.  We have 790 migration guide here https://community.se.com/t5/DCE-downloads/Complete-upgrade-process-for-DCE-7-9-0/ta-p/446776 

DCE OVA downloads and upgrade links are available here https://community.se.com/t5/DCE-downloads/tkb-p/dce-downloads 

If you go the 790 migration route, it's highly recommended maintain your DCE mac address to keep licensing, ITA device associations and any soap API integrations since they rely on DCE's mac address. 

[jake.l]

Thanks for your quick response. Our server uptime is low, possible a day or two, as we have restarted to try and resolve issues. Our current plan is to deploy a modern version of DCE and migrate devices over, however, we need to keep 7.8.1 operational during this process. 

Are there specific devices that could be causing the error in the log I posted? It referenced netbotz, but we have removed all new netbotz devices and the problem persists. 

Thankyou.

[ATrabert]

this error
Caused by: ClientAbortException:  java.net.SocketException: Broken pipe (Write failed)

is the desktop client disconnecting, but it doesn't include a reason why.  The nbc.xml log on the server web page under status or if you run capturelogs from DCE https://<DCE_IP>/capturelogs then download and extract that log file twice with a program like 7zip or winrar.  Once extracted, go to /data/logs/apache and open the ssl error log. 

look for errors like this one
[Wed Feb 28 11:59:18.058538 2024] [proxy:error] [pid 8219:tid 139655005591296] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8009 (localhost) failed
[Wed Feb 28 11:59:18.058603 2024] [proxy_ajp:error] [pid 8219:tid 139655005591296] [client 10.x.x.x:62382] AH00896: failed to make connection to backend: localhost

That's specifically the running out of connections to backend issue thats fixed by reboots and upgrading to 8.3+.  If it is this issue, how many simultaneous users do you have connecting to the desktop client and/or soap api/ITA connections?  How big is your server?  How many devices total are you monitoring?

Other things to look for is ensuring your server is correctly sized at 4 cpu/4gb ram for 500 devices, 8cpu/8gb of ram for 2000 devices, and 16 cpu/16 gb of ram for 4000 devices.  we have a sizing and performance troubleshooting guide here https://community.se.com/t5/Data-Center-Expert-Virtual/DCE-sizing-guide/ta-p/446647 and here https://community.se.com/t5/Data-Center-Expert-Virtual/DCE-performance-troubleshooting-guide/ta-p/44...