desktop client restarted while updating devices

[DCIM_Support]

DCO desktop client restarted while updating devices, all work got lost due to the restart. User confirmed that he didn't update his password. 

User application log shows pretty good amount of errors. Please help diagnose the root cause. 

Below is a snippet of the errors:

2019-12-30 09:50:46,129 ERROR [com.apc.config.hibersync.client.internal.HiberSyncClientExistingModelSynchronizer] (Model Thread 0) java.io.IOException: Unable to call getUpdates and read result
2019-12-30 09:50:46,166 WARN [com.apc.config.hibersync.client.internal.HiberSyncClientExistingModelSynchronizer] (Model Thread 0) Something went wrong, dumping modelChangesSnapshot to help debugging:
[com.apc.config.productcatalog.ui.internal.service.CustomCatalogClientMergeStrategy-snapshot:com.apc.config.productcatalog.impl.service.CustomCatalogSnapshot@37cb4f7e],
2019-12-30 09:50:46,189 ERROR [com.apc.sma.client.ExceptionHandlerImpl] (Model Thread 0) Unable to authenticate (typically causes include: user password expired or user account disabled). Restarting client.
2019-12-30 09:50:46,191 ERROR [com.apc.sma.client.ExceptionHandlerImpl] (Model Thread 0) Unable to authenticate (typically causes include: user password expired or user account disabled). Restarting client.
2019-12-30 09:50:46,192 WARN [com.apc.config.workbench.threads.impl.ExtendedThreadFactory] (Model Thread 0) Exception in thread Model Thread 0
java.lang.RuntimeException: java.util.concurrent.ExecutionException: javax.xml.ws.soap.SOAPFaultException: OutgoingCommunicationSwitch is off, so skipping Web Service call to http://services.product.apc.com/ -> public abstract java.util.Collection com.apc.product.services.ProductServicesWebService.getLicenseInformationIncludeExpired() throws com.apc.product.services.ProductServicesException !
at com.apc.dcoverview.internal.save.DefaultSaveService.saveChanges(DefaultSaveService.java:100)
at com.apc.dcoverview.internal.handlers.DCOverviewSaveHandler$SaveCallable.call(DCOverviewSaveHandler.java:156)
at com.apc.dcoverview.internal.handlers.DCOverviewSaveHandler$SaveCallable.call(DCOverviewSaveHandler.java:1)
at com.apc.config.workbench.threads.impl.wrappers.CallableWithProgressWrapper.call(CallableWithProgressWrapper.java:25)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.apc.config.workbench.threads.impl.ModelModificationThread.run(ModelModificationThread.java:10)
Caused by: java.util.concurrent.ExecutionException: javax.xml.ws.soap.SOAPFaultException: OutgoingCommunicationSwitch is off, so skipping Web Service call to http://services.product.apc.com/ -> public abstract java.util.Collection com.apc.product.services.ProductServicesWebService.getLicenseInformationIncludeExpired() throws com.apc.product.services.ProductServicesException !
at com.apc.config.workbench.threads.impl.ModelModificationThreadServiceImpl.syncExecExceptionSafe(ModelModificationThreadServiceImpl.java:192)
at com.apc.dcoverview.internal.save.DefaultSaveService.saveChanges(DefaultSaveService.java:94)
... 10 more
Caused by: javax.xml.ws.soap.SOAPFaultException: OutgoingCommunicationSwitch is off, so skipping Web Service call to http://services.product.apc.com/ -> public abstract java.util.Collection com.apc.product.services.ProductServicesWebService.getLicenseInformationIncludeExpired() throws com.apc.product.services.ProductServicesException !
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:161)
at com.sun.proxy.$Proxy58.getLicenseInformationIncludeExpired(Unknown Source)
at com.apc.external.client.serveraccess.ServerAccessNodeCountHelperImpl.updateLicenseCountFromServer(ServerAccessNodeCountHelperImpl.java:76)
at com.apc.external.client.serveraccess.ServerAccessNodeCountHelperImpl.access$0(ServerAccessNodeCountHelperImpl.java:74)
at com.apc.external.client.serveraccess.ServerAccessNodeCountHelperImpl$1.onDoneSaving(ServerAccessNodeCountHelperImpl.java:33)
at com.apc.dcoverview.internal.save.DefaultSaveService$SaveCallable.fireOnDoneSaving(DefaultSaveService.java:166)
at com.apc.dcoverview.internal.save.DefaultSaveService$SaveCallable.call(DefaultSaveService.java:155)
at com.apc.dcoverview.internal.save.DefaultSaveService$SaveCallable.call(DefaultSaveService.java:1)
at com.apc.config.workbench.threads.impl.ModelModificationThreadServiceImpl.syncExecExceptionSafe(ModelModificationThreadServiceImpl.java:189)
... 11 more
Caused by: com.apc.dsmf.outgoingcommunication.OutgoingCommunicationSwitchedOffException: OutgoingCommunicationSwitch is off, so skipping Web Service call to http://services.product.apc.com/ -> public abstract java.util.Collection com.apc.product.services.ProductServicesWebService.getLicenseInformationIncludeExpired() throws com.apc.product.services.ProductServicesException !
at com.apc.dsmf.outgoingcommunication.OutgoingCommunicationInterceptor.handleMessage(OutgoingCommunicationInterceptor.java:53)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:514)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:423)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:324)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:277)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:139)
... 19 more
Caused by: Error (401) ProductServicesWebServiceService.getProductVersion() @ https://dco.ms.com:443/isxo/core/product/ProductService.
at com.apc.dsmf.HttpCommunicationInterceptor.handleFault(HttpCommunicationInterceptor.java:38)
at org.apache.cxf.phase.PhaseInterceptorChain.unwind(PhaseInterceptorChain.java:491)
at org.apache.cxf.phase.PhaseInterceptorChain.wrapExceptionAsFault(PhaseInterceptorChain.java:342)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:324)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:514)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:423)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:324)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:277)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:139)
at com.sun.proxy.$Proxy57.getProductVersion(Unknown Source)
at com.apc.sma.client.ServerVersionChecker.getServerVersion(ServerVersionChecker.java:42)
at com.apc.sma.client.ServerVersionChecker.checkServerVersion(ServerVersionChecker.java:51)
at com.apc.sma.client.ServerVersionChecker.access$1(ServerVersionChecker.java:48)
at com.apc.sma.client.ServerVersionChecker$JobThread.run(ServerVersionChecker.java:76)

(CID:153093421)

[DCIM_Support]

Observations when looking at the provided stacktrace:
"Unable to authenticate (typically causes include: user password expired or user account disabled)" and "OutgoingCommunicationSwitch is off".
So the root cause is that the client could not get successful network communication to the server.

We show this by setting the communication icon in the bottom right corner to a red symbol.

(CID:153093562)

[DCIM_Support]

well, I followed your path and went back double checked the log, found the log warned once  the user logged in. Please see as below.

my questions:

1. can we say the user's login was not "successful" - well logged in, but no communication out?
2. if yes, how did this happen? i checked server log during that time, no error indicating communication with LDAP server . i do see some LDAP related warnings once in a while, but far outside that time frame
3. if no, how can i understand below warning message? i mean, no one would expect to check the communication icon after they login to make sure the login is successful, right? once you logged in, you logged in. 

2019-12-29 08:43:47,575 INFO [com.apc.sma.client.Application] (ModalContext) Using log levels: Eclipse log level: WARN, Application log appender level: INFO, Analytics appender level: WARN
2019-12-29 08:43:47,587 INFO [com.apc.sma.client.Application$EclipseToLog4jLogListener] (ModalContext) JVM args: [-Xms256M, -Xmx768M, -XX:+UseConcMarkSweepGC, -XX:+CMSIncrementalMode, -XX:+CMSIncrementalPacing, -XX:+HeapDumpOnOutOfMemoryError, -Dfile.encoding=utf-8, -DbuildType=release, -Dosgi.bundlefile.limit=0, -DsyncProduct=OPERATIONS, -Dosgi.nls.warnings=ignore, -DSWT_GTK3=0] [org.eclipse.core.runtime]
2019-12-29 08:43:49,413 INFO [com.apc.sma.client.Application] (ModalContext) Running server version 8.2.11.79512 (release)
2019-12-29 08:43:51,706 INFO [com.apc.sma.client.Licenses] (Model Thread 0) Server and client running with: colo,change,highavailability,insight,capacity
2019-12-29 08:44:27,579 WARN [com.apc.config.impact.impl.cfg.ImpactConfigurationServiceImpl] (Model Thread 0) No such profile Default. Has com.apc.config.impact.cfg.ImpactConfigurationBootstrapper been called? Anyway, creating now.
2019-12-30 08:41:35,507 WARN [com.apc.dsmf.outgoingcommunication.OutgoingCommunicationSwitch] (Thread-4) Outgoing communication switch turned off

(CID:153093564)

[DCIM_Support]

The user in this case was (presumedly) logged in successfully.

Before any calls to the server we re-authenticate, this is standard API implementation. It was this re-authentication that got an error. But this is not the root cause, as the next line in the stack trace shows that all out going communication is failing. So the re-authentication call just happened to be the first.
 

The OutGoingCommuncationSwitch is removed in the latest version of DCO and ITA, to avoid the problems you describe.

(CID:153550874)

[DCIM_Support]

Good to know how it works under the hood. 

Let's go back to your conclusion, "So the root cause is that the client could not get successful network communication to the server."

I personally think this is still a symptom. We still don't know what caused the "unsuccessful communication", could it be caused by timed-out session, or network glitch, or LDAP problem?

(CID:153550976)

[DCIM_Support]

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.