DCO 7.4.5 - Error Collecting Data

[DCIM_Support]

I upgraded my DCO server from 7.4 to 7.4.5 and now I am getting an "Error Collecting Data" alarm in DCO in regards to my DCE server.

The alarm mentions about increasing the respond time but I have never had a issue before. Any idea what might cause this?

Regards,

J

(CID:97485972)

[DCIM_Support]

Hi John,

Currently DCO 7.5 is the latest version, that contains many enhancements. So I would suggest considering the product upgrade to 7.5.

The error you are seeing/describing is usually for the case if DCO is unable to get data from DCE. There could be several reasons for this, possibly it might be because the connection between the two servers has been lost or interrupted. It might also occur if the DCE server is unable to handle the webservice requests from the DCO server. However, DCO will collect any data it missed when the connection is restored.

Just in case, if it should be needed to check or re-configure the polling interval and timeout settings: "System Setup > External Systems Configuration > and then edit the DCE server":

(CID:97485990)

[DCIM_Support]

Thanks for the response. I have increased the timeout to 10 seconds. Which is double the default and still have the same issue. However I have had the DCE & DCO servers connected and running for over a year and have not had this issue.

It arose after I upgraded the DCO server to 7.4.5 while the DCE server is and is still running 7.2.4.

Regards,

J

(CID:97485994)

[DCIM_Support]

Would this issue be due to the power strip hotfix when upgrading to 7.4? I never did this however I was already on 7.4 when I did the 7.4.5 upgrade so I didn't think anything about it. It is really annoying. Are you saying that upgrading to 7.4 should fix my issue? Because upgrading to 7.4.5 caused this issue...

(CID:99156028)

[DCIM_Support]

Hi John,

I would suggest to test the connection between DCO and DCE:

go to System Setup > External Systems Configuration > edit the DCE server, and then push the "Test" button, do the test both for port 80 and for port 443 ("use encryption").

This test, specially if the port 443 has been used for the configuration, should update the SSL certificate information between the external system (DCE) and DCO. It is important to accept the test of the server connection if successful, otherwise no changes will be saved.

I would also suggest to verify if the DCE user credentials are correct.

Furthermore, if there should be several DCE servers in the configuration, then do the test for each DCE server.

(CID:97485997)

[DCIM_Support]

Re-posting since I don't think comments notify everyone:

Would this issue be due to the power strip hotfix when upgrading to 7.4? I never did this however I was already on 7.4 when I did the 7.4.5 upgrade so I didn't think anything about it.

It is really annoying. Are you saying that upgrading to 7.4 should fix my issue? Because upgrading to 7.4.5 caused this issue...

(CID:99156202)

[DCIM_Support]

Hi John,

Sorry for not getting back to you sooner. I don't think that the upgrade itself (from 7.4 to 7.4.5) might have caused the issue, but it seems there might be sort of communication problem between the two servers (DCO and DCE). The communication issue might be related to the DCE user credentials or update of the ssl certificate. That's why I've suggested to test the connection between DCO and DCE.

If the issue basically should be user credential or ssl certificate related, then the additional upgrade (to 7.5) will not resolve it.

I'm actually very interested to know about the outcome of your tests, would it be possible for you to share some details.

(CID:99156209)

[DCIM_Support]

Maybe, What details do you need to know? The alarm I am getting is as follows:

______

Error Collecting Data

The Server is unable to collect device and alarm data from DCE. If this is caused by the server taking too long to respond, you can increase the Response Timeout in System Setup > External System Configuration. See the server log file for details.

_______

I keep adjusting the setting you mentioned but no luck. Also the server "Tests" fine. I looked through the log but don't see anything specific. Any idea where to look?

The only thing I see in the error log that is related is 'ERROR - Failed to get device sensors from ISX Central'.

Regards,

J

(CID:99156250)

[DCIM_Support]

Hi John, Many thanks for the info. Would it be possible to ask for the latest server logs to be shared with me? I can send (by direct email) an invite to my Schneider Electric box, if it is okay with you.

(CID:99156261)

[DCIM_Support]

The system is on a stand alone network so I can't share the logs, but if you tell me what files to look in, I might be able to print/scan them or type them out for you. We can take this offline. Where do I email you because your email is blocked in your profile?

(CID:99156284)

[DCIM_Support]

Hi John, I will send you an email very soon,

(CID:99156290)

[DCIM_Support]

So I am still having this issue. I really can't send the logs but I get this "Error" in the application logs.

ERROR [com.apc.sma.client.Application$EclipseToLog4  (main) Part already exists in page layout; com.apc.isx.advisor.ui.PlacementAdvisor. (org.eclipse.core.runtime)

ERROR [com.apc.sma.client.Application$EclipseToLog4  (main) Invalid preference page path: Web Browser (org.eclipse.core.runtime)

ERROR [com.apc.sma.client.Application$EclipseToLog4  (main) Invalid preference page path: Network Connections (org.eclipse.core.runtime)

In the communications log it has the following error:

Error (come.apc.monitoring.central.CentralServerFailureHandler.DETAILS) (ISXOSchedulerIO_Worker-4) Failed to get device sensors from ISX Central

(CID:99156901)

[DCIM_Support]

Hi John, Sorry to hear that the issue is not resolved yet - a similar issue has been posted by another community user some time ago, and it seems rebooting the DCE could also be an option to try out: https://community.exchange.se.com/t5/forums/searchpage/tab/message?advanced=false&allow_punctuation=....

(CID:99158096)

[DCIM_Support]

Hi John, What is the version of DCE? You have kindly shared the following line from the communication logs: "Error (come.apc.monitoring.central.CentralServerFailureHandler.DETAILS) (ISXOSchedulerIO_Worker-4) Failed to get device sensors from ISX Central" It would be useful to see the messages (at least 5 more lines) right after the above mentioned line.

(CID:99158141)

[DCIM_Support]

Sorry it took so long to respond. The lines are as follows: javax.xml.ws.WebServiceException: Could not send message. at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:135) at com.sun.proxy.$Proxy1748.getAllDeviceGroups (Unknown Source) at com.apc.monitoring.central.devices.DeviceGroupProxyImpl.getCentralServer (DeviceGroupProxy.Impl.java:69) at com.apc.monitoring.central.devices.CentralDataRetriever.retrieveDeviceData (CentralDataRetriever.java:16) at com.apc.monitoring.central.devices.DeviceGroupRetrievalJobExecutor.retrieveData (DeviceGroupRetrievalJobExecutor.java:58) at com.apc.monitoring.central.devices.DeviceGroupRetrievalJobExecutor.retrieveData (DeviceGroupRetrievalJobExecutor.java:1) Hope that helps you help me!

(CID:99158230)

[DCIM_Support]

Hi John, Many thanks, info is now added to our investigations and I will get back to you asap.

(CID:100339775)

[DCIM_Support]

Hi John, Can you please check your communication log and verify if it contains something like: java.lang.NullPointerException         at com.apc.monitoring.central.devices.sensors.SensorDataProxyImpl$SensorData.setTimestamp(SensorDataProxyImpl.java: xxx)

(CID:100340104)

[DCIM_Support]

I don't see that exact error. I searched for "SensorDataProxy" and I kept seeing the following error: Error (come.apc.monitoring.central.CentralServerFailureHandler.DETAILS) (ISXOSchedulerIO_Worker-4) Failed to get device sensors from ISX Central javax.xml.ws.soap.SOAPFaultException: Unmarchalling Error: [was class java.io.IOException] Premature EOF at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:146) at com.sun.proxy.$Proxy1738.getMultipleSensorData(Unknown Source) at com.apc.monitoring.central.devices.sensors.SensorDataProxyImpl.getMultipleSensorData(SensorDataProxyImpl.java:112) at com.apc.monitoring.central.devices.sensors.SensorDataProxyImpl.getDeviceSensors(SensorDataProxyImpl.java:62) Let me know if you need more of the error. Thanks for looking into this!

(CID:100340400)

[DCIM_Support]

Hi John, My pleasure, and many thanks for the verification and sharing the additional error messages - I will get back to you as soon as I have something to share, thanks.

(CID:100340408)

[DCIM_Support]

So I upgraded to DCE 7.2.7 (VM) and DCO 7.5 (VM) today and I still have the same error. Interesting enough, I also have a DCE 7.2.0 (Appliance) running and I am not getting any error from that. The DCE and DCO VM are running on the same host machine, just two different VMs.

(CID:100340421)

[DCIM_Support]

Hi John, Many thanks for getting back and sharing the update. The "Premature EOF" message in the logs could be caused by the connection to the DCO server being lost. That could be caused by the DCO being too slow to retrieve the response - or the DCO can give up waiting for the response. In general when DCO gives up waiting for the response we see "read timed out" messages instead of "premature EOF". The DCO read timeout setting can be altered on the second page of the DCO external server configuration of the DCE integration ("Stop waiting for data after..."). It might be worth to try to raise the timeout if it is not already high (several minutes). In the same configuration page, I would also note that the "Retrieve sensors" option is enabled. (And just in case, I would also check the server access settings on DCE (System > Server Administration Settings > Server Access...) to see if the communication port(s) are enabled.) There could also be network infrastructure between the two servers, that would cause the response to be cut-off.

(CID:100340506)

[DCIM_Support]

So i checked the setting you mentioned. Keep in mind I have TWO DCE servers. And only one is giving me an error AND it only happened when I upgraded DCO to 7.4.5. I compared the two servers and made the settings the same, both in DCE and DCO. The only difference was the DCE (v7.2.7) giving me problems had "SSH Server" options checked. So I unchecked them and reboot the server. I also went into DCO and changed the timeout settings to 120s and 900s to match the other server. Waited the 15m and the alert reappeared. So still no joy on fixing this error. I haven't updated my second DCE (v7.2.0) server yet because of these issues. The only thing I haven't tried was to remove the DCE error server from DCO and re-add it because I don't want to go through an re-link everything if I don't have too. The "Retrieve Sensors" option is check for both server. They are now setup identical in DCO and DCE. Any other ideas?

(CID:100340517)

[DCIM_Support]

One of my co-workers mentioned I should also note that DCE 7.2.7 runs SNMPv3 between discovered devices and DCE 7.2.0 runs mainly SNMPv1 between discovered devices. I increased the Global Scan Settings to 10 minutes instead of 5 to see if that fixes the issue.

(CID:100340697)

[DCIM_Support]

That did not fix the problem...

(CID:100340826)

[DCIM_Support]

Sharing the info with Community, This case will be investigated off-line.

(CID:102236387)

[DCIM_Support]

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.