EcoStruxure IT forum
Schneider Electric support forum about installation and configuration for DCIM including EcoStruxure IT Expert, IT Advisor, Data Center Expert, and NetBotz
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:20 PM . Last Modified: 2024-04-03 03:02 AM
I have 3 ETL integrations and they aren't working. They were running on 2 or 6 hour schedules, but for some reason they have stopped (no configuration changes were made). I logged on this morning and on the ETL page they all show green (last job run successfully) but they haven't run since between 19 hours and a day ago. The ETL logs show this as the last line from the normal job runs:
2019-03-25 00:00:59,229 INFO [ecE2t3T/Qz6azX4K8TPjrg] [de63772bbef6] [org.pentaho.di] (SNC_05_Update ServiceNow newly imported tickets - Build PUT http) Build PUT http - Finished processing (I=0, O=0, R=29, W=29, U=0, E=0)
Then these are the only lines after even when I do a manual trigger of the job:
2019-03-25 12:07:07,490 INFO [7kc+vt0dS76unqh+WgxgWA] [d1b7db16b02a] [org.pentaho.di] (default task-111 7kc+vt0dS76unqh+WgxgWA) RepositoriesMeta - Reading repositories XML file: FromInputStream
It just changes the default task number.
I also looked through the server logs and see lines every 2 -6 hours that are saying:
2019-03-25 10:59:01,388 INFO [] [] [org.quartz.impl.jdbcjobstore.JobStoreCMT] (QuartzScheduler_ISXOSchedulerMem-sqelpdco01v011552971494192_MisfireHandler) Handling 17 trigger(s) that missed their scheduled fire-time.
Earlier in the log it says 3 missed triggers, then 9 missed triggers, then 10, and then a bunch of 17s in a row. I am happy to provide any logs, etc needed to figure out what is going on.
(CID:141266815)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:21 PM . Last Modified: 2024-04-03 03:02 AM
Did you check whether the ETL tasks are in a paused state? This would be visible in the ETL configuration view which is in webmin in DCO 8.2.7 and earlier versions or in the settings of the DCO web client in DCO 8.2.12 and newer.
Can you share log files? If you cannot download and share the log files using the download log files from the struxureware data center operation menu in webmin, please start with the files server.log and etl.log from the folder /opt/jboss-as/standalone/log on the DCO server.
Regards
Greg Sterling
(CID:141266837)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:21 PM . Last Modified: 2024-04-03 03:02 AM
They are all enabled and running. Yes, I can download the log files from the webclient. How should I send them to you?
(CID:141266847)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:21 PM . Last Modified: 2024-04-03 03:02 AM
I sent you an invite for a box folder.
Regards
Greg Sterling
(CID:141266855)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:21 PM . Last Modified: 2024-04-03 03:01 AM
Ok, it would appear the job last started around midnight this morning (March 25th), the etl.log shows it got to this point and seems to have stopped ... like its hung at this step. I have forwarded the log file box folder to engineering for additional feedback.
2019-03-25 00:00:59,229 INFO [ecE2t3T/Qz6azX4K8TPjrg] [de63772bbef6] [org.pentaho.di] (SNC_05_Update ServiceNow newly imported tickets - Build PUT http) Build PUT http - Finished processing (I=0, O=0, R=29, W=29, U=0, E=0)
Regards
Greg Sterling
(CID:141266863)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:21 PM . Last Modified: 2024-04-03 03:01 AM
Any updates from engineering? This is effecting our production systems.
(CID:141267824)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:21 PM . Last Modified: 2024-04-03 03:01 AM
Sorry. I will request an update. I know an engineer was looking at the logs yesterday.
Regards
Greg Sterling
(CID:141267842)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:21 PM . Last Modified: 2024-04-03 03:01 AM
I'm rechecking again with dev to get an update.
Regards
Greg Sterling
(CID:141268583)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:21 PM . Last Modified: 2024-04-03 03:01 AM
Hello Christopher. I'm told the error appears to be a timeout error of some kind. In general the connection attempts from the DCO server to the servicenow server will wait about 30 seconds for the connection to complete. Is there a reason the connection may not be working to SNOW or taking a long time to complete?
Also, the servicenow integration/connector you're using. Is this the out of box connector available from Schneider or was it custom made for you?
Regards
Greg Sterling
(CID:141268833)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:21 PM . Last Modified: 2024-04-03 03:01 AM
So we have 3 integrations, 2 with ServiceNow 1 out of box (Asset) and 1 custom (Change), but the third integration just exports to a MS SQL server. So if it was a timeout error caused by connectivity to ServiceNow wouldn't only two out of the three integrations be affected? I have checked with our ServiceNow team and there were no configuration changes or outages during that time period when the integrations stopped.
(CID:141269680)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:21 PM . Last Modified: 2024-04-03 03:01 AM
I agree with your comment. Can you resend recent versions of the etl.log and server.log files from your DCO server and posted them to the box folder I previously shared? Both files are in the same folder: /opt/jboss-as/standalone/log
I assume the other two ETL tasks are running properly?
Regards
Greg Sterling
(CID:141270183)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:22 PM . Last Modified: 2024-04-03 03:01 AM
Ok the ETL log and Server logs are updated. And no none of the 3 are working. They all seem to be stuck at the same point 3 weeks ago, even though they are all turned on and scheduled for every 2 hours.
(CID:141270282)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:22 PM . Last Modified: 2024-04-03 03:01 AM
Thanks for uploading the newer log files. You are correct, the etl.log shows no activity since the servicenow job tried to execute. I have forwared this info to dev for a recommendation.
Regards
Greg Sterling
(CID:141270326)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:22 PM . Last Modified: 2024-04-03 03:01 AM
Christopher,
There is no easy method to cancel the current jobs and restart the ETL scheduler. Please plan to reboot your DCO server. This process will re-initialize the job scheduler and will cause the tasks to restart based on the defined schedules.
Regards
Greg Sterling
(CID:141271536)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:22 PM . Last Modified: 2024-04-03 03:01 AM
Ok so I rebooted and then I manually had to trigger the jobs, but they were all able to run. One weird thing though now is that they all say they are primed to run in seven hours, even though when I click into them, 2 are scheduled every two hours and one is scheduled every 6 hours. I will keep an eye and see if it runs in 7 hours.
(CID:141271643)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:22 PM . Last Modified: 2024-04-03 03:01 AM
Yes. Keep us posted.
Thanks
Greg Sterling
(CID:141271684)
Link copied. Please paste this link to share this article on your social media post.
Link copied. Please paste this link to share this article on your social media post.
Posted: 2020-07-05 06:22 PM . Last Modified: 2023-10-22 01:58 AM
This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.
Link copied. Please paste this link to share this article on your social media post.
Create your free account or log in to subscribe to the board - and gain access to more than 10,000+ support articles along with insights from experts and peers.