ETL integrations not running

[DCIM_Support]

I have 3 ETL integrations and they aren't working. They were running on 2 or 6 hour schedules, but for some reason they have stopped (no configuration changes were made). I logged on this morning and on the ETL page they all show green (last job run successfully) but they haven't run since between 19 hours and a day ago. The ETL logs show this as the last line from the normal job runs:

2019-03-25 00:00:59,229 INFO  [ecE2t3T/Qz6azX4K8TPjrg] [de63772bbef6] [org.pentaho.di] (SNC_05_Update ServiceNow newly imported tickets - Build PUT http) Build PUT http - Finished processing (I=0, O=0, R=29, W=29, U=0, E=0)

Then these are the only lines after even when I do a manual trigger of the job:

2019-03-25 12:07:07,490 INFO  [7kc+vt0dS76unqh+WgxgWA] [d1b7db16b02a] [org.pentaho.di] (default task-111 7kc+vt0dS76unqh+WgxgWA) RepositoriesMeta - Reading repositories XML file: FromInputStream

It just changes the default task number. 
I also looked through the server logs and see lines every 2 -6 hours that are saying:


2019-03-25 10:59:01,388 INFO  [] [] [org.quartz.impl.jdbcjobstore.JobStoreCMT] (QuartzScheduler_ISXOSchedulerMem-sqelpdco01v011552971494192_MisfireHandler) Handling 17 trigger(s) that missed their scheduled fire-time.

Earlier in the log it says 3 missed triggers, then 9 missed triggers, then 10, and then a bunch of 17s in a row. I am happy to provide any logs, etc needed to figure out what is going on. 

(CID:141266815)

[DCIM_Support]

Did you check whether the ETL tasks are in a paused state? This would be visible in the ETL configuration view which is in webmin in DCO 8.2.7 and earlier versions or in the settings of the DCO web client in DCO 8.2.12 and newer.

Can you share log files? If you cannot download and share the log files using the download log files from the struxureware data center operation menu in webmin, please start with the files server.log and etl.log from the folder /opt/jboss-as/standalone/log on the DCO server.

Regards

Greg Sterling

(CID:141266837)

[DCIM_Support]

They are all enabled and running. Yes, I can download the log files from the webclient. How should I send them to you?

(CID:141266847)

[DCIM_Support]

I sent you an invite for a box folder.

Regards

Greg Sterling

(CID:141266855)

[DCIM_Support]

Ok, it would appear the job last started around midnight this morning (March 25th), the etl.log shows it got to this point and seems to have stopped ... like its hung at this step. I have forwarded the log file box folder to engineering for additional feedback.

2019-03-25 00:00:59,229 INFO [ecE2t3T/Qz6azX4K8TPjrg] [de63772bbef6] [org.pentaho.di] (SNC_05_Update ServiceNow newly imported tickets - Build PUT http) Build PUT http - Finished processing (I=0, O=0, R=29, W=29, U=0, E=0)

Regards

Greg Sterling

(CID:141266863)

[DCIM_Support]

Any updates from engineering? This is effecting our production systems.

(CID:141267824)

[DCIM_Support]

Sorry. I will request an update. I know an engineer was looking at the logs yesterday.

Regards

Greg Sterling

(CID:141267842)

[DCIM_Support]

I'm rechecking again with dev to get an update.

Regards

Greg Sterling

(CID:141268583)

[DCIM_Support]

Hello Christopher. I'm told the error appears to be a timeout error of some kind. In general the connection attempts from the DCO server to the servicenow server will wait about 30 seconds for the connection to complete. Is there a reason the connection may not be working to SNOW or taking a long time to complete?

Also, the servicenow integration/connector you're using. Is this the out of box connector available from Schneider or was it custom made for you?

Regards

Greg Sterling

(CID:141268833)

[DCIM_Support]

So we have 3 integrations, 2 with ServiceNow 1 out of box (Asset) and 1 custom (Change), but the third integration just exports to a MS SQL server. So if it was a timeout error caused by connectivity to ServiceNow wouldn't only two out of the three integrations be affected? I have checked with our ServiceNow team and there were no configuration changes or outages during that time period when the integrations stopped. 

(CID:141269680)

[DCIM_Support]

I agree with your comment. Can you resend recent versions of the etl.log and server.log files from your DCO server and posted them to the box folder I previously shared? Both files are in the same folder: /opt/jboss-as/standalone/log

I assume the other two ETL tasks are running properly?

Regards

Greg Sterling

(CID:141270183)

[DCIM_Support]

Ok the ETL log and Server logs are updated. And no none of the 3 are working. They all seem to be stuck at the same point 3 weeks ago, even though they are all turned on and scheduled for every 2 hours. 

(CID:141270282)

[DCIM_Support]

Thanks for uploading the newer log files. You are correct, the etl.log shows no activity since the servicenow job tried to execute. I have forwared this info to dev for a recommendation.

Regards

Greg Sterling

(CID:141270326)

[DCIM_Support]

Christopher,

There is no easy method to cancel the current jobs and restart the ETL scheduler. Please plan to reboot your DCO server. This process will re-initialize the job scheduler and will cause the tasks to restart based on the defined schedules.

Regards

Greg Sterling

(CID:141271536)

[DCIM_Support]

Ok so I rebooted and then I manually had to trigger the jobs, but they were all able to run. One weird thing though now is that they all say they are primed to run in seven hours, even though when I click into them, 2 are scheduled every two hours and one is scheduled every 6 hours. I will keep an eye and see if it runs in 7 hours. 

(CID:141271643)

[DCIM_Support]

Yes. Keep us posted.

Thanks

Greg Sterling

(CID:141271684)

[DCIM_Support]

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.