ETL - Best practices for CPU/RAM management

[Sam_Vosch]

Hello,

We have a single PME system with 500+ devices spread over several sites and buildings in-which most of the data is entered into PME 2021 via ETL. Many of the devices are single measurement water devices, basic iEM meters (logging only energy delivered) and some other 3rd part building related devices. This setup has separate SQL and Application servers.

Data is published hourly from Com'X devices and ETL is run hourly. Therefore, every hour the system CPU gets slammed, which in-turn cripples the PME web application's response times.

What is the best way to manage CPU loading in ETL?

Should we make multiple small 'Jobs' in ETL or a couple of really large 'jobs'?

-Does ETL que jobs or run them co-currently

Should we limit 'Thread Pool Size' and Data Pipline Size' in the Advanced tab of ETL Administration Tool?  

Are there any 'Best practices' documentation for ETL on medium sized sites?

I know it's a lot of questions, but I am having a hard time finding ETL resources for PME so my understanding of ETL and how it processes jobs is limited.

Thank you.

[Pavel_Melnik]

Hi,

can't speak for ETL, but we used similar utility which is EDC. For a large project of 8-16 meters per site and over 200 hundreds of them we created separate jobs for each site and scheduled to run them at different times during each hour. Not that we were experiencing any performance issue, but just in case.