[Imported] period merge of remote clearscada historic points db with local clearscada historic db

[sbeadle]

>>Message imported from previous forum - Category:ClearSCADA Software<<
User: Roughneck42, originally posted: 2019-04-17 17:23:13 Id:409
we have two pump stations running 5k single clearscada servers. approximately 3k historical points per station. our main plant is running two 25k hot standby clearscada servers with approximately 15k historical points. We would like to merge the pump station historical data up to the main plant scada servers, comms between the two may be interrupted occasionally but is normally a megabit link. I was looking at EWS but this seems to want me to add in every single pump station point individually (groan) I am also worried that ews may be too slow as it seems to use xml.

I'd appreciate some alternative suggestions thanks in advance

[sbeadle · Accepted solution]

>>Responses imported from previous forum

Reply From User: adamwoodland, posted: 2019-04-18 00:03:41
https://community.se.com/t5/Geo-SCADA-Knowledge-Base/Comparison-of-Methods-to-Transfer-Point-Data-Be... might be a good starting point for requirement discussions

Reply From User: sbeadle, posted: 2019-04-18 07:53:02
Ways to make living with a multi-tier system simpler:
1 - use a script to export/import config between tiers and convert object types, setting the properties
2 - use templates on the lower and upper tiers, similar but with the appropriate types. (You can use Template Expressions to set the EWS address property automatically).

Reply From User: du5tin, posted: 2019-04-18 17:23:01
EWS is crazy fast until it isn't. If you try to pull too much historical data it will timeout and crash, then restart. We are starting to use it more for these multi-tiered systems though. For points with minimal historic data it works great and will backfill as required.

Like sbeadle suggests, you can probably export/import, do some sort of bulk conversion to EWS points and then update the tags with a script.

Reply From User: Roughneck42, posted: 2019-04-22 15:56:59
thanks for the response. The comparison table is a good start but how much is too much data when it comes to EWS?

Reply From User: sbeadle, posted: 2019-04-23 16:22:27
'It all depends'!
Allow (say) 50 bytes per value change/time and do a calculation. (This is my guess, I have not looked but it is the right order of magnitude).
Arrange 'scanning' so the client rarely asks for more than 50K of data per point (again, a guess).

Reply From User: du5tin, posted: 2019-04-23 16:41:32
I do not have a good rule of thumb for what is the best configuration. I do know anecdotally that EWS handled syncing history on points that were polled every 30 minutes for a time span of a year without issue (and this was fairly quick!). But we had trouble with facility points that had history records every 2-3 seconds (as some HMIs have) and 6 months of history. We saw high dbserver memory usage on the source system and EWS driver timeouts/freezes on the destination system.

Reply From User: Roughneck42, posted: 2019-04-23 17:48:26
awesome, thanks for the info.