Importing Historic Data

[DavidSkilbeck]

Hi, I have been running through the article below.

https://tprojects.schneider-electric.com/GeoSCADAHelp/Geo%20SCADA%202020/Content/ServerAdministratio...

For the last example 2.

Initially used the data provided as below

2016,07,03,16,23,23,0

2016,07,03,16,24,23,12

and got a message below

I changed the data to below to get it to work

(earliest I could go back).

2021,02,22,16,11,38,000,4.0218887
2021,02,22,16,11,53,000,3.1718216
.......
It did work, however the date in the history was today's date and not that of the file.

Is there a way of adjusting the time window (and if so how far back can we go).
Is there a setting for the correct date / time  to be loaded in to the history, as to that of the file ( I notice there is AM listed).

Thanks,

[AndrewScott · Accepted solution]

The writable life for historic data is configured using the Server Configuration tool:

https://tprojects.schneider-electric.com/GeoSCADAHelp/Geo%20SCADA%202020/Default.htm#ServerAdministr... l

The timestamps in the CSV file are in UTC, but ViewX is displaying the times in local time of the client PC, which appears to be UTC+10:00, so you are probably located in Australia.

[DavidSkilbeck]

Hi, do you refer to this section as below

as it seems to be set to 520 weeks.

I figures it would be something about UTC / Local.

Thanks,

[BevanWeiss]

Those sections that you have indicated only apply to 'Synchronisation Data'.. not to the data that the Main server is actively able to write to.

The section that applies for the historical writes is at the top.  If you look at the help for 'Archive after' you will see that this is what triggered the inability to update/add new historical data.

The help comments are:

Define the Archive Time for Historic Data

Alarm Summary, Configuration Changes Auditing, Event Journal or Historic Data is stored online for a specified duration (see Specify the Duration to Retain Historic Data Online), then the data is deleted. You can define a period of time after, which data that is stored online can be archived. When historic data is marked as eligible for archiving, it can be archived for example to CD, DVD and so on, (it has to be archived before the duration of the online data expires).

During the period that data is stored online, the data can be altered (by having new values reported for the time period) unless it is marked as eligible for archiving. When historic data is marked as eligible for archiving, it can no longer be altered on the disk—the new records for that period / time cannot be written to; it can only be read.

The concept here is that once you say that a historical period is 'archivable' then it should no longer be modified.  Since it may no longer be on the computer itself, and may actually be on a CD / DVD or write-once media stored for archiving.

For some reason I always 'think' that having an archive after time configured will help performance.  But I'm pretty sure it doesn't.  It obviously cuts down on when re-indexing might be required, but if you're trying to insert historical data into your system, then you probably need that historical data in there, and the cost of indexing is just a burden you need to wear.  As such, I'd say for 'most' systems, probably not having the Archive after tick box ticked is a reasonable approach (I think the default is ticked and 2 weeks).  If you will archive off to DVD etc, then you will need to have this configured.  But Geo SCADA Expert archiving is a little confusing anyway... so I suspect most people aren't using it.

[geoffpatton]

I have always kept years of history, but never actually archived any of it.

If I am remembering correctly the Archive after setting helps when you have redundancy and are restarting one of the machines.

This could have changed over the years too. I have been using ClearSCADA since version 2005.

[BevanWeiss]

I think this behaviour might have changed in recent versions.

I did a sync yesterday, and having the 4 week 'archive after' setting did NOT help in any way... the full 4 months of historic data was synchronised... which was quite painful over the two tin cans, and low value twine used for the inter-server connection.

I would have hoped that it would have done a hash over the current data blocks, and only requested sync where the data block hash didn't match... but it seems that's not the case either (or the behaviour is buggy..)

[AdamWoodland]

I think the methodology of how far back it syncs was changed around 2017 R1, based on the earlier change...

> would have done a hash

It used to do a CRC comparison up to 2013R1 or R2, they changed it as the CRC was too intensive to a much faster size and last modified date. If those match it doesn't re-sync the file.

Does cause problems if you copy/restore files to a server and that copy changes the last modified date, as the resync will just resync everything anyway.

[geoffpatton]

Good to know, I like your networking description. I think all of the servers I work on have lots of resources and a good gigabit connection between them, and I rarely am the one that restarts one.

[BevanWeiss]

Hmm... thanks for the refresher on that new system @AdamWoodland .  Unfortunately doesn't match my experience from yesterday.

The primary server had only been off since the 20th Feb, so it should have only been missing 4 days worth of data (or the last 1 week granules needing sync).  But it definitely resynchronised the whole history.

I do have a scheduled historic optimise, but that's only every Sunday... I guess I'll need to keep an eye on it, and report if it happens again.

[DavidSkilbeck]

Hi, these are the settings for my Geo SCADA.

This a sample of the data I want to import
4.0218887,2020,07,22,16,11,38,010,192,8
3.1718216,2020,07,22,16,11,53,010,192,8

This is what I am getting in Geo SCADA

If I use the below it wont import historic values

dt = new DateTime(2020, 2, 26, 11, 55, 52);

If I change the 2020 to 2021 it works.

[DavidSkilbeck]

It has been working. I just needed to select last year for it to display.

Even with offset property for the point set to "Hour - 2 Years", yo have to change search to last year.

Thanks,

[BevanWeiss]

It's possible that there are bugs around the offset property (and just Historic List behaviour in certain scenarios).

It would be worth reviewing the help documentation, and if you see behaviour that differs from the help documentation then raising a case with Tech Support.  That way they know about it, and can get it fixed for the future.

But always good to double check what the actual query is being used for the lists.