[Imported] How to lessen time in acquiring data from database

[sbeadle]

>>Message imported from previous forum - Category:Scripts and Tips<<
User: amanshow, originally posted: 2019-01-03 05:14:42 Id:341
Good day.

My partner and I created a VBScript code that gets data from CDBHistoric database and transfer it to an excel file every hour through the Windows Scheduler. At first, we used an sql query condition which filters the data acquired by time and transfers the data to a buffer table and then use the code below to filter data from that buffer table and transfer it to a new final excel file and rename it with the date for today so the operators can look at it for analysis. So for example, we wanted a data at this current hour, data is obtained and put into buffer table and then this code is used to the buffer table:

`Set oRS = objConn.Execute("SELECT TOP( " & TDN1 & " ) * FROM CDBHistoric WHERE ( RecordTime BETWEEN { OPC 'M' } AND { OPC 'H+1H' } ) ")`

However, when we got the results, we found out that it doesn't exactly get everything since some tags aren't updated every minute. For example, I have voltage, current, and power. Voltage and current change a lot so it's easy to get its updated value through that sql query but power only updates two times a week so say it updated 3 days ago and I wanted to get data right now, voltage and current will have values but the table for power is blank.

So what we did was we decided to change the sql query to this:

`Set oRS = objConn.Execute("SELECT TOP(1) RecordTime, Id, FormattedValue FROM CDBHistoric WHERE Id = "+CStr(counter) + " ORDER BY RecordTime DESC")`

It worked well for a few weeks until we noticed that it takes longer minutes for it to transfer.

For example, during the first few weeks, as soon as the computer clock turns to 10:00 A.M. the code executes and the excel file is generated at 10:01 A.M. But now it takes longer time, and instead of 10:01 A.M., it has reached to 10:30 A.M. for the file to be generated.

My partner and I have a hunch that it may have something to do with the `ORDER BY RecordTime DESC` code. I've been researching about it and is met with disappointment at zero results. Any help would be very much appreciated.

[sbeadle · Accepted solution]

>>Responses imported from previous forum

Reply From User: sbeadle, posted: 2019-01-03 08:53:15
querying all historic records will get slower and slower as more data is stored, eventually hitting memory, performance or server configured limits. You should add a time constraint to all historic queries, and also add an Id constraint (e.g. use a JOIN to CDBPoint and constrain that by name).

Reply From User: tfranklin, posted: 2019-01-03 18:14:51
As Steve suggests, it'll get slower as time goes on because you don't have a time constraint. And I'd agree with you and your partners hunch regarding the ORDER BY being a culprit for the slowdown. The order by is sorting the records for you, however since no time constraint is applied the amount of work is amplified as more records are included.

The TOP(1) record is telling the application to give you the first record in the returned record set, but you're also telling the application to sort the records first. As such, it has to query every single record, sort it, then pull the one on the top. This is where the big delay is happening.

I'd take a few steps back and reconsider what it is you're trying to do. You mentioned needing hourly data. If that's the case I'd recommend using a historic view instead of raw history. ClearSCADA will give you processed values which will return much faster than raw data in the way that you're looking for it. If you want the value at the top of the hour, try adding an End Last algorithm to a historic view and querying that. Since this would group values up by time, you could manage to write a single query to get all of the values you want as columns and each row would represent an hourly value.

Reply From User: BevanWeiss, posted: 2019-01-03 22:52:41
I think that you should look into HistoricViews. End / Start options might do what you want.
The performance of these will be drastically better than you trying to perform the same thing after you get all of the raw samples yourself.

Reply From User: amanshow, posted: 2019-01-04 06:20:54
I'm going to look into your suggestions regarding HistoricViews and Time Constraints. Thank you so much for the ideas and for replying. I truly appreciate it.

So for example, I went for HistoricViews and put 1H on time interval. What if some of the tags have static values or maybe they only change every month or two, will that affect anything? Will I still get a value?

Before we used this code

`Set oRS = objConn.Execute("SELECT TOP(1) RecordTime, Id, FormattedValue FROM CDBHistoric WHERE Id = "+CStr(counter) + " ORDER BY RecordTime DESC")`

We opted using `WHERE ( RecordTime BETWEEN { OPC 'M' } AND { OPC 'H+1H' } )` just so we could get hourly values. But our results lacked some of the tags and when we crosschecked, we saw that these said tags are static or don't frequently change,, so they don't show in the excel file.

Reply From User: tfranklin, posted: 2019-01-04 16:52:52
If you use a historic view and do something with End Last or similar, the value that it last held as it passes through the time slot will be shown.

Ex: You mentioned a value only changing 2x a month. If I use End Last and use a view for hourly values, the value for every hour would be the exact same and show whatever it was until it changes later in the month.

If you're using BETWEEN { OPC 'M' } AND { OPC 'H+1H' } then you're probably not getting everything you want because your start time is 'M' which is the current minute.

Lets say I want all of the hourly averages for a point this month. My query would read:

SELECT * FROM HISHOURLYAVERAGE WHERE RECORDTIME {OPC 'Mo'} and ID = 12345

or subsequently, use RECORDTIME BETWEEN {OPC 'Mo'} AND {OPC 'Mo+1Mo'}
Meaning the start of the current month through next month.

Do note the difference between some of the algorithms that can be run though. There's an End, End Last, Average, Average Last, etc... The main difference is if you don't use the ones ending in last and the data value didn't update for whatever the timespan is then it will return a null since there's nothing to evaluate. The LAST designation will carry over the current value through the timespan regardless of a new value update.

Reply From User: amanshow, posted: 2019-01-05 01:26:37
[at]tfranklin said:
If you use a historic view and do something with End Last or similar, the value that it last held as it passes through the time slot will be shown.

Ex: You mentioned a value only changing 2x a month. If I use End Last and use a view for hourly values, the value for every hour would be the exact same and show whatever it was until it changes later in the month.

If you're using BETWEEN { OPC 'M' } AND { OPC 'H+1H' } then you're probably not getting everything you want because your start time is 'M' which is the current minute.

Lets say I want all of the hourly averages for a point this month. My query would read:

SELECT * FROM HISHOURLYAVERAGE WHERE RECORDTIME {OPC 'Mo'} and ID = 12345

or subsequently, use RECORDTIME BETWEEN {OPC 'Mo'} AND {OPC 'Mo+1Mo'}
Meaning the start of the current month through next month.

Do note the difference between some of the algorithms that can be run though. There's an End, End Last, Average, Average Last, etc... The main difference is if you don't use the ones ending in last and the data value didn't update for whatever the timespan is then it will return a null since there's nothing to evaluate. The LAST designation will carry over the current value through the timespan regardless of a new value update.

Yeah I forgot to change the `{ OPC 'M' }` to `{ OPC 'H' }`. My bad :smiley:

I've only been playing around ClearSCADA for a few months so this is my first time to learn about HistoricViews and time constraints. Thank you so much for the replies. Duly noted and appreciated!

I might come back for more questions soon! Thank you all!

Reply From User: amanshow, posted: 2019-01-07 02:02:55
Hello, I have a follow up question. I found this:

The Historic Views feature allows you to perform SQL Queries on **processed historic data**.

while I was trying to research on historic views. In my system, my senior has instructed me to use raw historic data instead of processed historic data. Will historic views still work?

Reply From User: sbeadle, posted: 2019-01-07 09:48:52
Views are, by definition, a process on top of raw data. Most views create a new value for a time period, such as the average. However, many of the views will show the raw value of a sample within the time period, such as Min, Max or Last. Using the End Last type goes beyond the time period if the value is not found, making it more similar to raw values, and quite useful if data is sparse.

Aside: Going back to the original need - to have the latest values each hour exported to a spreadsheet - why not consider reading current value from the CDBPoint table?

Reply From User: amanshow, posted: 2019-01-23 08:52:57
It's been a while yes but I have a follow up question:

What do I put on the default length if I want it to go on indefinitely?

I've finally tried historic views and it's going so well so far. I'm thinking of using it in my vbscript, just like what I explained in the above post. I have a new project and I have to use the same code but _with_ historic views.

Any responses would be very much appreciated!

Reply From User: sbeadle, posted: 2019-01-23 10:07:32
The default time/period of a Historic View just affects what happens when you query without time constraints. If you add a time constraint then the defaults are ignored.

Reply From User: amanshow, posted: 2019-01-24 01:37:58
[at]sbeadle said:
Views are, by definition, a process on top of raw data. Most views create a new value for a time period, such as the average. However, many of the views will show the raw value of a sample within the time period, such as Min, Max or Last. Using the End Last type goes beyond the time period if the value is not found, making it more similar to raw values, and quite useful if data is sparse.

Aside: Going back to the original need - to have the latest values each hour exported to a spreadsheet - why not consider reading current value from the CDBPoint table?

By reading from CDBPoint table, I won't have to worry about points that updated like two weeks ago or anything?

Reply From User: BevanWeiss, posted: 2019-01-27 01:09:29
[at]amanshow CDBPoint only contains the latest value (CurrentValue), so it doesn't sound like it's what you want. Unless your spreadsheet would be 'executed' each hour.
As I understand you want the spreadsheet to hold the last value retrieved every hour, for every hour requested from the spreadsheet report. For this, HistoricViews are a good fit.
You should look at what options are available for the HistoricViews, by reading the help.
I believe that what you want is the 'Last' option, which may produce NULL results where no samples were retrieved within an hour. If this isn't the behaviour that you'd want, then as Steve said, End Last may suit you better.