Historic Data and Event Journal File Format

[sbeadle1]

📖 Home  Back  
ClearSCADA stores historic data as files on the computer's hard disk. The location of those files is configured in the Location section of the Server Configuration Utility. So that the ClearSCADA can find the data as required, it is stored in folders within those locations, with each folder being named according to certain rules.

When objects are created they are automatically assigned a unique ID which starts at 0, the root group object, and incrementing for each new object. If an object is removed from the database its unique ID is not reused.

Both event journal and historic data is stored in what is known as a stream. A stream (or historic group) specifies how many different folders (and files) are used for storage of historic data. When the ClearSCADA server needs to search through this historic data, it is able to search a smaller subset of data when it is split into different streams and hence performance can be improved. A stream size of 8 (the default) will mean that a maximum of 256 (2 to the power of "stream size") objects will be placed in a sub-directory. i.e. Objects with the unique IDs of 0 to 255 will go in directory JnI00000, objects in the range 256 to 511 will go in JnI00001. The number of segments will depend upon the highest object id in the database and is not a predefined number.

Event Journal

The Event Journal data files are stored in a folder named Journal. This location is user definable in the Server Configuration. Under this directory, the data directory names are made up of 8 characters. The first two characters are always 'Jn'. The third character is a capital letter in the range D to O which represents the stream size.

Letter	Stream Size	Objects Per Folder
D	3	8
E	4	16
F	5	32
G	6	64
H	7	128
I	8	256
J	9	512
K	10	1024
L	11	2048
M	12	4096
N	13	8192
O	14	16384

The remaining 5 letters are digits which represent the stream number. The stream number is a number generated from the Object Id. The stream size is configurable from the Server Configuration Dialog and specifies the number of objects to store in one 'stream' of data. A stream can be used to improve performance by keeping smaller subsets of data together. This means that searches for data may only need to search subsets of all data if the data required is across only one or two streams.

For new installations of ClearSCADA, the stream size field defaults to 8. A stream size of 0 is equivalent to older style event journal storage (previous versions). On changing this setting and restarting the server, the file format of the data on disk will be changed to place the data into the appropriate streams. File formats have not changed between versions, only the location of the files.

Event Journal filenames start with 'H' (for hour) followed by a decimal number that is the number of hours since 1-Jan-1601 00:00:00 UTC. The file extension is 'MWJ'.

Historic Data

The directory structure is similar to that used for the event journal described above. The root level folder is named Historic and under this are a number of folders with 8 characters in the folder name. The first two characters are always 'Hd'. The third character is a capital letter the same as that used by the journal. The remaining 5 letters are digits that represent the stream number (based on the historic group size).

For new installations of ClearSCADA, the historic group size field defaults to 8. This setting is controllable via the registry key

HKEY_LOCAL_MACHINE\Software\Serck Controls\SCX6\DB\HistoricGroupSize

If the value of this key is changed, the server will need to be restarted, and on restart, the server will rearrange the folder structure on the hard disk to match the new group size. File formats do not change between versions, only the location of the files.

On upgrading from older systems, the value of this parameter will be maintained as 0. The downside of this setting is that potentially there will be many thousands of files in a single Windows folder and it becomes very difficult to browse the folder structure as Windows struggles to browse that many files. If you need to browse the historic data file folders, then it is strongly recommended to configure this registry key to 8 and restart the server.

If performance tuning of the database is required (typically only on very large databases ie. >50k points), reasonable values for the historic group size based on the number of objects in the database are

• 7 - 16,000

• 8 - 65.000

• 9 - 250,000

• 10 - 1,000,000

It is not recommended to set this setting outside the range of 7 to 10, and strongly recommended to change this setting only on the advice of your Support Representative.

Historic filenames start with 'Wk' (for week) followed by a decimal number that is the number of weeks since 1-Jan-1601 00:00:00 UTC. There are several file extensions in use for different types of historic data:

File Extension	File Contents
HRD	Historic Raw Data
HMD	Historic Modified Data
HAD	Historic Annotation Data
HSD	Historic Suppression Data (introduced in ClearSCADA 2010)
HUD	Historic Units Data (deprecated)

In most servers, almost all the historical data will be in HRD files.

Other Comments

It is not necessary for two servers in a main-standby arrangement to have the same historic and event journal stream sizes. If however the historic data is manually copied from one machine to another and the sizes do not match, ClearSCADA will reformat the historic data on startup of the second server to make the formats different based on the stream sizes. This is not a problem as far as redundancy is concerned. It simply means that the physical file structure on the two servers will be different.

Data that is archived from a server from a particular stream size can be mounted on a server with a different stream/group size without problems.

Go: Home Back