Subscribing is a smart move!
You can subscribe to this forum after you log in or create your free account.
Unexpected Historic Database Logging
Geo SCADA Knowledge Base
Access vast amounts of technical know-how and pro tips from our community of Geo SCADA experts.
Invite a Co-worker
Send a co-worker an invite to the Exchange portal.Just enter their email address and we?ll connect them to register. After joining, they will belong to the same company.
Send Invite Cancel
📖 Home Back
If your system's historic database is being flooded with minor point value changes, it is likely that:
Or
To prevent unnecessary point updates, you should try to filter out (compress) the data at the outstation/PLC. By compressing the data at the outstation/PLC, you reduce the amount of data that is transmitted across the SCADA communications network (which may be low bandwidth) and you also reduce the amount of data that needs to be logged in ClearSCADA. Another advantage of Historic Compression is that it reduces the amount of data that is stored in the outstation/PLC (and so there is less use of the outstation/PLC's resources).
Depending on the features that your outstation supports, you may be able to reduce the amount of data logged (and transmitted to ClearSCADA) by configuring significant change deadbands (change deviations). The significant change deadbands set the outstation to only log values that change by a defined amount or percentage. Working in a similar way to the Historic Compression settings in ClearSCADA, the significant change deadband features are designed to reduce traffic between the outstation and master station and reduce the amount of data stored in the outstation. The significant change deadbands are set in the outstation itself or are set in ClearSCADA and are downloaded to the outstation.
Most PLCs do not support any kind of compression as they do not store historic data for later upload to the SCADA system. When using PLCs that are causing an excessive amount of historic events to be logged, you should:
If filtering logic cannot be used, you may be able to apply compression at the master station (see below).
When you have compressed the data at the outstation/PLC, you can apply further filters at the master station by using the Historic Data Filter setting and the Historic Compression settings in ClearSCADA.
Every point update is classified as either current data or logged data (except updates for Override/Release Override actions). Some drivers support both current and logged data (where logged data is usually enabled or disabled on a per-point basis), but other drivers support only current data (such as ICMP and SNMP) or only logged data (such as Primayer, Technolog, Radcom, and Isodaq).
For each point update, there is a reason for logging, which can be:
By using the Historic Data Filter setting, you can specify which points have their updates stored historically---you set the updates to be stored according to their reason for update. For example, for DNP3 points you can choose whether static data (polled current) or event data (significant data such as state changes, crossed thresholds etc.) or both static and event data is logged in the historic database. By filtering according to the reason for update, you can reduce the amount of point data that is stored historically.
Please note that the Historic Data Filter setting is only available on certain advanced drivers.
When configuring the Historic Data Filter setting, you can usually choose which reasons for logging are to be applied. The reasons may vary according to the driver and points (and the configuration of the point itself). For more information on the Historic Data Filter settings, please refer to the documentation for the relevant driver in ClearSCADA Help.
You can also apply Historic Compression settings in ClearSCADA. The Historic Compression settings allow you to define an additional filter for point updates that are stored historically and are defined in the point configuration.
The Historic Compression settings include:
ClearSCADA compresses the data that it stores historically for this point. You can choose to compress:
For example, the Historic Data Filter on a DNP3 point is set to allow its static (current) data and DNP3 event data to be logged historically (static and event data). The same point has its Historic Compression Apply To setting set to Current Data Only (i.e. static data only). This means that DNP3 events for the point are always stored historically as they are not compressed. Only the point's current data is affected by the Historic Compression settings, and so the current data for the point is compressed (not stored historically).
ClearSCADA compresses all data, so current data and any other data is stored historically for the point.
ClearSCADA does not compress any data for the point and so all of the point's data is stored historically.
Note that for points that have Logged Data storage enabled, ClearSCADA usually only stores logged data historically---see the relevant driver documentation in ClearSCADA Help for more details.
The Significant Change for Historic Compression feature allows you to set ClearSCADA so that it only stores a point value historically if the point value changes by at least the defined amount.You can choose from three different types of Significant Change:
The value of the point must change by a defined amount for it to be stored historically.
The value of the point must change by a defined percentage of the span if it is to be stored historically. The span is the point range (Zero Scale to Full Scale). The Percentage of Span option is available for points that are, or are similar to analog points (i.e. points with a range of values).
The value of the point must change by a defined percentage of the last stored value if it is to be stored historically.
Allows you to define the minimum amount of time between the values that ClearSCADA stores historically. Any values that are reported more frequently than the defined Minimum Interval are not stored historically.
The Historic Compression settings can be used to reduce the amount of data that is stored historically. However, the Historic Compressions settings DO NOT APPLY in the following situations:
So, if the point is updated, compression is only applied if the state of the point is the same as the last historically logged state. If the state of the point is different to the last historically logged state, compression is not applied, even if the Historic Compression settings are in place. For example, if an analog point crosses an alarm limit so its state changes from Normal to Low, the new data will always be logged irrespective of the Historic Compression settings.
Changes to state, quality, status, master station state and reasons for updates are usually related to important changes on your system that need to be recorded historically. That is why for these types of change, ClearSCADA ignores the Historic Compression settings - it is important that there is a record of these types of change and so they should not be filtered out by historic compression.
UnexpectedDataLoggingExample1.png
In the Trend shown above, the trace shows the previously stored historic point value, the last value that was discarded due to historic compression (but was saved in the current database) and the new historic value. As the value that had previously been discarded by historic compression is included, the Trend is able to show the true relationship of the values i.e. that the point value had been close to 30 for a certain amount of time and then it suddenly increased to 45.
If the Historic Compression settings had been applied in this situation, the Trend would not have included the value that was previously discarded by the historic compression feature. This would mean that the trace would have shown a gradual increase from the previously stored historic value of 30 to the new historic value of 45 (as shown below). This does not represent the actual values that were reported.
UnexpectedDataLoggingExample2.png
Go: Home Back
If your system's historic database is being flooded with minor point value changes, it is likely that:
- Your outstation is reporting minor point changes
Or
- All changes in a polled PLC are being logged historically.
To prevent unnecessary point updates, you should try to filter out (compress) the data at the outstation/PLC. By compressing the data at the outstation/PLC, you reduce the amount of data that is transmitted across the SCADA communications network (which may be low bandwidth) and you also reduce the amount of data that needs to be logged in ClearSCADA. Another advantage of Historic Compression is that it reduces the amount of data that is stored in the outstation/PLC (and so there is less use of the outstation/PLC's resources).
Compressing Data at the Outstation/PLC
Depending on the features that your outstation supports, you may be able to reduce the amount of data logged (and transmitted to ClearSCADA) by configuring significant change deadbands (change deviations). The significant change deadbands set the outstation to only log values that change by a defined amount or percentage. Working in a similar way to the Historic Compression settings in ClearSCADA, the significant change deadband features are designed to reduce traffic between the outstation and master station and reduce the amount of data stored in the outstation. The significant change deadbands are set in the outstation itself or are set in ClearSCADA and are downloaded to the outstation.
Most PLCs do not support any kind of compression as they do not store historic data for later upload to the SCADA system. When using PLCs that are causing an excessive amount of historic events to be logged, you should:
- Create a user program in the PLC to 'smooth' the value for the point (register). This is often achieved by using 2 registers, the first register containing the raw value and the second register containing the 'smoothed' value. The program then copies the value from the first register to the second register, but only when the difference between the two values exceeds a threshold. ClearSCADA is configured to read the 'smoothed' value from the second register.
If filtering logic cannot be used, you may be able to apply compression at the master station (see below).
Compressing Data at the Master Station
When you have compressed the data at the outstation/PLC, you can apply further filters at the master station by using the Historic Data Filter setting and the Historic Compression settings in ClearSCADA.
Every point update is classified as either current data or logged data (except updates for Override/Release Override actions). Some drivers support both current and logged data (where logged data is usually enabled or disabled on a per-point basis), but other drivers support only current data (such as ICMP and SNMP) or only logged data (such as Primayer, Technolog, Radcom, and Isodaq).
For each point update, there is a reason for logging, which can be:
- Current - The point has a new current value
- Significant Change - The point has changed value by an amount that exceeds the defined significant change deadband
- State Change - The point has changed state
- Timed Report - The point update is timed to occur (the update is a result of regular periodic logging of the point value)
- End of Period - The point update is timed to occur at the end of a calculated period
- End of Period + Reset - The point update is timed to occur at the end of a calculated period and the point's value has been reset
- Override - The point has been overridden by a user selecting the Override action.
- Release - The point has had its override removed by a user selecting the Release Override action.
By using the Historic Data Filter setting, you can specify which points have their updates stored historically---you set the updates to be stored according to their reason for update. For example, for DNP3 points you can choose whether static data (polled current) or event data (significant data such as state changes, crossed thresholds etc.) or both static and event data is logged in the historic database. By filtering according to the reason for update, you can reduce the amount of point data that is stored historically.
Please note that the Historic Data Filter setting is only available on certain advanced drivers.
When configuring the Historic Data Filter setting, you can usually choose which reasons for logging are to be applied. The reasons may vary according to the driver and points (and the configuration of the point itself). For more information on the Historic Data Filter settings, please refer to the documentation for the relevant driver in ClearSCADA Help.
You can also apply Historic Compression settings in ClearSCADA. The Historic Compression settings allow you to define an additional filter for point updates that are stored historically and are defined in the point configuration.
The Historic Compression settings include:
- Apply To
ClearSCADA compresses the data that it stores historically for this point. You can choose to compress:
- Current Data Only - ClearSCADA only compresses the current data that is stored historically for the point. It does not compress any other data that it stores historically for the point.
For example, the Historic Data Filter on a DNP3 point is set to allow its static (current) data and DNP3 event data to be logged historically (static and event data). The same point has its Historic Compression Apply To setting set to Current Data Only (i.e. static data only). This means that DNP3 events for the point are always stored historically as they are not compressed. Only the point's current data is affected by the Historic Compression settings, and so the current data for the point is compressed (not stored historically).
- All Data
ClearSCADA compresses all data, so current data and any other data is stored historically for the point.
- No Data
ClearSCADA does not compress any data for the point and so all of the point's data is stored historically.
Note that for points that have Logged Data storage enabled, ClearSCADA usually only stores logged data historically---see the relevant driver documentation in ClearSCADA Help for more details.
- Significant Change
The Significant Change for Historic Compression feature allows you to set ClearSCADA so that it only stores a point value historically if the point value changes by at least the defined amount.You can choose from three different types of Significant Change:
- Absolute
The value of the point must change by a defined amount for it to be stored historically.
- Percentage of Span
The value of the point must change by a defined percentage of the span if it is to be stored historically. The span is the point range (Zero Scale to Full Scale). The Percentage of Span option is available for points that are, or are similar to analog points (i.e. points with a range of values).
- Percentage of Value
The value of the point must change by a defined percentage of the last stored value if it is to be stored historically.
- Minimum Interval
Allows you to define the minimum amount of time between the values that ClearSCADA stores historically. Any values that are reported more frequently than the defined Minimum Interval are not stored historically.
The Historic Compression settings can be used to reduce the amount of data that is stored historically. However, the Historic Compressions settings DO NOT APPLY in the following situations:
- Compression is not applied to every point update. The following changes are exempt from compression to ensure that important changes are not discarded:
- Change of State (including Master Station state, though Master Station state only applies to analog points that are on systems that have Master Station Alarm Limits enabled)
- Change of Quality
- Change of Reason for logging
- Change of Status
So, if the point is updated, compression is only applied if the state of the point is the same as the last historically logged state. If the state of the point is different to the last historically logged state, compression is not applied, even if the Historic Compression settings are in place. For example, if an analog point crosses an alarm limit so its state changes from Normal to Low, the new data will always be logged irrespective of the Historic Compression settings.
Changes to state, quality, status, master station state and reasons for updates are usually related to important changes on your system that need to be recorded historically. That is why for these types of change, ClearSCADA ignores the Historic Compression settings - it is important that there is a record of these types of change and so they should not be filtered out by historic compression.
- Compression is only applied to point updates that are in sequential time order. If a point updates with a time that is before the last historically logged time, the data for the point is not compressed. This is called an 'out of sequence' point update (out of sequence point updates rarely occur).
- There is a sudden 'jump' in the point's value. When Historic Compression filters out values, the last value that was discarded is stored in the current database (but not in the historic database). If the next point update to be compressed contains a point value that has suddenly increased or decreased by a considerable amount, ClearSCADA ignores the Historic Compression settings and stores both the last discarded value and the new point update value historically. This is to ensure that when the historic data is displayed on a Trend, the trace indicates that the point value hardly changed over a period of time and then suddenly increased or decreased rapidly at a specific time. If the last discarded value was not also stored in the historic database, the trace on the Trend would indicate a gradual increase in the value from the time of the previously stored value to the value in the latest update.
UnexpectedDataLoggingExample1.pngIn the Trend shown above, the trace shows the previously stored historic point value, the last value that was discarded due to historic compression (but was saved in the current database) and the new historic value. As the value that had previously been discarded by historic compression is included, the Trend is able to show the true relationship of the values i.e. that the point value had been close to 30 for a certain amount of time and then it suddenly increased to 45.
If the Historic Compression settings had been applied in this situation, the Trend would not have included the value that was previously discarded by the historic compression feature. This would mean that the trace would have shown a gradual increase from the previously stored historic value of 30 to the new historic value of 45 (as shown below). This does not represent the actual values that were reported.
UnexpectedDataLoggingExample2.png
Go: Home Back
Author
Create your free account or log in to subscribe to the forum - and gain access to more than 10,000+ support articles along with insights from experts and peers.