Subscribing is a smart move!
You can subscribe to this board after you log in or create your free account.
Invite a Co-worker
Send a co-worker an invite to the portal.Just enter their email address and we'll connect them to register. After joining, they will belong to the same company.
Send Invite Cancel
Link copied. Please paste this link to share this article on your social media post.
📖 Home Back
The way that integrity polls, event polls, scan groups, logged data etc happens in the DNP3 driver is very complicated. This document attempts to explain how all these things fit together and which ones impact the others.
In DNP3, data can be in one of a number of classes. Which class it is in determines the behaviour of the RTU as well as how the data is retrieved from the device. Data can be in class 0, 1, 2 or 3.
Class 0 is static (current) data. This means that it simply has a current value. The RTU does not record time-stamped historic data for this point, and the only values returned for this point to ClearSCADA are the current values when a poll is performed.
The data classes (Class 1,2,3) can be used for a number of functions (such as data sets or file transfer), but for the purposes of this discussion I will limit this to the storage of historic, time-stamped data. Each time a point changes according to some conditions in the RTU (typically all changes for digitals and exceeding some percentage change for analogs), an event is generated which will contain the time at which the event occurred and the value of the point (among other things). This event will be stored in the RTU for later retrieval by the SCADA master. There is no real difference between the three classes in the way that they operate. Many customers use different classes of data to separate more critical data from the less critical data (so they can retrieve them at different rates if required).
An integrity poll is the major way that a DNP3 device is interrogated. An integrity poll is actually made up of two main components - the class polls (1, 2 and 3) and a static (current) data poll (or class 0 poll).
Integrity_Poll_Form.jpgIntegrity_Poll_Form.jpg
When polling a DNP3 device, the protocol rules stipulate that all events MUST be returned to the master prior to retrieving the static (current) values of points. This ensures that all data must be received in time series order (i.e. oldest first). If a device returns static (current) data before all event data is uploaded, it is actually breaking the DNP3 standard.
So we poll for all events first and process those, then the response includes the current values of all the points defined in the outstation. The master does not specifically request all point addresses like in protocols such as Modbus. The master simply says give me all the data that you have and the RTU will return all the data that is configured to be returned to the master. If a point is returned from the RTU to the master that is not defined in the master station database, it will simply be ignored.
When the static (current) data is received by the master, it has no timestamp (because all the RTU returned was the value of all points NOW). So in this case, the master will typically timestamp it according to the current time at the server now. This can cause problems if the time in the server and RTU is different because it means that data from the same source is being time-stamped at two different locations. If an event has already been received for the point with a timestamp ahead of the current server time, static (current) data point updates will use that time as their time-stamp until the current server time passes the time-stamped event time.
This can always be a source of problems, so in ClearSCADA 2007 R1 (SCX 66), a new feature has been added that allows a time request to be added in to the integrity poll request. This request simply asks the RTU to return the current time. If the RTU then returns it current time, the static (current) data will be time-stamped with the time returned from the RTU, rather than the potentially different server time. Supporting this function is part of the DNP3 level 3 subset and so all RTUs will not support this. For this reason, the setting will be left off by default, but should be switched on for the SCADAPack ES and ER (eNET and Magna) RTU, and left off for the rest of the SCADAPack range. For compatibility with other devices, refer to their device profile document.
Because the RTU can record event data as significant changes occur in the RTU, it is not always necessary to poll the device frequently.
It in not uncommon for polling intervals to be in the order of 30 minutes or more. Integrity polls are configured at a slow rate because the static data response can return a lot of data (which may not have changed) and can be expensive on the wire. Individual event polls or unsolicited messages only contain changes, so if there have been no (in the case of event polls) or few changes then the cost on the wire is small.
Different RTUs handle event generation and transmission in different ways. The SCADAPack ES / eNet and SCADAPack ER / Magna, allow events to be configured as either buffered or triggered. Buffered events are, as the name implies, simply buffered in the RTU for transmission to the master at a later date. Triggered events are used to initiate (or trigger) communications to the master station for upload of important events. For example, a power failure or security alarm may be configured as a triggered event so that operators using the SCADA system are very quickly notified of a security breach. A triggered event causes an "Unsolicited Message" or "Unsolicited Response" from the RTU to be generated and sent to the master.
The rest of the SCADAPack range of RTUs is a little different in that each data class has a Hold Time and a Hold Count. The time and count determines when events should be uploaded to the master, but again the idea is to reduce the amount of data that needs to be transmitted, or to allow critical updates to be transmitted to the master without needing to do a full (potentially expensive) integrity poll.
When an event is received at the master in a message from the RTU and the application layer is complete (one of the communications layers), the message is acknowledged by the master. When the RTU receives the acknowledgement from the master that it successfully received the events, the RTU will delete those events from its buffers. Different RTUs will do that in different ways, but suffice to say that the events are no longer available to be returned to the master.
There is an option in the integrity poll section named Poll Type. This often causes confusion as people get Class 0 polls confused with integrity polls. An integrity poll is a poll for all event data followed by all static (current) data. In the Poll Type selector, this is shown as "Class 0, 1, 2, 3".
Changing the Poll Type to "Class 0 Only" means you are no longer performing integrity polls of the device, but rather, are only doing static (current) data polls. No event data is requested or received at the master.
Consider the scenario where an RTU is polled by two masters. A local master (local display station) and remote master (main SCADA master for the system) both connect to the device. The local master is interested in the current values of the points so that local operators can monitor and control that site as needed and would usually poll the RTU on a direct link. The remote master is responsible for the entire SCADA network (maybe a main control centre) and polls for (maybe via a radio) and processes the event data (for reporting and alarming etc.).
If the RTU only supported one master, when the remote master receives the events, it will acknowledge them and they will be deleted from the RTU. If the local master then polls the RTU (Class 0,1,2,3), it will also receive events and acknowledge them causing them to be deleted from the RTU. This will mean that some events go to the local master and some go to the remote master station and neither gets the complete picture. In this case, we would set the local master to do Class 0 Only polls. This means that the local master will only request static (current) data. No event data will be received and so all events generated can go to the remote master station. This may mean that the local master should poll more quickly to get up to date values for the points. In this case, the DNP3 set address in the local master should be different to the remote master station.
In the situation where the RTU supports multi-master (eNet and Magna), this is not an issue because the RTU will maintain multiple lists of events. An event is not deleted until it has been retrieved by all configured masters. This would mean that both the local master and remote master station could be configured to do full integrity polls as long as they had different master addresses configured on the outstation set object.
It is not recommended DNP3 practice to perform a Class 0 poll periodically along with individual class polls. This breaks DNP3 standard recommendations so it is advised you do not configure your system in this way.
DNP3_Class_Polling.jpgDNP3_Class_Polling.jpg
Given the way that DNP3 RTUs store data, it is often common to use the full integrity poll as a background "sanity check", and to use a combination of class polls from the master and unsolicited messages from the RTU (for important events) to get the full picture of what is happening. Event polls simply ask for all events from a particular class from the RTU. They have an independent poll interval and sensibly would always be set to poll at a faster rate to the integrity poll as the integrity poll does a class poll as well.
When an RTU communicates with the master (in any response or unsolicited message), if it has events available in any of its classes, it will report this information to the master by way of the IIN status flags. When ClearSCADA sees this, it will then issue a poll for that class of event so that these events are also received. This is done according to recommendations for polling in the DNP3 standard.
Ticking the "Ignore Events Available IIN Bit" will ensure that a class event poll will not occur as a result of processing another message. This functionality could be useful when you wish to control polling strictly because of, say, limited bandwidth.
As event and static data is retrieved from the RTU, there are a number of settings that determine what gets done with the data.
Historic Logging
DNP3_Logged_Data.jpgDNP3_Logged_Data.jpg
This "Historic Data Filter" determines which types of data get passed through for processing by the historian. The default of "Event Data Only" means that static (current) data is not stored historically. This is good for points where events are being generated at the RTU, but obviously completely useless for points being polled only as part of an integrity poll i.e. no events.
Use the four options to determine which data is passed for processing by the historian based on the type of data being recorded in the RTU. See online help for more details.
Historic_Compression.jpgHistoric_Compression.jpg
Once data has been passed to the historian, the compression setting determine what is stored and what isn't. This is based on the setting of the Apply To field.
For the majority of telemetry systems using DNP3, we would discourage the "All Data" option as this may result in RTU events being lost. Using "No Data" is perfectly acceptable if the historic data filter is set to "Event Data Only". If you need less data to be stored in the historian, then it is better to reduce the amount of data being recorded and transmitted by the RTU which will also reduce the traffic on the comms network.
Alarm Generation
The "Alarms From" option under Logged data determines how alarms are processed when static (current) data and historic events are received from the RTU. This section is covered in detail in the online help and the reader should refer to the online help for discussion of the four fields below:
Point Significant Change
Significant change on a point applies to static (current) data only i.e. events reported by the RTU are considered significant and bypass this processing. If a value has not changed by more than the significant change amount defined for that point, it will not be passed to the database and hence will not be recorded in history. Screens will not update because the database, and hence clients are not updated.
If you have a point that you want to log, even though it is not changing, you can use either scan groups or level 3 scans (see below for more information on both).
Note that if a static (current) data value causes a state change i.e. from Low to Normal, then this is considered significant and will bypass significant change filtering.
An outstation Refresh will always update the current value of the points regardless of any settings.
Scan groups are used to force current values of points to be updated even when significant change has not been exceeded. Historic compression settings still apply to determine whether this means the point gets stored in the historic database.
Scan groups work differently for different protocols. In DNP3, unlike SCADAPack Modbus, a scan group expiring does NOT cause communications to the device. The criteria for scan groups is essentially this: after polling the device, has the scan group in which this point is configured expired. If so, update the current value of the point even if it has not changed significantly.
For the value to be then stored in the historic database, historic compression criteria must be met.
Scan groups cannot be used to update point values more frequently than the integrity poll interval. For example, if you have a 1 hour integrity poll and you require a 15 minute update of the point value (which may or may not be stored in the historian), then scan groups will not do this for you. You could consider the following:
A Level 3 scan is a feature that requires the RTU to support individual point reads rather than requesting all data. This is a function of the level 3 subset of the DNP3 standard and so is not supported by all RTUs. The scan group is configured on the channel, and a point is assigned into one of those scan groups. When you then tick the Level 3 scan option on the point, this will now force a poll of the device at the configured scan group interval.
DNP3_Level_3_Scan.jpgDNP3_Level_3_Scan.jpg
Please Note:
SCADAPacks (excluding the ES and ER) are Level 2 devices, not Level 3. Performing a Level 3 Scan on a SCADAPack will result in "Device Trouble" alarms being generated on the outstation object.
We previously discussed Event Class Polls. These will get all types of event (binary, analog, counter) from the RTU that are configured within a particular class. Event Scanning can be used to get particular types of event (eg. all binary) from all classes.
DNP3_Event_Scanning.jpgDNP3_Event_Scanning.jpg
Go: Home Back
The way that integrity polls, event polls, scan groups, logged data etc happens in the DNP3 driver is very complicated. This document attempts to explain how all these things fit together and which ones impact the others.
Data Classes
In DNP3, data can be in one of a number of classes. Which class it is in determines the behaviour of the RTU as well as how the data is retrieved from the device. Data can be in class 0, 1, 2 or 3.
Class 0 is static (current) data. This means that it simply has a current value. The RTU does not record time-stamped historic data for this point, and the only values returned for this point to ClearSCADA are the current values when a poll is performed.
The data classes (Class 1,2,3) can be used for a number of functions (such as data sets or file transfer), but for the purposes of this discussion I will limit this to the storage of historic, time-stamped data. Each time a point changes according to some conditions in the RTU (typically all changes for digitals and exceeding some percentage change for analogs), an event is generated which will contain the time at which the event occurred and the value of the point (among other things). This event will be stored in the RTU for later retrieval by the SCADA master. There is no real difference between the three classes in the way that they operate. Many customers use different classes of data to separate more critical data from the less critical data (so they can retrieve them at different rates if required).
Integrity Polls
An integrity poll is the major way that a DNP3 device is interrogated. An integrity poll is actually made up of two main components - the class polls (1, 2 and 3) and a static (current) data poll (or class 0 poll).
Integrity_Poll_Form.jpgIntegrity_Poll_Form.jpg
When polling a DNP3 device, the protocol rules stipulate that all events MUST be returned to the master prior to retrieving the static (current) values of points. This ensures that all data must be received in time series order (i.e. oldest first). If a device returns static (current) data before all event data is uploaded, it is actually breaking the DNP3 standard.
So we poll for all events first and process those, then the response includes the current values of all the points defined in the outstation. The master does not specifically request all point addresses like in protocols such as Modbus. The master simply says give me all the data that you have and the RTU will return all the data that is configured to be returned to the master. If a point is returned from the RTU to the master that is not defined in the master station database, it will simply be ignored.
When the static (current) data is received by the master, it has no timestamp (because all the RTU returned was the value of all points NOW). So in this case, the master will typically timestamp it according to the current time at the server now. This can cause problems if the time in the server and RTU is different because it means that data from the same source is being time-stamped at two different locations. If an event has already been received for the point with a timestamp ahead of the current server time, static (current) data point updates will use that time as their time-stamp until the current server time passes the time-stamped event time.
This can always be a source of problems, so in ClearSCADA 2007 R1 (SCX 66), a new feature has been added that allows a time request to be added in to the integrity poll request. This request simply asks the RTU to return the current time. If the RTU then returns it current time, the static (current) data will be time-stamped with the time returned from the RTU, rather than the potentially different server time. Supporting this function is part of the DNP3 level 3 subset and so all RTUs will not support this. For this reason, the setting will be left off by default, but should be switched on for the SCADAPack ES and ER (eNET and Magna) RTU, and left off for the rest of the SCADAPack range. For compatibility with other devices, refer to their device profile document.
Events
Because the RTU can record event data as significant changes occur in the RTU, it is not always necessary to poll the device frequently.
It in not uncommon for polling intervals to be in the order of 30 minutes or more. Integrity polls are configured at a slow rate because the static data response can return a lot of data (which may not have changed) and can be expensive on the wire. Individual event polls or unsolicited messages only contain changes, so if there have been no (in the case of event polls) or few changes then the cost on the wire is small.
Different RTUs handle event generation and transmission in different ways. The SCADAPack ES / eNet and SCADAPack ER / Magna, allow events to be configured as either buffered or triggered. Buffered events are, as the name implies, simply buffered in the RTU for transmission to the master at a later date. Triggered events are used to initiate (or trigger) communications to the master station for upload of important events. For example, a power failure or security alarm may be configured as a triggered event so that operators using the SCADA system are very quickly notified of a security breach. A triggered event causes an "Unsolicited Message" or "Unsolicited Response" from the RTU to be generated and sent to the master.
The rest of the SCADAPack range of RTUs is a little different in that each data class has a Hold Time and a Hold Count. The time and count determines when events should be uploaded to the master, but again the idea is to reduce the amount of data that needs to be transmitted, or to allow critical updates to be transmitted to the master without needing to do a full (potentially expensive) integrity poll.
When an event is received at the master in a message from the RTU and the application layer is complete (one of the communications layers), the message is acknowledged by the master. When the RTU receives the acknowledgement from the master that it successfully received the events, the RTU will delete those events from its buffers. Different RTUs will do that in different ways, but suffice to say that the events are no longer available to be returned to the master.
Class 0 Poll
There is an option in the integrity poll section named Poll Type. This often causes confusion as people get Class 0 polls confused with integrity polls. An integrity poll is a poll for all event data followed by all static (current) data. In the Poll Type selector, this is shown as "Class 0, 1, 2, 3".
Changing the Poll Type to "Class 0 Only" means you are no longer performing integrity polls of the device, but rather, are only doing static (current) data polls. No event data is requested or received at the master.
Consider the scenario where an RTU is polled by two masters. A local master (local display station) and remote master (main SCADA master for the system) both connect to the device. The local master is interested in the current values of the points so that local operators can monitor and control that site as needed and would usually poll the RTU on a direct link. The remote master is responsible for the entire SCADA network (maybe a main control centre) and polls for (maybe via a radio) and processes the event data (for reporting and alarming etc.).
If the RTU only supported one master, when the remote master receives the events, it will acknowledge them and they will be deleted from the RTU. If the local master then polls the RTU (Class 0,1,2,3), it will also receive events and acknowledge them causing them to be deleted from the RTU. This will mean that some events go to the local master and some go to the remote master station and neither gets the complete picture. In this case, we would set the local master to do Class 0 Only polls. This means that the local master will only request static (current) data. No event data will be received and so all events generated can go to the remote master station. This may mean that the local master should poll more quickly to get up to date values for the points. In this case, the DNP3 set address in the local master should be different to the remote master station.
In the situation where the RTU supports multi-master (eNet and Magna), this is not an issue because the RTU will maintain multiple lists of events. An event is not deleted until it has been retrieved by all configured masters. This would mean that both the local master and remote master station could be configured to do full integrity polls as long as they had different master addresses configured on the outstation set object.
It is not recommended DNP3 practice to perform a Class 0 poll periodically along with individual class polls. This breaks DNP3 standard recommendations so it is advised you do not configure your system in this way.
Event Polls
DNP3_Class_Polling.jpgDNP3_Class_Polling.jpg
Given the way that DNP3 RTUs store data, it is often common to use the full integrity poll as a background "sanity check", and to use a combination of class polls from the master and unsolicited messages from the RTU (for important events) to get the full picture of what is happening. Event polls simply ask for all events from a particular class from the RTU. They have an independent poll interval and sensibly would always be set to poll at a faster rate to the integrity poll as the integrity poll does a class poll as well.
When an RTU communicates with the master (in any response or unsolicited message), if it has events available in any of its classes, it will report this information to the master by way of the IIN status flags. When ClearSCADA sees this, it will then issue a poll for that class of event so that these events are also received. This is done according to recommendations for polling in the DNP3 standard.
Ticking the "Ignore Events Available IIN Bit" will ensure that a class event poll will not occur as a result of processing another message. This functionality could be useful when you wish to control polling strictly because of, say, limited bandwidth.
Time-stamped Data, Alarms and Logging
As event and static data is retrieved from the RTU, there are a number of settings that determine what gets done with the data.
Historic Logging
DNP3_Logged_Data.jpgDNP3_Logged_Data.jpg
This "Historic Data Filter" determines which types of data get passed through for processing by the historian. The default of "Event Data Only" means that static (current) data is not stored historically. This is good for points where events are being generated at the RTU, but obviously completely useless for points being polled only as part of an integrity poll i.e. no events.
Use the four options to determine which data is passed for processing by the historian based on the type of data being recorded in the RTU. See online help for more details.
- None - don't record any historic data
- Static data only - only pass data to the historian that came back from the Class 0 poll (may be part of an integrity poll or specific Class 0 poll in the scenarios described above).
- Event Data Only - if a complete record is being generated by the RTU in events, then this is often all that is needed. This is useful most commonly in scenarios where regular logging is occurring in the RTU.
- Static and Event Data - passes to the historian both time-stamped events and static (current) data.
Historic_Compression.jpgHistoric_Compression.jpg
Once data has been passed to the historian, the compression setting determine what is stored and what isn't. This is based on the setting of the Apply To field.
- No Data - store everything - do not apply compression at all.
- All Data - apply compression settings to both event and static (current) data.
- Current Data Only - only apply compression to static (current) data. All events will be stored.
For the majority of telemetry systems using DNP3, we would discourage the "All Data" option as this may result in RTU events being lost. Using "No Data" is perfectly acceptable if the historic data filter is set to "Event Data Only". If you need less data to be stored in the historian, then it is better to reduce the amount of data being recorded and transmitted by the RTU which will also reduce the traffic on the comms network.
Alarm Generation
The "Alarms From" option under Logged data determines how alarms are processed when static (current) data and historic events are received from the RTU. This section is covered in detail in the online help and the reader should refer to the online help for discussion of the four fields below:
- All Data
- Logged Data Only
- Most Recent Data
- Most Recent Logged Data
Point Significant Change
Significant change on a point applies to static (current) data only i.e. events reported by the RTU are considered significant and bypass this processing. If a value has not changed by more than the significant change amount defined for that point, it will not be passed to the database and hence will not be recorded in history. Screens will not update because the database, and hence clients are not updated.
If you have a point that you want to log, even though it is not changing, you can use either scan groups or level 3 scans (see below for more information on both).
Note that if a static (current) data value causes a state change i.e. from Low to Normal, then this is considered significant and will bypass significant change filtering.
An outstation Refresh will always update the current value of the points regardless of any settings.
Scan Groups (Confidence Interval)
Scan groups are used to force current values of points to be updated even when significant change has not been exceeded. Historic compression settings still apply to determine whether this means the point gets stored in the historic database.
Scan groups work differently for different protocols. In DNP3, unlike SCADAPack Modbus, a scan group expiring does NOT cause communications to the device. The criteria for scan groups is essentially this: after polling the device, has the scan group in which this point is configured expired. If so, update the current value of the point even if it has not changed significantly.
For the value to be then stored in the historic database, historic compression criteria must be met.
Scan groups cannot be used to update point values more frequently than the integrity poll interval. For example, if you have a 1 hour integrity poll and you require a 15 minute update of the point value (which may or may not be stored in the historian), then scan groups will not do this for you. You could consider the following:
- Have the RTU record an event every 15 minutes (best approach).
- reduce integrity poll interval to 15 minutes
- use Level 3 scan to poll the RTU for this point every 15 minutes
Level 3 Scans
A Level 3 scan is a feature that requires the RTU to support individual point reads rather than requesting all data. This is a function of the level 3 subset of the DNP3 standard and so is not supported by all RTUs. The scan group is configured on the channel, and a point is assigned into one of those scan groups. When you then tick the Level 3 scan option on the point, this will now force a poll of the device at the configured scan group interval.
DNP3_Level_3_Scan.jpgDNP3_Level_3_Scan.jpg
Please Note:
SCADAPacks (excluding the ES and ER) are Level 2 devices, not Level 3. Performing a Level 3 Scan on a SCADAPack will result in "Device Trouble" alarms being generated on the outstation object.
Input Event Scanning
We previously discussed Event Class Polls. These will get all types of event (binary, analog, counter) from the RTU that are configured within a particular class. Event Scanning can be used to get particular types of event (eg. all binary) from all classes.
DNP3_Event_Scanning.jpgDNP3_Event_Scanning.jpg
Go: Home Back
Author
Create your free account or log in to subscribe to the board - and gain access to more than 10,000+ support articles along with insights from experts and peers.