ClearSCADA Channel Quantity

[geoffpatton]

All,

What is your experience with using a lot or a few channels? A customer that I work with was advised by support to only have one channel for all their sites. They are all ethernet DNP3 and currently they are using a channel per site, with somewhere around 150 channels. They are on one of the more recent versions of Windows Server.

For another customer, I was also told this but the system was already installed and running. They have a larger quantity of channels and for their system, all the DNP3 sites have their own channel. This system is running on Windows 7 and since 7 is out of support I am suggesting they move to server OS instead of Windows 10. Also, more processors if they are available from the VM Server they are running on 2.2G Xeon 2 cores with16GB RAM.

247 Total Channels (Ethernet)

23 Modbus Channels

221 DNP3 Channels

1 simple OPC Channel

As far as I can tell it really is not a problem as long as you have a system with enough resources. I think years ago you would run into resource limits much more quickly. Both have plenty of internet bandwidth, all the DNP3 sites are on Cellular Modems.

I think one channel will be problematic whenever there are communication failures, especially multiple sites failing.

[BevanWeiss · Accepted solution]

Hey Geoff,

I think the 'recent' changes in CS2017R2 to add 'Independent Connections' has changed the 'best practice' a little.

I would expect that 'Independent Connections' should mean that devices failing on such a channel would have (little to) no impact on other devices on that same channel.  It would also reduce down the number of objects within the database, and that is a positive thing for performance.

My recommendation would be:

1 channel for every 'bandwidth limited' network/pathway (so definitely one for every radio network, or one for each site that might only have a low speed WAN link).

and then 1 channel for all the other comms which is like a single outstation over 3G/4G or on a higher speed WAN link.

Given your list of drivers / channels, have you considered swapping from Simple OPC to Advanced OPC?

And you're already using the Advanced Modbus driver (instead of the Simple Modbus driver) right?

One thing to note with the client versions of the Windows OS is that there are actual EULA limits on their usage as a 'server', so it's entirely possible that they are breaching these limits if they have several other ViewX clients around.

This link seems to almost suggest that Windows client OSes have a limit on the number of incoming TCP connections that would be allowed, although I don't believe I've seen this in action at all

https://social.technet.microsoft.com/Forums/en-US/d33de0fe-ba31-459a-8a43-201875c9ebe0/max-concurren...

The section from the EULA seems to indicate that this is intended more for things other people would use.. like file services/print services.. but as to how they've implemented/hampered it technically to disincentive-ize using a 'cheap' client OS license for server purposes.. I don't know.

https://www.microsoft.com/en-us/Useterms/Retail/Windows/10/Useterms_Retail_Windows_10_English.htm

2.      Installation and Use Rights.

d.      Multi use scenarios.

(iii)    Device connections. You may allow up to 20 other devices to access the software installed on the licensed device for the purpose of using the following software features: file services, print services, Internet information services, and Internet connection sharing and telephony services on the licensed device. You may allow any number of devices to access the software on the licensed device to synchronize data between devices. This section does not mean, however, that you have the right to install the software, or use the primary function of the software (other than the features listed in this section), on any of these other devices.

a) I think your advice to the client is wise.  If it's being used as a server, then it should run a server-grade OS.

b) I think to keep resource usage low, it's better to have fewer objects (especially active objects) within the ClearSCADA database, and this can be relatively easily done by sharing a Channel/Outstation Set between multiple Outstations where it's appropriate (either by having them accessed sequentially, or by enabling the 'Independent Connections' option).

With an existing system having DNP3 Outstations, this should be relatively risk free.  You could just setup a new Outstation Set, and new Channel that will hold all the aggregated 3G/4G Outstations.  Then just one site per day swing them across into the new Outstation Set.  After a week of doing that, you could run some tests to ensure that failures with one Outstation don't impact on the other Outstations on the same channel.

[geoffpatton]

@BevanWeiss,

Good information.

The one running Windows 7 only has two ViewX clients and a WebX server for like 10 connections and VNC access, so I don't think they are breaking anything with Windows Client licensing. Server OS is still more robust and their system is only going to keep growing so I hope they just agree. I have not pried them off of CS2015R1 yet because of friction with IT in getting a new ActiveX on their Citrix server. Windows 7 EOL and, GS2019 with Virtual ViewX are moving that ball, just have to plan out that transition with the SCADA administrator.

The other system is on 2017R1 so neither can use the independent connections until they upgrade.

I did not mention it before because I am not confident I remember correctly, but I think I was told that having so many channels was a problem because of the number of connections being created. Since it had been working for at least a year I figured we'd fix it if/when it became a problem. Ignoring the impact of the number of database objects wouldn't the number of connections, using independent connections be virtually the same as having all the channels?

Both of these systems started as 2010 versions so we have been doing this for years and they have been slowly growing. I think they may start growing faster so it has become time to figure it out. As you said the DNP should be easy and the Bulk Edit will make it even easier to change them over.

Yes to Advanced Modbus. I tend to forget there is a Simple Modbus Driver, I hate it I can never easily figure out which address a point is using. I used the Simple Modbus at first back with 2005/7 and then discovered that the SCADAPack Modbus worked better on everything and made so much more sense.

The OPC is not a lot of stuff and I was told that we will be getting a new Allen Bradley driver which will eliminate most if not all of the OPC devices, so we will probably leave that alone.

[BevanWeiss]

Ohh, yes, sorry I should have mentioned that also.

Earlier versions of Windows (and I think Windows 7 might have been included) had this concept from Microsoft to try to slow the rate at which worms and malware could propagate, which meant by default it would only allow something like 20 TCP connections to be opened per second.  There was a registry entry you could change to alter this.

From what I've read, Microsoft decided it wasn't really working and got rid of this in later versions of Windows 7 (and later OSes).

I've been using GeoSCADA Expert 2019 for a little while now in my development machine.

I probably wouldn't recommend deploying it for a customer just yet.

I'd recommend going with ClearSCADA 2017R3.

The Virtual ViewX is somewhat useful, but you can do a similar thing with Microsoft RDS RemoteApps, and I feel the behaviour is more consistent with what you'd expect from a 'remote app' (with the RDS method).  Licensing is of course a factor, with Microsoft RDS not exactly being the cheapest around (although I actually haven't seen any pricing on the Virtual ViewX either...).

The help is absolutely broken in GSExpert.. if you have a modal dialog (as pretty all dialogs in ViewX are) then you can't access the help without closing all the dialogs.

I'm not convinced on the Allen Bradley story sorry.

Even Citect has rumours swirling around that it's going to lose the ABCLX driver in the not too distant future.  I think it just gets too difficult to keep up with the breaking changes that Rockwell do on the CIP with new PLCs.  One of my previous employers had a big project involving Citect and the (at the time) new L7x processors.  The ABCLX driver was doing something that the new CLX didn't like, and it would cause the CLX to enter a major fault and throw away all of its program.  It ended up with Citect engineers onsite investigating the issue.. which I'm pretty sure required a new ABCLX driver update.  With Rockwell now owning a part of the Kepware brand (PTC), I suspect they will be less willing to work with other vendors in communicating with their products.

[geoffpatton]

Since HTML5 is a must they are upgrading to Geo SCADA.

I do not like that the Help opens in a uselessly small docked window. I find it better to be a separate App Window. I like to leave it open and minimize it to the taskbar.

[BevanWeiss]

Yes.. the Help is now within the same process as ViewX.  This is apparently a requirement of Virtual ViewX 😞

Microsoft RDS handled having it as a separate process, and displayed everything really nicely when run as a Remote App.

Have you tried the help with the new Jan 2020 release?

[sbeadle]

Thanks for your feedback on the Help. We have allowed Help to be dragged out as a standalone window, and (in the Jan release) allow it to be a tab, so it can be flipped under another window. We're looking at behaviour when a dialog is shown..

[BevanWeiss]

@sbeadle Hi Steve, RE: "We have allowed Help to be dragged out as a standalone window", has this change been made after the Jan 2020 update?

I've had a few issues with the 'new help', SUP-11355 [unable to have ViewX in foreground whilst Help open], which saw a few changes made, and after testing of those changes in the Jan 2020 update, I've now got SUP-11404 [Unable to access help pane when launched whilst modal dialog active].

My understanding is that the ViewX launched help is NOT a 'standalone window' as this would have broken the Virtual ViewX capability.  It is a child window (via the ViewX Window Manager) of the ViewX window itself, which is where the issues come about.

The work-around that @Anonymous user provided was to launch the help via either the Server Configuration tool or the Server Status tool.

I'd still love if the default behaviour was as a separate process (as the old behaviour), and only when launched with the "/thinfinityvirtualui" argument would it be within the Window Manager.  I would have thought this to be a pretty easy change.  Bring back the old ViewX code launching the separate process help, and then just conditionally execute the new code if the /thinfinityvirtualui argument is present.  Then, when run under VirtualViewX things can work as intended for that environment, and in the non-VirtualViewX environment, we get to keep the old behaviour 🙂

[Anonymous user · Accepted solution]

I guess we're getting a bit off topic, but the guys have had a bit of a look and it is much more complex than just reverting the code and having it depend on the argument.

We'll take into account that @geoffpatton agrees with you, if anyone else reading also hates it feel free to chime in.

[sbeadle]

But create a new topic!

[geoffpatton]

@BevanWeissThere is an advantage to having more channels than less. When you turn the logging on if everything is on one channel it is harder to see the information for the site/device you are investigating. The more segregated the connections are the less data you end up with, in the log, to sort through.

I do not have to enable logging a lot, but If we do change from a channel per site I don't think I'll go with just 1 channel for all the ethernet sites on one protocol.

[BevanWeiss]

That is true..

Although for ethernet devices, which are the type that would typically be on the bulk shared channel I generally find Wireshark more useful than the Channel / IO logs from.ClearSCADA.

The ACK / SYN / RST items are normally involved in the issue, and they're not reported in the IO logs.

Once I see the problematic packet(s), then I'd use the translated IO logs to show a prettier picture of what went wrong..

Which is almost always been that a Site Sentinel / Metasphere unit has just turned off its modem terminating the connection mid data stream... I hate those devices 😛

[geoffpatton]

@BevanWeiss  I mostly work with Modbus and DNP devices. I almost never have to log a Modbus device and the DNP Comms logging has always worked well for the DNP comm errors I have had. My Modbus troubleshooting is usually done with the program Modbus Poll. Most of my comm problems are not network related.

Once on Kepware Wireshark did wonders when we were chasing our tail with SNMP. Turned out IT was Blocking the SNMP connection.

When we first started using Cellular Modems they were 3G and they would randomly disappear from the network for a few minutes, sometimes every few hours. I think that 4G phones were given priority. Now that those are on 4G this does not happen very often. Talk about maddening, we had to ignore comm alarms for alarm email notifications and make ST programs that checked if the comm alarm was active/inactive for more than 10 minutes. The ST controls an internal point that is used for the comm alarm notifications.

[AdamWoodland]

One of the drivers behind channel to OS ratio in the past was the number of cores in the server. A channel is a thread in the driver, so the more channels the more threads.

In the past with slow processors and limited processing cores having hundreds or thousands of threads was a significant performance hit as threads all competed for a time slice. I first ran a production version of ClearSCADA (SCX6) on a PIII-450MHz and then it was really important to get it right as that machine would get hammered with comms and be unusable if things were wrong.

These days, along with the new async processing, I doubt it really matters and the focus can be on other things. The only impact I syspect will be on virtual/cloud systems where the number of cores can have a direct impact on cost so the less cores needed the lower the hosting costs.

[geoffpatton]

@AdamWoodland Thanks for the info.

I think when 2005 came out all the computers we were using had at least Hyper-Threading on them and were a lot faster than 450MHz. I remember getting an SCX6 version as a Demo and then ClearSCADA 2005 was released.

I am pretty sure I still have an install for 2005.

Shoot I have a VM with 2007 in case I need to load something old, that some of it gets trashed because of the switch to Metadata.

[AdamWoodland]

Yeah, but then on the high tuned systems you had the impact of cache thrashing, so do you go more cores are lower performance or less cores with better performance. I don't miss those days, much easier now!

I did have a build from 2002 saved once with intentions of running it up in the future, but the NAS it was on failed so I lost it.

[Anonymous user]

So at the end is better to not to use the independent connections but go with separate channels instead?

[BevanWeiss]

@Anonymous user  I don't feel that was the take-away.

I think it's more complicated than just one way being better.

The system load of Independent Connections is better (less) than having lots of Channel Objects in the database.

But, if there are lots of devices on a single 'channel' then the IO / Channel logs can be busier, making deep communication investigations a little trickier (normally just requiring using find in translated log vs not).

So it's a trade-off.

I'd still recommend Independent Connections, but with some logical groupings, maybe by region or such, so there's not thousands of outstation on one channel.

[geoffpatton]

@Anonymous user,

My Takeaway was pretty much what @BevanWeiss Just said for All single IP devices.

If you are doing Ethernet to Multidrop Serial then using a channel per converter with merged connections is probably best.

[LorenR]

Thank you both!