Communication issues with Geo SCADA in virtual machine

[jorge.florez]

Hi:

We have an application with two physical redundant servers. Every server has one virtual machine in which Geo SCADA is installed. One virtual machine is the primary server and the other virtual machine is the secondary server. Geo SCADA clients are installed in other virtual machines inside the physical servers. The virtualization software we are using is Hyper-V.

The application is communicating with several field devices (PLCs) by Ethernet/IP and OPC UA protocols. We started communicating one device of OPC without having any issues but as we connect more than three devices we started to experience communication problems. In Ethernet/IP we have issues since we activated the first device. The status screen for Ethernet/IP shows messages such as: "Communications state-failed-alarm raised", "Lost comm with outstation on channel XXXX". For OPC UA it shows messages such as: "Connection state failed (the operation timed out.)-Alarm raised", "Connection state-Disconnected (Driver failed)-Alarm raised", "Connection state-Disconnected (Driver failed)-Alarm raised", Connection state-Failed (Connection Error API Reconnect)-Alarma raised". In all cases, for both protocols, after approximately one minute the alarm is cleared and the whole process begins again.

We have tested with Geo SCADA installed in a laptop connected to the network and we do not have any problems. That's the reason why we suspect that the problem may have something to do with the virtualization process.

Have you had any reports about similar problems with Geo SCADA installed in virtualized environments?

Do you know about special configurations in virtual machines to help Geo SCADA work properly?

Thanks in advance for any help someone can provide me.

[geoffpatton]

My customers have not noticed any problems like this with VMware. One customer did use Hyper-V, installed a couple years ago and that worked fine. I think they have one server with only one Geo SCADA server VM, and the other had a VM for Geo SCADA server and a VM for Virtual ViewX. they had one big local controller and like 8 small remote sites all using DNP3 but they were polling them like it was Modbus since they had the network bandwidth to do so.

You may have a network bandwidth allocation problem.

I don't know anything about Hyper-V myself but if there are any network bandwidth settings you might want to try adjusting those and give the Geo SCADA Server one the most.

[AdamWoodlandToo]

What version of Windows are you running? If not a server OS you might be suffering from the OS throttling your outbound sockets (something Microsoft added back in a service pack of Windows XP to deal with the Internet worms and has caused performance issues similar to this before but not seen this behaviour in a long time so maybe no longer in Windows!)

What is your netstat showing for those failing connections? May show where in the TCP lifecycle it is failing.

It is possible some security software on the VM or the host is throttling sockets too.

I've not seen anything specifically related to performance in virtualisation. You have to make sure the various resources are available and the host isn't having to deal with some over-allocation. Resource over-allocation or dynamic resource assignment can cause issues although that is typically seen on user experience (mimic load times, etc) rather than protocol performance.

> That's the reason why we suspect that the problem may have something to do with the virtualization process

Sounds like you P2Ved the machine from a physical box, or have I misunderstood and this is a fresh install that has always been virtual?

May be worth running some performance benchmarks (with Geo SCADA stopped) to check out the machine's performance in general

[AdamWoodlandToo]

Assuming none of my comment helps, you might need to do some packet capturing and seeing if the problem is on the packets going out or the packets coming in (if they ever do) using something like Wireshark or the inbuilt Windows features.

Channel logging within Geo SCADA could also be an option if that is easier, and you can check the raw txt file for the tx and rx messages or run that file through the driver translator but then you won't see any TCP flags that you'd get with Wireshark which might indicate a low-level problem.

[jorge.florez]

Hi guys: All of you have part of the solution. Last night I realized that I have a configuration problem with two redundant switches that communicate the physical servers with the devices in the field. I tested unplugging one of the switches and the problem disappeared, so I could inferred that the root and backup configuration were not working properly. Thank you very much for replying so soon.