Modbus Communication Errors

[EngYoug006]

Hi everyone,

We are constantly facing communication loss between Modbus Devices (TeSys, ATS48, PM8000, SEPAM...) and PLC (M580 Hotstanby) in our Ecostruxure Hybrid DCS system ,  with recurrent following alarms for instance; Device Failure, Comm Error, ... (refer to the attached images). Devices stop randomly; we have a yellow Bell on our SCADA (SCADA Citect) and you only need to acknowledge the alarm on SCADA to get the communication back. Notice that this issue occurs on different zones randomly and too many times. 

What could be the main reason, knowing that we explored many possibilities  (Earthing, harmonics, performed some software changes such as adjusting number of Active Message Requests on Egx Gateways settings or eNOC port settings; hardware issues such as cables, gateways or switches...) ?

Thank you

[Mnabil]

Hello @EngYoug006 

the error has many reasons such as cable, number of devices on each loop, number of requests,,, and setting parameters like timeout on EGX or gateway.
Please share with me the architecture or the network and the protocols that we are using.

on the other hand, please check the timeout setting on both sides of the M580 and EGX also check the repetitive rate in M580

[EngYoug006]

Hi @Mnabil 

Kindly find enclosed some pictures of our communication architecture as it appears on SCADA and a PDF of our main network.

As you will notice on the diagram, we opted for a Schneider standard topology which consists of Main RSTP FO ring (PLC and Field), VFDs Modbus TCP sub-rings, star topology (using hubs before gateways) for Modbus Serial devices. Main and redundancy ETH networks (Control room). We also used DRS Connexium Switches which ensure communication with eNOC ports ( I am also wondering if switches do not have something to do with these issues).

Regarding software config in Hybrid DCS, we adjusted Active Message Requests of gateways as per number of Modbus Clients connected to each gateway but still we have intermittent and unstable communication.

On the other hand, I will check timeout settings as you recommended.

I remain at your entire disposal for any detail.

Kind regards

[Mnabil]

I did not see any obvious issue in configuration or system architecture, but it seems the problems accrue more in the devices that are linked to EGX , so please check the EGX settings, I attached print screens for some recommended setting  

[EngYoug006]

Yes exactly, errors occur  with Modbus devices connected to EGX gateways. I will make these recommended changes and see what happens. 

Other thing, Should I put the same value (32) for "number of active sends" on hybrid DCS config noting that the max value should be 4 according to doc ? See attached pictures.

NB. A max of 8 Modbus clients is linked to each gateway in our case.

Regards

 

[EngYoug006]

PS. We have some EGX with firmware where I can't find max connections parameter, is 32 by default ?

Attached pictures.

[Mnabil]

it is Okay, try to set it to the max value and make the time out 1 sec or 0.5 sec

[EngYoug006]

Hi again @Mnabil ,

In fact, I applied recommended changes on EGX SP timeout  as well as for max connections and put  "number of active sends"  setting on its max value i.e 4, no improvement so far.

I have another probability even though it does not affect other Modbus devices in other MCCs, shall we use EOL resistors VW3A8306RC on X10 port (Out port on LU9GC3 Hubs with Star Topology) to prevent EMI on devices ? See attachments.

[Mnabil]

I tested to add the EOL resistor but usually no enhancement, could you advise what are the numbers of devices or the requests  in each Modbus \EGX loop also Please check the EGX diagnostic and the amount of data 

[EngYoug006]

We already checked and tried to adapt the famous setting on hybrid DCS number of active sends" as per the number of devices on each gateway but didn't get concluding results. Nonetheless, number of Modbus devices on EGX settings (webpage config) is set at default value: 10 devices starting with register 1000. Regarding diagnostics, we have communication errors on our EGX exactly where we lose com PLC-Clients. 

I will share with you some screens to make it clear.

[Robert_Willis]

The EGX150 is a Modbus TCP/IP to Modbus Gateway and per the Modbus specification the EGX150 is the Master, and the connected devices are the slaves. Each individual Modbus Message MUST complete with a good or bad reply, before the next message is sent. The EGX150 has some communication buffers but is incapable of keeping up with a I/O scanner that is opening all of the connections at one time and sending multiple Modbus messages to multiple slaves and results in total communication failure. There should be a setting in PES to indicate that the I/O scanner is operating through a bridge and this needs to be checked. This option will open a single socket to the EGX150 and then send the Modbus messages. 

To verify that the EGX150 is working correctly stop all communication from PES and use a Modbus Simulator to verify communication to each device. I typically use this to test the response time of the Modbus RS485 network and through the years I have used 150 - 200ms as a general rule of thumb per Modbus device as a starting point. Decrease the time on the simulator and determine when the slave device is unable to respond. to the request before the timeout occurs.

[EngYoug006]

@Robert_Willis Thank you for your response. I will first of all have to find the PES I/O setting then test RS485 by next plant shutdown. I'll certainly be back to you for more details.

Kind regards 

[EngYoug006]

Hi @Mnabil,

Here are some EGX screens about our yesterday's conversation.

Best regards

[EngYoug006]

HI @Robert_Willis,

I want to share with you some pictures and Communication Control Guide.PDF, Diagnostics Control Services Guide.PDF,  so that you have better understanding of our Modbus network  topology on PES and some examples with templates' settings and their FBDs.

Frist, notice that we used PES Foudation Library Templates for needed devices for instance $TSysUAdvCtlMB, $EGtwMB, $EMPortM...as it appears on pictures. These templates are already programmed and designed for Schneider products most in use, then it makes it simpler and easier to work.

So as a short abstract, Modbus devices communicate with M580 eNOC Port (Ethernet) through EGX (which converts modbus serial RS485 to Ethernet). See Communication Control Guide, Chap15, Pg 124.

Therefore I need your help to find the I/O scanner bridge setting (if applicable in our case or another setting). 

By the way, please advise some Modbus simulator software and its tuto and doc. Thank you.

I remain at your disposal for any details and suggestions.

Kind regards

[EngYoug006]

Hi everyone,

Could redundancy configuration have something to do with our issues ?

See attached pictures.

Regards

[DVarnerMinn]

I have seen similar issues in the past.

Have you found a solution to this?

[EngYoug06]

Hi @DVarnerMinn ,

Not really, we had to go for another solution, let's say a radical one: replacing EGX by PLC and we noticed remarkable enhancements and a better stability.