EGX100MG stops communicating with ModbusRTU devices when even 1 of them is powered off

[Aafaq_SAH]

Hi,

I'm using an EGX100MG to communicate data from ModbusRTU based power meters (ION6200) onto a PLC. The PLC is the modbus master and uses ModbusTCP to connect to the gateway whereas the power meters are modbus slaves and I have daisychained them onto the RS485 port of the EGX. The way the PLC establishes connections with the slave devices is by opening separate ModbusTCP connections through the gateway which is why the statistics page of the EGX shows 17 active connections as I have 17 slave devices.

Everything works perfectly but as soon as I power off even 1 of the power meters, communication with the rest of the slave devices terminates as well. The active connections start showing 32 (which is the maximum number of connections the EGX can open) and no data is retrieved by any slave device. As soon as I turn the power meter back on, everything starts working again. 

I'm new to this kind of work but I'm pretty sure that this shouldn't happen; if one of the slave devices gets disconnected, the rest should still communicate. Could the issue be from the PLC side? Or have I missed some configuration setting in the EGX? Any help would be appreciated.

Kind Regards,

Aafaq Shahzad.

[Charles_Murison]

Hello @Aafaq_SAH ,

The only way to know for sure what is happening would be to put a serial monitor on the RS-485 loop as well as having switch port mirror traffic to a computer running packet capture software like Wireshark. This is not always possible, timely or cost effective. 

What may be happening is timeouts. Will give an example of 3 devices but this can extend to multiple. Static condition is modbus TCP request comes in, EGX converts this to serial request and transmits on the serial loop. Meter A gets request and responds. Same for meter B and meter C. Because each unit ID has a separate connection, it is possible multiple requests come in at the same time or at least before meter A has had a chance to respond. The EGX will store this request in a buffer to send out after Meter A has responded.

Now Imagen that Meter A is offline, However request for Meter A still comes in. EGX sends request and waits for the response. The timeout is a setting on the EGX and I believe 3 seconds is default. So 3 seconds have gone by before EGX has sent the no response from downstream modbus Exception. During this time, any and all requests for meter B and C are in the buffer waiting to be sent. Depending on the PLC programming, it is possible that the original requests have timed out before the EGX has even sent the response and PLC will try again.

What the PLC does next I can't say, one possibility is PLC thinks connection is broken so starts a new session with a new request, this also times out, etc., etc. Repeat until all connections are used up.

Since there is only 1 serial line, having multiple TCP connections does not improve performance much, yes you may see a small benefit this is still quite small and could lead to other troubles. If only 1 connection is used, likely PLC will know not to try talking to other devices until Mete A responds.

Again I have not see serial traffic or packet captures so this is only a guess. Having 1 device on serial loop down can take 90%+ of the bandwidth just waiting for device to time out.

Regards,

Charles

[Aafaq_SAH]

Thanks for the feedback @Charles_Murison.

You are right. The timeout on the EGX is set to 3 seconds which is the default. However, in the PLC program, each device has been given a timeout of 2 seconds. So I think the PLC times out the message request before the gateway can send a timout response to the PLC and therefore the PLC tries again and again continously which causes all other devices to keep on waiting. 

I think this might be the issue but am not fully sure. Should the timeout for each device in the PLC program be more or should it be less than the timeout set in the gateway?

Regards,

Aafaq Shahzad.

[Charles_Murison]

Hello @Aafaq_SAH ,

Timeout in the PLC ideally would be gateway timeout * number of devices + small buffer. However this is not always possible, especially if there are a lot of devices. At a minimum the PLC timeout needs to be greater than gateway timeout. Just imagen if 16 of the 17 devices are timing out and 17th device is the last request in the queue. That is a long time required before you can confirm if device is talking or not.

Regards,

Charles

[Aafaq_SAH]

Hi @Charles_Murison 

But I think that is true if the PLC was treating the devices as ModbusRTU. In my case, the PLC is treating the devices as ModbusTCP (as the PLC does not know that the devices are ModbusRTU over gateway). That is why the PLC establishes separate connections to each as ModbusTCP allows this functionality (which is also why the active connections in the statistics of the EGX show 17). So in my case, I think I should increase the timeout of each device from 2 seconds to 5 seconds in the PLC.

Any thoughts?

Best Regards,

Aafaq Shahzad.

[Charles_Murison]

Hello @Aafaq_SAH ,

5 seconds would be a minimum. Imagen if 3 meters on the loop are down, the back to back timeouts would be 9 seconds at a minimum. 5 seconds would give you buffer for 1 device being down. 50+ seconds would give you 16 devices going down. Each setting has own pros and cons.

Higher timeout gives more reliability but much longer response time to confirm communications are down. Longer timeouts gives quicker response times however has risks if multiple devices are down comms may break.

I cannot give you an exact number. Depends on risks acceptable and performance requirements. All I can give you is timeout should be something between 5 and 50 seconds.

Regards,

Charles  

[Aafaq_SAH]

Hi @Charles_Murison,

I see. But the maximum timeout I can set on the EGX is 10 seconds meaning that I can only tell if 3 devices are down at max. Also, I will have to change the timeout setting in the PLC program to more than 10 seconds for each device otherwise it will keep retrying. So what do I do in this case? I can't decrease the timeout time to less than 2 seconds in the PLC program otherwise the devices start timing out. It seems that I can only know if 3 devices are down at a time. Any more and the devices won't communicate.

Best Regards,

Aafaq Shahzad.

[Charles_Murison · Accepted solution]

Hello @Aafaq_SAH ,

Want to have EGX timeout as low as possible to keep communications up and PLC with timeout as high as possible. PLC timeout has to be greater than EGX timeout.

Regards,

Charles

[Aafaq_SAH]

Hi @Charles_Murison,

I understand. I think I can decrease the timeout a little bit to make communication better. Nevertheless, I have a much better understanding of everything now.

Thanks a bunch for the help Charles. Have a great day!

Kind Regards,

Aafaq Shahzad.