M580, READ_VAR, time out after working for more than 8 hours

[plcode]

Hello community,

Rack setup:

BMEP584040, firmware v03.20

BMENOC0301, firmware v2.22

Having a state machine polling two distinct holding registers from a modbus tcp slave using a single READ_VAR block by indexing request parameters. The block is edge triggered by state number and NOT activity bit. 

The block consistently fails with communication report 16#01 (exchange stop on timeout) after ~8 hours of successful communication. Activity bit works as expected, goes true for GEST timeout period, then falls with request timeout report. The only way to recover communication is to power cycle PLC or slave.

I have tried the same logic using PLC service port and all NOC ports, the behavior is the same. Modbus slave is responsive when the error kicks in, I have tested it with a master simulator every single time, slave is not the problem.

I think I'm not hitting the READ_VAR block limit since I'm using a single block, my guess is that somehow tcp sockets are being exhausted after several hours but I'm running out of ideas on how to troubleshoot this.

Please advise.

Thanks.

[plcode]

Managed to capture the exact moment when communication fails using wireshark. Request was retransmitted several times by the modbus master (READ_VAR) and received duplicated ACK's from slave, after that, every single request goes to timeout with communication report value of 16#01 even though the slave is correctly responding to the request.

Tried changing slave ip address hoping that the PLC would close the connection but for my surprise, the PLC keeps the connection alive by sending a TCP keep-alive every 30 seconds. 

Is there a DFB or EF that can be used to close the TCP connection?

Thanks.

[dnordenberg1]

Tried the cancel bit in GEST? This definitely sounds like a bug as this should not be necessary.

Btw symptom sounds a bit similar to https://community.se.com/t5/Modicon-PAC-Forum/Modbus-Tcp-ip-configuration-for-Immr-relay/m-p/501809#...

[plcode]

Yes tried cancel bit in GEST and also CANCEL function without success. Per the documentation, cancel bit just cancels an on-going transaction but do not close the underlying TCP connection, also confirmed using wireshark, that TCP connection is kept alive by the PLC by sending a TCP keep-alive packet every 30 seconds.

[dnordenberg1]

I would contact Schneider support so they can escalate this and maybe pass it on to the developers.

But I would try 4.30 firmware first, not that it fix something that comes to my mind but it would probably be a thing they want you to do so they don't need to dig in older code that might have been changed anyway. There was a big code change 3.x->4.x so it might even be fixed without being explicitly mentioned in release notes. Otherwise 4.x is mostly about security features and I would now consider 4.x very stable, early version shortcomings have been fixed.

[MatthewM]

Good job on the analysis.  I can't see your snips, but I can imagine a bit what your data looks.

This is quite unusual, and I am going to offer some information and thoughts that might help.

We do work on several large sites running similar if not the same FW versions of CPU and NOCs, using WRITE_VARs for peer to peer comms on combinations of both the CPU and NOC cards.  ECE v14 and v15.  These are solid and the comms does not fall over.

1.  If the PLC architecture, FW, and SW versions are the same, why does the WRITE_VAR never fail?  Perhaps the issues could be with the READ_VAR FB as opposed to the hardware or FW?  What version of the READ_VAR FB is in your project?

2.  Because of what you have caught with Wireshark, it suggests to me that it may be deeper than the READ_VAR FB and sit somewhere with the CPU or NOC management of the connection.  But that disagrees with point 1 above because the WRITE_VARs have never (to my knowledge) failed without just cause (like network outage).

We have also used READ_VARs extensively (on M340s and Premiums mainly, before the M580 was available), and had not known of any issues until recently where one site mentioned to me that they were having regular issues with READ_VARs on a M340 to a remote device (via fiber, serial and radio link!), and had to make use of the cancel bit.  I have not dug into that deeper but as far as I know it worked fine for many years, and I think it is more likely flakey radio comms that is the issue there.

Can you jump on the web page (you may have to enable it first) for the CPU/NOC and see the list of connections it has open?

And to confirm, is this a single CPU or part of a hot standby arrangement?  Some other forum discussions suggest issues with the READ_VAR in hot standby if the variables are not configured correctly for synchronization.