BRX stops responding to MTCP client

[Controls Guy]

It's the week of nonsense problems!    Had a BRX suddenly stop accepting MTCP writes today, but still talks to other MTCP clients, and will even talk to those same nodes for DMD

System:

• Customer's PC sets MC bits in BRX to request specific recipes
• HMI talks MTCP to BRX
• No POM
• BRX configured for 10 MTCP sessions, 60 second timeout, port 502
• All nodes on main site network.   Estimate probably under 100 nodes, including PC, IP cameras, etc.

History and current observations:

• Been working fine for a couple months
• HMI still talking fine
• DMD, installed today for troubleshooting on another PC in subnet ("Manager's PC"), communicates with BRX fine
• Modbus Poll on Manager's PC reports no response from BRX to either write/poll MC's and continuous polling of MHRs.
• Tried cycling BRX with no improvement
• Customer Wiresharked the normal client PC requesting jobs.   There are some transactions both to and from BRX (ACKs, SYNs and so forth) , but no response to the actual MTCP write packets. PCAP file has been emailed.   BRX IP is 192.168.35.80.

So the BRX seems to be not accepting Modbus writes, but only selectively; HMI still writes, reads, everything.

ModbusTCPServer.Transactions is incrementing wildly, but the HMI polls pretty fast.   .Errors and .LastError both at 0.    Any ideas or anything else to check?   I proposed temporarily disconnecting the HMI but with the client PC not able to send jobs, the HMI is the only safety net, so they don't want to disconnect it, even temporarily.

[BobO]

Looks weird to me. Filter your trace on the two relevant IP addresses and port 502. There isn't really much Modbus traffic, but the thing that stands out is the client opens a connection to the server and then immediately opens another one, and then sends a Modbus request on the second one. The PLC seems a bit put out with it and closes the first and then second socket. If the client is doing this often, you are probably burning up your concurrent sessions. Check the server structure to see how many sessions are active.

[Controls Guy]

OK, will check it in the morning.   Thank you!

[Controls Guy]

Yup, active sessions is maxed out (at 10)

What terminates sessions?   I'd think one thing would be the timeout in the MTCP config, but I seriously doubt they're using 9 sessions regularly.

[BobO]

Yup, active sessions is maxed out (at 10)

What terminates sessions?   I'd think one thing would be the timeout in the MTCP config, but I seriously doubt they're using 9 sessions regularly.

The answer to your question is the Inactivity Timeout. The PLC shuts down any session after the server has gone silent (on that connection) for that period of time.

Your client created two sessions back to back without even using the first one. While I have no proof of it, I assume that wasn't the first time it did it, and if it is doing it often enough, there may not be enough sessions ever. Ideally clients open connections and then keep using them without closing and reopening. I think if the session is closed normally we'll free it up immediately, but if it creating two rapidly without sending a request, that would fall to the inactivity timeout to clear. If the timeout is long enough and the new sessions are created often enough, you'll run out.

[Controls Guy]

OK, changed timeout from 60 to 5 seconds and sessions from 10 to the max of 16.    Should work now, so I'll watch the active session count while they run and see what it does.

I might get a job/recipe request from them say every 30 seconds or so, so if I make the timeout less than that, will it cause problems for the client, not having an open connection the next time they want to use it?   Will it typically just loop back and open a new one (which will now be OK, since we're aggressively closing unused ones).

[BobO]

OK, changed timeout from 60 to 5 seconds and sessions from 10 to the max of 16.    Should work now, so I'll watch the active session count while they run and see what it does.

I might get a job/recipe request from them say every 30 seconds or so, so if I make the timeout less than that, will it cause problems for the client, not having an open connection the next time they want to use it?   Will it typically just loop back and open a new one (which will now be OK, since we're aggressively closing unused ones).

No clue what the client will do, but since it was the connection-happy one causing the issue, I assume it will keep at it until it's happy.

[Controls Guy]

No clue what the client will do, but since it was the connection-happy one causing the issue, I assume it will keep at it until it's happy.

That's my suspicion as well.  Customer is in contact with the mfgr tech support so we may get an answer.