EtherNet I/P scanner comms dropping

[JasonO]

Hello,

I just set up a machine with a BRX PLC and a Nitra PAL valve bank. The valve bank is controlled via Ethernet I/P. I set up the EtherNet I/P Scanner in Do-More (see attached screenshot). I'm not using any input modules, so I set the RPI under input at 250 ms. Under output I set the RPI to 30 ms.

This all works, but I've been having trouble with the communications randomly dropping between the BRX and the PAL. The NS light on the PAL blinks red a few times due to the lost scanner connection, and then it re-establishes connection and keeps on going. I also added a timer in the PLC logic, that increments a counter every time that $PAL.XferCount stays the same for more than 500 milliseconds. I configured the PAL to hold the last state of the valves if the connection is lost, so it doesn't cause serious problems when the communication is lost, but of course I want to get this resolved so it hopefully doesn't happen at all.

I need to do some more testing today, but wanted to post here and see if anyone has any tips for me.

The machine network consists of a network switch with the PLC, CMore HMI (poll time set to 50 ms), and the Nitra PAL connected. At the time when I observed the comms issues, I also had my laptop plugged in with DoMore connected to the PLC, so that would have added some load to the communication.

Is the network overloaded? Do I need to increase the RPI for the outputs? I want the communication to be as fast as possible, because any unnecessary loss of time will adversely affect the machine cycle time.

[MAEdwards]

A Wireshark capture showing the communication issue between the BRX PLC and the Nitra PAL unit would be the most helpful.  Without a Wireshark capture, I can only suggest getting the General Status and Extended Status codes from a Data View in Do-more Designer and providing those code values here.

$PAL.GenStatusCode (0x00)
$PAL.ExtStatusCode (0x0000)
$PAL.ExtStatusInfo1 (0x0000)
$PAL.ExtStatusInfo2 (0x0000)

[JasonO]

OK, I will get a mirroring switch so I can Wireshark the comms. For now I installed an ECOMEX in the POM slot, dedicated to the PAL manifold, to see if that makes a difference. Still getting some drops in the communication, but have not tested enough yet.

None of the status codes had anything in them. It did not appear that the Ethernet I/P scanner structure on the BRX was seeing any problems (ie. Error counter not increasing) even though the comms were stopping briefly.

Will try to test more and get wireshark info.

[JasonO]

I've attached Wireshark captures between the PLC and PAL, and a screenshot of the capture.

In the time I had Wireshark running, the connection dropped 3 times. I noticed a pattern; there is an Encapsulation Sequence Number, and every time it rolls over at 65535, the connection appears to close and then reopen. I don't know enough about this stuff to know what's going on or how it should be working, but this doesn't seem right.

I'd love some input on this. Thanks!

[MAEdwards]

Good job on the data collection and your analysis of the issue.  This helps a lot. 
There may be issues on both sides, but I need to dig into this a bit more.  The BRX is rolling over the Encapsulation Sequence count after 65535, but the Encapsulation Sequence count should be a UDINT.  Since this count is rolling over prematurely in relation to the maximum 32-bit unsigned value, I wonder if the Nitra PAL is interpreting these lower sequence count numbers as old messages and discarding them.  Receipt of older packets will not sustain a CIP connection.  Let me dig a bit more and see if I can get a clearer picture of the situation.

[MAEdwards]

My thoughts...
The Forward_Open request from the BRX specifies a CIP Connection Timeout multiplier of 4 x the RPI.  Once the O->T Encapsulation Sequence count rolls over from 65535 to 0 there are four O->T packets sent from the BRX (sequence count 0x00000000, 0x00000001, 0x00000002, and 0x00000003) before the Nitra PAL issues an ARP request of the BRX's MAC followed by a FIN flag to terminate the TCP connection between the two.  This leads me to believe the Nitra PAL sees this as a CIP Connection Timeout.

One might think, however, "but the BRX is still sending CIP data."  Yes, but the Encapsulation Sequence count values are smaller than the previously received packet with the Encapsulation Sequence count of 65535. If the Nitra PAL considers this to be old data and discards the packets, then this will cause a timeout event; which is what I think we are seeing in Wireshark captures.

Because I have these components, I was able to test and reproduce the reported behavior.  I was also able to swap out the BRX with another EtherNet/IP Scanner that uses the full value range of the UDINT.  With the other Scanner, when the O->T Encapsulation Sequence incremented from 65535 to 65536 (and up), the CIP connection was maintained and the devices continued to exchange their data without interrupt.

[BobO]

Fixed.

[MAEdwards]

BobO,
I can check 'new bits' if you need me to.  Just let me know.

[BobO]

BobO,
I can check 'new bits' if you need me to.  Just let me know.

Sent.

[BobO]

That apparently was the issue. The fix will be in 2.11.1 when it's rolled out here in a month or two. I could probably get you a beta sooner, but a few things are still being revised so it might be best to wait, if possible.

[JasonO]

Thanks!! I think we'll be fine waiting.