Network traffic and the EBC100

[stoic-one]

I am planning a new project with a field I/O network using a T1H-DM1E, 3 EBC100's, and a C-More O/I.  Each remote/EBC100 will have 3 I/O modules attached to it.

That said, previously I did a project with a T1H-DM1E, 1 EBC100, and a 2 C-More O/I's, which would get intermittent timeout errors on the EBC100 and C-More's.  I alleviated this by moving one of the C-More's to the serial port on the processor.  This was somewhat confusing to me as the C-More project was reasonably well optimized for communication purposes, the tags were somewhat memory concentrated, including alarm tags.

Is there a recommended Poll Rate/Timeout setting for the slaves to reduce traffic on the network?
Should I consider connecting the C-More serially so it doesn't interfere with the I/O portion of the network as I did previously?

Any suggestions are welcome.

[BobO]

There are many variables here. In general, we would prefer that when you use Ethernet I/O, you do not use the Ethernet port for any other function. It's simply impossible to guarantee that the I/O will work perfectly when the Ethernet port is exposed to any other traffic. Will it work well? Sure, and it will tolerate a great deal, but you do end up with retries and such.

Let me give you a bit of background and some things to check. First and foremost, DM1Es are PLCs. That means that the most important thing we do is to read inputs, solve logic, and write outputs. To that end, there are a couple of limitations imposed on Ethernet traffic. They are:

1) The number of hardware level input buffers built in the Ethernet core. That number is 6 and means that we can handle 6 frames without servicing the Ethernet port through an interrupt. If we get a 7th packet before pulling at least one packet out of the core, we'll end up dropping that packet. $EthMissedFrames (DST49) tracks the number of packets that were missed due to inability to keep up. With reasonable pacing we don't see too many of these, but they do happen when there is a very high network load.

2) The number of packets we will allow during a single PLC scan. This number is 16 + 16. The first 16 will be handled regardless of the rate they come in, but after hitting 16, we then start tossing packets on the floor if they come in faster than 1 every 100us. When we finally hit 32 total, we shut off the Ethernet port until the next PLC scan. You can see how many were dropped in $EthDroppedPkts (DST40) and the number of times the Ethernet was shut down in $EthStoppedIntr (DST41).

Why do we do it that way? In testing, we did some classic hacker 'denial of service' stress tests and found that you could virtually shut down the PLC with enough Ethernet traffic. Clearly that doesn't meet with the goal of solving logic, so the challenge became to come up with a methodology for managing Ethernet traffic in a way that would allow you to do some pretty heavy lifting on the comms, but still do your primary PLC function. The 16+16 approach is what we came up with and it works very well. Dialing back traffic obviously introduces the possibility of losing important traffic, but the defense mechanism is a necessity.

Limiting Ethernet to just the I/O function (on an isolated network) will guarantee that you don't have issues...however...if you are comfortable with the bumps that come from doing more with it, there is no harm in doing so. The best way to quantify how you are doing is to monitor $EthMissedFrames, $EthDroppedPkts, and $EthStoppedIntr.

[ADC Product Engineer]

The likely culprit here is the amount of "fast" traffic that the five devices were generating.

3 EBC's trying to update as quick as they can + 2 C-more's trying to update as fast as they can; something has to give.  While Ethernet comms are quite fast, there still has to be some interdelay of packets so that other devices can get a word in edge-wise.

Put a bit of poll delay in the C-more panels.  It is very unlikely that even at 100 to 200ms of poll delay that the operator will notice it unless the screens have tons of objects on them.

[BobO]

The likely culprit here is the amount of "fast" traffic that the five devices were generating.

3 EBC's trying to update as quick as they can + 2 C-more's trying to update as fast as they can; something has to give.  While Ethernet comms are quite fast, there still has to be some interdelay of packets so that other devices can get a word in edge-wise.

Put a bit of poll delay in the C-more panels.  It is very unlikely that even at 100 to 200ms of poll delay that the operator will notice it unless the screens have tons of objects on them.

EBCs are strictly slave devices and do not generate any of their own traffic. The only purely asynchronous and unregulated inputs are the C-more's and any other public traffic on the network. Don't know anything about the C-more's comm implementation, but if they are single request/single response, 5 total devices shouldn't be enough traffic to cause any issue at all. I am curious to see what values the OP has in those diagnostic regs. My hunch is that he has other traffic on the network and moving the C-more was enough to mitigate it. Pacing the C-more is a fine answer if the C-more is causing the issue, but since there shouldn't have been a problem at all, I'm more curious about other stuff on the wire.

[stoic-one]

Thanks Bob, I appreciate your detailed reply.

I found out the hard way that the I/O portion of the network can be a bit traffic sensitive.  The second project mentioned above(which I actually did 1st) was not one I designed, and I had to work around the physical configuration to some extent.  Monitoring the missed frames/dropped packets is exactly what I used as a reference to tweak the time-out and polling values.  When everyone on the network has a polling value of zero (0) and is screaming at the PLC for their data, things have a way of getting bogged down.

I don't have anything on these networks that is time critical as long as things appear, at an operational level(read: Guy pushing buttons on C-More or PLC controlling I/O), to start/stop when commanded to do so.  I'm not trying to optimize it to perform as fast as possible, just be stable and reliable.  I do understand it is the C-More that is the "wiener in the warp-drive" here... This is a completely isolated network, there are no other devices generating traffic, just the devices listed.

To that end, I was just curious if there were any reference values that others had found to work for them, sans the trial and error method I used previously.

Thanks much

[stoic-one]

The likely culprit here is the amount of "fast" traffic that the five devices were generating.

3 EBC's trying to update as quick as they can + 2 C-more's trying to update as fast as they can; something has to give.  While Ethernet comms are quite fast, there still has to be some interdelay of packets so that other devices can get a word in edge-wise.

Put a bit of poll delay in the C-more panels.  It is very unlikely that even at 100 to 200ms of poll delay that the operator will notice it unless the screens have tons of objects on them.

EBCs are strictly slave devices and do not generate any of their own traffic. The only purely asynchronous and unregulated inputs are the C-more's and any other public traffic on the network. Don't know anything about the C-more's comm implementation, but if they are single request/single response, 5 total devices shouldn't be enough traffic to cause any issue at all. I am curious to see what values the OP has in those diagnostic regs. My hunch is that he has other traffic on the network and moving the C-more was enough to mitigate it. Pacing the C-more is a fine answer if the C-more is causing the issue, but since there shouldn't have been a problem at all, I'm more curious about other stuff on the wire.

Just to clarify a bit.

The AD Tech support guys told me they didn't recommend having C-More panels on the same network as the I/O, I can only speculate where they derived that piece of information.  This is why I moved one of them to serial.

Edit:
I did increase the polling time on the remaining C-More still on the network to reduce some of the traffic.

[BobO]

I'd love to, but it's virtually impossible to distill it to a simple guideline. You could have just 8 or 10 packets per second cause an issue if they all came in at exactly the same time, or you might process 8000 per second without a single retry if they were perfectly spaced out and your PLC had a very fast scan time. I can say with confidence that on an isolated network you can run the full Ethernet I/O config with large I/O bases and basically never see a retry, but the moment you throw any public traffic into the mix, all bets are off. I suspect that you could run a handful of bases with some C-more traffic and again see no issues at all...as long as there was no public network traffic.

An anecdote here at Host: We had a managed switch of some sort that would occasionally burst out some kind of gratuitous ARP traffic like he got them in a Buy 1/Get 10 Free sale...10+ very small packets as fast as he could paint them on the wire...all were the same content. We were sure we had issues with the Ethernet core either in the DM1E or the EBC100 because at odd times we'd get timeouts for no meaningful reason. After much study of Wireshark traces we were finally able to catch that bit of broadcast nonsense and removed the offending devices. As expected, the I/O got stone cold perfect after that.

If the network is isolated and you don't have any rogue device, I am puzzled about how only 5 or 6 devices operating in a purely half duplex master/slave mode could be causing issues...unless C-more is actually running multiple outstanding requests and/or multiple sessions, which is really not the way we intended it to be done.

[BobO]

The AD Tech support guys told me they didn't recommend having C-More panels on the same network as the I/O, I can only speculate where they derived that piece of information.  This is why I moved one of them to serial.

They got that from us. My original intent was to put the port in a mode...Ethernet I/O *or* Programming...but never both. After studying it during testing, we concluded that both could be done safely (with caveats) and left it as is.

You have exposed something though. There is no reason for issues with only 5 devices on an isolated network. Are you running Designer on the same network? If so, the PC might be generating traffic and a Wireshark trace might be instructive. If not, perhaps C-more is doing stuff I'm not aware of...and a Wireshark trace might be instructive there too. Regardless, I'd like to understand this better.

[stoic-one]

As I noted, the project which gave me trouble was the one with two C-More panels, one controller, and one EBC100.

I have only been onsite with this customer a few times as I usually do the design(not in this case), offsite electrical drawings, offsite programming, and then onsite start-up, with maybe one follow-up visit to resolve any customer requested changes.  This problem was brought to my attention after onsite start-up, communication time-outs displayed on the C-More intermittently, and other comm errors shown in the Do-More software while online.  I will say that the processor error logs seemed to be reasonably accurate.

Not being onsite now, I can tell you that the switch used was an Automation Direct 5 port un-managed, based on the BOM for the project.  As I said, it's a closed network, everything has static IPs and there isn't even a router on the network. It's completely dedicated with only the following devices:
1 DM1E
1 EBC100
2 C-More 15" touch panels - Not a poorly done O/I, data is packed together pretty well.
1 Laptop running Do-More for programming, but only when I'm there doing it.  They DO NOT have the software onsite.

That was the original configuration prior to moving one of the C-More panels to the DM1E serial port.  They tell me it still displays timeout errors on the C-More that is still connected via Ethernet every once in a while.  I'm not there, who knows...

[BobO]

That's a very light load and shouldn't cause any trouble at all. What are all of the Do-more versions...firmware, gate array, etc? What is the PLC scan time and max scan time?

When you said you were using the diagnostic registers to tune, what were you seeing...missed frames, dropped packets, both?

A Wireshark trace would be a beautiful thing to see, although it would require a switch with port mirroring...

[stoic-one]

That's a very light load and shouldn't cause any trouble at all. What are all of the Do-more versions...firmware, gate array, etc? What is the PLC scan time and max scan time?

When you said you were using the diagnostic registers to tune, what were you seeing...missed frames, dropped packets, both?

A Wireshark trace would be a beautiful thing to see, although it would require a switch with port mirroring...

I am not onsite, so I can't access the C-More panels but they were EA7-T15C both with whatever is the latest published firmware, C-More SW 2.78.
Dropped packets and missed frames, both, one more prevalent and useful, cannot remember which.

Perhaps the next time onsite I can gather more information by taking an old hub and use Wireshark.  That kind of screws up the physical configuration...

Edit: My apologies, I wasn't planning on revisiting this problem in this thread or I would have come more prepared.  Also all cabling tested out at 100MB.
Fact of the matter is I didn't know at the time, or I would have contacted Host when I was fussing over this.  AD Tech guys helped, but I think it would have been resolved more smoothly here.

[BobO]

I want it to work right for you. We're willing to work as hard on it as you are.

[stoic-one]

I want it to work right for you. We're willing to work as hard on it as you are.

Understood.

Just to clarify, I'm a self employed contractor. Unless a problem manifests itself in a manner that prevents them from operating, they won't pay me to come onsite/travel to their facility. It's over 6 hours away and I have other irons in the fire, like the one initially mentioned above, which is a 9 hour drive...

Having got that out there, my earliest opportunity, I'll gather what information I can and provide it for you to analyze.

Thanks Bob

[BobO]

Bottom line is that with only 2 C-mores and an EBC100 on an isolated network, it shouldn't be possible to generate enough traffic to cause a missed frame, and unless your scan time is painfully high, you shouldn't see any dropped packets either. So the answer to the initial question should be "let 'er rip".

Since there does appear to be an issue which was mitigated to some extent by removing one of the C-mores, one or more of my assumptions are incorrect, and without more info to identify which, I can only speculate. Speculations and provably incorrect assumptions make the nerd in me pretty twitchy, but I certainly understand the economics of contract work and long road trips. I do want you to know that when you are in position to dig deeper, we are willing to invest as much time as it takes to get the actual answer and make corrections as necessary.

[stoic-one]

Thanks Bob, I'll get you the information, it will probably take some time, but I'll get it.