Do-More Communication lock up

[jbenek]

I have a total of 10 MODBUS TC/IP devices hooked to my Do-More

3 are I/O devices, 5 are data polling devices and 2 are device servers polling serial power meters.

How I wrote the program is I have 1 main program. This contains my ladder logic. Then I have sub programs. Each one has the Read / Write boxes in it for each device.

Then in the main program I have run commands for each sub program.

Initially I had the sub programs scrolling threw at a rate of one every 100 ms.

This caused communication lock up of the Do-More. I changed the poll time to 1 second. No help.

I talked to AD tech support and they suggested that I just turn all the programs on at the same time with an ST1, then have an ST1 on all the read boxes then use the Read box timers. Still locks up.

Then I tried getting rid of all my sub programs and just putting everything in the main. That also locked up communication.

Ive done this same logic with 5 devices in the past with no issues. It seems when I add the additional devices this killing my scan time and making everything lock up.

What is the best way to poll a bunch of different Modbus devices? Read and write?

[BobO]

What do you mean by locked up? We've done a great deal of testing with large numbers of devices and I have never had a CPU lock up. There are a lot of possible reasons why the comms could be unresponsive, but I'm going to need more information to make a guess at why and make a recommendation.

[BobO]

I'm going to take a swag here and guess that one or more of the MRX/MWX addresses are wrong or the addressed slave isn't responding. You didn't mention whether you added additional Modbus/TCP Client devices or whether all 10 Modbus devices are using the default @IntModTCPClient, but I'm also going to guess that you are using only the built-in device. If both guesses are true, then the issue is that the MRX/MWX instructions are taking a very long time to fail when attempting to talk to the one or more devices that aren't responding. Since you are likely using one connection to talk to all 10 devices, when one stalls, they all stall.

Having a single device do all the work is analogous to having one telephone that you are using to call 10 people. For each person, you make the call, exchange information, and hang up. In this case, one of the people isn't answering, and it takes you a while to make that determination before moving onto the next person. While most automation timeouts are on the order of 100s of milliseconds, when establishing a TCP connection, the TCP layer timeout is 45 seconds. We've lowered that to 15 seconds in later firmware, but it is still a very long time in automation terms.

There multiple possible solutions, but the very best is to create a dedicated Modbus/TCP Client device for each unique device you talk to. This prevents any single device from shutting down the comms to the other 9, even while taking a very long time to timeout. Bascially, use 10 phones to talk to 10 people, and after calling the first time, you never hang up. In addition to being robust, it is much higher performance.

[jbenek]

I'm working on the program this morning.

I will better explain here in the next 1/2 hour.

The "Lock Up" is the Do-More designer, not the processor. Sorry for the confusion. I'm getting the red "Error!" box at the bottom.

I am using individual Modbus client's

I'll confirm setting here shortly.

[BobO]

From the System Information dialog, what are your Do-more and OS versions?

[jbenek]

Do-more 1.0.0 and OS 1.0.7

[BobO]

You are essentially running initial release firmware. I would definitely be on the latest. There were some changes in the networking that could potentially affect this issue.

That might help, but I would still like to understand exactly what your program is doing. The biggest thing to consider as we work on this is that the CPU has a finite ability to process packets. The lowest level limitation is the receive FIFO, which is only 6 packets. We do process the FIFO by interrupt, by there are layers of interrupts and the Ethernet interrupt is not the highest priority. That only comes into play of you are using more than one comm port at a time, but even if you aren't, if we got 10 response packets at precisely the same time, we might start dropping packets. We had a device on our local network, it was a managed switch I think, that would send out flurries of 10 or 12 broadcast packets at a time, we effectively 0 time between packets. Our CPU wasn't amused.

Please start by updating the firmware. If it is still misbehaving, we will move to making sure that we are sequencing the devices, at least partially.

[jbenek]

Great, I'll try that now. I got stuck in a bunch of meetings today and didn't get to focus on this as much as I wanted to.

I'll let you know

[jbenek]

Thanks so much Bob, that solved the issue.

I'll be sure to update my firmware from here on out. I feel dumb but relieved at the same time 

[BobO]

Thanks so much Bob, that solved the issue.

I'll be sure to update my firmware from here on out. I feel dumb but relieved at the same time 

Feel dumb? Good gracious there's no reason for that! Do-more has been very solid, be we've had our share of bugs. I'm just glad it fixed the issue. I thought it might, but wasn't sure. There were some issues with the default length of the socket queues causing higher than acceptable error rates in some situations. Unfortunately tests can be so sanitary that bugs escape testing but are instantly found by real world apps. Sometimes we catch them in house before they are reported. Those may get fixed before anyone is even aware...so it is best to stay up to date. We might have snuck some good stuff in there.

[jbenek]

Do-more is very solid. I've been extremely happy with it. Way more intuitive than direct soft ever was. Its actually enjoyable to use. I currently have about 20-30 Do-more units deployed. Some have been running energy plants for over a year with no issues at all.