Host Engineering Forum
General Category => Do-more CPUs and Do-more Designer Software => Topic started by: jcb on March 08, 2021, 10:14:43 AM
-
New Issue today. Redlion G3 to Do-More via Modbus TCP.
I had some MHR Values that started getting larger than 16 bit unsigned values so I converted them to Doubles at the end of last week. I have been polling this Do-More PLC along with others, DL205's and Allen Bradley Contrologix. This morning I come in and All my other PLC data is online on the Redlion but this PLC reading Longs. I rebooted the redlion and everything came back but this PLC.
I added ModbusTCPServer to a data view and It shows 1 Session with 43758607 Transactions and No errors or last errors. The only unit connecting to this as a server is the Redlion G3. The rest of the TCP Modbus this unit is the Slave and the requests are handled in Ladder Logic (those are all working fine). The PLC is working great and remote IO and everything else is working I have no errors in the system information.
When I unplug the Redlion for 10 minutes the ActiveSession is staying at 1 and the Transactions haven't changed at all.
I then added a MRX to another PLC and attempted to read and Just got an error bit set no change in the Active Sessions or Transactions.
Is it possible the Modbus Server is Locked up?
How would I check this and is there anyway to restart it without power cycling the PLC.
Are there any other things I should check.
-
I don't have any specific knowledge of the Modbus TCP server locking up, but it's certainly possible.
If you can transition to program mode, you can try Re-initialize PLC IO, from the PLC menu. If that doesn't work, you can try tweaking a Modbus TCP server setting (change the timeout or number of sessions) and write the PLC. Either of those might unstick what's stuck.
And you might check $QueuesFlushed (ST145) and $CommStackOverrn (ST146) just to make sure something bad didn't happen in the TCP stack.
-
Redlion recommends only 4 connected devices / Modbus driver. If you have more than that when the HMI is the Master, then split them up on additional Drivers. Otherwise set up the HMI with Modbus Slave blocks and let your connected devices push the data to the HMI.
-
Not sure what it was no errors in the stack.
I had to power cycle the PLC and now everything is working fine?
I converted my reads back to 16 bit and I am just shifting <<16 the MSW and adding the LSW in the Redlion.
-
Not sure what it was no errors in the stack.
I had to power cycle the PLC and now everything is working fine?
I converted my reads back to 16 bit and I am just shifting <<16 the MSW and adding the LSW in the Redlion.
If you can dupe it, I definitely want to know about it.
-
Not sure if this is even remotely related.
Have been running BRX communicating Modbus TCP with ABB VFD. System has been running for about 2 months. BRX lost comms with VFD at some point in the past few days (why, I don't know). Program is set-up to read from drive at all times. Started troubleshooting. Neither success or error bits on the MRX were toggling at 50ms read time. No comm warning in DMD either. Transitioned from Run to Program a couple of times. Communication did not restart. Inserted a PING into the program and the BRX could PING the drive in <10ms. Tried Program to Run a couple more times in DMD with no results. Power cycled the drive with no comms restoration. Tried increasing the MRX read time to 1 sec. No results. Power cycled the controller, comms restored. Thoughts?
-
Not sure if this is even remotely related.
Have been running BRX communicating Modbus TCP with ABB VFD. System has been running for about 2 months. BRX lost comms with VFD at some point in the past few days (why, I don't know). Program is set-up to read from drive at all times. Started troubleshooting. Neither success or error bits on the MRX were toggling at 50ms read time. No comm warning in DMD either. Transitioned from Run to Program a couple of times. Communication did not restart. Inserted a PING into the program and the BRX could PING the drive in <10ms. Tried Program to Run a couple more times in DMD with no results. Power cycled the drive with no comms restoration. Tried increasing the MRX read time to 1 sec. No results. Power cycled the controller, comms restored. Thoughts?
Sounds like a driver state machine was hung. Not a great thing. Would love to know how to dupe it. Generally very easy fi
If it happens again, try "Re-initialize PLC I/O...". If that fixes it, or not, gives me clues.
-
As an additional piece of information, three additional Programs are setup to talk to three additional like drives. These three drives were powered off at the time of troubleshooting with no comms warning in DmD either. I didn't check success/error bits on these MRX commands but I assume they probably weren't toggling either.
Also, I forgot to mention I was not running the most recent firmware/booter. Since I was troubleshooting, I went ahead and updated to most recent.
-
For each new Modbus slave, are you creating new Modbus/TCP Client devices (i.e. one per Modbus/TCP slave), or are some of them sharing the same Modbus/TCP Client device?
It is recommended that due to the overhead of TCP (not Modbus/TCP, but TCP), you create a different TCP client per server (slave) IP Address, in this case, it would be a Modbus/TCP Client device.
-
They all have their own client.
-
Unfortunately have the same issue.
My older 2.7 deployments have not had the issue on poor links, but my 2.8 unit experienced this same lockup.
Sent an email to Host & AD support. I'm not glad, but am anxious to see an update now that others are also experiencing it.
We changed out Communications monitoring from the internal Set Bit structure to a Heartbeat register (clock) from the slave device.
Have not had a lockup where the BRX is the Slave device, so that may help narrow it down.
All slaves are a separate client, polls range from every 5-30 seconds per device. 4 slave devices.
Dedicated ethernet wireless link, typically around 20Mbps with 4-10ms pings.
No links reported down from SCADA (polls the slaves and master independently).
Slaves are all DL-06 with ECOM100.
All transactions use the same registers (like remote IO).
-
Unfortunately have the same issue.
My older 2.7 deployments have not had the issue on poor links, but my 2.8 unit experienced this same lockup.
Sent an email to Host & AD support. I'm not glad, but am anxious to see an update now that others are also experiencing it.
We changed out Communications monitoring from the internal Set Bit structure to a Heartbeat register (clock) from the slave device.
Have not had a lockup where the BRX is the Slave device, so that may help narrow it down.
All slaves are a separate client, polls range from every 5-30 seconds per device. 4 slave devices.
Dedicated ethernet wireless link, typically around 20Mbps with 4-10ms pings.
No links reported down from SCADA (polls the slaves and master independently).
Slaves are all DL-06 with ECOM100.
All transactions use the same registers (like remote IO).
It could be one of a couple of things. The easiest answer is the Modbus driver state machine is hanging. The more concerning answer is that the TCP stack tweaks we made to support the ECOMEX POM is causing issues. If the Success/Error bits are toggling but nothing is going out the port, that points to the stack. If the bits aren't toggling, that points to the driver state machine. Not concrete, but suggestive. Of course there might be hole in the state machine exposed by a stack failure...so both could be true. :o
I would very much like to get to the bottom of this, but doing so would require diagnostic firmware and some time from people who are experiencing the issue.
-
The State bits do not toggle (Success/Fail), we were initially only monitoring the Success bit to reset a fail timer (Off Delay).
They stayed healthy and never produced a Fail set bit either.
-
The State bits do not toggle (Success/Fail), we were initially only monitoring the Success bit to reset a fail timer (Off Delay).
They stayed healthy and never produced a Fail set bit either.
Well, after reviewing the driver code, it could easily hang due to a problem in the stack, so that might not really tell me what I hoped.
How often/quickly does it fail?
-
Ran for approximately 2 months before it failed in April. Sent an email 4/22:
The BRX is also monitored by Geo SCADA (ClearSCADA; HMI/SCADA server), at intervals from 5-60 seconds depending on item requirements.
As of today we monitor/control ~500 various items within the BRX, mix of registers & coils/inputs
The BRX program is 14,720 words in size. Scan times 38ms typically, up to 90ms on occasion.
It has been in operation for about 1.5 months without a power cycle, and suddenly stopped processing the MRX & MWX instructions.
The BRX was failing to respond at various intervals to the HMI/SCADA server.
Using the Domore Designer to go from Run-Program-Run, the physical switch (Run-Stop-Run) did not restart the communications.
Only a power cycle was able to restart instructions/communications and has not failed to respond the server since doing so.
The only error we found was within DST38, set as 11 (Gateway Target Device Failed to Respond).
The server machine also collects some data directly from the slave PLCs, and at no point lost communications.
The BRX and server machine are on the same switch, within the same enclosure, so a true loss of communications to a specific PLC would have been captured by the server as well.
Firmware: 2.8.3
Bootloader: 1.0.9
All were updated to the latest release in Do-more as of yesterday, 4/21/21, but revision history did not seem to mention anything we thought related.
I am sure there may be some other troubleshooting items you may need and would be happy to provide as we can, but had some limited diagnostics time to get the process back online.
-
Ran for approximately 2 months before it failed in April. Sent an email 4/22:
The BRX is also monitored by Geo SCADA (ClearSCADA; HMI/SCADA server), at intervals from 5-60 seconds depending on item requirements.
As of today we monitor/control ~500 various items within the BRX, mix of registers & coils/inputs
The BRX program is 14,720 words in size. Scan times 38ms typically, up to 90ms on occasion.
It has been in operation for about 1.5 months without a power cycle, and suddenly stopped processing the MRX & MWX instructions.
The BRX was failing to respond at various intervals to the HMI/SCADA server.
Using the Domore Designer to go from Run-Program-Run, the physical switch (Run-Stop-Run) did not restart the communications.
Only a power cycle was able to restart instructions/communications and has not failed to respond the server since doing so.
The only error we found was within DST38, set as 11 (Gateway Target Device Failed to Respond).
The server machine also collects some data directly from the slave PLCs, and at no point lost communications.
The BRX and server machine are on the same switch, within the same enclosure, so a true loss of communications to a specific PLC would have been captured by the server as well.
Firmware: 2.8.3
Bootloader: 1.0.9
All were updated to the latest release in Do-more as of yesterday, 4/21/21, but revision history did not seem to mention anything we thought related.
I am sure there may be some other troubleshooting items you may need and would be happy to provide as we can, but had some limited diagnostics time to get the process back online.
I'm seeing two things in that description. 1) MRX/MWX not working, 2) BRX wasn't responding to another device. That sounds like both client and server functions.
-
We first noticed the issue as we were getting random Bad Quality for objects inside Geo SCADA (all from the BRX unit, the items from the DL-06 units were good). These errors happened every few minutes and would last for a short period. No official comm fail ever happened with the SCADA portion (5 minutes). Again, remote units responded to the SCADA during that time without issue.
Then we had the first lock up when we had a well fail to start and the tank level alarms for No Change. This indicated that the MRX/MWX items were failing to operate.
Only solution was a power cycle. Didn't not work with Run-Stop-Run or within Designer (Stop/Program-Run).
-
Another location that might have relevant information is DST411. It is a retentive location so even after a power fail it retains the value. The TCP stack can do exception dumps and panics which we catch and store the return address in that location. We also stick a "panic @XXXXX: msg" or ""dtrap @XXXX: msg" into the system error (ERR, LastERR0-7) location when that happens. That information can be very helpful.
We'd like to fix this.
-
Only solution was a power cycle. Didn't not work with Run-Stop-Run or within Designer (Stop/Program-Run).
Another level of kicking it in the head is Re-Initialize PLC I/O from the PLC menu. That forces the drivers to remount. That fixing it is another clue.
-
Will add that to the SOP ;)
Currently DST411 and DST51 show a 0. We did a firmware update that day to the latest 2.8.5, that reset the values?
-
Currently DST411 and DST51 show a 0. We did a firmware update that day to the latest 2.8.5, that reset the values?
No. They are retentive and only cleared on Clear PLC.
So nothing has crashed, trapped, or panicked. That's good. Not super helpful, but good.
-
For what it's worth, mine is reporting: 9330172.
-
For what it's worth, mine is reporting: 9330172.
I don't have a map file from the actual release but what I have should be close. There is a trap call in the function that this address maps to. The failure could relate to the new ECOMEX. I'm thinking it is probably not a big deal, but it could be. Do you use the ECOMEX?
-
Define "use". On this controller, the ECOMEX is installed in the BRX but is not in use (has not been used, is not configured and no ethernet cable is plugged into it).
-
Define "use". On this controller, the ECOMEX is installed in the BRX but is not in use (has not been used, is not configured and no ethernet cable is plugged into it).
That might cause it. Something appears to be requesting access to the secondary adapter, but it isn't mounted.
-
There's definitely an issue:
So, I've got another system I'm building. In this system I have a Master BRX communicating EIP with two slave BRX's. The master BRX is also communicating (as a client) with four other Modbus devices. At present time the master has lost comms with the two slave BRX's over EIP and 3 of the 4 modbus devices. On the one functioning modbus device the success bit for the read/write commands is setting/resetting as expected. For the other three devices the status bits are not doing anything on the read or write. I don't often use emoji's but: >:(
I haven't touch this system yet in an effort to restore comms. Let me know if there's information that you need.
-
So both Modbus and EIP functions have hung, but one Modbus/TCP client is still functional?
Please try Re-initialize PLC I/O.
-
So....I reinitialized I/O.
Prior to re-initializing, I could see no success/fail bits triggering (MWX or MRX) on the 3 non-responsive Modbus devices. It did look like both the EIP devices messages were periodically toggling the read/write fail bits (they weren't frequent so it was hard to catch them visually).
After reinitializing the I/O, the EIP came back immediately. The working Modbus device is still working and the 3 initially un-responsive Modbus devices are still un-responsive. I'm still not seeing fail toggle bits on the MRX and MWX. I am also not receiving any warning messages about the non-responsive modbus devices.
Edit: Additional Information
Under Device Status, the Modbus device that is working will show it working through all "Locked" MRX/MWX instructions for that device. Periodically the state will transition to "Closed", it will re-open and start cycling through the instructions again. For the other 3 devices, the state shows "Open" but the there's no movement under the "Locked Instruction" category for these devices.
-
How long was the system running before it failed? If it is quick-ish and repeatable, I can easily make firmware that will give me more information about the internals of the devices and IP stack.
-
Not quickish. It had been running for several weeks.
-
I plan on restarting the controller in a little while to see if it restores comms. Is there anything else you need me to do prior to this?
-
I plan on restarting the controller in a little while to see if it restores comms. Is there anything else you need me to do prior to this?
I don't know anything else I can do without custom firmware. Restart it.
-
Power cycle restored comms without additional action.
-
I see some things I can't yet explain. Don't know if it is relevant, but it is certainly abnormal. Investigating.
-
Just out of curiosity, of those having problems, do you have a Gateway defined for your primary Ethernet adapter, or not?
-
The two systems I'm having trouble with have gateways defined for both. I work across different subnets and have to have the gateway defined.
-
The two systems I'm having trouble with have gateways defined for both. I work across different subnets and have to have the gateway defined.
Well, I found a thing, not sure if it the thing.
When we added the ECOMEX, it required enabling code that enabled some routing functions. For outbound packets this is required, but it went the extra mile for us and enabled an internal gateway function which was causing legit havoc in certain conditions. I don't know exactly what those conditions are, but the trigger is for something to send a subnet broadcast, mostly likely to a subnet not configured on either the PLC's NICs. I won't bore y'all with the details, but basically the gateway function commits a faux pas and improperly forwards a broadcast packet. By itself, that's no big deal...but if he tells everybody, and everybody else tells everybody, etc., it's geometric. The TTL in the packet eventually extinguishes the flames, but not before annoying the stack greatly. The immediately obvious symptom is brief comm dropouts and $QueuesFlushed (ST145) getting set.
We'll get a fix out shortly.
As I said, I don't know if this fixes the Modbus issue, but it may well do it. I don't think the stack enjoys being assaulted repeatedly, and after weeks or months of abuse it might have leaked enough resources to become non-responsive.
For reference, the thing that was generating the subnet broadcast was a Win7 PC doing a NETBIOS name resolution. It could be about anything though.
-
Would this cause issues with the EIP messaging? I assume it would.
-
Would this cause issues with the EIP messaging? I assume it would.
Easily. And depending on how frequently a trigger event occurred, comm could be very unreliable, even before things fully broke.
How many BRX do you have in the system?
-
I've currently got 6 BRX's running on my network.
-
So I haven't had any issues with this for a while but also haven't made any changes to the HMI or PLC.
I had to update a report trigger to add a manual trigger function on the PLC. After I wrote the online change to the PLC I lost the ModbusTCPServer communications.
All Modbus and EIPMSG clients are working fine but I can't connect to the Modbus TCP Server from the HMI or my workstation. It shows no active sessions or transaction changes.
I also have an unused BX-P-ECOMEX in the POM slot. I was going to use it for dedicated Remote IO but didn't see any performance increase over a single network connection so I unplugged it but never pulled it.
DST411 is 0
DST51 is 0
OS: 2.8.2
They are running production on the line until the 11:30 PST lunch break at which time I am going to try in the following order
Reinitialize PLC I/O
Check for Comms restoration.
Firmware update
Check for Comms restoration.
Finally a Power Cycle if none of the above work.
Thanks
John.
-
Reinitialize of PLC I/O didn't change anything.
Update to 2.8.5 and after firmware update power on reboot all comms are working.
I will post again if the problem persists.
-
Reinitialize of PLC I/O didn't change anything.
Update to 2.8.5 and after firmware update power on reboot all comms are working.
I will post again if the problem persists.
We'll be dropping 2.8.6 shortly, probably tomorrow.
-
We'll be dropping 2.8.6 shortly, probably tomorrow.
Just days after I finally got with the program and figured out I wasn't on 2.8.5! :P
-
Is there a way to deploy firmware updates to multiple controllers at the same time or is this an update each one individually type of thing?
-
Is there a way to deploy firmware updates to multiple controllers at the same time or is this an update each one individually type of thing?
One at a time. Batch would be nice, but that's kinda tricky.
-
2.8.6 is live.
Please let me know if you have issues.
-
I don't see it under the firmware section on your site. I can't get to the firmware directly.
-
I don't see it under the firmware section on your site. I can't get to the firmware directly.
It's on LiveUpdate. Can you not access it?
-
I am operating on a restricted LAN and cannot perform "live" updates. I have to get the binaries from the website. Have the binaries been updated on the website? If so, the release notes haven't been.
-
I am operating on a restricted LAN and cannot perform "live" updates. I have to get the binaries from the website. Have the binaries been updated on the website? If so, the release notes haven't been.
Emailed to you.
-
Are there any other changes in this release in addition to disabling the broadcast storming?
-
Are there any other changes in this release in addition to disabling the broadcast storming?
Not on purpose.
-
Good line! :D I lol'ed.
-
I always give that line to my wife.
-
Are there any other changes in this release in addition to disabling the broadcast storming?
Not on purpose.
I'm dying over here ;D
Updated the controller we first noticed this on, will update if it has any problems.
Thank you for the fast response to this issue
-
Well, I hate to bring this back up, but I'm not sure this issue is entirely resolved. Came in earlier in the week and I had an EIP message fault. Slave device was online (when I went to operate the equipment) and it took a controller power cycle on the BRX master to restore comms. Just now power cycled a modbus slave device. The BRX showed loss of comms warning. When modbus slave was powered back up, I cleared the warnings in the BRX master. Reporting no warning, the BRX wasn't communicating with the slave. Re-initialized I/O without restoration of comms to slave. Power cycled the BRX master and comms came back immediately. :(
-
Well, I hate to bring this back up, but I'm not sure this issue is entirely resolved. Came in earlier in the week and I had an EIP message fault. Slave device was online (when I went to operate the equipment) and it took a controller power cycle on the BRX master to restore comms. Just now power cycled a modbus slave device. The BRX showed loss of comms warning. When modbus slave was powered back up, I cleared the warnings in the BRX master. Reporting no warning, the BRX wasn't communicating with the slave. Re-initialized I/O without restoration of comms to slave. Power cycled the BRX master and comms came back immediately. :(
Same boat unfortunately.
We added a forced reboot on first Wednesday of month after 11AM so long as no pumps in operation. Also added a method for supervisor to force a reboot (authorization code "5150" ;D)
-
To add to the investigation.
We have two units that stopped responding today, one would not work through DmD and required hard disconnect.
Both on the latest FW.
PLC1:
VPN connection primary office, connected to RS485 flowmeter.
This has two SCADA machines that collect data (across the VPN), PLC is the Server (Slave) via Modbus TCP.
Additionally onsite there is an AVG (EZTouch) HMI on the Ethernet system, all wired.
PLC had 8 connections in config, all 8 were occupied and would not clear. We forced the two SCADA servers to disconnect (Out of Service).
The connections would not timeout on PLC status. After 5 minutes we performed a software REBOOT instructions.
Connections reestablished, shows 3 active now.
PLC2:
VPN Connection to primary office, connected via Ethernet to MDS SD4 (1) radio and RS232 to additional SD4 (2).
Single SCADA server on VPN connected to PLC Modbus TCP Server. Five remote sites polled by the PLC, Modbus TCP Client, via SD4 (1).
Slave connection, Modbus RTU, on SD4 (2); built in port.
Site stopped responding to Client requests from SCADA and also stopped polling slaves at the same time via TCP/Ethernet.
Could not connect to DmD, no response.
PLC continued to respond via serial connection as Modbus RTU slave.
Required a physical power disconnect.
So virtually all of our BRX units with Ethernet connections are beginning to exhibit these problems.
All are on the latest firmware.
Unfortunately, it is not happening on only MWX/MRX configured devices now, it is even on slave only units. Albeit different types of failure (max connections vs. unresponsive device)
-
I'm bumping this again to see if those who have installed Version 2.9 have seen this issue resolved (or at least apparently resolved). I've upgraded some of my stuff to 2.9 but haven't had things on long enough to see if there are still comms issues.
-
I'm bumping this again to see if those who have installed Version 2.9 have seen this issue resolved (or at least apparently resolved). I've upgraded some of my stuff to 2.9 but haven't had things on long enough to see if there are still comms issues.
I'm curious as well, but we're pretty sure we know what was causing that. With the right test we were easily able to dupe it.
-
So, we are still having this issue on 2.9.6. Perhaps not as frequent, but it is still interrupting our processes.
Two BRX on 2.9.6 on a licensed radio link (9.6kbps...) using Modbus TCP. Request is sent every minute, it has had more interruptions due to poor weather. Causes are not limited to "no reponse", can also add old/non-updated data.
Additionally, have other units on 2.9.6 with Modbus TCP links over WiFi type radios, that when we get a series of comms fails it stops updating. These are typicall 10-20Mbps, but do experience a random loss here and there. These stop responding to update requests from the Master unit.
EDIT: Only solution is to powercycle or force REBOOT
-
So, we are still having this issue on 2.9.6. Perhaps not as frequent, but it is still interrupting our processes.
Two BRX on 2.9.6 on a licensed radio link (9.6kbps...) using Modbus TCP. Request is sent every minute, it has had more interruptions due to poor weather. Causes are not limited to "no reponse", can also add old/non-updated data.
Additionally, have other units on 2.9.6 with Modbus TCP links over WiFi type radios, that when we get a series of comms fails it stops updating. These are typicall 10-20Mbps, but do experience a random loss here and there. These stop responding to update requests from the Master unit.
EDIT: Only solution is to powercycle or force REBOOT
We identified an issue in the TCP stack where unstable radios could cause the listening socket to get hung for a long time, on the order of 20 minutes. If the issue that got us there to start with continued constantly, the socket would appear to freeze indefinitely. The specific condition that causes it is to receive two duplicate TCP SYNC frames, and then the client give up on the connection. It doesn't appear to be as common when the client connects to the server and stays connected, but since many radio remotes are connecting and disconnecting every time they read, it appears to happen more frequently there. This never happens on wired connections, which is why it has been almost impossible for us to duplicate.
We have a fix in beta if you'd like to try it.
-
Kinda already running beta now so... Send it over :P