Host Engineering Forum
General Category => Do-more CPUs and Do-more Designer Software => Topic started by: Mike@Forshock on September 16, 2021, 12:52:16 AM
-
Have created a topic and support ticket about this before, it still remains. No seems worse
We got an alert the comms for a specific site had failed (from the BRX).
Site is running perfectly fine, an old DL-06 with ECOM100.
Cannot connect to DmD on local PC to Ethernet. (Connects, Reads program, fails on system config)
Responds to Modbus TCP (Server) from local PC (SCADA)
NetEdit3 can connect, make changes from same local PC
Disconnect/close SCADA connections, allowed the DmD to connect.
Device will not show Success/Error status. The other Modbus TCP devices are working just fine!
Created a new device while it was online, connected the instructions (MWX/MRX) to new device, came up instantly.
This cant keep happening every 2 months...
-
Latest available firmware.
-
It sounds like it is running out of IP stack resources. That could be due to a memory leak (that I don't know about) or a flood of packets from a bad actor. It would have been helpful to know whether $QueuesFlushed (ST145) or $CommStackOverrn (ST146) were active. If it also helpful to know what is in $EthDroppedPkts (DST40) and $EthStoppedIntr (DST41). This give some insight into what might be failing.
Unless we can dupe this, we can't begin to fix it. I understand your frustration, but problems like this are the absolute hardest, and other than a description of the failure mode, this gives us very little to go on. Even with your program, unless we dupe all of the comm activity, we aren't likely to see the issue.
It isn't a good answer, and it may not be viable, but if you are able to detect (from the PLC) that the PLC comms are hung, you could have the PLC reboot itself.
-
I appreciate quick and honest answer, I do understand the difficulty to recreate the problem.
The remote PLC is on a wireless link that used to be very good, but now has some issues (New buildings built in the last year or two).
While the site generally works well, it does get latency and dropped packets. So the low quality of the link is very likely the cause of the problem.
The other four remote units (all DL-06 /w ECOM100) are also wireless, but have not seen same loss of network.
As for the ability to cycle power on the unit, this is not a solution that is reasonable. This unit has a hardwired connection for controlling a water well/pump, so the outputs dropping out will also interrupt that process.
For me, finding the reason the Success/Error bits do not update would be the first check. On a device failure/hang, the error bit should at least be on. When this happens, both remain false. We changed our communications health detection to use a rising edge Success and heartbeat (pseudorandom and analog input; make sure PLC is in run mode) from remote unit.
The DST40 & 41 (Added all $Eth items) are now on our list of things to watch, but would these be from ALL ethernet devices? Currently, these are at 0 for each for the last 10 hours.
-
I appreciate quick and honest answer, I do understand the difficulty to recreate the problem.
The remote PLC is on a wireless link that used to be very good, but now has some issues (New buildings built in the last year or two).
While the site generally works well, it does get latency and dropped packets. So the low quality of the link is very likely the cause of the problem.
The other four remote units (all DL-06 /w ECOM100) are also wireless, but have not seen same loss of network.
As for the ability to cycle power on the unit, this is not a solution that is reasonable. This unit has a hardwired connection for controlling a water well/pump, so the outputs dropping out will also interrupt that process.
For me, finding the reason the Success/Error bits do not update would be the first check. On a device failure/hang, the error bit should at least be on. When this happens, both remain false. We changed our communications health detection to use a rising edge Success and heartbeat (pseudorandom and analog input; make sure PLC is in run mode) from remote unit.
The DST40 & 41 (Added all $Eth items) are now on our list of things to watch, but would these be from ALL ethernet devices? Currently, these are at 0 for each for the last 10 hours.
80ms scan time? That's pretty stout. It shouldn't hurt anything, but high scan times are never helpful for comm. Is it always that high?
It wouldn't require power cycling, just invoking the REBOOT instruction, but it will interrupt the outputs.
The instruction is clearly not working right if it's never timing out. It's a joke here at Host, and I mean no disrespect with this, but when there is a failure in the controller and the mode of failure doesn't satisfy someone, my response is generally "what about 'exception condition' do you not understand?" The bottom line is it shouldn't be happening, and we make every effort to keep things from happening, but bugs are bugs and they don't always behave the way we want them to behave...cause they're bugs.
Given the unstable link, I'm now at 90% confidence this is a network stack resource issue. Wondering if some of the low level networking code isn't accepting a missing/unavailable resource as a valid basis for timing out. That isn't great and shouldn't happen, but when is it ever OK to run out of memory? Never. Exception. If it is that, I would expect ST145 and/or ST146 to be on. They indicate that when the Ethernet hardware is trying to inject a packet into the stack, there were no free buffers. That generally should never happen. While that by itself shouldn't cause something else to hang, I'm wondering if there is some pregnant moment in the TCP stack that's causing a low level hang up.
DST40 and DST41 count up in response to excessive packet loads. They aren't problems necessarily, but they give a glimpse into whether there are packet bursts happening that could be causing issues.
I'll put some thought into this to see if I can come with reasons this might be happening and hopefully some diagnostic checks that would let us see what might be going on.
-
We're formulating a plan to build a random/repetitive hardware failure stress test that would approximate the behavior of bad networking hardware to see if we can force a similar environment to bad radios. Hopefully we can learn something quickly.
-
Scan times are high, we found our own bug in one of our pointer based subroutines. This PLC hasn't received an update yet, but when the BRX gets an invalid pointer (Block of 24, send request for 25...) the scan times shout way up. Had a unit with ~90ms, fixed logic and now below 30ms!
Code for the exception, not the rule :P
Will add ST145 & 146 to monitors and see if we can create an automatic invoked REBOOT or at least user initiated (with failsafes)
--
Edit:
Thanks, hope we can find something.
-----
-
Code for the exception, not the rule :P
Yeah, well, you have to be careful with that. When I say exception I don't mean rare event, I mean gravity quit working or the speed of light changed...things that must never happen. The only real recourse to such a thing is to reboot, which may be fine for some, but probably very bad for others. Your PLC is still functioning even though you've had a failure. For most that is preferable to rebooting. We put the REBOOT instruction in there for exactly these cases, if that's determined to be the best action and safe to do.
The real right answer is to fix the bug and eliminate the exception. We'll do our best.
-
Can you change the PLC to pump run signal to three-wire? Then you could survive a REBOOT.
-
Control Guy: That is something we will plan if this isnt fixed. For now we stick with making a new Device remotely until we can do a reboot...
Good News / Bad News: Only took 1.5 days to cause another. When the device (comms) stop working, they will not reset even if we stop using them. Thus we made a new device, the old device will not work if reused after any time duration.
Attached screenshot of the devices and the $Eth registers suggested. No help from what I can tell, no counts.
EDIT: We used the IntModTCTPClient this time to get comms back. The Moonbeam1 is original, N is the new we created to get comms back. They will not work still and had Idle time of 46 when used.
-
Control Guy: That is something we will plan if this isnt fixed. For now we stick with making a new Device remotely until we can do a reboot...
Good News / Bad News: Only took 1.5 days to cause another. When the device (comms) stop working, they will not reset even if we stop using them. Thus we made a new device, the old device will not work if reused after any time duration.
Attached screenshot of the devices and the $Eth registers suggested. No help from what I can tell, no counts.
EDIT: We used the IntModTCTPClient this time to get comms back. The Moonbeam1 is original, N is the new we created to get comms back. They will not work still and had Idle time of 46 when used.
Still would like to know what the state of ST145 and ST146 is.
If you are making new devices, you are going to program mode. If you are going to program mode, why can you you not REBOOT?
-
Still would like to know what the state of ST145 and ST146 is.
If you are making new devices, you are going to program mode. If you are going to program mode, why can you you not REBOOT?
That would be funny right? I made extra drivers from last time so I could check as requested previously without a restart. ;)
This morning I could not as the well was in operation. It will be off in the next few hours and will implement a REBOOT instruction with our planned safety checks.
All off, see attachment
EDIT: I did reattempt to use the devices, no change in statuses
-
All off, see attachment
And you have not rebooted since the device hung?
-
No, devices are still locked.
EDIT: $DriverError just changed to ON
-
No, devices are still locked.
EDIT: $DriverError just changed to ON
So your network infrastructure (radios, etc) is pretty sloppy, as you suggested. The test we are running may well expose the issue then.
The fact that ST145/146 were not on gives me hope that it isn't in the stack. That's really preferable. Just gotta find it.
-
Understood.
The unfortunate reality of radio, it will go down at times. When first deployed it was great, then houses were built everywhere and short of a 40ft+ tower we are not getting over them and there are powerlines directly over the site :o
Good hunting!
FYI: Reboot instruction added and brought devices back online.
-
I do see something that bothers me a bit at one of the lower layers (there's like 5 layers). I don't know for certain that it's an issue, but I'm also not certain that it isn't, and it was changed at some point in the last few years. Developing...
-
Not recent. The thing I found was changed in Do-more 1.3.
Might still be a hole though, but not new.
-
Thanks for the update and for getting testing started quickly.
-
Thanks for the update and for getting testing started quickly.
It's entirely possible that this is an old problem that has only become obvious due to your network infrastructure changes/issues. If we can find a way to duplicate it, we can feel much better about the fix, so that is the focus for now.
-
Yes wireless links are bad
Run two do-mores sending two data streams and two wireless links if the pump cant be off for the 5 seconds it takes to reboot :)
This will also test for a hardware failure in case this is a new symptom and you will now have something to test the old system against
Having two systems you can use the instruction for failed communications to enable the second system just like our eastern pipeline should have had before it got hacked
"ST153
$EthMasterError
Status
Will be ON when the Ethernet I/O Master detects an error with one of its Ethernet I/O Slaves - most likely an Ethernet I/O Slave is offline
"
When ST153 is on enable 2nd Do-more System
If not used contact on PING error
"Colonial Pipeline Cyber Attack: Hackers Used Compromised ...
https://www.bloomberg.com ? news ? articles ? hackers...
Jun 4, 2021"
"Two is one....one is none"
This is used in Marine's training
"https://www.forbes.com/sites/work-in-progress/2011/06/21/two-is-one-and-one-is-none/?sh=559c7d968a8c"
Here is how to program it and two professional examples of how and why it should be used
-
Yes wireless links are bad
Run two do-mores sending two data streams and two wireless links if the pump cant be off for the 5 seconds it takes to reboot :)
Not sure how that helps in any way?
Links are not down for long durations, simple low quality (retries).
The DoMore running the comms is the PLC controlling the well output (locally). When it reboots, the relay drops. It is unaffected by the wireless link.
At this time DoMore does not support "native" hot-standby, so sending two commands from two units across two wireless links is asking for command collisions.
-
Why not latch the relay and add a second output to unlatch when you want the pump to go off? Reboots won?t shut the pump down.
-
Potentially related to this topic and the previous topic with the non-responsive Modbus/EIP comms that was attempted to be addressed with the latest firmware:
I hooked a system up this morning with a BRX serving as an EIP slave to an BRX EIP master (whatever the correct terminology is for explicit EIP master/slave relationship). The master has been powered on for about two months and the slave has been powered off during this time. I reapplied power to slave. Could see the slave immediately in DmD but the master explicit EIP messaging was unresponsive. Power cycled the master and comms to the slave came back immediately. Seems like a continuation of the seemingly "fixed" problem from a little while back.
-
Potentially related to this topic and the previous topic with the non-responsive Modbus/EIP comms that was attempted to be addressed with the latest firmware:
I hooked a system up this morning with a BRX serving as an EIP slave to an BRX EIP master (whatever the correct terminology is for explicit EIP master/slave relationship). The master has been powered on for about two months and the slave has been powered off during this time. I reapplied power to slave. Could see the slave immediately in DmD but the master explicit EIP messaging was unresponsive. Power cycled the master and comms to the slave came back immediately. Seems like a continuation of the seemingly "fixed" problem from a little while back.
The most recent corrected issues are related to adapter routing, and were a side effect of adding the new ECOMEX. As far as I know, those are all resolved.
With regard to the Modbus client instructions not completing, there are a couple of possibilities as to why that might be happening, but so far we've been unable to duplicate it. I am nearly certain we can come up with a fix, even if it is a higher level bandaid, but I would prefer to dupe the problem and determine exactly what isn't working right first. I know that as users it really doesn't matter as long as it works, but I'm never confident I've fixed an issue until I understand the issue.
-
With regard to the Modbus client instructions not completing,
I'm assuming EIP messaging would also be affected?
-
With regard to the Modbus client instructions not completing,
I'm assuming EIP messaging would also be affected?
Depends on what the cause is, but likely yes. The issue is almost certainly in lower level code that would be common to both.