News:

  • October 16, 2026, 11:23:57 AM

Login with username, password and session length

Author Topic: BRX Looses Comms, No response to DmD, Modbus TCP  (Read 34321 times)

Mike@Forshock

  • Sr. Member
  • ****
  • Posts: 59
BRX Looses Comms, No response to DmD, Modbus TCP
« on: September 16, 2021, 12:52:16 AM »
Have created a topic and support ticket about this before, it still remains. No seems worse

We got an alert the comms for a specific site had failed (from the BRX).
Site is running perfectly fine, an old DL-06 with ECOM100.

Cannot connect to DmD on local PC to Ethernet. (Connects, Reads program, fails on system config)
Responds to Modbus TCP (Server) from local PC (SCADA)
NetEdit3 can connect, make changes from same local PC
Disconnect/close SCADA connections, allowed the DmD to connect.
Device will not show Success/Error status.  The other Modbus TCP devices are working just fine!
Created a new device while it was online, connected the instructions (MWX/MRX) to new device, came up instantly.

This cant keep happening every 2 months...

Mike@Forshock

  • Sr. Member
  • ****
  • Posts: 59
Re: BRX Looses Comms, No response to DmD, Modbus TCP
« Reply #1 on: September 16, 2021, 01:02:12 AM »
Latest available firmware.

BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 6177
  • Yes Pinky, Do-more will control the world!
Re: BRX Looses Comms, No response to DmD, Modbus TCP
« Reply #2 on: September 16, 2021, 10:17:14 AM »
It sounds like it is running out of IP stack resources. That could be due to a memory leak (that I don't know about) or a flood of packets from a bad actor. It would have been helpful to know whether $QueuesFlushed (ST145) or $CommStackOverrn (ST146) were active. If it also helpful to know what is in $EthDroppedPkts (DST40) and $EthStoppedIntr (DST41). This give some insight into what might be failing.

Unless we can dupe this, we can't begin to fix it. I understand your frustration, but problems like this are the absolute hardest, and other than a description of the failure mode, this gives us very little to go on. Even with your program, unless we dupe all of the comm activity, we aren't likely to see the issue.

It isn't a good answer, and it may not be viable, but if you are able to detect (from the PLC) that the PLC comms are hung, you could have the PLC reboot itself.
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO

Mike@Forshock

  • Sr. Member
  • ****
  • Posts: 59
Re: BRX Looses Comms, No response to DmD, Modbus TCP
« Reply #3 on: September 16, 2021, 11:24:36 AM »
I appreciate quick and honest answer, I do understand the difficulty to recreate the problem.

The remote PLC is on a wireless link that used to be very good, but now has some issues (New buildings built in the last year or two).
While the site generally works well, it does get latency and dropped packets. So the low quality of the link is very likely the cause of the problem.
The other four remote units (all DL-06 /w ECOM100) are also wireless, but have not seen same loss of network.

As for the ability to cycle power on the unit, this is not a solution that is reasonable.  This unit has a hardwired connection for controlling a water well/pump, so the outputs dropping out will also interrupt that process.

For me, finding the reason the Success/Error bits do not update would be the first check. On a device failure/hang, the error bit should at least be on.  When this happens, both remain false.  We changed our communications health detection to use a rising edge Success and heartbeat (pseudorandom and analog input; make sure PLC is in run mode) from remote unit.

The DST40 & 41 (Added all $Eth items) are now on our list of things to watch, but would these be from ALL ethernet devices? Currently, these are at 0 for each for the last 10 hours.


BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 6177
  • Yes Pinky, Do-more will control the world!
Re: BRX Looses Comms, No response to DmD, Modbus TCP
« Reply #4 on: September 16, 2021, 12:18:45 PM »
I appreciate quick and honest answer, I do understand the difficulty to recreate the problem.

The remote PLC is on a wireless link that used to be very good, but now has some issues (New buildings built in the last year or two).
While the site generally works well, it does get latency and dropped packets. So the low quality of the link is very likely the cause of the problem.
The other four remote units (all DL-06 /w ECOM100) are also wireless, but have not seen same loss of network.

As for the ability to cycle power on the unit, this is not a solution that is reasonable.  This unit has a hardwired connection for controlling a water well/pump, so the outputs dropping out will also interrupt that process.

For me, finding the reason the Success/Error bits do not update would be the first check. On a device failure/hang, the error bit should at least be on.  When this happens, both remain false.  We changed our communications health detection to use a rising edge Success and heartbeat (pseudorandom and analog input; make sure PLC is in run mode) from remote unit.

The DST40 & 41 (Added all $Eth items) are now on our list of things to watch, but would these be from ALL ethernet devices? Currently, these are at 0 for each for the last 10 hours.

80ms scan time? That's pretty stout. It shouldn't hurt anything, but high scan times are never helpful for comm. Is it always that high?

It wouldn't require power cycling, just invoking the REBOOT instruction, but it will interrupt the outputs.

The instruction is clearly not working right if it's never timing out. It's a joke here at Host, and I mean no disrespect with this, but when there is a failure in the controller and the mode of failure doesn't satisfy someone, my response is generally "what about 'exception condition' do you not understand?" The bottom line is it shouldn't be happening, and we make every effort to keep things from happening, but bugs are bugs and they don't always behave the way we want them to behave...cause they're bugs.

Given the unstable link, I'm now at 90% confidence this is a network stack resource issue. Wondering if some of the low level networking code isn't accepting a missing/unavailable resource as a valid basis for timing out. That isn't great and shouldn't happen, but when is it ever OK to run out of memory? Never. Exception. If it is that, I would expect ST145 and/or ST146 to be on. They indicate that when the Ethernet hardware is trying to inject a packet into the stack, there were no free buffers. That generally should never happen. While that by itself shouldn't cause something else to hang, I'm wondering if there is some pregnant moment in the TCP stack that's causing a low level hang up.

DST40 and DST41 count up in response to excessive packet loads. They aren't problems necessarily, but they give a glimpse into whether there are packet bursts happening that could be causing issues.

I'll put some thought into this to see if I can come with reasons this might be happening and hopefully some diagnostic checks that would let us see what might be going on.
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO

BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 6177
  • Yes Pinky, Do-more will control the world!
Re: BRX Looses Comms, No response to DmD, Modbus TCP
« Reply #5 on: September 16, 2021, 01:56:43 PM »
We're formulating a plan to build a random/repetitive hardware failure stress test that would approximate the behavior of bad networking hardware to see if we can force a similar environment to bad radios. Hopefully we can learn something quickly.
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO

Mike@Forshock

  • Sr. Member
  • ****
  • Posts: 59
Re: BRX Looses Comms, No response to DmD, Modbus TCP
« Reply #6 on: September 16, 2021, 02:03:01 PM »
Scan times are high, we found our own bug in one of our pointer based subroutines.  This PLC hasn't received an update yet, but when the BRX gets an invalid pointer (Block of 24, send request for 25...) the scan times shout way up.  Had a unit with ~90ms, fixed logic and now below 30ms!

Code for the exception, not the rule  :P

Will add ST145 & 146 to monitors and see if we can create an automatic invoked REBOOT or at least user initiated (with failsafes)


--
Edit:
Thanks, hope we can find something.
-----

BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 6177
  • Yes Pinky, Do-more will control the world!
Re: BRX Looses Comms, No response to DmD, Modbus TCP
« Reply #7 on: September 16, 2021, 02:27:32 PM »
Code for the exception, not the rule  :P

Yeah, well, you have to be careful with that. When I say exception I don't mean rare event, I mean gravity quit working or the speed of light changed...things that must never happen. The only real recourse to such a thing is to reboot, which may be fine for some, but probably very bad for others. Your PLC is still functioning even though you've had a failure. For most that is preferable to rebooting. We put the REBOOT instruction in there for exactly these cases, if that's determined to be the best action and safe to do.

The real right answer is to fix the bug and eliminate the exception. We'll do our best.
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO

Controls Guy

  • Internal Dev
  • Hero Member
  • ****
  • Posts: 3628
  • Darth Ladder
Re: BRX Looses Comms, No response to DmD, Modbus TCP
« Reply #8 on: September 17, 2021, 12:05:23 AM »
Can you change the PLC to pump run signal to three-wire?  Then you could survive a REBOOT.
I retract my earlier statement that half of all politicians are crooks.  Half of all politicians are NOT crooks.  There.

Mike@Forshock

  • Sr. Member
  • ****
  • Posts: 59
Re: BRX Looses Comms, No response to DmD, Modbus TCP
« Reply #9 on: September 17, 2021, 09:46:01 AM »
Control Guy:  That is something we will plan if this isnt fixed.  For now we stick with making a new Device remotely until we can do a reboot...

Good News / Bad News:  Only took 1.5 days to cause another.   When the device (comms) stop working, they will not reset even if we stop using them.  Thus we made a new device, the old device will not work if reused after any time duration.

Attached screenshot of the devices and the $Eth registers suggested.  No help from what I can tell, no counts.
EDIT: We used the IntModTCTPClient this time to get comms back.  The Moonbeam1 is original, N is the new we created to get comms back.  They will not work still and had Idle time of 46 when used.
« Last Edit: September 17, 2021, 09:48:06 AM by Mike@Forshock »

BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 6177
  • Yes Pinky, Do-more will control the world!
Re: BRX Looses Comms, No response to DmD, Modbus TCP
« Reply #10 on: September 17, 2021, 10:39:40 AM »
Control Guy:  That is something we will plan if this isnt fixed.  For now we stick with making a new Device remotely until we can do a reboot...

Good News / Bad News:  Only took 1.5 days to cause another.   When the device (comms) stop working, they will not reset even if we stop using them.  Thus we made a new device, the old device will not work if reused after any time duration.

Attached screenshot of the devices and the $Eth registers suggested.  No help from what I can tell, no counts.
EDIT: We used the IntModTCTPClient this time to get comms back.  The Moonbeam1 is original, N is the new we created to get comms back.  They will not work still and had Idle time of 46 when used.

Still would like to know what the state of ST145 and ST146 is.

If you are making new devices, you are going to program mode. If you are going to program mode, why can you you not REBOOT?
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO

Mike@Forshock

  • Sr. Member
  • ****
  • Posts: 59
Re: BRX Looses Comms, No response to DmD, Modbus TCP
« Reply #11 on: September 17, 2021, 11:29:23 AM »
Still would like to know what the state of ST145 and ST146 is.

If you are making new devices, you are going to program mode. If you are going to program mode, why can you you not REBOOT?

That would be funny right?  I made extra drivers from last time so I could check as requested previously without a restart.   ;)
This morning I could not as the well was in operation.  It will be off in the next few hours and will implement a REBOOT instruction with our planned safety checks.

All off, see attachment
EDIT: I did reattempt to use the devices, no change in statuses
« Last Edit: September 17, 2021, 11:35:03 AM by Mike@Forshock »

BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 6177
  • Yes Pinky, Do-more will control the world!
Re: BRX Looses Comms, No response to DmD, Modbus TCP
« Reply #12 on: September 17, 2021, 11:35:15 AM »
All off, see attachment

And you have not rebooted since the device hung?
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO

Mike@Forshock

  • Sr. Member
  • ****
  • Posts: 59
Re: BRX Looses Comms, No response to DmD, Modbus TCP
« Reply #13 on: September 17, 2021, 11:44:42 AM »
No, devices are still locked.
EDIT:  $DriverError just changed to ON
« Last Edit: September 17, 2021, 11:54:46 AM by Mike@Forshock »

BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 6177
  • Yes Pinky, Do-more will control the world!
Re: BRX Looses Comms, No response to DmD, Modbus TCP
« Reply #14 on: September 17, 2021, 12:18:32 PM »
No, devices are still locked.
EDIT:  $DriverError just changed to ON

So your network infrastructure (radios, etc) is pretty sloppy, as you suggested. The test we are running may well expose the issue then.

The fact that ST145/146 were not on gives me hope that it isn't in the stack. That's really preferable. Just gotta find it.
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO