News:

  • February 11, 2025, 12:48:04 AM

Login with username, password and session length

Author Topic: BRX won't come out of PROGRAM mode  (Read 1366 times)

Bolt

  • Hero Member
  • *****
  • Posts: 573
BRX won't come out of PROGRAM mode
« on: February 11, 2024, 09:37:24 PM »
I have a BX-DM1E-10ED23-D that kept sending me email alerts that the MQTT data wasn't posting to the cloud, so I VPN'd into it. Seeing nothing really wrong other than it kept queuing new data points, I lazily tried a Mode change from DmD, RUN to PROGRAM. Now, when changing it from PROGRAM back to RUN, it says Low level hardware initialization failed.

OS 2.9.7, mode switch TERM. I don't see much wrong in the status logs, just one ST152  which refers to an I/O system checks RTU module's channel that's disconnected, that's true, and no big deal. What else can I try?

BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 6068
  • Yes Pinky, Do-more will control the world!
Re: BRX won't come out of PROGRAM mode
« Reply #1 on: February 12, 2024, 12:56:38 PM »
I have a BX-DM1E-10ED23-D that kept sending me email alerts that the MQTT data wasn't posting to the cloud, so I VPN'd into it. Seeing nothing really wrong other than it kept queuing new data points, I lazily tried a Mode change from DmD, RUN to PROGRAM. Now, when changing it from PROGRAM back to RUN, it says Low level hardware initialization failed.

OS 2.9.7, mode switch TERM. I don't see much wrong in the status logs, just one ST152  which refers to an I/O system checks RTU module's channel that's disconnected, that's true, and no big deal. What else can I try?

Interesting.

I dug into that and there is only one thing that could cause that error when returning to RUN mode: all of the TCP/IP stack packet buffers are in use. If you have a TCP client, TCP server, or UDP device in the project that is receiving packets but the PLC app isn't processing them, this can happen. There are some spots in the code where we forcibly purge them to prevent this issue, but it doesn't appear that we do that going from PROGRAM to RUN, we just check to see the PLC is healthy before going to RUN.

If you don't have anything like that in your program, then I'm wondering if there is some unidentified/expected network activity that is getting hung in the stack. As an example, one of the TCP server hangs we had was fixed by tweaking some stack internals, but even though it was only reported recently, the stuff we changed was there since Rel 1. We would only see the problem with radio modems that apparently would duplicate some frames in a way that caused the stack some serious heartburn. If there is another similar issue we haven't identified, it might cause the problem you are seeing.

I think I might add some network diagnostics to the next update. I don't generally leave things like that in production builds due to performance concerns, but it would allow us to track this before it caused the PLC to hang. Might be helpful in identifying the cause.

As for now, you could try to re-init the I/O or bump something small in the sysconfig and rewrite the program. Barring that, reboot.
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO

Bolt

  • Hero Member
  • *****
  • Posts: 573
Re: BRX won't come out of PROGRAM mode
« Reply #2 on: February 13, 2024, 09:33:34 AM »
This unit has no comms of any sorts on it's network, it is the only device behind the stride linx VPN router.
I did try to write a simple ladder change to the PLC with no new ability to go RUN..
I was within a few miles yesterday, so I stopped in and power cycled the whole panel yesterday and it went into RUN without a problem.
Weird, I think in the future I'll add a REBOOT ladder rung to all my programs for simpler reboot capabilities without having to go to PROGRAM mode.
Thanks

BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 6068
  • Yes Pinky, Do-more will control the world!
Re: BRX won't come out of PROGRAM mode
« Reply #3 on: February 13, 2024, 09:59:27 AM »
This unit has no comms of any sorts on it's network, it is the only device behind the stride linx VPN router.
I did try to write a simple ladder change to the PLC with no new ability to go RUN..
I was within a few miles yesterday, so I stopped in and power cycled the whole panel yesterday and it went into RUN without a problem.
Weird, I think in the future I'll add a REBOOT ladder rung to all my programs for simpler reboot capabilities without having to go to PROGRAM mode.
Thanks

Which actually begs the question of whether the router is generating/passing some traffic that is causing us trouble. If this is a persistent issue, I'd really like to dig in deeper. After the black eye over the radios, I've become cognizant of the potential for a changing network environment to cause difficult to duplicate issues. If we see problems in a place like this where there are very few devices, it is much easy to debug. We have some bug fixes in 2.9.9, so I'm going to look into adding some network diags that can be switched on as needed.
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO

Bolt

  • Hero Member
  • *****
  • Posts: 573
Re: BRX won't come out of PROGRAM mode
« Reply #4 on: February 13, 2024, 10:13:20 AM »
Yeah, in hindsight I should have tried rebooting the router remotely as well.

BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 6068
  • Yes Pinky, Do-more will control the world!
Re: BRX won't come out of PROGRAM mode
« Reply #5 on: February 13, 2024, 11:08:43 AM »
Yeah, in hindsight I should have tried rebooting the router remotely as well.

I'm not suggesting that the router is at fault (although rebooting them has cured things before), just that we've seen multiple cases where network infrastructure has generated traffic that we failed to handle well. It's on us not to fail, but when the conditions are unique to an installation, it's hard to dupe everything we might see.
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO

BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 6068
  • Yes Pinky, Do-more will control the world!
Re: BRX won't come out of PROGRAM mode
« Reply #6 on: February 20, 2024, 03:23:52 PM »
Just a heads up. We added some network diagnostics to the 2.9.9 release, due out imminently.

The most immediately relevant data point is DST72 (when DmD is released it will be $EthFreePackets). Keep an eye on it. It should hover around 375. If you see it walking down toward zero, we need to be involved ASAP. There is more that we can do if this is true, but I'd prefer that conversation to be with a support tech.
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO