ModbusTCPServer - Possibly Locked up

[jcb]

New Issue today. Redlion G3 to Do-More via Modbus TCP.

I had some MHR Values that started getting larger than 16 bit unsigned values so I converted them to Doubles at the end of last week. I have been polling this Do-More PLC along with others, DL205's and Allen Bradley Contrologix. This morning I come in and All my other PLC data is online on the Redlion but this PLC reading Longs. I rebooted the redlion and everything came back but this PLC.

I added ModbusTCPServer to a data view and It shows 1 Session with 43758607 Transactions and No errors or last errors. The only unit connecting to this as a server is the Redlion G3. The rest of the TCP Modbus this unit is the Slave and the requests are handled in Ladder Logic (those are all working fine). The PLC is working great and remote IO and everything else is working I have no errors in the system information.

When I unplug the Redlion for 10 minutes the ActiveSession is staying at 1 and the Transactions haven't changed at all.
I then added a MRX to another PLC and attempted to read and Just got an error bit set no change in the Active Sessions or Transactions.

Is it possible the Modbus Server is Locked up?

How would I check this and is there anyway to restart it without power cycling the PLC. 

Are there any other things I should check.

[BobO]

I don't have any specific knowledge of the Modbus TCP server locking up, but it's certainly possible.

If you can transition to program mode, you can try Re-initialize PLC IO, from the PLC menu. If that doesn't work, you can try tweaking a Modbus TCP server setting (change the timeout or number of sessions) and write the PLC. Either of those might unstick what's stuck.

And you might check $QueuesFlushed (ST145) and $CommStackOverrn (ST146) just to make sure something bad didn't happen in the TCP stack.

[ATU]

Redlion recommends only 4 connected devices / Modbus driver.  If you have more than that when the HMI is the Master, then split them up on additional Drivers. Otherwise set up the HMI with Modbus Slave blocks and let your connected devices push the data to the HMI.

[jcb]

Not sure what it was no errors in the stack.

I had to power cycle the PLC and now everything is working fine?

I converted my reads back to 16 bit and I am just shifting <<16 the MSW and adding the LSW in the Redlion.

[BobO]

Not sure what it was no errors in the stack.

I had to power cycle the PLC and now everything is working fine?

I converted my reads back to 16 bit and I am just shifting <<16 the MSW and adding the LSW in the Redlion.

If you can dupe it, I definitely want to know about it.

[RBPLC]

Not sure if this is even remotely related.

Have been running BRX communicating Modbus TCP with ABB VFD. System has been running for about 2 months. BRX lost comms with VFD at some point in the past few days (why, I don't know). Program is set-up to read from drive at all times. Started troubleshooting. Neither success or error bits on the MRX were toggling at 50ms read time. No comm warning in DMD either. Transitioned from Run to Program a couple of times. Communication did not restart. Inserted a PING into the program and the BRX could PING the drive in <10ms. Tried Program to Run a couple more times in DMD with no results. Power cycled the drive with no comms restoration. Tried increasing the MRX read time to 1 sec. No results. Power cycled the controller, comms restored. Thoughts?   

[BobO]

Not sure if this is even remotely related.

Have been running BRX communicating Modbus TCP with ABB VFD. System has been running for about 2 months. BRX lost comms with VFD at some point in the past few days (why, I don't know). Program is set-up to read from drive at all times. Started troubleshooting. Neither success or error bits on the MRX were toggling at 50ms read time. No comm warning in DMD either. Transitioned from Run to Program a couple of times. Communication did not restart. Inserted a PING into the program and the BRX could PING the drive in <10ms. Tried Program to Run a couple more times in DMD with no results. Power cycled the drive with no comms restoration. Tried increasing the MRX read time to 1 sec. No results. Power cycled the controller, comms restored. Thoughts?   

Sounds like a driver state machine was hung. Not a great thing. Would love to know how to dupe it. Generally very easy fi

If it happens again, try "Re-initialize PLC I/O...". If that fixes it, or not, gives me clues.

[RBPLC]

As an additional piece of information, three additional Programs are setup to talk to three additional like drives. These three drives were powered off at the time of troubleshooting with no comms warning in DmD either. I didn't check success/error bits on these MRX commands but I assume they probably weren't toggling either.

Also, I forgot to mention I was not running the most recent firmware/booter. Since I was troubleshooting, I went ahead and updated to most recent.

[franji1]

For each new Modbus slave, are you creating new Modbus/TCP Client devices (i.e. one per Modbus/TCP slave), or are some of them sharing the same Modbus/TCP Client device?

It is recommended that due to the overhead of TCP (not Modbus/TCP, but TCP), you create a different TCP client per server (slave) IP Address, in this case, it would be a Modbus/TCP Client device.

[RBPLC]

They all have their own client.

[Mike@Forshock]

Unfortunately have the same issue.

My older 2.7 deployments have not had the issue on poor links, but my 2.8 unit experienced this same lockup.
Sent an email to Host & AD support.  I'm not glad, but am anxious to see an update now that others are also experiencing it.

We changed out Communications monitoring from the internal Set Bit structure to a Heartbeat register (clock) from the slave device.

Have not had a lockup where the BRX is the Slave device, so that may help narrow it down.
All slaves are a separate client, polls range from every 5-30 seconds per device. 4 slave devices.
Dedicated ethernet wireless link, typically around 20Mbps with 4-10ms pings.
No links reported down from SCADA (polls the slaves and master independently).
Slaves are all DL-06 with ECOM100.
All transactions use the same registers (like remote IO).

[BobO]

Unfortunately have the same issue.

My older 2.7 deployments have not had the issue on poor links, but my 2.8 unit experienced this same lockup.
Sent an email to Host & AD support.  I'm not glad, but am anxious to see an update now that others are also experiencing it.

We changed out Communications monitoring from the internal Set Bit structure to a Heartbeat register (clock) from the slave device.

Have not had a lockup where the BRX is the Slave device, so that may help narrow it down.
All slaves are a separate client, polls range from every 5-30 seconds per device. 4 slave devices.
Dedicated ethernet wireless link, typically around 20Mbps with 4-10ms pings.
No links reported down from SCADA (polls the slaves and master independently).
Slaves are all DL-06 with ECOM100.
All transactions use the same registers (like remote IO).

It could be one of a couple of things. The easiest answer is the Modbus driver state machine is hanging. The more concerning answer is that the TCP stack tweaks we made to support the ECOMEX POM is causing issues. If the Success/Error bits are toggling but nothing is going out the port, that points to the stack. If the bits aren't toggling, that points to the driver state machine. Not concrete, but suggestive. Of course there might be hole in the state machine exposed by a stack failure...so both could be true. 

I would very much like to get to the bottom of this, but doing so would require diagnostic firmware and some time from people who are experiencing the issue.

[Mike@Forshock]

The State bits do not toggle (Success/Fail), we were initially only monitoring the Success bit to reset a fail timer (Off Delay).
They stayed healthy and never produced a Fail set bit either.

[BobO]

The State bits do not toggle (Success/Fail), we were initially only monitoring the Success bit to reset a fail timer (Off Delay).
They stayed healthy and never produced a Fail set bit either.

Well, after reviewing the driver code, it could easily hang due to a problem in the stack, so that might not really tell me what I hoped.

How often/quickly does it fail?

[Mike@Forshock]

Ran for approximately 2 months before it failed in April.  Sent an email 4/22:

The BRX is also monitored by Geo SCADA (ClearSCADA; HMI/SCADA server), at intervals from 5-60 seconds depending on item requirements.
As of today we monitor/control ~500 various items within the BRX, mix of registers & coils/inputs
The BRX program is 14,720 words in size. Scan times 38ms typically, up to 90ms on occasion.

It has been in operation for about 1.5 months without a power cycle, and suddenly stopped processing the MRX & MWX instructions. 
The BRX was failing to respond at various intervals to the HMI/SCADA server.
Using the Domore Designer to go from Run-Program-Run, the physical switch (Run-Stop-Run) did not restart the communications. 
Only a power cycle was able to restart instructions/communications and has not failed to respond the server since doing so.

The only error we found was within DST38, set as 11 (Gateway Target Device Failed to Respond).
The server machine also collects some data directly from the slave PLCs, and at no point lost communications.
The BRX and server machine are on the same switch, within the same enclosure, so a true loss of communications to a specific PLC would have been captured by the server as well.

Firmware: 2.8.3
Bootloader: 1.0.9
All were updated to the latest release in Do-more as of yesterday, 4/21/21, but revision history did not seem to mention anything we thought related.

I am sure there may be some other troubleshooting items you may need and would be happy to provide as we can, but had some limited diagnostics time to get the process back online.