ModbusTCPServer - Possibly Locked up

[BobO]

Ran for approximately 2 months before it failed in April.  Sent an email 4/22:

The BRX is also monitored by Geo SCADA (ClearSCADA; HMI/SCADA server), at intervals from 5-60 seconds depending on item requirements.
As of today we monitor/control ~500 various items within the BRX, mix of registers & coils/inputs
The BRX program is 14,720 words in size. Scan times 38ms typically, up to 90ms on occasion.

It has been in operation for about 1.5 months without a power cycle, and suddenly stopped processing the MRX & MWX instructions. 
The BRX was failing to respond at various intervals to the HMI/SCADA server.
Using the Domore Designer to go from Run-Program-Run, the physical switch (Run-Stop-Run) did not restart the communications. 
Only a power cycle was able to restart instructions/communications and has not failed to respond the server since doing so.

The only error we found was within DST38, set as 11 (Gateway Target Device Failed to Respond).
The server machine also collects some data directly from the slave PLCs, and at no point lost communications.
The BRX and server machine are on the same switch, within the same enclosure, so a true loss of communications to a specific PLC would have been captured by the server as well.

Firmware: 2.8.3
Bootloader: 1.0.9
All were updated to the latest release in Do-more as of yesterday, 4/21/21, but revision history did not seem to mention anything we thought related.

I am sure there may be some other troubleshooting items you may need and would be happy to provide as we can, but had some limited diagnostics time to get the process back online.

I'm seeing two things in that description. 1) MRX/MWX not working, 2) BRX wasn't responding to another device. That sounds like both client and server functions.

[Mike@Forshock]

We first noticed the issue as we were getting random Bad Quality for objects inside Geo SCADA (all from the BRX unit, the items from the DL-06 units were good). These errors happened every few minutes and would last for a short period. No official comm fail ever happened with the SCADA portion (5 minutes). Again, remote units responded to the SCADA during that time without issue.

Then we had the first lock up when we had a well fail to start and the tank level alarms for No Change. This indicated that the MRX/MWX items were failing to operate.

Only solution was a power cycle. Didn't not work with Run-Stop-Run or within Designer (Stop/Program-Run).

[BobO]

Another location that might have relevant information is DST411. It is a retentive location so even after a power fail it retains the value. The TCP stack can do exception dumps and panics which we catch and store the return address in that location. We also stick a "panic @XXXXX: msg" or ""dtrap @XXXX: msg" into the system error (ERR, LastERR0-7) location when that happens. That information can be very helpful.

We'd like to fix this.

[BobO]

Only solution was a power cycle. Didn't not work with Run-Stop-Run or within Designer (Stop/Program-Run).

Another level of kicking it in the head is Re-Initialize PLC I/O from the PLC menu. That forces the drivers to remount. That fixing it is another clue.

[Mike@Forshock]

Will add that to the SOP

Currently DST411 and DST51 show a 0.  We did a firmware update that day to the latest 2.8.5, that reset the values?

[BobO]

Currently DST411 and DST51 show a 0.  We did a firmware update that day to the latest 2.8.5, that reset the values?

No. They are retentive and only cleared on Clear PLC.

So nothing has crashed, trapped, or panicked. That's good. Not super helpful, but good.

[RBPLC]

For what it's worth, mine is reporting: 9330172.

[BobO]

For what it's worth, mine is reporting: 9330172.

I don't have a map file from the actual release but what I have should be close. There is a trap call in the function that this address maps to. The failure could relate to the new ECOMEX. I'm thinking it is probably not a big deal, but it could be. Do you use the ECOMEX?

[RBPLC]

Define "use". On this controller, the ECOMEX is installed in the BRX but is not in use (has not been used, is not configured and no ethernet cable is plugged into it).

[BobO]

Define "use". On this controller, the ECOMEX is installed in the BRX but is not in use (has not been used, is not configured and no ethernet cable is plugged into it).

That might cause it. Something appears to be requesting access to the secondary adapter, but it isn't mounted.

[RBPLC]

There's definitely an issue:

So, I've got another system I'm building. In this system I have a Master BRX communicating EIP with two slave BRX's. The master BRX is also communicating (as a client) with four other Modbus devices. At present time the master has lost comms with the two slave BRX's over EIP and 3 of the 4 modbus devices. On the one functioning modbus device the success bit for the read/write commands is setting/resetting as expected. For the other three devices the status bits are not doing anything on the read or write.  I don't often use emoji's but:    

I haven't touch this system yet in an effort to restore comms. Let me know if there's information that you need.

[BobO]

So both Modbus and EIP functions have hung, but one Modbus/TCP client is still functional?

Please try Re-initialize PLC I/O.

[RBPLC]

So....I reinitialized I/O.

Prior to re-initializing, I could see no success/fail bits triggering (MWX or MRX) on the 3 non-responsive Modbus devices. It did look like both the EIP devices messages were periodically toggling the read/write fail bits (they weren't frequent so it was hard to catch them visually).

After reinitializing the I/O, the EIP came back immediately. The working Modbus device is still working and the 3 initially un-responsive Modbus devices are still un-responsive. I'm still not seeing fail toggle bits on the MRX and MWX. I am also not receiving any warning messages about the non-responsive modbus devices.

Edit: Additional Information
Under Device Status, the Modbus device that is working will show it working through all "Locked" MRX/MWX instructions for that device. Periodically the state will transition to "Closed", it will re-open and start cycling through the instructions again. For the other 3 devices, the state shows "Open" but the there's no movement under the "Locked Instruction" category for these devices.

[BobO]

How long was the system running before it failed? If it is quick-ish and repeatable, I can easily make firmware that will give me more information about the internals of the devices and IP stack.

[RBPLC]

Not quickish. It had been running for several weeks.