Host Engineering Forum

General Category => Do-more CPUs and Do-more Designer Software => Topic started by: belias on October 01, 2023, 08:16:21 AM

Title: TCP Comm Stops on Certain TCP Devices Until Power Cycle
Post by: belias on October 01, 2023, 08:16:21 AM

I'm running a DM1E using DMD 2.9.4 (PLC OS 2.9.7, Bootloader 1.1.0).

I'm trying to troubleshoot a problem where the BRX gets into a state where it will not communicate to certain TCP devices.  This happens infrequently (about 3-4 times per year), but when it does happen it requires a power cycle to correct.  This happened again last night (on the latest FW), and I'm trying to figure out a cause.

The BRX has the following ethernet based devices in-use:

@MQTT - MQTT Client
@GenPanelMR - Modbus TCP Client
@ISY_TCP_Client - TCP Client *
@PedModbus1 - Modbus TCP Client
@SolarEdge1 - Modbus TCP Client *
@SolarEdge2 - Modbus TCP Client *
@Weather_TCP - TCP Client *
@WFieldIO - Modbus TCP Client


Some additional notes / comments:

1.  All devices on the list above are in the LAN (segregated VLAN) with the exception of Weather_TCP.  The Weather_TCP device is part of an HTTPCMD code block that parses the response from a cloud-based weather provider.

2.  When the issue occurs only the devices marked with a * above fail to communicate.  The other devices remain functional.  I can't see an obvious connection between the failed devices that would not also apply to the functioning devices.

3.  When the failure is occuring, the PLC has the following status bit conditions:

ST143 - On
ST145 - Off
ST146 - Off
DST411 - 9418964
DST51 - 0

4.  When the failure occurs and I look at (for example) the MRX instruction for the @SolarEdge1 device, the "On Error JMP to Stage 90" is highlighted, and the program goes to Stage 90 (and then attempts to re-run every 5 seconds per a timer instruction).  However, the exception response output for the MRX instruction remains at 0.  This is the same for code blocks for the other failing TCP devices.

5.  During the failure, the end devices are online and responding normally.  For example, I can read coils from the @SolarEdge1 device (using a separate program), and I can manually request an HTTP GET string from the cloud weather provider used by @Weather_TCP.

6.  Running a DEVCLEAR or a CLOSE on the various devices does not solve the problem.

7.  Running a Reset PLC I/O does not solve the problem.

8.  The only solution I have found is to either physically power cycle the PLC or to use the REBOOT instruction.


Any ideas on what could be causing this?  Or what to look for the next time this happens prior to power cycling?


Thanks.

Title: Re: TCP Comm Stops on Certain TCP Devices Until Power Cycle
Post by: belias on October 04, 2023, 07:43:36 AM

Bump.  Any suggestions on how to troubleshoot this?  I know if I just let it go, it?s going to happen again in a few months and I still won?t be any closer to figuring it out.  I hate intermittent issues?