Hi all, I've been on the intermittent hardware watchdog struggle bus for quite a while at this point, about a year or two, it's livable it's just annoying and I haven't been able to put a nail in it yet. It continues to rear its ugly head at me every week or two, typically overnight which stops unattended production, if it blips during the day, I can trigger a quick restart, I have quite a few things that won't auto-restart on power restore but will once manual intervention is completed (intentional), I previously had it auto-continue but there were enough other issues that happened in the 10ish second blip as it reboots now that we run closer to 110% on most every piece of equipment.
I've gotten a lot of first-hand experience in the past few years on the BRX platform and in my un-studied (background in IT, but have a bent for it) opinion it's fantastic, I have 4 different PLCs programmed and 3/4 are working perfectly (minus my own mistakes usually). This PLC had 0 issues for about half its life; the most annoying part is I'm not really sure when exactly the issue began because we make some changes often to increase efficiency, throughput, and add functionality we didn't know we wanted. I've physically rewired this specific PLC about 10 times in the last 5 years and am often adding/changing something. We're in the Ag industry so it's always changing or needing streamlined.
The only single thing I can pick out in my head is that I think it may have begun around the time that 2.10 was released, but with the number of changes we make, it may be entirely unrelated, and I've slept since it started.
Current add-in cards are
(7/8 occupied)BX-SERIO - BX-32ND3 - BX-32ND3 - BX-08AD-3 - BX-16TRZ - BX-16TRZ - BX-16TRZ
On to the troubleshooting, I've done a lot of looking at the forums here so I *think* I've checked most of the suggestions, but I figured I'd ask here as well.
- Is there environment noise that could be causing electrical spikes? Relatively speaking, I'd say yes, but actually less than when this was initially deployed, I've been able to streamline a lot of it, lots of small additions and tweaks but nothing that I could point at and say "maybe this was a problem". most outputs are to 110v fuji contactors, 24v interposing relays, or 24v inputs on another device (VFD FWD/REV, interposing relay inputs on another device, etc.)
- Are there VFD's nearby? Yes, however other PLC's are closer to the VFD's than this one, this is connected to less VFD's than the other PLCs as well.
- Is there any consistency to timing or equipment running? I don't think so, the whole cabinet is active/inactive often, we use the PLC to a pretty full extent in my opinion.
- Is the incoming power clean? I can't say with 100% certainty, but there are other PLCs on the same 110v power feed that aren't having the same issues.
- Tried different PLC? Yes, I actually bought a remanned, it's a 36ED23 with 7x I/O cards, I haven't replaced all the IO cards to further diagnose.
- Have you checked network traffic? I have the system on an isolated VLAN that contains all 4 PLC's, 5x HMI's, and a solenoid block that is not connected to this PLC, other than my programming machine crossing into the VLAN there aren't any other comms but NTP and SMTP outs coming from the HMI's so pretty low load. I attached a screenshot from my computer crossing from another VLAN over wifi and I think these networks timeouts are specific to that route. All PLC related hardware is connected to 2x din mounted 100mb/s switches in the cabinets, and there are 2 PLC's connected to the switch this one is on, the other PLC hasn't showed problems but is running some more simple hardware.
- Is it a serial comms issue? Maybe, I had some bugs from a scale head infeed and not formatting the data properly which resulted in a Driver Error ST143, that ended up I think being rooted in a STR2INT seeing negative values, and my inexperience of not handling the data in errors properly, but I was able to get that resolved so the error no longer occurs.
- Have you tried flashing a blank program, then flashing the current program back? Yep, a few times, and in the PLC change, that also did the same thing.
- Are the scan times excessive? It depends... I'm at a Min 4.5ms, Avg 5.4ms, and Max 8.4ms the max was higher, around 9.2, but I figured out how to do some good streamlining using Yield instructions on a lot of my monitoring instructions and INT2STR (for HMI display) that took those numbers down.
- What is the Watchdog Timeout set to? the default of 1000ms
- Are there any com errors? No, but I did happen to notice I have some Modbus I/O Scanner timeouts on the intsermodbusclient coming from an air compressor that I recently tied in (in the past 3 weeks) that I'm going to troubleshoot, seems like .4% are timing out which is odd to me, could be wiring related.
Right now, I'm more looking for a "what haven't I tried"? Outside of replacing all the add on I/O modules I can't think of where to go from here. Maybe someone else can see something from the screenshots, happy to provide any additional data requested.
Thanks for your thoughts!
Topic: Seemingly random hardware watchdog timeouts (Read 12 times)