Hardware Trips

[pbw]

Hi, I have a 36ER3 that is running a cement batching plant.
During yesterdays production run, I noticed 2 instances where the PLC Crashed, Rebooted, and resumed operation.
I attached the system Log. What other information could we gather to pinpoint the issue. We have been lucky so far, in that no huge mishaps took place, but this could be potentially disastrous if it happened during certain times in production. Not life threating, but severe monetary loss.

[Greg]

Questions:

• Is there radiated electrical noise nearby? (e.g. VFDs)?
• Is the I/O wire routing in wireways with higher voltage?
• Do you have snubbers on all your relay I/O?
• Do you have dirty power sources? Spikey? Or brownouts?
• Any changes to the physical machine lately?
• Any programming changes lately?
• What kind of operation was the machine doing at the time of the reboots as shown in the date/time of the System Log? Potentially an electrically noisey event?

Some things to check:

• DST385 ($WatchdogReboots)
• DST400-409 values: These could contain values that may help us help you. They are not Nicknamed, but you can display them in a Data View

[pbw]

1. There is a small VFD, but it's not near the PLC or the PLC control Cabinet.
2. Next to no low Voltage wires with High Voltage wires mixed, as the IO is for the most part AC. The Low voltage wires are in their separate conduits.
3. Snubbers on all Contactors, Interposing relays have 120 volt coils, and I know of no snubber that ADC sells for them. https://www.automationdirect.com/adc/shopping/catalog/relays_-z-_timers/electro-mechanical_relays/square_-z-_cube_relays,_plug-in,_3a_-_15a_(78x-z-_qxx-z-h78x_series)/general_purpose,_15a_(781_-z-_782_-z-_783_-z-_784_series)/781-1c-120a
4.Power seems to be very stable for the most part.
5.No changes to the machine
6.I have been adding software options lately.
7. It seems to happen random.

I was present at the machine when it happen yesterday, we have an old Air Compressor in the building, and it happened right after the compressor Motor Shutdown.
There is a snubber on the contactor coil, but I figured the Motor was the problem, so we took it offline and hooked up an external diesel compressor to try to eliminate the issue.
But it happen after that as well. Now I'm thinking, there is a Hydraulic pump, that stops and starts roughly every 5 minutes, that might be the cause.
I'll put the PLC behind a UPS with Surge protection and see what happens.
The PLC is AC Powered. There are a few other devices (C-More Screen, Ethernet Switch, 2- Rice Lake Scale indicators) that are not showing an signs of abuse, but they are also behind a Rhino Power Supply.
Could I have jumbled the code that bad to cause a Hardware Watchdog Trip?

[Greg]

Could I have jumbled the code that bad to cause a Hardware Watchdog Trip?

Well, it doesn't have to be "jumbled," as in, programming error. It could be something as simple as, turning on a new motor that is needed, and that motor happens to generate enough EMI to glitch a critical signal somewhere. So, not a programming mistake, just a simple change that inadvertently causes the problem.

But I will tell you from experience, that weird things like hardware reboots are statistically caused by the things that both you and I suspect... EMI somewhere. And I'm sure you know that a motor that is turned off, if it has enough inertia, can suddenly become a generator of backward EMF and EMI. So, I think you are looking in the right direction.

I'll put the PLC behind a UPS with Surge protection and see what happens.

Excellent. Let us know.

BTW, the codes in DST400-409 seem pretty normal, so they are not much help. Also, the $WatchdogReboots = 6, you need to know, means not only that it has rebooted 6 times, but that when that number reaches 10, it will not reboot. Instead, it will go to STOP mode and have to be manually put back into RUN. If this is not optimal, you can actually write a zero to this to keep it from reaching 10. Rebooting may be better than STOP, or vice versa, in case of failure. That's up to you. Of course, neither is good and we need to figure out what is causing it.

[pbw]

I will definitely need to zero it, because switching to stop could potentially be bad.
Thank you so much for pointing that out.

[pbw]

Had the PLC behind a UPS Surge/Battery, and still happened.
The PLC was the only device powered from the UPS, no relays, valves Etc..

[rlp122]

I'd bet a beer that there is noise is coming in on one of the serial or ethernet lines.

[Controls Guy]

I'd bet a beer that there is noise is coming in on one of the serial or ethernet lines.

Maybe put in some fiber or at least an optical coupler, maybe.

[BobO]

Had the PLC behind a UPS Surge/Battery, and still happened.
The PLC was the only device powered from the UPS, no relays, valves Etc..

Yeah...as rlp122 (hi Richard!) suggested, anything wired to the outside world becomes an antenna and a potential noise injection point.

Noise used to be a far smaller issue when 5v-TTL-crunchy-slow-chips were the norm. With 1.5v core voltages and parts that are orders of magnitude faster, it's a constant fight.

[Controls Guy]

Have you tried replacing the PLC?   Maybe there's some marginal component or solder joint.

[pbw]

I have not tried substituting a PLC, I'll see if I can squeeze one in tomorrow.
5 Serial Coms being used. All RS-485.
2 are to Rice Lake Indicators, 1 to a GS20 VFD, 1 to a Banner Wireless Radio, and 1 to a View Marq Display.
On the Ethernet side, I'm utilizing a 5 Port Stride Switch. Connected is, The BRX, C-More, IP Phone and the Plant network for FTPing Batch records to User's PCs.
Seems to happen once or twice a day. I can remove the View Marq and the GS20 without affecting production. I think I'll try that and if it still happens I'll try replacing the PLC.

[pbw]

I'm not using the PLC's on Board 24V DC Source.
In an effort to try to narrow it down, what if I remove the 120 V source and powered the PLC by Back-Feeding 24vdc into it's 24v circuitry. Would it even turn on? Is this something you guys would try?
The reasoning is that, everything behind the 24V Power supply is acting normal, but I guess it could be due to a host of other issues.

[pbw]

I'd bet a beer that there is noise is coming in on one of the serial or ethernet lines.

I could remove the ethernet Network and just have the PLC directly connected to the C-More to try to rule out the Ethernet noise.
If the noise where to be coming in on the Serial ports, what are some options to mitigate it?

[Controls Guy]

Serial to serial optical isolator.  Forget who sells them, feels like s Black Box thing.  Or, if it's a long run, convert the first segment to fiber with two media converters.

Before I did that I'd swap the PLC, possibly with a DC power one

[pbw]

Good idea.