6 Yellow LED's

[eman5oh]

I have a T1H-DM1E that has been problematic since it was installed, it reboots with watchdog errors about once or twice a month. In the last week now it has stopped running and did not reboot twice. When I go to check on it it has all six LED's lit Yellow, the only way I have found to bring it back is to power cycle it. What does this mean? We have many other PLC from you guys that run for years without skipping a beat, so I know you guys make a great product, just trying to get this problem child figured out.

[BobO]

The yellow LEDs are the initial reset state, which it apparently didn't come out of.

What is your power like? Are there drives or other electrically noisy stuff nearby?

[eman5oh]

I think we have ground loop problem, been trying to track the exact cause of it but it has been elusive.  We have measured 2.5 amps on the ground conductor at the step-down transformer that feeds the section of the building that this PLC is installed in. The power distribution for the building is fairly complex and we can't turn it all off. We have added an EMI filter and surge suppressor to the input power in the cabinet and the 24 volt supply that runs this CPU along with two others has  one of the automation direct buffer modules on it so we can go for around 10 seconds with no power at all. We have used a fluke data logger to look at the incoming power and found nothing of interest when looking at times when we have had issues with one of the plc's. The odd thing is we have 3 plc's in this cabinet and all 3 CPU's are powered from the same power supply and buffer, but only one at a time ever reboots on a watchdog. We are looking into pulling power form a different part of the building at this point but want to be sure that it will fix the issue. I will gladly take any advice or suggestions at this point.

[ADC Product Engineer]

I'll corroborate that it it most likely a power or grounding issue of some sort.  That 2.5 amps on the transformer ground is not a good sign. 

[ATU]

Anything connected to the USB port?

[Garyhlucas]

A large part of the problem with grounding is because electronics engineers don't quite grasp that grounding for safety per NEC rules is totally different than signal grounding.  Finding current flow on a grounding conductor at a transformer means almost nothing and absolutely nothing to a PLC if the grounding in the PLC cabinet is done properly.  Most people don't understand that wires leak current to ground, motors leak current to ground, transformers leak to ground, lights leak to ground.  NEC rules REQUIRE ground loops everywhere!   The purpose is to ensure that no two pieces of metal in a building every have a potential difference large enough to harm some.  When that motor fails and dumps 5000 amps into the grounding system it will follow every path to the power source to dissipate and hopefully all the paths are good enough to handle what they see.

If power is the cause a more likely issue is that if a transformer is involved the electrician didn't bond the neutral to the ground at the transformer.  This can create a very dangerous situation if one phase goes to ground and doesn't trip a breaker.  Now the other phases are at a much higher potential to ground and if your PLC is seeing that you could have a problem.   When I was an industrial electrician I saw the transformer neutral bonding issue many times.  If you have a control transformer in the PLC cabinet the neutral needs to be bonded to ground there, and yes that is another ground loop!

[ADC Product Engineer]

It's not always power grounding that is the issue.  Sometimes a DC supply is floating.  Rare, but it happens and causes some really strange things when it does.

Other times it is a communications line where the HMI is at a different ground potential than the PLC cabinet and trying to equalize through the comm line.

If I had a dollar for every time I solved an issue that was caused by power and/or grounding I'd be retired on a beach with a nice beverage right now.  And I am only slightly joking.  It happens a lot. 

[eman5oh]

Anything connected to the USB port?

No the USB port os not connected.

[ATU]

What all do you have connected on the Network that is communicating with the PLC's ? What is your longest run?

[eman5oh]

It's not always power grounding that is the issue.  Sometimes a DC supply is floating.  Rare, but it happens and causes some really strange things when it does.

Other times it is a communications line where the HMI is at a different ground potential than the PLC cabinet and trying to equalize through the comm line.

If I had a dollar for every time I solved an issue that was caused by power and/or ground I'd be retired on a beach with a nice beverage right now.  And I am only slightly joking.  It happens a lot.

We have been through everything that we could think of, that is why I am asking for advice. Looking for that one thing that we have overlooked. All of the DC supplies returns are connected together in a block and that block is earthed. I have also removed the earth connection and checked for continuity between earth and DC return, it was good, nothing the fluke could measure. The HMI is in the same cabinet and grounded as well. We are using Ethernet to the HMI and I believe that ethernet uses isolation transformers at each end to eliminate the ground loop problem. Still feel free to offer up other odd, likely or unlikely possibilities as I am still trying to figure this one out. If it would just break and stay broke I could fix that.

[BobO]

Have you replaced either the Terminator P/S or the CPU?

[eman5oh]

Have you replaced either the Terminator P/S or the CPU?

Both were replaced while troubleshooting, about 8 months ago, we have a lot oF Automation direct hardware so we have stuff on hand to swap out. The CPU and supply that was then run at my desk for months without any issues. This issue is definitely related to this specific installation.

[eman5oh]

What all do you have connected on the Network that is communicating with the PLC's ? What is your longest run?

We have 3 PLC and the HMI in the cabinet. Two PLC connect to one and other with a crossover ethernet cable and the third connects to one of those with a serial cable and transfers data via mod bus. We also have 4 remote HMI's with the furthest one around 600 ft away. We have fiber run between the two connected buildings.

[ATU]

Elaborate on the serial link.  How is that connected and to what? How long is the run? What kind of cable are you using?

[eman5oh]

We have a serial Modbus connection between two of the PLC's in the cabinet. The cable is about 2 feet long and is like a telephone cable except it has 6 conductors. The cable plugs into the serial port on the bottom of each cpu. The two plc's share the same power supplies, one for I/O and one for the CPU power. All power supply DC (-) are connected together in a block of terminal strips and that block is earth grounded.