DL06 Redundancy

[PlcThatWorks]

We are getting involved in our customers requiring redundancy on the PLC, so now I am getting to install two DL06’s with ECOM100 cards and a hub. 

Here is my question:  How can I (without having a master) make either PLC read the others watchdog time, and if it has timed out to shut it down (STOP command)?  The kicker is that we are also using port 2 Modbus, and if the talking PLC gets shut down the other has to pick up the talking.

[ATU]

Can you provide a little more detail. How is this setup, its not very clear. What watchdog are you refering to? What network are you refering to "talking". Is it on the ethernet or on the serial port that you are concerned about?

[PlcThatWorks]

Good point – We deal with electronics for generators on ships and mega yachts.  We put in DL06’s to do some of the external control logic.  If the DL06 in the panel goes south, and starts smoking, we need another DL06 to pick up the slack.  I was hoping to make a setup so there was no master or slave PLC.  No primary or secondary PLC.  But I am having to shelf the notion that neither is a leader or follower, based on some of the following.  I encounter the following problem:

The communication the PLC’s are using is Modbus out of Port 2 on the DL06 to our external generator devices.  There can only be one PLC that “request data” from our devices on the generators at a time.  Two DL06’s talking at the same time and neither will get answers.  Shutting up one PLC unless the other has failed (by failure I mean any of the special relays “SP36 – SP56” gets set) is where I get stuck.  I can send a STOP routine to the PLC that has failed, but how do I tell the other to start communicating on Port 2?

Further the PLC that is not communicating via Port 2 to the Generator devices then needs it’s v-memory updated from the PLC that is communicating (updates are done via Ethernet).  My thought is that both PLC’s has to come to the same output configuration before any external actions can be taken.  If the two disagree (most likely because of internal failure in the PLC) we shut the failed PLC down (assuming the failed PLC is the leader) with a STOP command, but now the PLC that has not failed should stop inquiring about V-memory statuses and start Port 2 communication for V-memory updates.

[ATU]

I have never done redundant systems before, but here is a way that I would approach the problem. First I would consider adapting your program to use stages at least for the communications part.  The key is determining when the serial communications hang up in either PLC or the program state is undetermined. Stage programing is great for doing this.  You set up all of your serial comunications in a loop of stages and at the bottom of this loop you write a number from a counter to a memory location in the other PLC. Next time through the loop you increment the counter. The other PLC monitors this location and checks for it to change. If it doesn't change within a certain period of time, then you know that PLC is either stuck in the loop somewhere else or the program is not in RUN.
     To really shut down the communications ( I am assuming 422/485 multidrop) from the other PLC,  you need to either isolate those lines or disable them.  However, the simplest method may be to put a NC relay in the power supply line. Have the relay controlled by the other PLC. So each PLC can detect the other having a problem, kill the power,then allow it to reboot and be put in the standby state. Most of the time this cures the problem anyway.  You have to set up some sort of method to let each PLC know who is active. Status registers, etc. This could be an IO line or a memory location that actually disables the other PLC from taking control on startup, putting it into the monitor mode. 
 One question that I would have for Host is if you have 2 PLC's being an Ethernet master like this, are you assured that the writes will happen if both occur exactly at the same time or what is the best way to handle this?

[AZRoger]

PldThatWorks,

The methods being discussed so far on this topic only catch a small percentage of the ways a PLC and things it connects to can go bad. In the earliest FAA "fail safe" systems, they used 3 processors. They all read the same inputs. They all computed the outputs. The two that came up with the same result were OK. If the third matched, it was OK too. With only two processors, there was really no way to tell whether the accusor (falsly claiming that the OTHER processor is bad) or the accusee is actually bad. When two are checking on each other, there's no way to tell who's right!

Now for you specific application, are there failures that could crash them both? Eg. electrical glitches? Fire? Lightning? A software bug that both experience because conditions were right? How do they share or handoff the I/O that isn't modbus or ethernet?

It might be best to back off a little and put in some self-diagnostic logic that can report status. If all is well, it could have  a clock as part of the display. If it stops timing, then the PLC has quit for any number of reasons. It the display goes dark, it has failed. And so on. Let the PLC self diagnose (if it can) and report errors on the screen. And keep a spare around.

Really and truly fail-safe operation needs to look at the big picture. I little redundancy doesn't help much, and might even hurt by adding complicated processes that are hard to test and therefore more likely than the mainline to be wrong. I'd be very cautious with this one. 

Roger

[ATU]

Maybe what is needed is just a backup without the complication of doing the hot swap. Perhaps a hardwired system where the person operating the system has to physically turn a switch that completely switches between 2 isolated controllers. As AZRoger has suggested, give the user diagnostics and let them decide. However, sounds like the user may not be inclined to do the repairs and could be stranded out at sea somewhere.

[PlcThatWorks]

Thanks guys - all great ideas.  Three processors - how fun would that be.  I need to go find a customer who will pay for the program for that one.  The end of it all looks to be that one is master, and does the Modbus coms.  The backup gets updates via Ethernet, and if the master fails it will de-energize and output, that is wired to an input on the slave.  The slave detects the input missing, and it stops the Ethernet coms and picks up the Modbus coms.  Both PLC's will re-energize and output if they fail, to alarm of a failure in the control system.  This should warn the engineer to be on watch in the engine room.

Our programs are really not complicated, 100 rungs or so, not much that can go wrong there, but if a $10M yacht is docking and all power is lost because of a PLC going down - well let's just say that is where insurance comes in