Power Outage Issues

[BobO]

Another location that might shed light is DST410. Any time there is a mode change failure, that location gets written. That was a relatively recent addition, but I think it was in 2.9.6.

[RBPLC]

DST410 = 0

[BobO]

So the PLC didn't see a mode transition failure. That fits with the log and post mortem regs.

Would still like to see the log prior to the reboots.

[RBPLC]

Do you have more than one PLC on the same network? Is it possible that we aren't looking at the correct one? Is it possible that an IP address has gotten reused?

The only way 1400 gets into that post mortem state is in program, going to run leaves different values, and the log shows no mode transitions to either mode. The log and the post mortem appear to agree and I've never seen a case when they didn't. I know of no possible way the PLC could be running with both of those at the state they are at. That doesn't mean it isn't, just that it would require multiple failures in different systems for that memory state to be present when the PLC is in run mode and I've never seen such a thing. That suggests another explanation.

Just for reference, what are the log entries prior to the three power cycles?

There are multiple PLCs on the same network.
This is the correct one. This is the only one that is programmed remotely like it. I have limited network troubleshooting capability but I pinged the controller, got a response, pulled the network cable, pinged and got no response, plugged back in, pinged and got a response. I suspect that there are no conflicting IP addresses.

I have emailed the log file.
I physically confirmed that the lights on the front of the PLC indicate:
ON - PWR
OFF - RUN
ON -MEM
OFF - ERR

This corresponds to what I am seeing in DMD.

Also, I didn't really press the issue at the time, but S477 from the original post shows a switch back to run mode. I cannot with complete certainty confirm that after I made changes to programming that I didn't do something with the log file, but I don't think so. The forces that are shown in that original post were correct as I was testing a few things. If you look at the log now, those entries appear to have been overwritten.

[BobO]

According to the log, the PLC was changed to program mode through DmD by the Default User in April. It was never changed to run mode after that. There are three empty log entries (probably due to older DmD) that are most likely mode restoration ("PLC mode restored to program on startup.") entries that we added in the last release.

Please try the following.
1. Switch the PLC to RUN via the keyswitch.
2. Confirm that the mode goes to RUN (led on). Make note of DST400.
3. Set switch back to TERM.
4. Switch the PLC back to program via DmD. 
5. Confirm the mode goes to PROGRAM (led off). Make note of DST400.
6. Open the log in DmD and see if the mode transitions are getting logged.

[BobO]

That new log entry was added in OS 2.9.6, which was after the last DmD release, so it's blank.

I was gonna suggest using the web page to access the log...but realized the new entry hasn't been added to it. Need to fix that...

[Garyhlucas]

So from this conversation I am guessing my habit of always putting a UPS in to feed a PLC is a good one.  I had a jobsite get hit with lightning, 3 times in one month!  First one we found out that the building service had no surge protection. Second one showed us that the costly AB UPS was specially built with no surge protection!  They came to install the building surge protection and found out they had the wrong voltage unit.  So the 3rd strike did 8K worth of damage to the brand new SLC500 rack.

So my practice now surge suppressor on the incoming panel power, a shielded control transformer powering a surge protected UPS that supplies a 24vdc power supply to run a 24vdc PLC.  Knock on a large tree but I haven't lost any more PLCs. I only allow customer Ethernet through fiber optic connections.  One customer strung 7 routers together to cross a campus.  Lightning took out their server and all the routers.

[RBPLC]

According to the log, the PLC was changed to program mode through DmD by the Default User in April. It was never changed to run mode after that. There are three empty log entries (probably due to older DmD) that are most likely mode restoration ("PLC mode restored to program on startup.") entries that we added in the last release.

Please try the following.
1. Switch the PLC to RUN via the keyswitch.
2. Confirm that the mode goes to RUN (led on). Make note of DST400.
3. Set switch back to TERM.
4. Switch the PLC back to program via DmD. 
5. Confirm the mode goes to PROGRAM (led off). Make note of DST400.
6. Open the log in DmD and see if the mode transitions are getting logged.

What about the RUN mode transition shown in the original post and the overwritten log entries? The attached does not correspond to the log file that I sent you.

[BobO]

According to the log, the PLC was changed to program mode through DmD by the Default User in April. It was never changed to run mode after that. There are three empty log entries (probably due to older DmD) that are most likely mode restoration ("PLC mode restored to program on startup.") entries that we added in the last release.

Please try the following.
1. Switch the PLC to RUN via the keyswitch.
2. Confirm that the mode goes to RUN (led on). Make note of DST400.
3. Set switch back to TERM.
4. Switch the PLC back to program via DmD. 
5. Confirm the mode goes to PROGRAM (led off). Make note of DST400.
6. Open the log in DmD and see if the mode transitions are getting logged.

What about the RUN mode transition shown in the original post and the overwritten log entries? The attached does not correspond to the log file that I sent you.

That is an excellent question. The logs are ring buffers and limited to 256 entries. If there is some busted wrap logic on the write or read that could explain missing log entries. Don't think that explains the post mortem DSTs though.

[RBPLC]

According to the log, the PLC was changed to program mode through DmD by the Default User in April. It was never changed to run mode after that. There are three empty log entries (probably due to older DmD) that are most likely mode restoration ("PLC mode restored to program on startup.") entries that we added in the last release.

Please try the following.
1. Switch the PLC to RUN via the keyswitch.
2. Confirm that the mode goes to RUN (led on). Make note of DST400.
3. Set switch back to TERM.
4. Switch the PLC back to program via DmD. 
5. Confirm the mode goes to PROGRAM (led off). Make note of DST400.
6. Open the log in DmD and see if the mode transitions are getting logged.

What about the RUN mode transition shown in the original post and the overwritten log entries? The attached does not correspond to the log file that I sent you.

That is an excellent question. The logs are ring buffers and limited to 256 entries. If there is some busted wrap logic on the write or read that could explain missing log entries. Don't think that explains the post mortem DSTs though.

I have sent you additional log files/images from troubleshooting this morning. I did a number of things including the above requested. The file names are in order and pretty self explanatory.

1) It is apparent that there is definitely an issue in the Sys log file and end values are being overwritten.
2) The controller absolutely does not go back to RUN after a power outage. Confirmed this twice.

[BobO]

2) The controller absolutely does not go back to RUN after a power outage. Confirmed this twice.

Consistently?!? Does it do it when you cycle the panel?

[RBPLC]

2) The controller absolutely does not go back to RUN after a power outage. Confirmed this twice.

Consistently?!? Does it do it when you cycle the panel?

I have a breaker in-line with power to the controller. I turned this off twice while the controller was in RUN mode. After about 15 seconds I turned it back on and neither time did it go back to RUN.

Also, it does look like it's dumping "some" retentive memory. The issue reported on initially seems to have occurred again. Namely, it looks like some retentive bits are not resuming keeping their state and a portion of my retentive UDT memory is being dumped.

[BobO]

I have a breaker in-line with power to the controller. I turned this off twice while the controller was in RUN mode. After about 15 seconds I turned it back on and neither time did it go back to RUN.

Also, it does look like it's dumping "some" retentive memory. The issue reported on initially seems to have occurred again. Namely, it looks like some retentive bits are not resuming keeping their state and a portion of my retentive UDT memory is being dumped.

Starting to wonder if it is a hardware issue. The retentive ram write sounds dodgy. It might be instructive to run your code on similar hardware and see if we can dupe it, but barring that I am gonna want to swap hardware with you and see if we can dupe it with your hardware.

[RBPLC]

I have a breaker in-line with power to the controller. I turned this off twice while the controller was in RUN mode. After about 15 seconds I turned it back on and neither time did it go back to RUN.

Also, it does look like it's dumping "some" retentive memory. The issue reported on initially seems to have occurred again. Namely, it looks like some retentive bits are not resuming keeping their state and a portion of my retentive UDT memory is being dumped.

Starting to wonder if it is a hardware issue. The retentive ram write sounds dodgy. It might be instructive to run your code on similar hardware and see if we can dupe it, but barring that I am gonna want to swap hardware with you and see if we can dupe it with your hardware.

Retentive memory is being dumped (not sure the extent, yet). I have a custom memory area called GC (bit) that is 256 in size. I'm troubleshooting this first because use of my UDTs has pointers and I could be missing something that is inadvertently resetting values.

GC200 is used nowhere in any part of the program. I can turn it on via a data view and when rebooted, the bit is off. When doing the same thing with C1000 (which is used nowhere in the program, and the retentive range is left at default), it too is reset. I just tested this with the REBOOT instruction. Using the REBOOT instruction also results not going to RUN.

For reference, I did the same REBOOT with a BRX on my desk with C1000 turned on via data view and after REBOOT, C1000 was retained as expected. 

[BobO]

I doubt that this is an issue, but make sure $WatchdogReboots is zero. There is a mechanism in the PLC to prevent runaway crash and reboot cycles.