Host Engineering Forum
General Category => Do-more CPUs and Do-more Designer Software => Topic started by: RBPLC on May 11, 2023, 11:50:32 AM
-
Looking for general guidance.
I had a power outage this morning. Power was out for about 45 minutes, BRX PLC lost power during this time. At time of outage, controller was in RUN mode and was actively running piece of equipment. After power was restored, controller was in PROGRAM mode. I can't figure out why it didn't go back to Run mode when power was restored. The BRX has 6 remote EBC100s. I checked the options and the BRX is set to allow RUN mode even if remotes aren't connected. PLC log shows power off event and power on event. Any thoughts?
-
Did the power outage cause a surge? electrical noise and surges have cause problems like this for me in the past. (not just BRX AB also) I always install a power line filter and surge protectors with only the plc connected to them. Could be the problem maybe??
JW
-
Looking for general guidance.
I had a power outage this morning. Power was out for about 45 minutes, BRX PLC lost power during this time. At time of outage, controller was in RUN mode and was actively running piece of equipment. After power was restored, controller was in PROGRAM mode. I can't figure out why it didn't go back to Run mode when power was restored. The BRX has 6 remote EBC100s. I checked the options and the BRX is set to allow RUN mode even if remotes aren't connected. PLC log shows power off event and power on event. Any thoughts?
Nothing else in the log?
Newest version of the firmware should be giving more info on the startup modes transitions, but earlier versions didn't give as much. Generally it's a bad battery resulting in loss of retentive state, I/O config change, or the Ethernet master issue you referred to.
-
Looking for general guidance.
I had a power outage this morning. Power was out for about 45 minutes, BRX PLC lost power during this time. At time of outage, controller was in RUN mode and was actively running piece of equipment. After power was restored, controller was in PROGRAM mode. I can't figure out why it didn't go back to Run mode when power was restored. The BRX has 6 remote EBC100s. I checked the options and the BRX is set to allow RUN mode even if remotes aren't connected. PLC log shows power off event and power on event. Any thoughts?
Nothing else in the log?
Newest version of the firmware should be giving more info on the startup modes transitions, but earlier versions didn't give as much. Generally it's a bad battery resulting in loss of retentive state, I/O config change, or the Ethernet master issue you referred to.
Attached is the event log. As best I can tell, it looks like retentive memory for some UDTs was lost as well. I've been trying to figure out today if the UDT loss was a result of the power outage or some programming thing and I want to blame it on the power issue. A retentive non UDT bit was off as well which I would have expected to be on. I'm running OS 2.9.6. $BatteryLow bit is not on.
-
Attached is the event log. As best I can tell, it looks like retentive memory for some UDTs was lost as well. I've been trying to figure out today if the UDT loss was a result of the power outage or some programming thing and I want to blame it on the power issue. A retentive non UDT bit was off as well which I would have expected to be on. I'm running OS 2.9.6. $BatteryLow bit is not on.
According to the log the PLC was already in program mode.
-
Attached is the event log. As best I can tell, it looks like retentive memory for some UDTs was lost as well. I've been trying to figure out today if the UDT loss was a result of the power outage or some programming thing and I want to blame it on the power issue. A retentive non UDT bit was off as well which I would have expected to be on. I'm running OS 2.9.6. $BatteryLow bit is not on.
According to the log the PLC was already in program mode.
Is there a scenario where the log file would be incorrect? The controller/equipment had been running continuously since 4/28/23 in RUN mode until the power outage.
-
Is there a scenario where the log file would be incorrect? The controller/equipment had been running continuously since 4/28/23 in RUN mode until the power outage.
I'll never say never, but I don't know of any way to get to run mode without creating a log entry.
-
Same issue. This morning, my facility power very briefly went out twice. The attached log file time is correct. After the first outage, I noticed (through other means) that the controller didn't appear to be running. Before I could debug, the power went out again. After the second outage, I got a chance to remote in and look at the log file. This is similar to the first post about this issue. At present time, I haven't done anything with the controller other than log in to look at the Sys Info Log. It's currently in Program mode.
Edit: I just noticed from the first post that the log file entries appear to have been overwritten. I don't remember doing anything to clear the System Log the first time around.
-
What is your hardware version?
-
$HardwareRev = 16696.
-
I'm a bit puzzled. Everything I can identify that would go from from run to program as a result of a power cycle should be leaving something in the log.
Can you dump DST400-409 into a Data View and post a picture of that?
-
Values attached.
-
Those values are a push down stack of the current state and the last 4 power cycles. Basically at startup we grab the top 8 values (4 states) and copy them to the lower 8 values. The top 2 values contain the current state. During initialization, mode changes, etc. the PLC OS writes values after each operation. Those values allow us to know the last initialization state, thus have a pretty good idea where the PLC is or failed at before the restart.
The 1400 values indicates the PLC was in program mode when it was power cycled.
The 608 value got reused (that shouldn't be, but ::) ) and the PLC was either validating the program during goto run, or was in program mode and was updating the sys config flash. One might have left the PLC flash corrupted (incomplete write to flash), and the other would have caught the PLC in transitional limbo which wouldn't corrupt but wasn't a good spot to reboot.
The 806 value is the last state published on a normal transition to run mode.
So the PLC appeared to be running, then power was cycled and while initializing or writing flash was power cycled again, and after that was in program mode three times.
Per the log and those post mortem registers I see no evidence that the PLC has been in run more for a while. If retentive memory was wiped that would be in the log. If retentive memory wasn't wiped, the log and those post mortem states should be believable.
-
I don't know what to tell you. After the first instance of this I went in and fixed the UDT memory that appeared to have been dumped. It was absolutely in Run when I left it. We have been running this piece of equipment almost continuously since May when this first happened. It was running this morning as well. Y8 of the BRX is tied into main control power for the piece of equipment. If that output is not on, the piece of equipment doesn't run. Also, WY0 is outputting a process variable that would cause erratic operation if the BRX wasn't running. I can't explain the log file, just that something isn't correct.
-
Do you have more than one PLC on the same network? Is it possible that we aren't looking at the correct one? Is it possible that an IP address has gotten reused?
The only way 1400 gets into that post mortem state is in program, going to run leaves different values, and the log shows no mode transitions to either mode. The log and the post mortem appear to agree and I've never seen a case when they didn't. I know of no possible way the PLC could be running with both of those at the state they are at. That doesn't mean it isn't, just that it would require multiple failures in different systems for that memory state to be present when the PLC is in run mode and I've never seen such a thing. That suggests another explanation.
Just for reference, what are the log entries prior to the three power cycles?
-
Another location that might shed light is DST410. Any time there is a mode change failure, that location gets written. That was a relatively recent addition, but I think it was in 2.9.6.
-
DST410 = 0
-
So the PLC didn't see a mode transition failure. That fits with the log and post mortem regs.
Would still like to see the log prior to the reboots.
-
Do you have more than one PLC on the same network? Is it possible that we aren't looking at the correct one? Is it possible that an IP address has gotten reused?
The only way 1400 gets into that post mortem state is in program, going to run leaves different values, and the log shows no mode transitions to either mode. The log and the post mortem appear to agree and I've never seen a case when they didn't. I know of no possible way the PLC could be running with both of those at the state they are at. That doesn't mean it isn't, just that it would require multiple failures in different systems for that memory state to be present when the PLC is in run mode and I've never seen such a thing. That suggests another explanation.
Just for reference, what are the log entries prior to the three power cycles?
There are multiple PLCs on the same network.
This is the correct one. This is the only one that is programmed remotely like it. I have limited network troubleshooting capability but I pinged the controller, got a response, pulled the network cable, pinged and got no response, plugged back in, pinged and got a response. I suspect that there are no conflicting IP addresses.
I have emailed the log file.
I physically confirmed that the lights on the front of the PLC indicate:
ON - PWR
OFF - RUN
ON -MEM
OFF - ERR
This corresponds to what I am seeing in DMD.
Also, I didn't really press the issue at the time, but S477 from the original post shows a switch back to run mode. I cannot with complete certainty confirm that after I made changes to programming that I didn't do something with the log file, but I don't think so. The forces that are shown in that original post were correct as I was testing a few things. If you look at the log now, those entries appear to have been overwritten.
-
According to the log, the PLC was changed to program mode through DmD by the Default User in April. It was never changed to run mode after that. There are three empty log entries (probably due to older DmD) that are most likely mode restoration ("PLC mode restored to program on startup.") entries that we added in the last release.
Please try the following.
1. Switch the PLC to RUN via the keyswitch.
2. Confirm that the mode goes to RUN (led on). Make note of DST400.
3. Set switch back to TERM.
4. Switch the PLC back to program via DmD.
5. Confirm the mode goes to PROGRAM (led off). Make note of DST400.
6. Open the log in DmD and see if the mode transitions are getting logged.
-
That new log entry was added in OS 2.9.6, which was after the last DmD release, so it's blank.
I was gonna suggest using the web page to access the log...but realized the new entry hasn't been added to it. Need to fix that...
-
So from this conversation I am guessing my habit of always putting a UPS in to feed a PLC is a good one. I had a jobsite get hit with lightning, 3 times in one month! First one we found out that the building service had no surge protection. Second one showed us that the costly AB UPS was specially built with no surge protection! They came to install the building surge protection and found out they had the wrong voltage unit. So the 3rd strike did 8K worth of damage to the brand new SLC500 rack.
So my practice now surge suppressor on the incoming panel power, a shielded control transformer powering a surge protected UPS that supplies a 24vdc power supply to run a 24vdc PLC. Knock on a large tree but I haven't lost any more PLCs. I only allow customer Ethernet through fiber optic connections. One customer strung 7 routers together to cross a campus. Lightning took out their server and all the routers.
-
According to the log, the PLC was changed to program mode through DmD by the Default User in April. It was never changed to run mode after that. There are three empty log entries (probably due to older DmD) that are most likely mode restoration ("PLC mode restored to program on startup.") entries that we added in the last release.
Please try the following.
1. Switch the PLC to RUN via the keyswitch.
2. Confirm that the mode goes to RUN (led on). Make note of DST400.
3. Set switch back to TERM.
4. Switch the PLC back to program via DmD.
5. Confirm the mode goes to PROGRAM (led off). Make note of DST400.
6. Open the log in DmD and see if the mode transitions are getting logged.
What about the RUN mode transition shown in the original post and the overwritten log entries? The attached does not correspond to the log file that I sent you.
-
According to the log, the PLC was changed to program mode through DmD by the Default User in April. It was never changed to run mode after that. There are three empty log entries (probably due to older DmD) that are most likely mode restoration ("PLC mode restored to program on startup.") entries that we added in the last release.
Please try the following.
1. Switch the PLC to RUN via the keyswitch.
2. Confirm that the mode goes to RUN (led on). Make note of DST400.
3. Set switch back to TERM.
4. Switch the PLC back to program via DmD.
5. Confirm the mode goes to PROGRAM (led off). Make note of DST400.
6. Open the log in DmD and see if the mode transitions are getting logged.
What about the RUN mode transition shown in the original post and the overwritten log entries? The attached does not correspond to the log file that I sent you.
That is an excellent question. The logs are ring buffers and limited to 256 entries. If there is some busted wrap logic on the write or read that could explain missing log entries. Don't think that explains the post mortem DSTs though.
-
According to the log, the PLC was changed to program mode through DmD by the Default User in April. It was never changed to run mode after that. There are three empty log entries (probably due to older DmD) that are most likely mode restoration ("PLC mode restored to program on startup.") entries that we added in the last release.
Please try the following.
1. Switch the PLC to RUN via the keyswitch.
2. Confirm that the mode goes to RUN (led on). Make note of DST400.
3. Set switch back to TERM.
4. Switch the PLC back to program via DmD.
5. Confirm the mode goes to PROGRAM (led off). Make note of DST400.
6. Open the log in DmD and see if the mode transitions are getting logged.
What about the RUN mode transition shown in the original post and the overwritten log entries? The attached does not correspond to the log file that I sent you.
That is an excellent question. The logs are ring buffers and limited to 256 entries. If there is some busted wrap logic on the write or read that could explain missing log entries. Don't think that explains the post mortem DSTs though.
I have sent you additional log files/images from troubleshooting this morning. I did a number of things including the above requested. The file names are in order and pretty self explanatory.
1) It is apparent that there is definitely an issue in the Sys log file and end values are being overwritten.
2) The controller absolutely does not go back to RUN after a power outage. Confirmed this twice.
-
2) The controller absolutely does not go back to RUN after a power outage. Confirmed this twice.
Consistently?!? Does it do it when you cycle the panel?
-
2) The controller absolutely does not go back to RUN after a power outage. Confirmed this twice.
Consistently?!? Does it do it when you cycle the panel?
I have a breaker in-line with power to the controller. I turned this off twice while the controller was in RUN mode. After about 15 seconds I turned it back on and neither time did it go back to RUN.
Also, it does look like it's dumping "some" retentive memory. The issue reported on initially seems to have occurred again. Namely, it looks like some retentive bits are not resuming keeping their state and a portion of my retentive UDT memory is being dumped.
-
I have a breaker in-line with power to the controller. I turned this off twice while the controller was in RUN mode. After about 15 seconds I turned it back on and neither time did it go back to RUN.
Also, it does look like it's dumping "some" retentive memory. The issue reported on initially seems to have occurred again. Namely, it looks like some retentive bits are not resuming keeping their state and a portion of my retentive UDT memory is being dumped.
Starting to wonder if it is a hardware issue. The retentive ram write sounds dodgy. It might be instructive to run your code on similar hardware and see if we can dupe it, but barring that I am gonna want to swap hardware with you and see if we can dupe it with your hardware.
-
I have a breaker in-line with power to the controller. I turned this off twice while the controller was in RUN mode. After about 15 seconds I turned it back on and neither time did it go back to RUN.
Also, it does look like it's dumping "some" retentive memory. The issue reported on initially seems to have occurred again. Namely, it looks like some retentive bits are not resuming keeping their state and a portion of my retentive UDT memory is being dumped.
Starting to wonder if it is a hardware issue. The retentive ram write sounds dodgy. It might be instructive to run your code on similar hardware and see if we can dupe it, but barring that I am gonna want to swap hardware with you and see if we can dupe it with your hardware.
Retentive memory is being dumped (not sure the extent, yet). I have a custom memory area called GC (bit) that is 256 in size. I'm troubleshooting this first because use of my UDTs has pointers and I could be missing something that is inadvertently resetting values.
GC200 is used nowhere in any part of the program. I can turn it on via a data view and when rebooted, the bit is off. When doing the same thing with C1000 (which is used nowhere in the program, and the retentive range is left at default), it too is reset. I just tested this with the REBOOT instruction. Using the REBOOT instruction also results not going to RUN.
For reference, I did the same REBOOT with a BRX on my desk with C1000 turned on via data view and after REBOOT, C1000 was retained as expected.
-
I doubt that this is an issue, but make sure $WatchdogReboots is zero. There is a mechanism in the PLC to prevent runaway crash and reboot cycles.
-
I doubt that this is an issue, but make sure $WatchdogReboots is zero. There is a mechanism in the PLC to prevent runaway crash and reboot cycles.
$WatchdogReboots = 0
-
$WatchdogReboots = 0
Good. Just wanted to rule it out.
If $WatchdogReboots hits 10 (I think) it will drop into program mode on reboot. That prevents an endless boot loop from keeping you out of remote access.
-
Not to tell you your business but it looks like the issue is somehow tied to the Sys log entries/common logic.
I had two slaves on this setup that were always offline. I removed these to see if that could potentially be causing it to not go back to RUN (even though they appeard to be configured properly). After removing, I REBOOTed again and no dice. Same Program mode, no retention.
After doing this, for the heck of it, I cleared the Sys log entries. I then REBOOTed and this time, the retentive bits mentioned above keep their state as expected and the controller returned to RUN as expected.
The UDT issue I'm still trying to track down to see if that's on me. The retention and RUN appear to be correct now.
-
Not to tell you your business but it looks like the issue is somehow tied to the Sys log entries/common logic.
I had two slaves on this setup that were always offline. I removed these to see if that could potentially be causing it to not go back to RUN (even though they appeard to be configured properly). After removing, I REBOOTed again and no dice. Same Program mode, no retention.
After doing this, for the heck of it, I cleared the Sys log entries. I then REBOOTed and this time, the retentive bits mentioned above keep their state as expected and the controller returned to RUN as expected.
The UDT issue I'm still trying to track down to see if that's on me. The retention and RUN appear to be correct now.
Oooh....that does give me some clues...
-
Not to tell you your business but it looks like the issue is somehow tied to the Sys log entries/common logic.
I had two slaves on this setup that were always offline. I removed these to see if that could potentially be causing it to not go back to RUN (even though they appeard to be configured properly). After removing, I REBOOTed again and no dice. Same Program mode, no retention.
After doing this, for the heck of it, I cleared the Sys log entries. I then REBOOTed and this time, the retentive bits mentioned above keep their state as expected and the controller returned to RUN as expected.
The UDT issue I'm still trying to track down to see if that's on me. The retention and RUN appear to be correct now.
Oooh....that does give me some clues...
I didn't have a good Memory Image and that's one issue that was throwing me off. I fixed my Memory Image and have backed it up. This was my UDT memory backup. Since getting the Memory values right, I've REBOOTed multiple times and all UDT retention seems to be correct.
I'm almost positive whatever the issue is was messing with the UDT retention as well and not some errant pointer overwriting values unknowingly.
-
I didn't have a good Memory Image and that's one issue that was throwing me off. I fixed my Memory Image and have backed it up. This was my UDT memory backup. Since getting the Memory values right, I've REBOOTed multiple times and all UDT retention seems to be correct.
I'm almost positive whatever the issue is was messing with the UDT retention as well and not some errant pointer overwriting values unknowingly.
I can't see any issue and none of this code has changed since day 1 of 205 Do-more. Still, it's interesting. The last PLC mode (that gets restored on powerup), the image reg, the system log, and
and a bunch of other things are all packed into the battery backed ram. If a pointer got loose (OS, not logic) and started spraying over that area any/all of those things could happen. I don't see how clearing the SysLog would affect anything else, unless the SysLog buffer logic got dumb and the log entries were getting stuck elsewhere.
Does your logic use any MEMCLEAR instructions with UDTs?
-
I didn't have a good Memory Image and that's one issue that was throwing me off. I fixed my Memory Image and have backed it up. This was my UDT memory backup. Since getting the Memory values right, I've REBOOTed multiple times and all UDT retention seems to be correct.
I'm almost positive whatever the issue is was messing with the UDT retention as well and not some errant pointer overwriting values unknowingly.
I can't see any issue and none of this code has changed since day 1 of 205 Do-more. Still, it's interesting. The last PLC mode (that gets restored on powerup), the image reg, the system log, and
and a bunch of other things are all packed into the battery backed ram. If a pointer got loose (OS, not logic) and started spraying over that area any/all of those things could happen. I don't see how clearing the SysLog would affect anything else, unless the SysLog buffer logic got dumb and the log entries were getting stuck elsewhere.
Does your logic use any MEMCLEAR instructions with UDTs?
There are no MEMCLEAR instructions used with UDTs.
-
There are no MEMCLEAR instructions used with UDTs.
We had a bug in MEMCLEAR with UDTs where in certain circumstances it could get loose and go places it shouldn't. It was fixed, but I was just trying to rule it out as the source of trouble.
At this point I'm not sure where to go. Clearly something got loose in your retentive memory and clearing the SysLog seems to have cured it. I wasn't able to dupe any weirdness though and until we can there's nothing to fix. If it shows up again we can poke at it some more. Problems like this are frustrating for both of us.
-
Are you using the Memory Image Manger with UDT based Regions?
If you have modified the UDTs without deleting their Memory Image Region on the Disk Project, that data in the disk project is corrupt, i.e. you modified the UDT's memory layout, added fields, moved fields around, changed field data types, changed data sizes, of any field in a UDT. The Memory Image Region is just RAW data, so if you write that Region on top of the new UDT schema layout, it will not be correct/valid.
-
Are you using the Memory Image Manger with UDT based Regions?
If you have modified the UDTs without deleting their Memory Image Region on the Disk Project, that data in the disk project is corrupt, i.e. you modified the UDT's memory layout, added fields, moved fields around, changed field data types, changed data sizes, of any field in a UDT. The Memory Image Region is just RAW data, so if you write that Region on top of the new UDT schema layout, it will not be correct/valid.
I had not been using the Memory Image Manager to restore any memory elements. I probably did have a bad memory copy but I didn't attempt to restore from it at any point while the above was going on.