H2-DM1E: All lights on steady yellow

[plcnut]

I got to work this morning to find a note that "the system turned unresponsive at about 11:30 pm last night". After attempting to connect with DMD unsuccessfully, I opened the main cabinet to find all the H2-DM1E lights on steady yellow. I power cycled the system and it re-started in run mode and all is well.
I do not know what caused the controller to go to the error state. Any ideas? (the help file was not much help: "All of the LEDs are ON and the color of each LED is YELLOW indicates the boot loader has started running")

I am running the latest version of DMD, and upgraded the OS on the PLC to the latest, yesterday afternoon.

Background FYI's:
We have a 24 port managed Switch in the network with some bad ports due to lightning, that has caused some sporadic network problems since last week.

The dedicated LAN for this system is comprised of: 2- H2-DM1E's, 2- CMore HMI's (a 15" EA9, and a 10" EA7), 6- Windows7 64 bit servers, 11- network printers (Zebra GX420d's), 1- managed 24 port switch (Netgear Prosafe), 1- un-managed 8 port switch (Stride), and 1 router (Linksys EA6900) with the WAN tied to our main network.

[BobO]

That is actually the state at a power-on reset. Normally that is what you see for an instant before the loader takes off. I have seen very rare occasions of the reset hanging like that. I'm not thrilled about it, but the more important question is "why did it reset?"...

[plcnut]

but the more important question is "why did it reset?"...

That is what I'm wondering about.

I made a minor program change late last night, as you can see in this log, but the controller did not log anything when it locked up (I would assume this would eliminate a power failure as the culprit).

S387, Thu Apr 10 09:44:16 2014, Default User changed mode to PROGRAM
S388, Thu Apr 10 09:44:48 2014, Default User updated the PLC operating system
S389, Thu Apr 10 09:45:06 2014, System was turned off
S390, Thu Apr 10 09:45:06 2014, System was turned on
S391, Thu Apr 10 09:45:44 2014, Default User changed mode to RUN
S392, Thu Apr 10 12:46:57 2014, System was turned off
S393, Thu Apr 10 13:17:59 2014, System was turned on
S394, Thu Apr 10 19:51:59 2014, PLC rebooted following hardware watchdog timeout.
S395, Thu Apr 10 19:51:59 2014, System was turned off
S396, Thu Apr 10 19:51:59 2014, System was turned on
S397, Thu Apr 10 20:59:12 2014, PLC rebooted following hardware watchdog timeout.
S398, Thu Apr 10 20:59:12 2014, System was turned off
S399, Thu Apr 10 20:59:12 2014, System was turned on
S400, Thu Apr 10 21:09:35 2014, System was turned off
S401, Thu Apr 10 21:09:48 2014, System was turned on
S402, Thu Apr 10 21:37:07 2014, System content changed by Default User (Program)
S403, Fri Apr 11 06:48:38 2014, System was turned off
S404, Fri Apr 11 06:48:48 2014, System was turned on

EDIT:
I added some older log entries to include the OS update.
This log shows where there were hardware Watchdog timeouts as well :/
Looking back through the log since February, there was only 1 other watchdog timeout...

[BobO]

There are a couple of image register values to allow programmatic tracking of watchdogs: $WatchdogReboot (ST13) and $WatchdogReboots (DST385). ST13 will be 'on' following a reboot, and DST385 will increment each time it watchdogs. Track those. If this continues following the power cycle, we're gonna have to dig deeper. The hard part is that with as much comm as you are doing, it will be nearly impossible to duplicate the conditions that's causing your issues...but if there is an issue, we gotta find it.

[plcnut]

DST385 = 6. I don't think that I have ever reset the memory on this controller, (So this would be lifetime for this controller).
According to the Log, I had one timeout on April 1st, and then 2 yesterday. The others must have been considerably older, because my System log was last cleared on February 25th.
Also, this is the first DoMore CPU I bought, and it has undergone all the testing/upgrades of this system (You guys have seen some of the code I've thrown at it). I do know that in the past I did some testing with a Strenuous FOR loop in $MAIN that caused some WatchDog issues, so they would be included in that count.
I also replaced the Managed switch with an un-managed prosafe about an hour ago, so the faulty switch has now been eliminated from the equation.

[BobO]

I do know that in the past I did some testing with a Strenuous FOR loop in $MAIN that caused some WatchDog issues, so they would be included in that count.

That would only cause a software watchdog...not a hardware watchdog. Reboots are a function of a CPU in the weeds, or frozen due to a backplane access issue.

[plcnut]

Bob,
We had another hardware watchdog timeout this morning (the system restarted itself though, so I didn't know it happened until I checked the log).
What could I monitor that would help you know what is causing it?

[BobO]

The two locations I mentioned in the previous post will allow you know that it happened without checking the log. The time it happens might be interesting...if it is recurring, what are you doing at that time?

There is currently nothing else in the controller that will give me useful info, but let me think about this a bit. I think I can put together some special firmware that will allow us to know what instruction we crashed in.

[ATU]

What about the rack?. I've had unexplainable DL CPU faults in the past, replaced the rack and it all goes away.