Host Engineering Forum

General Category => Do-more CPUs and Do-more Designer Software => Topic started by: Bolt on May 31, 2022, 11:30:01 AM

Title: Hardware Watchdog Timeout
Post by: Bolt on May 31, 2022, 11:30:01 AM
Over the weekend I had a BRX on 2.9.3 with a hardware watchdog timeout resulting in the PLC dropping to PROGRAM mode.  I have over a year's worth of entries in my Event Logs, and no other watchdogs to be found. Do you think I just unknowingly reached the 10 timeout limit over the last 5ish years?

Can you help me decipher the DST400 records to learn anything?
DST 400 806
DST401 1075183616
DST402 518
DST403 1075183616
DST404 1603
DST405 1802249
DST406 806
DST407 1835023
DST408 518
DST409 16918021
Title: Re: Hardware Watchdog Timeout
Post by: BobO on May 31, 2022, 12:01:21 PM
We report the count in $WatchdogReboots (DST385). You can reset it from code. The purpose of it is to prevent a continuous reboot (from corruption, busted program execution, etc) from making it impossible to communicate with the PLC. By dropping into program, you have a much better chance of remote recovery.

All of the codes look normal, so the reboot didn't come during a transitional state. We have more codes for development, but we only leave the transitional state codes in product.
Title: Re: Hardware Watchdog Timeout
Post by: Bolt on May 31, 2022, 12:29:50 PM
Yeah, I've since thought about resetting the count if no change in a certain number of days, etc.  I have already added the count to my "PLC Restarted" email.
Title: Re: Hardware Watchdog Timeout
Post by: PLCwannabe on June 06, 2022, 12:58:49 AM
I had the same issue happen yesterday on a plc controlling a milk tank. Plc unexpectedly went into program mode. I cycled the run/program/off switch and the 24vdc power supply, and it started working again. However after reading this thread I checked dst385, and it was at zero. The event log showed 4 or 5 watchdog timeouts in the last couple months since it was last cleared. Is there any more info to be gleaned from the attached screenshot?
 Is it ok to just clear dst385 every day?
Title: Re: Hardware Watchdog Timeout
Post by: BobO on June 06, 2022, 10:10:54 AM
I had the same issue happen yesterday on a plc controlling a milk tank. Plc unexpectedly went into program mode. I cycled the run/program/off switch and the 24vdc power supply, and it started working again. However after reading this thread I checked dst385, and it was at zero. The event log showed 4 or 5 watchdog timeouts in the last couple months since it was last cleared. Is there any more info to be gleaned from the attached screenshot?
 Is it ok to just clear dst385 every day?

That's odd. DST385 should be incrementing each time it watchdogs. It absolutely shouldn't be zero. Which is kinda beside the point...shouldn't be watchdogging either.

Seems like something else is going on there.
Title: Re: Hardware Watchdog Timeout
Post by: PLCwannabe on June 07, 2022, 12:25:32 AM
Does a power cycle perhaps clear dst385?
I've had a lot of buffer overflow errors on this specific plc. Could that have anything to do with this issue?
Title: Re: Hardware Watchdog Timeout
Post by: BobO on June 07, 2022, 08:40:38 AM
Does a power cycle perhaps clear dst385?
I've had a lot of buffer overflow errors on this specific plc. Could that have anything to do with this issue?

No. It's retentive.

We should be preventing overflows from breaking anything, but if there is something uncaught, it could definitely cause unpredictable behavior.
Title: Re: Hardware Watchdog Timeout
Post by: BobO on June 08, 2022, 09:40:37 AM
I had the same issue happen yesterday on a plc controlling a milk tank. Plc unexpectedly went into program mode. I cycled the run/program/off switch and the 24vdc power supply, and it started working again. However after reading this thread I checked dst385, and it was at zero. The event log showed 4 or 5 watchdog timeouts in the last couple months since it was last cleared. Is there any more info to be gleaned from the attached screenshot?
 Is it ok to just clear dst385 every day?

Can you post a screen shot of the general info page with firmware and hardware revisions?
Title: Re: Hardware Watchdog Timeout
Post by: PLCwannabe on June 08, 2022, 03:07:13 PM
Done.
Title: Re: Hardware Watchdog Timeout
Post by: BobO on June 08, 2022, 04:11:33 PM
Done.

There are definitely fixes in 2.9.x. We'll be dropping to 2.9.5 within a week or so. Might wanna update to that after we do.
Title: Re: Hardware Watchdog Timeout
Post by: eman5oh on June 27, 2022, 02:17:35 PM
Any word on when 2.9.5 will be out?
Title: Re: Hardware Watchdog Timeout
Post by: JeffS on November 15, 2022, 06:20:59 PM
I have a PLC that seems to have watchdog hardware timeouts about twice a day.  I am on firmware 2.9.6. 

I saw this error "panic @008EBA6C: exit_hook" on the status page and figured it could possibly be contributing.  I have no idea what this error means though, can anyone shed some light on what this could mean?
Title: Re: Hardware Watchdog Timeout
Post by: brucek on November 16, 2022, 08:32:52 AM
Just and observation but in the second screen shot of the list of reboots it seems to happen every 11ish hours. Do you have a task or something that happens every 11ish hours?
Title: Re: Hardware Watchdog Timeout
Post by: JeffS on November 16, 2022, 09:58:38 AM
Nope, I don't see anything that would occur every 11ish hours.   I had thought it could be a comms issue that was causing this, but I have since disabled all the comms and it still giving watchdog reboots.
Title: Re: Hardware Watchdog Timeout
Post by: BobO on November 16, 2022, 01:42:24 PM
It is a fault in the TCP/IP stack, apparently failing during some kind of termination. Never heard of it before and have no idea what could cause it. If it is an option, disconnect the Ethernet cable from the PLC. I'd like to rule out any external input as the cause.
Title: Re: Hardware Watchdog Timeout
Post by: JeffS on November 16, 2022, 06:10:06 PM
Looks like the panic error happens once a minute.  I can't really disconnect the ethernet as it is required for operation.  What I will try to do is get by the site at some point and put my computer in place of the PLC for a couple minutes and get a wireshark log.