Host Engineering Forum

General Category => Do-more CPUs and Do-more Designer Software => Topic started by: ATU on September 30, 2020, 10:28:57 AM

Title: MWX Hang up
Post by: ATU on September 30, 2020, 10:28:57 AM
I started having an issue with an MWX instruction that just does nothing. It doesn't fault or give me a success. It just sits there all alone in a stage. I made sure the program is not looping back through the stage. I don't have any conflicts on the port or the device. The device status shows the instruction as "Locked" .  What does that tell me? The idle time toggles from 0 to 1.  I have never seen this before. This was working. If I shut the stage off, I get a fault "Device instruction terminated before completion @0000081A (MWX at Mod3_RL_Axis4@27)" Help

Title: Re: MWX Hang up
Post by: ATU on September 30, 2020, 10:58:12 AM
I went back and changed the device to another one already in the program and it worked. So I changed all the other uses for the device, 3 in that program to the other device and deleted that one. Downloaded, saved. Then I recreated the device and used the new one as before. All works fine. Somehow that device got corrupted. I have a copy of that program, do you want to look at it?
Title: Re: MWX Hang up
Post by: BobO on September 30, 2020, 11:06:45 AM
I went back and changed the device to another one already in the program and it worked. So I changed all the other uses for the device, 3 in that program to the other device and deleted that one. Downloaded, saved. Then I recreated the device and used the new one as before. All works fine. Somehow that device got corrupted. I have a copy of that program, do you want to look at it?

Sometimes the problem is something in retentive memory that gets flushed in the process of recreating stuff, so the project itself may not be the issue, but, if putting that project back into a unit results in the failure coming back, then I definitely want to see it.
Title: Re: MWX Hang up
Post by: BobO on September 30, 2020, 11:10:39 AM
Another thing that you can try in the future is to export the project without the SysConfig, and then import again. That has been known to clean things up.

One of the biggest strengths and weaknesses of Do-more is our continuous improvement of it. From version to version we are patching stuff in the background. I think sometimes that process isn't as clean as we would like, and older projects acquire some entropy over time.
Title: Re: MWX Hang up
Post by: ATU on September 30, 2020, 11:22:27 AM
I loaded the old program in and it worked fine.
Title: Re: MWX Hang up
Post by: BobO on September 30, 2020, 11:26:35 AM
I loaded the old program in and it worked fine.

Probably something stuck in retentive memory then. Clearing probably would have fixed it.
Title: Re: MWX Hang up
Post by: ATU on September 30, 2020, 02:19:13 PM
The original problem was 5 modbusTCP devices that were on the same TCP port. The device I was talking with could only handle 4 at one time.  2 of the MWX instructions were in this lock up mode. So I broke off  the 5th device on its own port. However, the 2 that locked up continued to do so even after I changed the port.  I had to recreate both devices before it started working again.  I didn't know about the load/reload trick.  Maybe  a utility to clean up and initialize old revisions would be nice to have? Working on long projects, I go through several revisions before its finished.
Title: Re: MWX Hang up
Post by: BobO on September 30, 2020, 03:51:01 PM
Maybe  a utility to clean up and initialize old revisions would be nice to have? Working on long projects, I go through several revisions before its finished.

I don't mean to imply that there is any known junk in the project to fix. There are already internal integrity checks being run, and we fix everything we learn about. We have seen some occasions where dumping the documentation has helped, and I'm speculating that going through an export/import without the binary copy of the SysConfig might restore something amiss. About the only time we ever do that in house is when alpha or beta development versions break something and we are trying to recover code.

Title: Re: MWX Hang up
Post by: ATU on September 30, 2020, 04:08:07 PM
Sorry, didn't intend to infer that. I didn't even know the export utility existed until today.  That is probably a better way to migrate an old project that was done several revisions ago.

Title: Re: MWX Hang up
Post by: BobO on September 30, 2020, 04:24:37 PM
Sorry, didn't intend to infer that. I didn't even know the export utility existed until today.  That is probably a better way to migrate an old project that was done several revisions ago.

We make every effort to transparently migrate projects when you open them. That process is usually 100% once released. If for any reason it isn't, export/import might be helpful.

The Export and Import are located on the File menu. The export format is mostly human readable text, with the exception of the SysConfig text-encoded binary. Much of the SysConfig content is exported outside of the binary, and the binary section can be omitted.
Title: Re: MWX Hang up
Post by: franji1 on September 30, 2020, 04:47:04 PM
The best way to do it is to re-configure all your configurable devices in a new project exactly like you had.

Then do Tools->Insert Instructions from File at the Code-Block level into that NEW project.  The device names must match.

Only simple devices get imported without a #BEGIN SYSCONFIG section.  Anything with any kind of complexity requires that you have them defined via the UI.  File->Import Project would require the SYSCONFIG section to exist for any complex devices.  Hence, the Tools->Insert Instructions from File is required to get anything to import via TXT file w/non-trivial devices without a SYSCONFIG section.
Title: Re: MWX Hang up
Post by: ATU on December 02, 2020, 02:18:26 PM
I had this same issue again on an MWX instruction. It just sits there. It ran for weeks without issue and now its just stuck on an MWX instruction. No fault, no System error or warning. Anything you want me to check before I reboot the CPU?
Title: Re: MWX Hang up
Post by: BobO on December 02, 2020, 02:25:24 PM
I had this same issue again on an MWX instruction. It just sits there. It ran for weeks without issue and now its just stuck on an MWX instruction. No fault, no System error or warning. Anything you want me to check before I reboot the CPU?

Can you send me the project, so I maybe can offer some guidance?
Title: Re: MWX Hang up
Post by: ATU on December 02, 2020, 02:33:25 PM
Yup
Title: Re: MWX Hang up
Post by: ATU on December 02, 2020, 03:45:10 PM
Did you receive it?
Title: Re: MWX Hang up
Post by: BobO on December 02, 2020, 04:22:09 PM
Did you receive it?

I did. Been fighting fires.
Title: Re: MWX Hang up
Post by: ATU on December 02, 2020, 06:55:30 PM
Great!   Let me know if you see anything.  Don't burn your fingers.
Title: Re: MWX Hang up
Post by: BobO on December 03, 2020, 10:41:20 AM
Great!   Let me know if you see anything.  Don't burn your fingers.

How do these program blocks re-RUN once they've complete?
Title: Re: MWX Hang up
Post by: ATU on December 03, 2020, 10:54:52 AM
STAGE 0 in the CLX modules monitor a word in the Data Packet being sent to the BRX from the Rockwell PLC via EIP.  The Rockwell PLC reads that word back and looks for it to be cleared by the BRX when its done. The EIP explicit message server is enabled.
Title: Re: MWX Hang up
Post by: BobO on December 03, 2020, 11:24:21 AM
I'm wondering if somehow the Modbus driver thinks it's locked by an instruction. I think the wait for driver to become available has no timeout, so it would sit there indefinitely. The Device status view might you you a sense of that.
Title: Re: MWX Hang up
Post by: ATU on December 03, 2020, 11:36:14 AM
They needed to run the machine this morning, I rebooted and it cleared this time.  Is there any way for me to check that programmatically and clear it  without a reboot? I think there should be some kind of timeout or else it sits there for days stuck.  I can look next time it does it. I should of thought to look this time, I just didn't see any system faults.
Title: Re: MWX Hang up
Post by: BobO on December 03, 2020, 11:44:56 AM
They needed to run the machine this morning, I rebooted and it cleared this time.  Is there any way for me to check that programmatically and clear it  without a reboot? I think there should be some kind of timeout or else it sits there for days stuck.  I can look next time it does it. I should of thought to look this time, I just didn't see any system faults.

If it is doing what I described, the firmware is broken. A timeout is just masking that, and probably making it worse, since there is never a good time to steal a locked driver from another instruction. No way to know programmatically right now, other than timing the program. If (when) it happens again, please pull up the Device status view and see what the device thinks it's doing. I don't envision this to be a particularly difficult thing to fix if it can be duplicated, or can learn something about how it got where it did.
Title: Re: MWX Hang up
Post by: ATU on December 03, 2020, 12:14:42 PM
Is that the button that says devices? You get a list.
Title: Re: MWX Hang up
Post by: franji1 on December 03, 2020, 12:23:12 PM
Is that the button that says devices? You get a list.
double click on the specific device in that list
Title: Re: MWX Hang up
Post by: ATU on April 27, 2022, 03:05:31 PM
I had this happen again. Unfortunately, they needed the machine to run and I couldn't debug it. However, I did reboot the device that it was talking to and no change. I tried putting a long watchdog timer on the rung and restart the instruction but no effect. No faults nothing. It just doesn't move on until I do a Pgm/Run transition. I did put the new firmware in. It is just so infrequent. It has been months since it happened last and the machine ships to Mexico next week. Have you seen anything like this? Why doesn't it reset when I break the input line? Sorry I didn't check the device status, it has been so long I forgot to look for it.

Title: Re: MWX Hang up
Post by: BobO on April 27, 2022, 03:19:48 PM
I had this happen again. Unfortunately, they needed the machine to run and I couldn't debug it. However, I did reboot the device that it was talking to and no change. I tried putting a long watchdog timer on the rung and restart the instruction but no effect. No faults nothing. It just doesn't move on until I do a Pgm/Run transition. I did put the new firmware in. It is just so infrequent. It has been months since it happened last and the machine ships to Mexico next week. Have you seen anything like this? Why doesn't it reset when I break the input line? Sorry I didn't check the device status, it has been so long I forgot to look for it.

Are you running 2.9.x firmware?
Title: Re: MWX Hang up
Post by: ATU on April 27, 2022, 03:27:00 PM
I upgraded now, but I think it was 2.8.  I would have upgraded the firmware the prior time it happened with the latest. 12/2/2020.
Title: Re: MWX Hang up
Post by: BobO on April 27, 2022, 03:32:26 PM
I upgraded now, but I think it was 2.8.  I would have upgraded the firmware the prior time it happened with the latest. 12/2/2020.

2.8.x had client issues that were resolved in 2.9.1.
Title: Re: MWX Hang up
Post by: ATU on April 27, 2022, 03:35:48 PM
Fingers crossed that fixes it amigo.
Title: Re: MWX Hang up
Post by: BobO on April 27, 2022, 03:43:10 PM
Fingers crossed that fixes it amigo.

The symptom you describe sounds exactly like it.

When establishing a TCP connection to the server, there were situations where a failure would cause a permanent driver level hang. Once established it continued to work fine, so it tended not to show up in clean networks. Radios and other semi-reliable networks demonstrated it pretty quickly. Once we knew that it was pretty easy to dupe and fix.
Title: Re: MWX Hang up
Post by: ATU on April 27, 2022, 04:27:05 PM
it tended not to show up in clean networks.

What's a clean network? It took 2 years to show up again.
Title: Re: MWX Hang up
Post by: BobO on April 27, 2022, 04:33:08 PM
What's a clean network? It took 2 years to show up again.

Normal wired Ethernet that isn't being overwhelmed by noise. Noisy or unreliable (radio) networks tended to show it quickly. We duped it by constantly power cycling a switch. In the right conditions you can get it fail in minutes or hours. In a clean network it might never fail. Sounds like your network was pretty clean.