Writing to PLC concerns/complaints

[Bolt]

Sometimes when writing a (minor) update a running PLC, it takes allot longer to write than others.  It stays on the Monitoring ROM update status in PLC Project switch pending; Documentation File ROM update pending; status box for quite a while, sometimes over a minute.  During the Write Process, the rest of the PC is not always usable.  Some mouse clicks are ignored, some are held and turn into select actions, etc.  So, I tend to leave the PC or do something else.  Then, it jumps to the New Edge/Instruction IDs Assigned; Save Project to Disk prompt.  As long as I don't acknowledge this, it doesn't really complete the write process.  As such, all Trend Views are on hold until you click Yes or No.  So, after when you come back the PC, there might be 10 minutes of Trend View data "missing", and you still can't tell if your update addressed whatever you were trying to address!

So, my requests:
1.  Let the PC be usable on another program/monitor while writing to PLC
2.  Speed up the Monitoring ROM stage
3.  Auto Save to Disk with new Instructions IDs (or at least end the writing process to the Trend Views get back to work

Thanks for reading.  This is currently in DmD 2.6.2, but I've experienced it for years.  Windows 7 Pro 64 bit machine.

[franji1]

Very strange.  What is your PLC scan time?

It could be your Flash ROM writes may be taking longer than typical writes, possibly due to excessive Flash ROM writes.  Flash ROM gets slower and slower over time, but this must be after THOUSANDs of writes, which is actually do-able with certain instructions.

Do you see this behavior on any other PLCs, specifically a new(er) one?

[Bolt]

3 ms min, 5 ms avg, 11 ms max.  Always been in that range, I've been keeping an eye on it over time as my code expands.  This is on a BX-DM1E-36ER3-D, spring 2017 purchase that I do most of my monkeying with.

I have probably done over a thousand program writes from the PC, but I doubt I have any instructions in the PLC that are writing to the ROM.  How can I check?

Subsequent ROM writes are faster than the slow ones, they just vary.  It seems to be faster if I don't hit Save before Write.

[franji1]

3 ms min, 5 ms avg, 11 ms max.  Always been in that range, I've been keeping an eye on it over time as my code expands.  This is on a BX-DM1E-36ER3-D, spring 2017 purchase that I do most of my monkeying with.

That's OK.  If it was 50-100ms (10-20x slower than what you normally run), that would be a red flag.  Very large/string heavy programs can have long scan times.  Long scan times affect update time.

I have probably done over a thousand program writes from the PC, but I doubt I have any instructions in the PLC that are writing to the ROM.  How can I check?

It's specific instructions - I'd have to ask...

Subsequent ROM writes are faster than the slow ones, they just vary.  It seems to be faster if I don't hit Save before Write.

If you have a ton of documentation (rung comments, nicknames, etc.) that doesn't change, then that might be the issue.  Also, run mode edits take longer, since the updates are being slowly written to Flash ROM every scan, but your scan time isn't that long, so this should not be that much of a hit like you are seeing.

I would like you to try something (do NOT do any "Save" or "Save As" for the following)
1. Measure how long a run mode edit takes where you only change documentation (e.g. edit 1 nickname).
2. Then measure how long a run mode edit where you only change a single contact (no doc or sysconfig change).  See how long that takes.
3. Then measure how long an edit where you only change the System Configuration (tweak something) - that should only download the program and SysConfig, but not documentation.  See how long that takes.  This will NOT be a run mode edit since you changed the SysConfig.  That's expected.

Report back those 3 measurements.

[franji1]

First glance, SETUPNOD and SETUPIP instructions update the Flash ROM.  So for example - bad example, but still an example, if you have a program doing this with ST7 (scan toggle), you will be doing a few thousand writes to Flash in the twinkle of an eye.  Not that you are doing THAT, but ill-formed logic could easily cause these to occur more frequent than you think.

PLCs can be SPAM generators without even trying.  One customer had his PLC send an email to the purchasing department whenever the level of a specific material measured "low".  Well, when it was teetering on that threshold, the purchasing agent got a whole bunch of emails all at once.  Program just needed some high water mark/low water mark logic.  If this was doing SETUPIP/SETUPNOD, it may have gone un-noticed.

[BobO]

We are too heavy handed with grabbing system level modality on downloads. I know why it was done, but it is extremely poor form. I routinely get stuck by it too, and I say mean thing about my own product when it happens.

[Bolt]

Nearly 6 years later, and this is still extremely aggravating. Problem has persisted across OS's and PC's. Can't do much else while connecting, writing, etc. And don't get me started on disconnecting.

When a VPN connection is lost/dropped/otherwise disconnected, you need to click Disconnect. Okay, fair enough.

"Are you sure? Disconnect from PLC?" I'm actually not even connected anymore, so, Yes.

"Attention! There are differences between the program in memory and the program in the PLC. Write the changes before disconnecting?" No, I couldn't even if I wanted to.

"Error! Error closing comm session. PLC will automatically terminate session after timeout. Would you like to see the detailed error list for this communication link?"

The last message likes to come back twice sometimes, and if you're extra unlucky, DmD just crashes and needs a End Task. But then, most of the time, I can't re-open DmD. I have restarted my PC over a handful of times today already just trying to deal with some very minor updates to a customer with spotty internet. When I do restart Windows, I see a brief popup stating DmDesigner.exe - Application Error The instruction at 0x00000000100633C2 referenced memory at 0x0000000005EA5DE0. The memory could not be read. Click on Ok to terminate the program.

All that work just to disconnect from a PLC that has already been forcefully disconnected by external means.

Meanwhile, a remote desktop connection to the same site, same internet barely misses a beat. I'm not saying the connection was never compromised, but it handled it quite gracefully.

[franji1]

As you have figured out, a Remote Desktop is MUCH MORE RELIABLE than actually doing real-time comm via a flaky VPN routed PC to PLC over the Internet over our specific 0x7070 UDP port.

If there's any client VPN code on your PC, this adds another layer of "what's wrong?" (we've had to deal with 3rd party VPN software that works "most" of the time when things get flakey - sometimes it's their software/driver that gets a little wonky, which causes our software to crash).

Are you using ADC's VPN product?  We might be able to work with them to try to resolve any specific issues.

If it's 3rd party, you could try asking them to work with us to figure it out.

Like I said, Remote Desktop works MUCH BETTER!

[BobO]

Nearly 6 years later, and this is still extremely aggravating. Problem has persisted across OS's and PC's. Can't do much else while connecting, writing, etc. And don't get me started on disconnecting.

When a VPN connection is lost/dropped/otherwise disconnected, you need to click Disconnect. Okay, fair enough.

"Are you sure? Disconnect from PLC?" I'm actually not even connected anymore, so, Yes.

"Attention! There are differences between the program in memory and the program in the PLC. Write the changes before disconnecting?" No, I couldn't even if I wanted to.

"Error! Error closing comm session. PLC will automatically terminate session after timeout. Would you like to see the detailed error list for this communication link?"

The last message likes to come back twice sometimes, and if you're extra unlucky, DmD just crashes and needs a End Task. But then, most of the time, I can't re-open DmD. I have restarted my PC over a handful of times today already just trying to deal with some very minor updates to a customer with spotty internet. When I do restart Windows, I see a brief popup stating DmDesigner.exe - Application Error The instruction at 0x00000000100633C2 referenced memory at 0x0000000005EA5DE0. The memory could not be read. Click on Ok to terminate the program.

All that work just to disconnect from a PLC that has already been forcefully disconnected by external means.

Meanwhile, a remote desktop connection to the same site, same internet barely misses a beat. I'm not saying the connection was never compromised, but it handled it quite gracefully.

I'm sorry it has caused you issues.

From DmD's perspective, there is no VPN. We have no visibility into the internals of the TCP/IP stack or the underlying physical connection. The UDP packets get through or they don't. We don't know the difference.

A remote connection to a PC is largely stateless. Dropping it is much like unplugging a mouse. No big deal. The programming session is not like that. The effort that DmD makes to keep things in sync is to prevent your PLC from ending up corrupted, of particular concern given that the PLC may be running something critical, and to keep you from losing work. Remote Desktop just closes. I'm sure we could do that too, but you wouldn't like it if we did.

The other thing that is happening is layers. The various layers between the user and the PLC all have their shot at resolving things and unfortunately they all like to speak up.

Given that we tend to test in a stable network environment, we don't see any crashes. We probably need to put some effort into testing with bad connections.

The other thing we could look at is allowing the layers to be aware of each other and once we see a failure at one layer bypassing higher layer checks.

And for the record, I spent a couple of days trying to identify the source of startup modality hangs. I couldn't find anything that we were doing that should be causing it. I'm sure we're involved, but I still don't know why.

Again, I'm sorry.

[Bolt]

As you have figured out, a Remote Desktop is MUCH MORE RELIABLE than actually doing real-time comm via a flaky VPN routed PC to PLC over the Internet over our specific 0x7070 UDP port.

If there's any client VPN code on your PC, this adds another layer of "what's wrong?" (we've had to deal with 3rd party VPN software that works "most" of the time when things get flakey - sometimes it's their software/driver that gets a little wonky, which causes our software to crash).

Are you using ADC's VPN product?  We might be able to work with them to try to resolve any specific issues.

If it's 3rd party, you could try asking them to work with us to figure it out.

Like I said, Remote Desktop works MUCH BETTER!

Yes, this is all via AD's Stridelinx VPN router. Most customer sites it works great. A few others, not so much. I'm trying to avoid the remote desktop solution, as then I'm always poking around on customer's onsite PC (and I'd need to open up the VPN router to program PLC from the outside).

[Bolt]

I'm sorry it has caused you issues.

From DmD's perspective, there is no VPN. We have no visibility into the internals of the TCP/IP stack or the underlying physical connection. The UDP packets get through or they don't. We don't know the difference.

A remote connection to a PC is largely stateless. Dropping it is much like unplugging a mouse. No big deal. The programming session is not like that. The effort that DmD makes to keep things in sync is to prevent your PLC from ending up corrupted, of particular concern given that the PLC may be running something critical, and to keep you from losing work. Remote Desktop just closes. I'm sure we could do that too, but you wouldn't like it if we did.

The other thing that is happening is layers. The various layers between the user and the PLC all have their shot at resolving things and unfortunately they all like to speak up.

Given that we tend to test in a stable network environment, we don't see any crashes. We probably need to put some effort into testing with bad connections.

The other thing we could look at is allowing the layers to be aware of each other and once we see a failure at one layer bypassing higher layer checks.

And for the record, I spent a couple of days trying to identify the source of startup modality hangs. I couldn't find anything that we were doing that should be causing it. I'm sure we're involved, but I still don't know why.

Again, I'm sorry.

Well the remote desktop doesn't really disconnect, I'm sure it buffers or something in the background, but and keeps the connection open. A TeamViewer window can be left open for days and it will still be there as if nothing happened. Meanwhile, either DmD will be in a disconnected state, or the Stridelinx connection will drop entirely.

I have never had much luck (outside of a direct LAN connection) with the Re-open Session option. I always figured it had something to do with the VPN connection, but you state that DmD is largely unaware of VPN connection, and just sends/receives packets. Maybe it's due to a new internal VPN connection's updated IP address?

Yes, if the layers were aware of each other, then the popups and retry delays, etc could be greatly reduced.

[BobO]

Well the remote desktop doesn't really disconnect, I'm sure it buffers or something in the background, but and keeps the connection open. A TeamViewer window can be left open for days and it will still be there as if nothing happened. Meanwhile, either DmD will be in a disconnected state, or the Stridelinx connection will drop entirely.

I have never had much luck (outside of a direct LAN connection) with the Re-open Session option. I always figured it had something to do with the VPN connection, but you state that DmD is largely unaware of VPN connection, and just sends/receives packets. Maybe it's due to a new internal VPN connection's updated IP address?

Yes, if the layers were aware of each other, then the popups and retry delays, etc could be greatly reduced.

The PLC uses UDP, rather than TCP. There are a dozen good reasons for this, but the one negative is that there is no session in UDP. To get around this, we have our own session built into the PLC programming protocol. If the PLC and DmD can't talk for 30 seconds or so, the PLC drops the session to prevent sessions from hanging indefinitely. DmD constantly attempts to re-open the session as necessary, using the last password that was provided, but on occasion it can't restore it. The re-open session function is a higher level attempt to fix it.

DmD is entirely unaware of the physical connection. It opens a UDP socket and sends and receives packets. I can plug/unplug the PLC's Ethernet cable all day long without issue, but we have definite challenges with power management and VPNs. It is likely doing something in the socket layer and that is causing us some heartburn.

I don't offer this as an excuse, but to help frame the problem. DmD's Comm Server is a separate app that was designed to allow sharing of serial, USB, Ethernet devices over multiple programming instances, as well as non-programming apps like DDE servers and HMIs. Much of that capability is really no longer required since multi-drop serial isn't used for programming these days, but rearchitecting it isn't trivial. I suspect that some of the issues we face here are related to the architecture, and more specifically, Windows being unhappy with some of what we are doing. That said, we can do better.

[MarkTTU]

Bolt, I stumbled across this today looking for something else, but wanted to let you know you're not the only one who gets frustrated at these issues. I do too and have been for longer than I can remember (I think my first customer VPN setup was circa 2004 and these issues existed in DirectSOFT too). I've had a few conversations over the years with both BobO, franji1, and others and I get the behind the scenes stuff that is at the root of it. Still wish it could be better handled, but at the end of the day I still prefer the BRX PLCs over all others.

[Bolt]

The latest version of DmD has been giving me a few issues.

1. I could connect to (OS 2.10.3) PLC and write program fine several times, but when I tried to update my test bench PLC to OS 2.11 over LAN, and it kept freezing up DmD on the update screen. As soon as I would press "Update PLC Operating System", it wouldn't continue, had to kill the program via Task Manager, couldn't reconnect upon re-opening as "Comm Link already in use in another programming session". Restart PC (install Windows updates...), try again, same problem. Restart PC again, try again, same problem. Restart PC, uninstall 2.11 (and 2.10 for good measure). Re-install 2.11, and I could connect and update no problems.

2. Now today, I'm trying to connect to a customer's site via VPN connection, and it prompts for the PLC's password, frozen. It's like it doesn't accept focus on the password entry field. Force close in Task Manager, but the password box never goes away. Try again in DmD, comms link in use. Restart PC, try again, same results. Restart PC again, re-install 2.10, connect, password prompt, enter password, and I'm in.

Quick "fix" question, is there a way to force all comm links to break to avoid the drawn out process or restarting PC?

[franji1]

The latest version of DmD has been giving me a few issues.

See if you have any .dmp files.  Email them to use if you do (support at hosteng.com)

1. We had a similar issue dealing with powering up bench PLCs, some with USB connections, some with Ethernet USB ports.  Make sure Designer is closed.  Power up your bench PLC - let everything (USB switches whatever hardware in your office) start up.  Then start up Designer.  Make sure you do NOT see a "Retry" dialog when trying to connect to your bench PLC.

2. Possibly similar to #1?

"Quick fix" - Run Task Manager and kill every instance of DmDesigner.exe and CSMainDm.exe.  CSMainDm.exe is the Comm Server that is "hung" and must be killed.