Host Engineering Forum
General Category => Do-more CPUs and Do-more Designer Software => Topic started by: Bolt on June 10, 2020, 10:41:15 PM
-
I've been sending email with a BRX via Gmail for quite some time now, and using the TLS security for over a year now.
I'm running 2.7.2 on my PLC currently, and have been since shortly after that came out.
My last email successfully sent about 6 hours ago, and have missed several since then. Forcing a stage with email instruction to run ultimately results in System Information -> System Status -> ERR: "Operation timed out in EMAIL @00002BEC" . This takes almost (exactly?) 60 seconds to occur.
Going to Configure -> Device Configuration -> @Email -> Edit Device -> Test settings and verify server credentials results in "Your settings are good!"
My scan time ranges from 3.8 to 4.1 ms Avg, and the Max increased from 7.7 to 8.1 ms during the test period. I do know a few weeks ago I was in the lower 3's for average scan time, but surely that's still a good enough time.
What can I do to further troubleshoot/log what's occurring?
-
I've only used the email instruction in 1 project but i had to use the DNS lookup command before the email instruction to get the ip address and move to the email instruction to get it to work. Could gmail have changed the address and you are using a direct address and that is why? Just a thought not sure if that is it or not.
-
What can I do to further troubleshoot/log what's occurring?
Debug->Do-more Logger, and you want to turn ON ST36 so that the EMail instruction will log its status out the Ethernet port and DmLogger will display it.
-
If you have Designer 2.7, the other thing you could do is float your cursor over the wrench in the top right corner of the EMAIL instruction when online to the PLC. This lists tools specific for THAT instruction. Select the top one for EMAIL which is Enable PLC Message Dump for EMAIL and Launch Do-more Logger.
See the attachment for a screen shot of the tool menu.
-
I only get a response after 1 minute of "Timed Out! Disconnecting SMTP..." "Operation timed out in EMAIL @00002BEC"
I've tried loading a previous version of my program into the PLC, making no difference in the success of the email instruction.
-
I only get a response after 1 minute of "Timed Out! Disconnecting SMTP..." "Operation timed out in EMAIL @00002BEC"
I've tried loading a previous version of my program into the PLC, making no difference in the success of the email instruction.
If you've enabled the dump as Franj suggested, and run DMLogger, you should be getting quite a bit of info in DMLogger that would give us a clue.
-
I did. All I gets returned are those two lines.
-
I did. All I gets returned are those two lines.
Set $SysMsgLevel (DST61) to 2.
-
If you have Designer 2.7, the other thing you could do is float your cursor over the wrench in the top right corner of the EMAIL instruction when online to the PLC. This lists tools specific for THAT instruction. Select the top one for EMAIL which is Enable PLC Message Dump for EMAIL and Launch Do-more Logger.
See the attachment for a screen shot of the tool menu.
Are we setting $SysMsgLevel to 2 as part of this? Won't return much useful info without it.
-
Okay, it was set to 1. Here's the results with 2.
Ignore the 2 lines in the middle, they were triggered by an HTTP request that logs continuously.
-
Are we setting $SysMsgLevel to 2 as part of this? Won't return much useful info without it.
That will be in Designer 2.8, but it is not 2.7. User must set DST61 manually in 2.7 via Data View or Change Value dialog.
-
Are we setting $SysMsgLevel to 2 as part of this? Won't return much useful info without it.
That will be in Designer 2.8, but it is not 2.7. User must set DST61 manually in 2.7 via Data View or Change Value dialog.
I did set DST61 in a Data View to 2 and got the increased info in Logger.
I have since successfully sent an email from the same Gmail account with a new app password in DMSim!
-
Okay, it was set to 1. Here's the results with 2.
Ignore the 2 lines in the middle, they were triggered by an HTTP request that logs continuously.
That trace suggests that the CPU wasn't ever able to establish the TCP connection to the server. Could be a routing/gateway thing. Make sure the gateway is set properly and the router is happy. Could also be that the server just didn't allow the connection, for whatever reason. It would take a Wireshark to know more, but you would need a switch with port mirroring to see more.
-
So, I can get it to work in the Simulator, but not in BRX. Same Gmail app password for each. Both test out fine in device setup. See attached screenshots of each program. Only difference is the BRX has a large program, the Sim only the email logic.
-
So, I can get it to work in the Simulator, but not in BRX. Same Gmail app password for each. Both test out fine in device setup. See attached screenshots of each program. Only difference is the BRX has a large program, the Sim only the email logic.
It's a fundamental network/routing thing. The program shouldn't be a factor.
-
It's a fundamental network/routing thing. The program shouldn't be a factor.
Weird. It worked for so long, and I haven't messed with the router settings at all in 6 weeks.
I'm pretty sure my switch does not support port mirroring, would have to figure out how to do it in my router.
-
Weird. It worked for so long, and I haven't messed with the router settings at all in 6 weeks.
I'm pretty sure my switch does not support port mirroring, would have to figure out how to do it in my router.
I'd cycle the power to any switches and routers. They sometimes get tummy aches.
-
I'd cycle the power to any switches and routers. They sometimes get tummy aches.
No luck there.
Today the PLC started acting even funnier, after hanging up on some Modbus TCP reads it lost after power cycling some of the network's access points.
I finally got to a point in the process where I could preform a physical Term -> Stop, Power Down, wait, Power Up, Stop -> Run -> Term. (Yesterday I only quickly did a Run to Stop transition while re-submitting the @Email Device settings).
Emails started rolling through after that...
Any ideas what could cause this spiral out of control?
-
There are cases where the controller can run out of stack buffers due to high network loads. It tries to clean things up, but it isn't always 100%. If you see $CommStackOverrn (ST146) or $QueuesFlushed (ST145) set, that's what's happened. You can also get clues of ugly network situations from $EthDroppedPkts (DST40) and $EthStoppedIntr (DST41). If you start seeing weirdness, check these.
-
Dang, I wished I would have checked those before power cycling.
-
Dang, I wished I would have checked those before power cycling.
Generally, none of these are issues. When PLCs are connected to more public networks, inevitably weird network traffic can show up. These locations can give insight into what might be happening.
-
... preform a physical Term -> Stop, Power Down, wait, Power Up, Stop -> Run -> Term. ....
I had to do this again today to remedy the SMTP not working
-
... preform a physical Term -> Stop, Power Down, wait, Power Up, Stop -> Run -> Term. ....
I had to do this again today to remedy the SMTP not working
There are cases where the controller can run out of stack buffers due to high network loads. It tries to clean things up, but it isn't always 100%. If you see $CommStackOverrn (ST146) or $QueuesFlushed (ST145) set, that's what's happened. You can also get clues of ugly network situations from $EthDroppedPkts (DST40) and $EthStoppedIntr (DST41). If you start seeing weirdness, check these.
You didn't check these did you?
-
No.. I noticed there was a production window in which I could shutdown the PLC, and I ran with that, before I re-read this thread and saw what I needed to check first. Oops.
-
Happened again this morning. DMLogger shows at last email attempt:
Port 29298(0x7272),10/08/20 10:51:53.377 ,"TCP connection failed or was unexpectedly closed in EMAIL @0000116F"
Port 29298(0x7272),10/08/20 10:51:53.376 ,"SMTP disconnected!"
Port 29298(0x7272),10/08/20 10:51:53.376 ,"Server connection lost. Shutting down..."
Currently, the following bits are:
DST40 $EthDroppedPkts == 35
DST41 $EthStoppedIntr == 0
ST145 $QueuesFlushed == OFF
ST146 $CommStackOverrn == OFF
-
Happened again this morning. DMLogger shows at last email attempt:
Port 29298(0x7272),10/08/20 10:51:53.377 ,"TCP connection failed or was unexpectedly closed in EMAIL @0000116F"
Port 29298(0x7272),10/08/20 10:51:53.376 ,"SMTP disconnected!"
Port 29298(0x7272),10/08/20 10:51:53.376 ,"Server connection lost. Shutting down..."
Currently, the following bits are:
DST40 $EthDroppedPkts == 35
DST41 $EthStoppedIntr == 0
ST145 $QueuesFlushed == OFF
ST146 $CommStackOverrn == OFF
That sounds like the server is dropping the connection, but everything looks fine. Not sure why the server would do that.
-
Well, without further user input subsequent emails started rolling again, I think it only didn't send the one in question at 10:51. Sorry, I didn't specifically try to send a test email before posting.
In such a scenario, do you reckon it activates the EMAIL instruction's On Error JMP to Stage function?
-
That sounds like the server is dropping the connection, but everything looks fine. Not sure why the server would do that.
Could BRX interpret slow/glitchy interwebs connection as a server hangup?
-
In such a scenario, do you reckon it activates the EMAIL instruction's On Error JMP to Stage function?
Yes.
-
Could BRX interpret slow/glitchy interwebs connection as a server hangup?
Maybe, but TCP timeouts are quite long. This sounds more like the server terminated the connection, but it is impossible to know for certain without a Wireshark packet trace.
-
OK, makes sense. What I liked about the theory was that it's something that can break and fix itself without intervention, which this does.
-
OK, makes sense. What I liked about the theory was that it's something that can break and fix itself without intervention, which this does.
Any theory that doesn't involve me fixing a bug is a good one. ;)
-
Any theory that doesn't involve me fixing a bug is a good one. ;)
Got that right! ;D
I guess server misbehavior checks both of those boxes too, though. (Spontaneous, AND you can't fix it!)
-
I don't know if this is related, but over the last week, my Linux Web server has a had a ton of updates being installed along with Watchdog timeouts which might indicate either a hangup or maybe a restart.