Host Engineering Forum
General Category => Do-more CPUs and Do-more Designer Software => Topic started by: Dean on September 17, 2015, 12:30:25 PM
-
I have a device outputting a serial ASCII stream, that I have set to send every 10 seconds. I have Do-More 1.4 H2-DM1E receiving on the internal serial port at 19200 via STREAMIN. the ASCII message length is 223 characters. Completion is the character count. I then parse out 5 instrument readings from it. For about 8 weeks it has been performing without a hitch, and now suddenly things have gone pear shaped. The STREAMIN function will randomly fail to get the entire string, and start somewhere in the middle of the string. It will right itself after a while. Its as if the partial strings keep getting processed until the magic number of 223 is hit, then the next transmission starts at the beginning again. Obviously this screws up all the parsing, as that is based on character counts as well using STRSUB. Nothing on the output device or the PLC has changed. Anyway, I would like to see what is going on, but I cant figure out a way to see what is happening with the string, and the STREAMIN instruction. Any ideas?
-
Is there any terminating delimiter on the incoming text, such as a CR/LF? This is the best way to "frame" an ASCII "packet" (vs. just character count).
-
You can also use the 'inQueue.length' for the STREAMIN message length. This way you will be sure to get everything that is there every time.
-
There are a bunch of carriage returns interspersed, but no line feed. There are no unique characters at the end of the string that aren't repeated somewhere else in the string. If you output it to a serial reader program it looks similar to this (mine is a bit different, but the format is the same);
(From the manual)
T01=09/13/22, 08:37:04
D01=A1 3.4685 Mo-cm 1B R= 1000000
D01=B1 21.4632 oC 09 R= 1000000
D01=K1 0.2930 uS/cm 13 R= 1000000
D01=L1 0.1100 PPM 56 R= 1000000
The 01 after the T or D indicates the unit address, mine is the default "00". the bolded characters (my formatting) are what the manufacturer calls
"Exclusive-or checksum of all preceding characters."
They seem to change each transmission.
So as you can see there are a number of carriage returns, one after the date stamp, and then after each instrument channel. What flummoxes me is, why does it run for 2 months straight without a single hiccup, and now suddenly this problem creeps in. As I say, the hardware all looks good, the software hasn't changed on either end of the line. It might run for 20 hours without a problem, or it might happen 5 times a day, and its totally random.
-
You could just use the CR as THE delimiter (instead of using LENGTH) with a timeout of 11 seconds (since you should be getting something every 10 seconds), then parse each line (instead of a bunch of lines). If you got any line that didn't parse perfectly, just throw it out, and continue on and just process the next line.
OR, somehow add some sort of validation on the whole 223 characters, and if it is ever fails, do NOT process any data, but wait 2 seconds, then flush the input queue (you probably got two partial packets in your 223 characters). Use DEVCLEAR to flush the @IntSerial device.
Realize that if you power-up your PLC in the middle of a transfer, you could get partial "packets". Similarly, if you power-down the device or just disconnect it, you could receive a partial packet. In that situation, half the packet is sitting in your serial port buffer, but then when it sends the next packet out, the first half of the new packet will appear as the last half of the old/partial packet. This is why framing (and checksums) are helpful in ASCII protocols.
-
I'm not sure I get what you are suggesting. If you get the chance can you explain in a little more detail?
-
There are a bunch of carriage returns interspersed, but no line feed. There are no unique characters at the end of the string that aren't repeated somewhere else in the string. If you output it to a serial reader program it looks similar to this (mine is a bit different, but the format is the same);
(From the manual)
T01=09/13/22, 08:37:04
D01=A1 3.4685 Mo-cm 1B R= 1000000
D01=B1 21.4632 oC 09 R= 1000000
D01=K1 0.2930 uS/cm 13 R= 1000000
D01=L1 0.1100 PPM 56 R= 1000000
The 01 after the T or D indicates the unit address, mine is the default "00". the bolded characters (my formatting) are what the manufacturer calls
"Exclusive-or checksum of all preceding characters."
They seem to change each transmission.
So as you can see there are a number of carriage returns, one after the date stamp, and then after each instrument channel. What flummoxes me is, why does it run for 2 months straight without a single hiccup, and now suddenly this problem creeps in. As I say, the hardware all looks good, the software hasn't changed on either end of the line. It might run for 20 hours without a problem, or it might happen 5 times a day, and its totally random.
There must be some mechanism for framing...idle time, special characters, data format, etc. If you are framing strictly on a number of characters, you will lose frame as soon as a bad character shows up, assuming you ever got framed to start with. The best way to do this is to have a framing step that determines the start of frame and flushes the receive buffer, then starts the receive sequence. If at any time the receive packet breaks, you return to the framing step to reframe.
Modbus, for instance, uses interpacket silence to frame. We sit in the framing state tossing characters until we've had at least 1 scan of silence, then jump to the receive sequence.
It's much easier with special characters, although silence isn't too bad. Finding data format in an unknown sequence can be painful, but it can be done.
My guess is that you dropped a character and lost frame.
-
Looking at your data now, it appears you will have to frame on the 'T', and there might be some idle time between packets, so you might be able to treat silence as the end-of-frame.
-
OK. Please teach me about "framing". How do I tell the PLC to look for that beginning T, and/or listen for silence?
-
Here is one way to do it. I did not overdo the documentation, so holler back with questions. I also would not recommend doing this in MAIN, but was just trying to give you an Idea of how you could do it. This is also un-tested...
-
plcnut,
Thank you for taking the time to break this down. After a brief read through this morning I think I understand the general concept. I will study this over the weekend, and attempt to create a sandbox to test the idea on my application. My application runs in a program with stages as well, but its more passive, and of course uses the character count for success. As everyone has advised, good framing will make things more robust, and I need to implement it. I did find a small hardware problem though. The 24VDC ground wire from the PLC to the power supply, while it wasn't disconnected, it was a bit loose on the PS end( it came right out when I pulled very gently). I wonder if losing that ground reference was part of why I started losing transmissions?
-
I wonder if losing that ground reference was part of why I started losing transmissions?
Yes, that could certainly cause the issues you are describing.
-
As an update, the fixing the loose ground wire appeared to have no effect, as the phenomenon continued. Weirdly, the problem just went away on its own, and I've gotten 4 days straight of good data. This just confirms what everyone has said, I need error checking. I've spent a few hours trying to break down plcnut's solution, and I'm afraid its pretty well over head, so I figured I would try to build a very simple first step. This is what I tried.
1. IF the InQueue is not equal to zero THEN STREAMIN
2. STFIND the letter "T"
3. IF "T" is at index 0 THEN do a little parsing
4. IF "T" is not at index zero THEN DEVCLEAR AND JMP to Step 1.
My problem as always is dealing with the STREAMIN instruction. I can't seem to find a way to get On Success. I've tried various network timeout values, but that never triggers. plcnut had 0x0A OR 100ms. I don't think there are any line feeds in the string, so that doesn't trigger it, and I don't no why I never get a network timeout. I opened a data view for the string used in my project, and viewed it as hex. I exported it and copied into word. Using a HEX to ASCII converter here, Hex to ASCII (http://www.rapidtables.com/convert/number/hex-to-ascii.htm) I was able to see that there is a CR after the date/time stamp, and after each sensor value EXCEPT the last one. I have attached an excerpt from the meter service manual, and as you can see, each sensor value line should be followed by a space and a CR. I wonder why that last one is getting cut off? For anyone that's interested, I have also attached my string in a Word doc in ASCII and HEX. Note that DMD Data View converts the CR to a period (.), and there is no period at the end of the ASCII string either. Short story long, any ideas how I terminate this STREAMIN instruction?
-
CR is 0x0D. Change the termination character from 0x0A (LF) to 0x0D (CR)
-
If the incoming byte length is known, I wait until the expected characters have arrived before invoking STREAMIN, and then reading just what I need. STREAMIN immediately returns success in that case. To do that, after framing, just wait until .InQueue is greater than or equal to what you are waiting for. While waiting, use a timer as a timeout. If the timeout fires prior to receiving the expected length, fail the read and go back to framing.
If you don't know the length, then a delimiter is required.
Please understand...it is possible for this to work with far less effort, but what we are describing is how to make it robust.
-
Please understand...it is possible for this to work with far less effort, but what we are describing is how to make it robust.
100% agree. This is the lesson I have taken from this situation. This segment needs error proofing. I guess I just need to start from scratch, and rebuild this segment. A couple questions if I may:
Lets say we have a string like this
abc123CR123abcCR
I Streamin when InQ >0 and with 1 delimiter 0xOD into SS0
On Success I run another STREAMIN 1 Delimiter 0x0D into SS1
I should get SS0 = abc123 and SS1 = 123abc Right?
According to the help files, once the first chunk is moved to the destination, then whatever is left in the buffer gets shifted to the beginning, so I think I'm interpreting this properly.
If so, I can separate each chunk into it's own string, and check to see if it starts with the correct character. If so, move on to parsing. Once parsing is done, clear the device, and go back to step 1. If not, clear all the strings, clear the device, and go back to step 1. Does this make sense as a strategy?
-
abc123CR123abcCR
I Streamin when InQ >0 and with 1 delimiter 0xOD into SS0
On Success I run another STREAMIN 1 Delimiter 0x0D into SS1
I should get SS0 = abc123 and SS1 = 123abc Right?
Yes.
I Streamin when InQ >0 and with 1 delimiter 0xOD into SS0
According to the help files, once the first chunk is moved to the destination, then whatever is left in the buffer gets shifted to the beginning, so I think I'm interpreting this properly.
A STREAM is a buffered sequence of characters. As you call STREAMIN, you are removing characters from that sequence. STREAMIN has up to three ways it can terminate: by receiving a count, by encountering a terminator, and by timing out. .InQueue tells you the total number of characters currently sitting in the queue, making it possible to build the basic protocol flow outside of the STREAMIN, and just using the STREAMIN as the way to access the data. If you know the length expected, you can spin on .InQueue until the expected number has arrived, then call STREAMIN. In your case, waiting until .InQueue is non-zero, then calling STREAMIN with the terminator works fine too. I like the length approach when possible, but the terminator approach is good too.
A PACKET is a chunk of data. Once read, the entire packet is removed, even if you only asked for the first byte.
If so, I can separate each chunk into it's own string, and check to see if it starts with the correct character. If so, move on to parsing. Once parsing is done, clear the device, and go back to step 1. If not, clear all the strings, clear the device, and go back to step 1. Does this make sense as a strategy?
You are tokenizing as you go. That should work fine.
-
Thanks to everyone for your guidance. I believe I'm on the right track here. I've built a quick sandbox application, and it appears to be working as described. If I get a partial transmission, or lose a character or two, one or more of the sub strings gets whacked out enough to fail the integrity testing. This causes the program to pitch the entire transmission, clear the buffer and start over. That said, I think my final application will only throw out whichever sub string(s) fail integrity testing and keep those that pass. This will help diagnose when problems come up. Again, I really appreciate the help. This forum is an invaluable resource.
-
Thanks to everyone for your guidance. I believe I'm on the right track here. I've built a quick sandbox application, and it appears to be working as described. If I get a partial transmission, or lose a character or two, one or more of the sub strings gets whacked out enough to fail the integrity testing. This causes the program to pitch the entire transmission, clear the buffer and start over. That said, I think my final application will only throw out whichever sub string(s) fail integrity testing and keep those that pass. This will help diagnose when problems come up. Again, I really appreciate the help. This forum is an invaluable resource.
Perfect. From not understanding the concept of 'framing' to a protocol expert in just a few days! That's awesome.
I've written comm code since shortly after the earth cooled and we designed Do-more comms with that experience in mind. We often joke about the fact that it's easier to develop a protocol in this PLC than in virtually any environment I've ever worked with. It's not optimal for some things, but the basics of getting data in and out of a serial or Ethernet port are far, far easier than doing so using C++ in a PC. It really makes me happy to see users get good results!
-
From not understanding the concept of 'framing' to a protocol expert in just a few days!
If only that were true, but with everyone's help, I have put a few more wrenches in the toolbox.
-
From not understanding the concept of 'framing' to a protocol expert in just a few days!
If only that were true, but with everyone's help, I have put a few more wrenches in the toolbox.
Don't sell yourself short. You came up to speed pretty quick.
With what you understand now, consider downloading the DirectNet server sample app. If you can follow what I've done there, you've added some nice tools to the toolbox.
-
Bob,
I spent a few minutes scanning through your example, and it mostly reminded me of how little I actually know about this stuff. Yikes! I have a couple questions if I may for anyone that may have a minute to look at what I'm doing. I have attached my most recent sandbox attempt. My question is you will notice that Stage 6 has a In Success Jump instruction to Stage 0. It never seems to make the jump to Stage 0. I know that the conditions are met on the STREAMIN, as there is a carriage return. Here is a typical example of SS55 as Quoted ASCII "D00=E4 0.0000 ppmO3 07 R= 25000 $0D"
. You'll notice the escaped CR, so it should trigger the On Success bit. I have also attached a screen shot of a trend of all 6 stages. You will notice that Stage 0 never fires. What am I not understanding here? I mean it works, but shouldn't I see Stage 0 actually execute?
-
It may only be in stage 0 for a fraction of a scan...and comms are performed at the bottom of the scan. This in addition to the fact that trends are polled...not guaranteed to get ever scan.
Put an INC instruction in each stage, conditioned with a rising edge contact.
-
Well that's an interesting tool. Thank you. I'll definitely remember this trick.
-
Well that's an interesting tool.
Edge triggered INCs work GREAT for seeing some kind of "heart beat" for code that may be executing just 1 scan (or even sub-scan!).