How to trend Serial Streamin Instruction

[Dean]

I have a device outputting a serial ASCII stream, that I have set to send every 10 seconds. I have Do-More 1.4 H2-DM1E receiving on the internal serial port at 19200 via STREAMIN. the ASCII message length is 223 characters. Completion is the character count. I then parse out 5 instrument readings from it. For about 8 weeks it has been performing without a hitch, and now suddenly things have gone pear shaped. The STREAMIN function will randomly fail to get the entire string, and start somewhere in the middle of the string. It will right itself after a while. Its as if the partial strings keep getting processed until the magic number of 223 is hit, then the next transmission starts at the beginning again.  Obviously this screws up all the parsing, as that is based on character counts as well using STRSUB. Nothing on the output device or the PLC has changed. Anyway, I would like to see what is going on, but I cant figure out a way to see what is happening with the string, and the STREAMIN instruction. Any ideas?

[franji1]

Is there any terminating delimiter on the incoming text, such as a CR/LF?  This is the best way to "frame" an ASCII "packet" (vs. just character count).

[plcnut]

You can also use the 'inQueue.length' for the STREAMIN message length. This way you will be sure to get everything that is there every time.

[Dean]

There are a bunch of carriage returns interspersed, but no line feed. There are no unique characters at the end of the string that aren't repeated somewhere else in the string. If you output it to a serial reader program it looks similar to  this (mine is a bit different, but the format is the same);
(From the manual)
T01=09/13/22, 08:37:04
D01=A1 3.4685 Mo-cm 1B R= 1000000
D01=B1 21.4632 oC 09 R= 1000000
D01=K1 0.2930 uS/cm 13 R= 1000000
D01=L1 0.1100 PPM 56 R= 1000000
The 01 after the T or D indicates the unit address, mine is the default "00". the bolded characters (my formatting) are what the manufacturer calls

"Exclusive-or checksum of all preceding characters."

They seem to change each transmission.
So as you can see there are a number of carriage returns, one after the date stamp, and then after each instrument channel. What flummoxes me is, why does it run for 2 months straight without a single hiccup, and now suddenly this problem creeps in. As I say, the hardware all looks good, the software hasn't changed on either end of the line. It might run for 20 hours without a problem, or it might happen 5 times a day, and its totally random.

[franji1]

You could just use the CR as THE delimiter (instead of using LENGTH) with a timeout of 11 seconds (since you should be getting something every 10 seconds), then parse each line (instead of a bunch of lines).  If you got any line that didn't parse perfectly, just throw it out, and continue on and just process the next line.

OR, somehow add some sort of validation on the whole 223 characters, and if it is ever fails, do NOT process any data, but wait 2 seconds, then flush the input queue (you probably got two partial packets in your 223 characters).  Use DEVCLEAR to flush the @IntSerial device.

Realize that if you power-up your PLC in the middle of a transfer, you could get partial "packets".  Similarly, if you power-down the device or just disconnect it, you could receive a partial packet.  In that situation, half the packet is sitting in your serial port buffer, but then when it sends the next packet out, the first half of the new packet will appear as the last half of the old/partial packet.  This is why framing (and checksums) are helpful in ASCII protocols.

[Dean]

I'm not sure I get what you are suggesting. If you get the chance can you explain in a little more detail?

[BobO]

There are a bunch of carriage returns interspersed, but no line feed. There are no unique characters at the end of the string that aren't repeated somewhere else in the string. If you output it to a serial reader program it looks similar to  this (mine is a bit different, but the format is the same);
(From the manual)
T01=09/13/22, 08:37:04
D01=A1 3.4685 Mo-cm 1B R= 1000000
D01=B1 21.4632 oC 09 R= 1000000
D01=K1 0.2930 uS/cm 13 R= 1000000
D01=L1 0.1100 PPM 56 R= 1000000
The 01 after the T or D indicates the unit address, mine is the default "00". the bolded characters (my formatting) are what the manufacturer calls

"Exclusive-or checksum of all preceding characters."

They seem to change each transmission.
So as you can see there are a number of carriage returns, one after the date stamp, and then after each instrument channel. What flummoxes me is, why does it run for 2 months straight without a single hiccup, and now suddenly this problem creeps in. As I say, the hardware all looks good, the software hasn't changed on either end of the line. It might run for 20 hours without a problem, or it might happen 5 times a day, and its totally random.

There must be some mechanism for framing...idle time, special characters, data format, etc. If you are framing strictly on a number of characters, you will lose frame as soon as a bad character shows up, assuming you ever got framed to start with. The best way to do this is to have a framing step that determines the start of frame and flushes the receive buffer, then starts the receive sequence. If at any time the receive packet breaks, you return to the framing step to reframe.

Modbus, for instance, uses interpacket silence to frame. We sit in the framing state tossing characters until we've had at least 1 scan of silence, then jump to the receive sequence.

It's much easier with special characters, although silence isn't too bad. Finding data format in an unknown sequence can be painful, but it can be done.

My guess is that you dropped a character and lost frame.

[BobO]

Looking at your data now, it appears you will have to frame on the 'T', and there might be some idle time between packets, so you might be able to treat silence as the end-of-frame.

[Dean]

OK. Please teach me about "framing". How do I tell the PLC to look for that beginning T, and/or listen for silence?

[plcnut]

Here is one way to do it. I did not overdo the documentation, so holler back with questions. I also would not recommend doing this in MAIN, but was just trying to give you an Idea of how you could do it. This is also un-tested...

[Dean]

plcnut,

Thank you for taking the time to break this down. After a brief read through this morning I think I understand the general concept. I will study this over the weekend, and attempt to create a sandbox to test the idea on my application. My application runs in a program with stages as well, but its more passive, and of course uses the character count for success. As everyone has advised, good framing will make things more robust, and I need to implement it. I did find a small hardware problem though. The 24VDC ground wire from the PLC to the power supply, while it wasn't disconnected, it was a bit loose on the PS end( it came right out when I pulled very gently). I wonder if losing that ground reference was part of why I started losing transmissions?

[plcnut]

I wonder if losing that ground reference was part of why I started losing transmissions?

Yes, that could certainly cause the issues you are describing.

[Dean]

As an update, the fixing the loose ground wire appeared to have no effect, as the phenomenon continued. Weirdly, the problem just went away on its own, and I've gotten 4 days straight of good data. This just confirms what everyone has said, I need error checking. I've spent a few hours trying to break down plcnut's solution, and I'm afraid its pretty well over head, so I figured I would try to build a very simple first step. This is what I tried.
1. IF the InQueue is not equal to zero THEN STREAMIN
2. STFIND the letter "T"
3. IF "T" is at index 0 THEN do a little parsing
4. IF "T" is not at index zero THEN DEVCLEAR AND JMP to Step 1.
My problem as always is dealing with the STREAMIN instruction. I can't seem to find a way to get On Success. I've tried various network timeout values, but that never triggers. plcnut had 0x0A OR 100ms. I don't think there are any line feeds in the string, so that doesn't trigger it, and I don't no why I never get a network timeout. I opened a data view for the string used in my project, and viewed it as hex. I exported it and copied into word. Using a HEX to ASCII converter here, Hex to ASCII I was able to see that there is a CR after the date/time stamp, and after each sensor value EXCEPT the last one. I have attached an excerpt from the meter service manual, and as you can see, each sensor value line should be followed by a space and a CR. I wonder why that last one is getting cut off? For anyone that's interested, I have also attached my string in a Word doc in ASCII and HEX. Note that DMD Data View converts the CR to a period (.), and there is no period at the end of the ASCII string either. Short story long, any ideas how I terminate this STREAMIN instruction?

[franji1]

CR is 0x0D.  Change the termination character from 0x0A (LF) to 0x0D (CR)

[BobO]

If the incoming byte length is known, I wait until the expected characters have arrived before invoking STREAMIN, and then reading just what I need. STREAMIN immediately returns success in that case. To do that, after framing, just wait until .InQueue is greater than or equal to what you are waiting for. While waiting, use a timer as a timeout. If the timeout fires prior to receiving the expected length, fail the read and go back to framing.

If you don't know the length, then a delimiter is required.

Please understand...it is possible for this to work with far less effort, but what we are describing is how to make it robust.