STREAMIN without a delimiter or known length

[davidjryan]

I am trying to talk to an external device using TCPIP and the STREAMOUT/STREAMIN instructions. My problem is that the external device (a robot) is not giving a delimiter of any kind at the end of its data stream, so the only way I receive the data from the robot in my String Structure is if I know the exact number of bytes in the expected response.  Everything works great in this situation, I always get "On Success" TRUE and my SL memory is filled with the robot's data. Unfortunately, my problem is that there are a few commands I send the robot (using STREAMOUTs) that can genereate responses of varying lengths (depending on the robot's state) and if I don't give the exact Length to STREAMIN, I either lose data (Length is too small and the returned data is truncated) or I get nothing (Length is too large, STREAMIN Timesout with ("On Error" TRUE)).

So, my questions are:
1) Is it possbile to determine the length of the data in the input buffer before I call STREAMIN so I can then call the exact Length of bytes in STREAMIN?
2) Is it possible to put whatver is in the input buffer during the STREAMIN call into the desired Data Destination after Timeout even if the result of STREAMIN is an Error?

I am guessing #1 would be a new instruction command, while #2 would be a change to STREAMIN's current behavior.

I could write a function to read a single byte at a time and append the byte to a destination string. When I see the first "On Error", I would know that I've emptied the buffer and my response string is complete.

Any ideas or comments on this one?

[plcnut]

You could monitor the length of the input buffer using the .inqueue.
STAGE
Have a stage with buffername.inqueue /=0 JMP next stage.
STAGE
DELTA contact buffername.inqueue INVERT POWERFLOW contact JMP next stage.
STAGE
MOVE buffername.inqueue to variable
STREAMIN using variable as length.

[franji1]

If you KNOW that the robot will ALWAYS stick ENTIRE RESPONSE in ONE TCP PACKET, then you only need to see when $MyTCPStream.InQueue > 0.

This is technically NOT "acceptable" TCP protocol (TCP is a network/framing layer, not app layer), but many "protocols" do it this way.  "Technically", your robot COULD send 1 byte at a time over multiple TCP packets, and Do-more would just accept the "packets", one character at a time.  HOWEVER, this means that the APP layer MUST have a "termination delimiter" such as <CR><LF> (see the typical TCP app layer messages).  However, since whoever developed your robot comm protocol does not understand "framing", they most likely implemented "a TCP packet" EQUALS "a command response frame".  (If you asked THEM if that's what they mean, I can tell you they would say "of course", BUT TECHNICALLY THAT IS BAD FORM, but Do-more can handle it!).  Hence, just seeing the .InQueue go from 0 to > 0 means you "got the whole response packet".

Note: This would NOT work with a serial comm link since there is no "TCP packet" layer in RS-232/422/485 - it's just raw characters.

[plcnut]

So what you're saying is, as soon as you see anything in the .inqueue, you could copy that length and use it for STREAMIN.
Correct?

[davidjryan]

I knew the Forum gods would be kind if I answered a question before asking one!

Thank you gents... I thought there would be a TCP Queue length variable somewhere, I just didn't know where to look for it. My apologies if this was buried in another forum question or blatently in the help/documents somewhere.

The robot manufacturer has realised they are not conforming to the TCPIP standard (after much packet sniffing and proof from my end) and have corrected the problem on their "newer" robots but there are refusing to go back and fix the firmware on their older robots (one of which I am trying to talk to).

All responses are in one packet so InQueue should work.

FYI, my 1-byte-at-a-time stage method worked, so all is good for now, but I will try the InQueue method right away so I only need to call one instance of STREAMIN.
 

[Forumulus]

I knew the Forum gods would be kind if I answered a question before asking one!

Yea, verily thine offering of an answered question has pleased Forumulus, lord of the forums, and my countenance has been made to shine upon thee!

[BobO]

I rarely call STREAMIN before determining that the expected/required character count is already buffered. Delimiter based reads are nice where possible, but very few raw binary protocols use variable lengths with delimiters...so you just get in the habit of waiting for the data to arrive and then call the read with a known length. Of course string centric or terminal type protocols are often exclusively delimiter based, so we support that as well.

[davidjryan]

.InQueue worked liked a charm!

This was my first foray into TCPIP with the DoMore hardware and I must say it's very impressive! Kudos Host Engineering!

Many thanks and praise be upon thee... 

[franji1]

Thank you gents... I thought there would be a TCP Queue length variable somewhere, I just didn't know where to look for it. My apologies if this was buried in another forum question or blatently in the help/documents somewhere.

Don't feel bad - it's not obvious.  For many Do-more device operations, we typically provide TWO primary "state" access mechanisms:
1. As parameter(s) in the specific device instruction
2. As a structure member in the associated structure of the device (not all devices have associated structures, however).

Typically, if there is a structure associated with a device, we create a "secondary parameter" in the instruction so that you can see it below the device.  For example, @MyTCPPort, $MyTCPPort, where @ designates the device name and $ designates the heap-structure-name.  Any corresponding heap-structure-name also shows up in the System Configuration dialog's Device Configuration table (last column, "Heap Item").

The robot manufacturer has realised they are not conforming to the TCPIP standard (after much packet sniffing and proof from my end) and have corrected the problem on their "newer" robots but there are refusing to go back and fix the firmware on their older robots (one of which I am trying to talk to).

This is actually GOOD!  Meaning that you can use the assumption PACKET == RESPONSE.  However, if you ever get their NEW drive, you will need to re-write.

[franji1]

For a complete list of various structures and structure members, bring up the Help system, Search tab, search for DMD0327 (this is the unique topic title key).  Or search for Structure Members.