Host Engineering Forum

General Category => Do-more CPUs and Do-more Designer Software => Topic started by: plcnut on October 02, 2012, 07:50:18 PM

Title: SS memory and pointers
Post by: plcnut on October 02, 2012, 07:50:18 PM
Maybe I'm going about this the wrong way, but I am trying to use the STRSUB instruction in a for loop to extract individual pieces of data (6 digit alphanumeric part numbers)  from a rather large STREAMIN (up to 550 bytes), and place the part numbers into individual strings. (This is part of a system that will sort products with barcodes on a conveyor, and divert them to a ten station packing section.) My problem is that I wanted to use a pointer (SS(v0)) in the output string of the STRSUB, but Do-more Designer won't let me. What am I doing wrong? Thank you in advance for the help! BTW the software is great so far :)
Title: Re: SS memory and pointers
Post by: plcnut on October 02, 2012, 08:22:05 PM
 :-[ I guess I should read the instructions a little closer... the pointer must be in BRACKETS not parenthesis.
Thanks guys.  ;D
Title: Re: SS memory and pointers
Post by: franji1 on October 02, 2012, 11:36:07 PM
I wrote a tokenization function that took a while to get working right.  It assumes the tokens are separated by white space, then sticks them in a new user data-block of STRINGs I called Token, so Token0 is the first token, Token1 is the second token, up to 10 tokens (Token9). It uses a repeat/until (maybe while?).  It doesn't convert any of the tokens to integers, but it wouldn't be that difficult to take the output of this "Tokenize" TASK to do that step.  Is this something that would be helpful, or do you have it working?
Title: Re: SS memory and pointers
Post by: plcnut on October 03, 2012, 07:53:27 AM
If you don't mind sharing, I would really like to have a look.
I think I have mine like I want it for receiving the data and entering it into tables, and I'm happy so far with how the program is organized, but I have some comparing to do that I haven't figured out yet. I will be receiving up to 10 orders of varying sizes at a time, each order is entered into a string block, I have to be able to make 4 separate lists from these orders, based on the picking sections, and organized by part numbers. If there are multiples of the same part numbers, I need to either put them together on the list, or add a quantity column to my table.
Title: Re: SS memory and pointers
Post by: BobO on October 03, 2012, 09:41:38 AM
As you finish this up, I would love to hear your experience with Do-more, with particular emphasis on how well the instruction set did the job and how well you could manage complexity. We still have very little feedback from actual applications.
Title: Re: SS memory and pointers
Post by: plcnut on October 03, 2012, 10:09:49 AM
This is a first for me to deal with large amounts of data coming in, and so far it has been really smooth and organized with the program. Way easier than DL!
Title: Re: SS memory and pointers
Post by: franji1 on October 03, 2012, 11:03:06 AM
I just copied the code-block (right click on Tokenize task in my Project Browser to bring up context menu, selected Copy Code Block), then pasted it below.  I would definitely recommend IMPORTING this into a NEW PROJECT (File->Import->Project) and then tweaking it, then copy/pasting it into YOUR project.

For example, I use SS1 as the "Line" (the input to the Tokenize function).  You may want to use a different source STRING element.  It's a little heavy on documentation, but that's a good thing.  ;D  This includes symbolic constants (e.g. NEVER_YIELD is the decimal constant 65,535) They're still a constant, but have a human readable name).  There's also a nickname for an array index (V100).

Also, my Tokens block of STRINGs is STRING32, meaning they are up to 32 characters long, so no single token can be longer than 32 characters.  You may want to tweak that.

So, before you do ENTASK Tokenize from the "calling" code-block with an edge-triggered, just stick the "line" into SS1.  Then look at Tokenize.Done (it probably will take multiple scans) and look at V100 to see how many tokens there are in the Tokens STRING data-block, then look at Tokens0 thru Tokens[V100-1] (although you can't do negative offsets, just positive offsets, e.g. R[V10+6]), so you'll need to "calculate" the last index (V100-1) and stick that in another V.

Let me know if you have any problems importing it or if you have any questions on its behavior.


Code: [Select]
// Do-more Designer Clip   Source Project: C:\Do-more Designer\Projects\~DmD11    Format: Do-more Designer 2.00+  DO-MORE Ladder
PLC H2-DM1E

$TSK Tokenize

#BEGIN COMMENT
"Support AT RUNTIME an unknown maximum number of tokens (see Tokens STRING data block)."
""
"Copy command line string to a ""working"" RemainderOfLine string which we will extract tokens from."
""
"Initialize our loop index (index to Tokens array)"
""
"Initialize our LoopToggle bit to ON (necessary in order to process EDGE-TRIGGED instructions from INSIDE a LOOP within the SAME LADDER SCAN)."
""
"Clear the White Space Found/Not Found flags."
#END

STRCLEAR Tokens0 D99
MEMCOPY SS1 SS99 0x20000 1
MOVE 0 V100
SET C1
RSTR C98 C99

#BEGIN COMMENT
"Just in case somebody has messed with this task's .TimeSlice member value and made me  NEVER yield, let me yield every 100 micro seconds during this ""time consuming"" string manipulation/tokenization code."
#END

STRE Tokenize.TimeSlice NEVER_YIELD
MOVE 100 Tokenize.TimeSlice

#BEGIN COMMENT
"This looks like an infinite loop, but the loop termination logic is INSIDE the loop (look for BREAK statement)"
#END

WHILE ST1

#BEGIN COMMENT
"If there's no more tokens to process, we're done."
"Or if the number of tokens in this line exceeds our capacity, we're done too."
#END

STRE SS99.Length 0
ORGE V100 D99
BREAK

#BEGIN COMMENT
"Because we're dealing with edge-triggered instructions inside a loop (this is the cause of all those PROGRAM CHECK WARNINGS!), just do this work ""every other time"" through the loop.  The LoopToggle bit makes this a little easier."
#END

STR C1
MOVE 0 D97
STRFIND SS99 0x2 D97 C99 C98 """ $09$0D$0A"""

#BEGIN COMMENT
"If we found whitespace, extract everything up to (but not including) the space and put it in the current Token."
""
"Since we're only executing REAL logic ""every other time"" through this loop because of the edge-triggered instructions, actively CLEAR the White Space Found flag."
""
"Now remove the extracted token from the line.  We also don't want the found White Space to be included, so increment the White Space Index value to move PAST the white space character, then use that as the ""starting point"" of the ""remainder of the line""."
""
"Conditionally, we want to ONLY use tokens that are NOT empty (e.g. a line with 5 spaces together, we need to ""eat"" those).  So, if the extacted token is empty, do NOT bother incremementing our ""current token"" index.  Code-wise, that means if we DO have a token string (i.e. its length is > 0), go ahead and increment the current index of our Tokens array."
#END

STR C99
STRSUB SS99 0 0x0 D97 Tokens[V100]
RST C99
INC D97
STRSUB SS99 D97 0x0 -1 SS99
ANDGT Tokens[V100].Length 0
INC V100

#BEGIN COMMENT
"If White Space was NOT found, that means that we're on the last token, all the way to the end of the line.  So just put the Remainder of the Line into the current Token."
""
"One of our termination conditions is that there's no more work to do (Remainder of Line is empty), since we know we just processed our last token, just clear the Remainder of Line string!"
""
"For completeness, reset the White Space NOT Found flag."
""
"Even though we found no white space, we don't want to count empty tokens in our token count, so again, for completeness-sake, increment the token count if there is something in the current token."
#END

STR C98
STRPRINT Tokens[V100] 0x0 "SS99"
STRCLEAR SS99 1
RST C98
ANDGT Tokens[V100].Length 0
INC V100

#BEGIN COMMENT
"Toggle our Loop Toggle bit!"
#END

STRN C1
OUT C1

WEND

#BEGIN COMMENT
"Edge Triggered TASKs are like subroutines, once the program counter reaches the bottom, the task is done!  No EXIT to call.  Even this END statement is optional (if END does not exist, the end is the last non-NOP).  But the END coil can be used to divide logic between ""used vs. unused code"".  Here, I needed it so I could have this rung comment (you should not put rung comments on trailing NOP rungs since they technically do not exist)."
#END

END

$TSKEND Tokenize

#BEGIN MEM_CONFIG
 Tokens STRING32 decimal 10 -1
 Tokenize TASK  0
#END

#BEGIN ELEMENT_DOC
"C98","WSNotFound","",""
"SS99","RemainderOfLine","",""
"C99","WSFound","",""
"D97","WSIndex","",""
"V100","TokenizeIndex0","",""
"SS1","Line","",""
"D99","MaxNumTokens","",""
"C1","LoopToggle","",""
"SK","NEVER_YIELD","65535",""
#END

Title: Re: SS memory and pointers
Post by: plcnut on October 04, 2012, 12:54:06 PM
Thank you franji1,
I look forward to importing your code later today to have a look.
Title: Re: SS memory and pointers
Post by: plcnut on October 06, 2012, 01:24:31 PM
I like your tokenize task franji1, it is similar to what I was doing, except I was extracting the data and updating a pointer to move further into the string, versus cutting the data out of the end like yours, Thank you :)
 I ran into a problem with my application though... I don't have experience with importing data from a pc over rs232, but some of the orders I will be receiving may total up to 2800 individual memory locations, (1400 6 digit Alphanumeric product numbers, 1400 4 digit integer product quantities). I'm still waiting on the IT dept. for an exported .csv file from the system to see what it will actually look like.
My problem is, how do I get that much data into my system, and am I crazy to be trying? (when they asked me to start this project, I understood it to be MUCH smaller orders ie 15 items per order average)
Title: Re: SS memory and pointers
Post by: franji1 on October 06, 2012, 02:49:58 PM
Is your question about how to write a program on the PC to send data to a PLC, or is it about how to store the records in the PLC?

I can answer the second question very easily - no problem.  I just allocated a new STRING data block 1400 in length called ProdName, each STRING max 8 characters long (it has to align on a DWORD boundary, since you need 6 chars, it had to be 8 ).  Then allocated a new UNSIGNED WORD data block 1400 in length called ProdQuant (range 0-65535).  You may want to bump that up to 1500, just in case.

Anyway, the unused memory from a default H2-DM1E (DM1 would be the same) went down from 159,072 to 139,472 bytes (so this used up a measely ~30K ~20K).

So, the index in one data-block corresponds to the matching item in the other, e.g. you found a matching part number ABC123 at index 599 in ProdName (i.e. ProdName599 == "ABC123"), then ProdQuant599 is 45, meaning ABC123's quantity is 45.

The first question, maybe write a VB application that dumps a "line" from the CSV in the form of "ABC123,45<CR><LF>" out the PC's serial port into a General Purpose serial port (internal or H2-SERIO port), so you can easily "add" the rows to your data blocks in the Do-more PLC (or lookup in ProdName then add that quantity to existing quantity in ProdQuant at the same index???).
Title: Re: SS memory and pointers
Post by: BobO on October 06, 2012, 02:53:49 PM
I was just doing the same math...you'll have plenty of room for the buffers. At 115200, it should only take a couple of seconds to transfer the data, and if it could be Ethernet instead of serial, much quicker than that. I honestly think you can do this no problem, and actually, our tasking makes it very easy to spread the work out over many scans, thus reducing the hit to the scantime.

Title: Re: SS memory and pointers
Post by: plcnut on October 06, 2012, 05:09:46 PM
Thanks guys.

I was trying to set up the serial port buffer and it had a maximum of 1024 bytes(if I am remembering correctly from yesterday), at a large order of 1400 part numbers, that would require a larger buffer than I can set up. Am I misunderstanding the buffer? Or can I set up a data buffer to use a Block versus a String? Or maybe I am misunderstanding the amount of ASCII data in a byte, a byte will only hold 2 ASCII charachters right?

Bobo, I would be happy to use the Ethernet port for the data, it would be easier than the serial from a hardware standpoint I'll just have to study into it...

Thank you for your time.
Title: Re: SS memory and pointers
Post by: BobO on October 06, 2012, 05:24:31 PM
The buffer is just the number of bytes that can be read at once...in one call to a STREAMIN or PACKETIN instruction. You would not blindly transfer tens of thousands of bytes without some form of packetizing or error checking, so I would expect that you would be reading a line at a time, or some other quantum of data. The 1K limit per read is quite generous in that context.

One ASCII character is one byte.

From the Do-more programming perspective, Ethernet and serial are about the same level of effort. From the PC app (or whatever) that is sending the data, Ethernet and serial are very different. Assuming that you don't need the full bunch of data very often, serial is more than adequate.

It sounds like you are wandering over into a custom protocol. If you are not familiar with developing such, you might want to seek the help of someone who is. While not 'hard' per se, the learning curve can be pretty frustrating.

It would be pretty straightforward to write a PC based TCP server to provide the data, and use a TCP Client device in Do-more to connect and download the information.
Title: Re: SS memory and pointers
Post by: plcnut on October 11, 2012, 08:04:02 PM
BobO,

I just spoke with the IT guy.
He says if I can do an HTTP request, that would be the best way for him to provide the data. Serial may be problematic. So it sounds like the custom protocol may be the way to go.

I think we are looking at something like this:

OPENTCP, (ip address, port no.)

STREAMOUT, (get/put, etc. in text form from a string structure)

STREAMIN, (place received text into string structure)

CLOSE, (close tcp connection)

Process received data...

Am I getting close?
I hope that he can get a dummy server set up that I can practice sending some requests and see how it will actually work.
Thanks again for your help guys!!!
Title: Re: SS memory and pointers
Post by: BobO on October 11, 2012, 08:57:44 PM
You will likely have many STREAMOUT/STREAMIN pairs in a protocol, but yes, those are the correct functions in the correct order.

Some additional thoughts:
1. It is not necessary to use string structures. If your data is primarily binary, you can allocate a byte buffer and access the individual elements directly.
2. Build your client protocol in a dedicated program block and use stage programming. Trust me on this, even if you are not normally a stage guy.
3. To simplify debug, push input and output strings out to DM Logger using STREAMOUT and the DMLogger device. It makes a very convenient protocol analyzer.
4. And have fun. Writing protocols can be frustrating, just relax and don't be in too much of a hurry.
Title: Re: SS memory and pointers
Post by: plcnut on October 13, 2012, 11:25:53 AM
Thanks, my new H2-DM1E will be here on tuesday, and the IT dept. is supposed to have the server set up where I can access the information I need. They are going to limit the size of data transmissions from their end so I won't have to worry about any of that. My IT tech seems to think this will be pretty easy... Well see next week.

In regards to your suggestion to seek the help of someone familiar with these things, I don't have anybody to ask about this stuff aside from web forums, I do know an AB guy, but that isn't much help with these problems(his suggestion is: "Get a ControlLogix") :o

So once again, Thank you for the help!
Title: Re: SS memory and pointers
Post by: BobO on October 13, 2012, 12:22:34 PM
In regards to your suggestion to seek the help of someone familiar with these things, I don't have anybody to ask about this stuff aside from web forums, I do know an AB guy, but that isn't much help with these problems(his suggestion is: "Get a ControlLogix") :o

We're happy to help you, and we'll be here as long as you need us. My point was driven by a sense that you were coloring outside the lines, and if this is a 'getting paid to do something' scenario, you might get bogged in the details and find that you aren't getting paid enough for the time you end up investing. I don't think the app is particularly difficult with Do-more...of course I am a bit biased though...and I've been writing comm code for about 25 years. ;)

As for ControlLogix, I'm not familiar with AB stuff, do they have a good platform for custom protocol development? I'm certain Do-more does, and I'm certain you can make this work.
Title: Re: SS memory and pointers
Post by: plcnut on October 13, 2012, 03:06:27 PM
"coloring outside the lines"!!!!! More like scribbling  :D I've used the basic CoPro for ASCII communications for a scale, printer, and barcode reader, but this is a whole new level, so scribbling is probably an understatement. This is a "paid by the hour" situation, at a rate that takes into account a good bit of learning. I do have a deadline for the easy part of the system (2-1/2 months), but the I can push the database part out a little longer if I have too. Thanks again!
Title: Re: SS memory and pointers
Post by: BobO on October 13, 2012, 03:35:13 PM
By the hour? Well that's a different story! ;)

I'm working on sample code to read a text file from an HTTP server into a large string. I think it is a good sample and others can benefit from it too. Have the server open and talking to me, now trying to read the file. Once I have it all clean and documented, I'll post it.
Title: Re: SS memory and pointers
Post by: BobO on October 13, 2012, 04:35:38 PM
In progress...
Title: Re: SS memory and pointers
Post by: b_carlton on October 13, 2012, 04:59:01 PM
I think this will be a massive help. There had been mentions of the 'capability' of building a custom protocol handler but to see a specific one in actual development will help many (read that as 'me'!).
Title: Re: SS memory and pointers
Post by: BobO on October 13, 2012, 08:10:17 PM
After studying the basic HTTP GET command, I think it would be useful to add an instruction for it, and possibly POST as well. It is very manageable as a custom protocol, but I suspect using an HTTP server as a simple way of providing production content or recipes or whatever to the PLC would be very popular. What I learn from rolling the protocol in Do-more will make it pretty simple to do the instruction. Not sure when, but maybe DmD 1.1.

Thinking I might like to add a DNS lookup instruction as well, although everything I've read tells me to run away. We'll see. I think it would be very handy.
Title: Re: SS memory and pointers
Post by: Controls Guy on October 13, 2012, 09:27:08 PM
As for ControlLogix, I'm not familiar with AB stuff, do they have a good platform for custom protocol development?

No.  It's a good PLC, even very good, but it ain't a protocol engine.  Even if AB has some prebuilt AOI's (Add-On-Instructions), I bet they're protocols, not protocol tools.

Among the majors, Omron is probably the one who's invested the most effort in protocol creation tools (even have a standalone product in their suite called CX-Protocol), but I think DM will be at least as good if not better.
Title: Re: SS memory and pointers
Post by: Controls Guy on October 13, 2012, 09:34:23 PM
After studying the basic HTTP GET command, I think it would be useful to add an instruction for it, and possibly POST as well. It is very manageable as a custom protocol, but I suspect using an HTTP server as a simple way of providing production content or recipes or whatever to the PLC would be very popular. What I learn from rolling the protocol in Do-more will make it pretty simple to do the instruction. Not sure when, but maybe DmD 1.1.

Thinking I might like to add a DNS lookup instruction as well, although everything I've read tells me to run away. We'll see. I think it would be very handy.

HTTP.Get and .Post would be very handy.  I'm doing a Get from the MMI on a project I'm doing now, and could easily see the PLC wanting to do one.

I think the lack of being able to talk to DNS servers shouldn't be a big deal.  If you really needed it, you could have some PC on the same network as the PLC resolve the IP address from the URL once a day or something and send the IP address to the PLC via Modbus or something.  They're not constantly changing or anything are they?
Title: Re: SS memory and pointers
Post by: ATU on October 13, 2012, 10:37:51 PM
If you provided Post and Get, you could write a simple script (like in PHP) to access a database.
Would be simple and royalty free.
Title: Re: SS memory and pointers
Post by: deep6ixed on October 15, 2012, 02:08:38 AM
If you provided Post and Get, you could write a simple script (like in PHP) to access a database.
Would be simple and royalty free.

As a former webmaster who is familiar with PHP, I'd be more than happy to help with that side of things to make a PHP lib for MySQL access...
Title: Re: SS memory and pointers
Post by: BobO on October 15, 2012, 02:26:55 PM
Here is the first shot.

Put desired file in RequestFilename (SL0).

Put server address in ServerAddress (D0).

Toggle C0 to run it.

ReadFileSuccess or ReadFileFailure will be set on completion. On failure ReadFileErrMsg may contain useful info.

Result is in HttpDataFile. Max size is 1024 bytes.

There are two methods of reading the actual file employed. When the server returns Content-Length header, I do a one-shot read. If it isn't and the length is unknown, I read until the socket is closed. Either is valid based on the what the server does.

Probably won't enhance it more since I am leaning toward adding HTTPGET and HTTPPOST to DmD 1.1.

Enjoy. Comments welcome.
Title: Re: SS memory and pointers
Post by: plcnut on October 16, 2012, 12:56:14 PM
Thank You very much Bob!

I'm looking forward to trying it out this afternoon.
Just waiting on FedEx...

I will let you know how it goes.
Title: Re: SS memory and pointers
Post by: plcnut on October 16, 2012, 05:30:56 PM
One little problem I have ran into. The dataview window, when set to ASCII, will not display, nor allow me to type an =(equal)sign. Is it just me or is something wrong?
Title: Re: SS memory and pointers
Post by: franji1 on October 16, 2012, 07:53:55 PM
The equal sign entry is a known bug.

You have to be online to the PLC to see the ASCII status, and turn status ON for the Data view.  Click on the Data View, then click on the Status sticky button to enable status for the current view (2nd toolbar from the top).
Title: Re: SS memory and pointers
Post by: plcnut on October 17, 2012, 10:25:53 AM
The equal sign entry is a known bug.

You have to be online to the PLC to see the ASCII status, and turn status ON for the Data view.  Click on the Data View, then click on the Status sticky button to enable status for the current view (2nd toolbar from the top).

Thanks franji1,
Title: Re: SS memory and pointers
Post by: plcnut on October 17, 2012, 10:32:41 AM
I have the Http GET working! ;D Thank you very much Bobo for the code. Because the server I am querying is set up just for this app, I was able to get rid of most of your coding (headers, and file length). Its a load off my mind to have this portion of the app working!
Thanks again
Title: Re: SS memory and pointers
Post by: BobO on October 17, 2012, 11:19:33 AM
Glad it's working!

If the data payload is known and arbitrary, then you can easily define the data in such a way as to greatly simplify the reading of the content...most obviously by limiting the length and using some form of delimiter. When you suggested that you were using port 80, I assumed that the server was going to be a standard web server sourcing files generated by something else. In that context the more generalized code is required.

It's kinda moot though. Once it became apparent to me that HTTP GET/POST were not particularly difficult to code and both could be used on a much broader basis, I wanted to do the sample app as a way to prototype an instruction for the controller. I think such instructions could be very powerful additions to the Do-more instruction set and we are planning to add them for Rel 1.1.
Title: Re: SS memory and pointers
Post by: Bolt on December 18, 2017, 11:17:08 AM
It's kinda moot though. Once it became apparent to me that HTTP GET/POST were not particularly difficult to code and both could be used on a much broader basis, I wanted to do the sample app as a way to prototype an instruction for the controller. I think such instructions could be very powerful additions to the Do-more instruction set and we are planning to add them for Rel 1.1.

Did GET/POST get added to Do More Designer?  I can't seem to find much on it, and am struggling making this program work successfully, am getting 400 bad request.  Was wondering if there were newer instructions to work with before I continue.
Title: Re: SS memory and pointers
Post by: BobO on December 18, 2017, 11:38:17 AM
Did GET/POST get added to Do More Designer?

No, sorry.
Title: Re: SS memory and pointers
Post by: plcnut on December 18, 2017, 03:23:20 PM
Did GET/POST get added to Do More Designer?  I can't seem to find much on it, and am struggling making this program work successfully, am getting 400 bad request.  Was wondering if there were newer instructions to work with before I continue.
If you can post some screenshots of what you have then I may be able to give you some pointers.
Title: Re: SS memory and pointers
Post by: Bolt on December 18, 2017, 07:37:46 PM
That would be great.  Here you go:

I am trying to get a weather file using an API key, in XML format

I am basically using BobO's ReadHttpFile program

I OPENTCP to client, it connects

I STREAMOUT "GET /api/.....xml HTTP/1.0$0D$0A$0D$0A'

I get 400 Bad Request Response

If I STREAMOUT with only 1 <CR><LF> at the end of the request I get a Timed out waiting for server after 10 seconds, and 10 seconds later the TCP.Connected drops




Title: Re: SS memory and pointers
Post by: plcnut on December 18, 2017, 08:40:45 PM
I believe you need to get rid of the HTTP/1.0 from you GET string, and you may need to add in the entire URL for your request ie: "http://myWebsiteUrl.com/api/theRestOfMyGetString$0D$0A"
Also, do you have a link to the api specs?
Title: Re: SS memory and pointers
Post by: Bolt on December 19, 2017, 01:19:00 PM
Using full address (numerical or url) with no HTTP/1.0 and just one set of <CR><LF> results in time out.

In short, the only way I get a 400 bad request response is with just the filename after .com, and with HTTP/1.0$0D$0A$0D$0A, all other options have led to timeouts.

The info page is https://www.wunderground.com/weather/api/d/docs

The link to their customer forum isn't working...

Here is a screenshot of the page in a web browser

Thanks for having a look!



Title: Re: SS memory and pointers
Post by: franji1 on December 19, 2017, 01:43:43 PM
Your string should look like the following:

GET api.wunderground.com/api/b4e16c90a27b013d/conditions/q/TN/Jonesborough.xml<CR><LF>
or
"GET api.wunderground.com/api/b4e16c90a27b013d/conditions/q/TN/Jonesborough.xml$0D$0A"
or possibly
"GET api.wunderground.com/api/b4e16c90a27b013d/conditions/q/TN/Jonesborough.xml HTTP/1.0$0D$0A"

based on their template in the info page link you provided
http://api.wunderground.com/api/Your_Key/conditions/q/CA/San_Francisco.json
or for your URL
http://api.wunderground.com/api/b4e16c90a27b013d/conditions/q/TN/Jonesborough.xml

Not sure if you need to insert
" HTTP/1.0" or " HTTP/1.1"
before the CR/LF

Can you get a screen shot of your entire string in a WIDE Data view using QUOTED format? (dock it on the bottom instead of the left side), or possibly use 4 Hex/ASCII format on a TALL Data View docked on the left?
Title: Re: SS memory and pointers
Post by: plcnut on December 19, 2017, 02:14:01 PM
Here is a working example.
I just used BobO's sample and added a little bit of code.
You can see that you will need to use multiple STREAMIN's to get all the data back. I did not have time to fiddle around with parsing the data, or even checking to see if it is all making it in, but I think I have enough here to get you going in the right direction.
(Don't forget to add in your key code to the URL).
Title: Re: SS memory and pointers
Post by: Bolt on December 19, 2017, 09:26:50 PM
Many thanks to plcnut for helping me out here.  I briefly tried franj1's suggestions, but quickly moved on. I was able to take plcnut's example program and make it work in my simulator.  It's a little messy, but proof of concept has been achieved.

I wouldn't mind a few tips on cleaning up my program.

I STREAMIN the api's data into SXL0-3.  1024 bytes each, SXL3 only 256.

I STRFIND each SXL string, searching for, in this example, "<temp_f>", and if found, do a STRSUB with the remainder of the string after the found <temp_f>

I do a STRFIND on the newly created string, and find "</temp_f>", and STRSUB the rest out to a SS.

I STR2REAL this SS to an R value, and have my result.


Questions,

Am I going about this the right way, or the long way?  I realize there are a few short cuts I could take in my STRFIND/STRSUB procedures, but I don't see many steps to save.

How do I cleanly determine the end of the final string?  The data set is ~3500 bytes, which varies slightly (but is published in the header data...)
EDIT:I was playing with it at fixed bytes per STREAMIN, all I need to do is use the .InQueue data to determine how much data to put in each string

What do I do if the text I am looking for falls partly between two SXL's, it would not be found.  Can I re-constitute all the SXL's back together before searching?  That would also simplify the multiple STRFIND steps.

Again, shout out to Jason for guiding, <pushing>, me in the right direction!
Title: Re: SS memory and pointers
Post by: Bolt on December 20, 2017, 06:39:41 PM
Here is what I have come up with.

It works pretty good, with a catch.

It will successfully fetch all the data via the API.

When I process the data, it fails about half the time.  On reattempt to process the same data, it works fine.

Please see attached program.

Load the DMD file

Import the Memory data CSV file

Write to Simulator

X14 turns on DMLogger
X0 fetches data via API - V1 will increment success
X15 process the data, but only the dataset corresponding to the appropriate X1-X9 bit status, so pick an X parameter, toggle X15.  V11 is success, V10 and V12 are error counts.  Should make sense in the data view.

I hope someone can tell me why the success rate of the STRFIND based program is not very high.

Any advice welcome.

Title: Re: SS memory and pointers
Post by: plcnut on December 20, 2017, 07:08:42 PM
I will try to look this over for you when I get a chance.
Title: Re: SS memory and pointers
Post by: Bolt on December 20, 2017, 08:01:29 PM
I changed my program up one more time, using strictly N100x=!-1 statements to advance through the STRFIND statements and it seemed to have cleared up, running reliably now.

Sorry for the false alarm.  Thanks for your help.
Title: Re: SS memory and pointers
Post by: Bolt on December 22, 2017, 03:48:17 PM
I have fully implemented the Calling for weather data and Processing it into usable data into my desired program.  Now I am having issues with incomplete data coming across in response to the HTTP GET request. 

A full response consists of 1 string of HTTP/1.0 200 OK<CR><LF>, followed by 4 strings of data, 3 at 1024 bytes, and the last one a little less.

I call for data every 3 minutes (well within terms of API license)

I always get a 200 OK response.

Sometimes, maybe 1 or 2 times an hour, I receive all the data.

The rest of the time I get partial data, either the first string is about 1000 bytes and then nothing else, or first string is 1024 bytes, and the second is a partial, maybe 500 bytes.

My program is basically: If .InQueue > 0, I do a STREAMIN (1).  Next STAGE.  If .InQueue > 0 I do a STREAMIN (2).  Repeat as needed.

Any idea where to be looking here?  Are my STREAMIN's too close together, timing wise?

Title: Re: SS memory and pointers
Post by: BobO on December 22, 2017, 03:53:12 PM
When is the connection being closed? Is the server closing it in response to a read?
Title: Re: SS memory and pointers
Post by: Bolt on December 22, 2017, 04:25:18 PM
The series of if .InQueue > 0 statements all have an inverted powerflow that advances to the Done SG, which executes a CLOSE on the Device.

Do I need to create a a statement along the lines of:

if .InQueue < 1024 wait 2 seconds, if it doesn't climb to 1024, run the STREAMIN?


I've tried putting a generic 1 or 2 second delay before every STREAMIN to try and wait for more bytes but that didn't help.
Title: Re: SS memory and pointers
Post by: plcnut on December 22, 2017, 04:27:34 PM
I think you need to add a timer in those stages where you check the queue, so that it waits for data instead of closing the connection right away.
Title: Re: SS memory and pointers
Post by: BobO on December 22, 2017, 04:35:31 PM
I tend to write my handlers reverse of that. I sit in a stage waiting for either the arrival of some quantity of data (.InQueue >= ??) or a timeout. If I get the expected data, I jump to the read handler. If I get a timeout, I jump to the timeout handler.

The read handler is usually just the STREAMIN with no input logic. OnSuccess jumps to next wait state.

SG Wait1
 STR .InQueue >= ExpectDataCount ---------- JMP Read1
 STR On ------------------------------------TMR T0
 STR T0.Done -------------------------------JMP Timeout

SG Read1
 -------------------------------------------STREAMIN
                                              OnSuccess JMP Wait2
                                              OnError JMP Error
SG Wait2

etc...
Title: Re: SS memory and pointers
Post by: Bolt on December 22, 2017, 05:36:50 PM
The best I can tell is the server drops my connection after less than 300 milliseconds.  So that is why my putting timers on did not help much.  Even using the timer at 10 milliseconds does not help it get the queue filled up on time.

Any more ideas I might try?  Is this an issue with their server that it is dropping me quite quickly?

Every time I poll the data via web browser it works for me, but it can of course load all the bytes at once.

Are there any work arounds for dealing with more than 1024 bytes at once?

Thanks for your continued help.
Title: Re: SS memory and pointers
Post by: BobO on December 22, 2017, 05:39:00 PM
Did you try it the way I described?
Title: Re: SS memory and pointers
Post by: Bolt on December 22, 2017, 06:44:58 PM
Yes, more or less.  See screenshot.
Title: Re: SS memory and pointers
Post by: BobO on December 22, 2017, 06:59:32 PM
Can you post the specific query you are using? I want to play with it some.
Title: Re: SS memory and pointers
Post by: plcnut on December 22, 2017, 07:29:42 PM
It looks like you are attempting to queue enough data so that you don't have to worry about piecing too much data back together. The problem is, that you are waiting for the buffer to be completely full before you empty it, and therefore much of it is getting dumped. You need to get the data in as soon as it hits the buffer, and then worry about piecing it later. I would help you with it, but cannot make any time commitments right now.
Title: Re: SS memory and pointers
Post by: plcnut on December 22, 2017, 07:32:49 PM
You may be able to use the hex equivalent of a "> " as the delimiter in your STREAMIN, which would allow you to STREAMIN the data for each xml element in its own string.
...maybe...
Title: Re: SS memory and pointers
Post by: Bolt on December 22, 2017, 08:36:23 PM
You may be able to use the hex equivalent of a "> " as the delimiter in your STREAMIN, which would allow you to STREAMIN the data for each xml element in its own string.
...maybe...

Well, there are 60+ 77 parameters in the XML file, so it would be quite a mess to dump the noise, keep the stuff I want, etc.  Not a bad idea though.
Title: Re: SS memory and pointers
Post by: Bolt on December 22, 2017, 08:43:18 PM
Can you post the specific query you are using? I want to play with it some.

So, I was going to copy the pertinent programs into a new file to run in the simulator, and post here.  One thing led to another, and before long I had an older simulator program open that I built it all on earlier.  It (still) works solid.  I cleaned it up a little, etc, still works great.  I go to my PLC program, delete the 3 programs involved, and copy the 3 from the simulator over.

It still will not work on the PLC.  The exact 3 programs work great on the simulator.  Where do I go from here?

This is my old version without the timers for each STREAMIN.  I could post my simulator program here, but it works, so that wouldn't help much?

Simulator Scan Times are 10 to 32,000 microseconds, min to max
PLC Scan Times are 2,000 to 28,000 microseconds, min to max
Title: Re: SS memory and pointers
Post by: BobO on December 22, 2017, 08:51:35 PM
I have a hunch. I'm not excited about it.

What may be happening is that the remote server is sending the data and immediately disconnecting. The problem is that when the socket is forceably closed by the remote connection, the TCP device grabs the data in the socket FIFO and stashes it into his buffer...but...his buffer is only 1024 bytes. Any additional data in the socket is getting tossed on the floor. The difference between how this is working in the Sim vs PLC is just speed.
Title: Re: SS memory and pointers
Post by: plcnut on December 22, 2017, 09:51:53 PM
I have reliably moved a LOT of data into a Do-more from local as well as remote servers over the past several years (ever since Do-more launched). I think that the way the timer rungs are structured are causing the issues. I *may* have time to do some testing some time next week.
Title: Re: SS memory and pointers
Post by: BobO on December 22, 2017, 10:23:18 PM
I have reliably moved a LOT of data into a Do-more from local as well as remote servers over the past several years (ever since Do-more launched). I think that the way the timer rungs are structured are causing the issues. I *may* have time to do some testing some time next week.

Maybe, but I rewrote the read to chunk through the data as fast as I know how and it's dropping data. I want to play with it some more though.
Title: Re: SS memory and pointers
Post by: BobO on December 23, 2017, 12:40:47 PM
It's what I thought. When the server drops the connection, that forces the PLC's socket closed. When the PLC's socket gets forced closed with data still in the buffer, we read that data and stuff into a cache so it's still available to the app. Since the PLC's are resource constrained and because we like to avoid dynamic allocation where possible (concerned about long term fragmentation), the buffer in the TCP Client is pre-allocated and only 1024 bytes. In this case there is well over 1K in the buffer, so it gets tossed on the floor.

Good news/bad news: The solution is simple (set cache size to match buffered data and dynamically allocate) and I'm willing to do it (low-ish fragmentation risk), but we are in the middle of development for a bunch of new modules we'll be rolling out next year, and the code is in no shape to release. Late Q1.

@Nut: You haven't had issues because your server isn't dumping you.
Title: Re: SS memory and pointers
Post by: Bolt on December 23, 2017, 01:25:55 PM
I see.  Now I know what I am chasing.

Thanks for your looking into it, and an eventual solution!

In case anyone cares, here's my code to do it with.

Like I said, it works fine in the simulator, not reliably in the PLC.  And I'm not fond of my logic as far as searching separate strings, if the data falls partially between 2 strings, it throws a wrench in everything.

Load the program, load the memory parameters, open the data view, toggle C1210.
Title: Re: SS memory and pointers
Post by: BobO on December 23, 2017, 01:46:22 PM
There really is a gray area here when the server forces the connection closed when the client hasn't closed it. I understand why they do it, and it isn't strictly prohibited by the spec, but it's the equivalent of hanging up on someone before they said goodbye. I had a workaround before, but the cache wasn't deep enough. I have a better workaround now.
Title: Re: SS memory and pointers
Post by: Bolt on January 19, 2018, 11:52:23 AM
I have come up with an easy work around for this problem.

I wrote a simple php script on a server local to the PLC that fetches the file from the www, processes it into an array, and then only echo's the relevant data to the file (with some labels for parsing purposes).  Now I can get the data I want out of the file in a short, ~100 character response when the PLC reads the php file.

Now I have a question.  I am running multiple programs that create a TCP connection with the same server.  Do I :

1. keep them as separate devices (TCP clients)?
2. use the same device for each program?

Same device for all will run into issues with the DEVCLEAR instruction if multiple programs are running.

3. write all the comms with the server into just one program?