News:

  • September 19, 2019, 07:03:12 AM

Login with username, password and session length

Author Topic: SS memory and pointers  (Read 15614 times)

Bolt

  • Hero Member
  • *****
  • Posts: 146
Re: SS memory and pointers
« Reply #45 on: December 22, 2017, 03:48:17 PM »
I have fully implemented the Calling for weather data and Processing it into usable data into my desired program.  Now I am having issues with incomplete data coming across in response to the HTTP GET request. 

A full response consists of 1 string of HTTP/1.0 200 OK<CR><LF>, followed by 4 strings of data, 3 at 1024 bytes, and the last one a little less.

I call for data every 3 minutes (well within terms of API license)

I always get a 200 OK response.

Sometimes, maybe 1 or 2 times an hour, I receive all the data.

The rest of the time I get partial data, either the first string is about 1000 bytes and then nothing else, or first string is 1024 bytes, and the second is a partial, maybe 500 bytes.

My program is basically: If .InQueue > 0, I do a STREAMIN (1).  Next STAGE.  If .InQueue > 0 I do a STREAMIN (2).  Repeat as needed.

Any idea where to be looking here?  Are my STREAMIN's too close together, timing wise?


BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 4680
  • Yes Pinky, Do-more will control the world!
Re: SS memory and pointers
« Reply #46 on: December 22, 2017, 03:53:12 PM »
When is the connection being closed? Is the server closing it in response to a read?
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO

Bolt

  • Hero Member
  • *****
  • Posts: 146
Re: SS memory and pointers
« Reply #47 on: December 22, 2017, 04:25:18 PM »
The series of if .InQueue > 0 statements all have an inverted powerflow that advances to the Done SG, which executes a CLOSE on the Device.

Do I need to create a a statement along the lines of:

if .InQueue < 1024 wait 2 seconds, if it doesn't climb to 1024, run the STREAMIN?


I've tried putting a generic 1 or 2 second delay before every STREAMIN to try and wait for more bytes but that didn't help.

plcnut

  • Hero Member
  • *****
  • Posts: 801
    • premiersi.com
Re: SS memory and pointers
« Reply #48 on: December 22, 2017, 04:27:34 PM »
I think you need to add a timer in those stages where you check the queue, so that it waits for data instead of closing the connection right away.
Circumstances don't determine who we are, they only reveal it.

~Jason Wolthuis
Premier Systems Integration, LLC
http://premiersi.com

BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 4680
  • Yes Pinky, Do-more will control the world!
Re: SS memory and pointers
« Reply #49 on: December 22, 2017, 04:35:31 PM »
I tend to write my handlers reverse of that. I sit in a stage waiting for either the arrival of some quantity of data (.InQueue >= ??) or a timeout. If I get the expected data, I jump to the read handler. If I get a timeout, I jump to the timeout handler.

The read handler is usually just the STREAMIN with no input logic. OnSuccess jumps to next wait state.

SG Wait1
 STR .InQueue >= ExpectDataCount ---------- JMP Read1
 STR On ------------------------------------TMR T0
 STR T0.Done -------------------------------JMP Timeout

SG Read1
 -------------------------------------------STREAMIN
                                              OnSuccess JMP Wait2
                                              OnError JMP Error
SG Wait2

etc...
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO

Bolt

  • Hero Member
  • *****
  • Posts: 146
Re: SS memory and pointers
« Reply #50 on: December 22, 2017, 05:36:50 PM »
The best I can tell is the server drops my connection after less than 300 milliseconds.  So that is why my putting timers on did not help much.  Even using the timer at 10 milliseconds does not help it get the queue filled up on time.

Any more ideas I might try?  Is this an issue with their server that it is dropping me quite quickly?

Every time I poll the data via web browser it works for me, but it can of course load all the bytes at once.

Are there any work arounds for dealing with more than 1024 bytes at once?

Thanks for your continued help.

BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 4680
  • Yes Pinky, Do-more will control the world!
Re: SS memory and pointers
« Reply #51 on: December 22, 2017, 05:39:00 PM »
Did you try it the way I described?
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO

Bolt

  • Hero Member
  • *****
  • Posts: 146
Re: SS memory and pointers
« Reply #52 on: December 22, 2017, 06:44:58 PM »
Yes, more or less.  See screenshot.

BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 4680
  • Yes Pinky, Do-more will control the world!
Re: SS memory and pointers
« Reply #53 on: December 22, 2017, 06:59:32 PM »
Can you post the specific query you are using? I want to play with it some.
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO

plcnut

  • Hero Member
  • *****
  • Posts: 801
    • premiersi.com
Re: SS memory and pointers
« Reply #54 on: December 22, 2017, 07:29:42 PM »
It looks like you are attempting to queue enough data so that you don't have to worry about piecing too much data back together. The problem is, that you are waiting for the buffer to be completely full before you empty it, and therefore much of it is getting dumped. You need to get the data in as soon as it hits the buffer, and then worry about piecing it later. I would help you with it, but cannot make any time commitments right now.
Circumstances don't determine who we are, they only reveal it.

~Jason Wolthuis
Premier Systems Integration, LLC
http://premiersi.com

plcnut

  • Hero Member
  • *****
  • Posts: 801
    • premiersi.com
Re: SS memory and pointers
« Reply #55 on: December 22, 2017, 07:32:49 PM »
You may be able to use the hex equivalent of a "> " as the delimiter in your STREAMIN, which would allow you to STREAMIN the data for each xml element in its own string.
...maybe...
Circumstances don't determine who we are, they only reveal it.

~Jason Wolthuis
Premier Systems Integration, LLC
http://premiersi.com

Bolt

  • Hero Member
  • *****
  • Posts: 146
Re: SS memory and pointers
« Reply #56 on: December 22, 2017, 08:36:23 PM »
You may be able to use the hex equivalent of a "> " as the delimiter in your STREAMIN, which would allow you to STREAMIN the data for each xml element in its own string.
...maybe...

Well, there are 60+ 77 parameters in the XML file, so it would be quite a mess to dump the noise, keep the stuff I want, etc.  Not a bad idea though.
« Last Edit: December 22, 2017, 08:46:44 PM by Bolt »

Bolt

  • Hero Member
  • *****
  • Posts: 146
Re: SS memory and pointers
« Reply #57 on: December 22, 2017, 08:43:18 PM »
Can you post the specific query you are using? I want to play with it some.

So, I was going to copy the pertinent programs into a new file to run in the simulator, and post here.  One thing led to another, and before long I had an older simulator program open that I built it all on earlier.  It (still) works solid.  I cleaned it up a little, etc, still works great.  I go to my PLC program, delete the 3 programs involved, and copy the 3 from the simulator over.

It still will not work on the PLC.  The exact 3 programs work great on the simulator.  Where do I go from here?

This is my old version without the timers for each STREAMIN.  I could post my simulator program here, but it works, so that wouldn't help much?

Simulator Scan Times are 10 to 32,000 microseconds, min to max
PLC Scan Times are 2,000 to 28,000 microseconds, min to max

BobO

  • Host Moderator
  • Hero Member
  • *****
  • Posts: 4680
  • Yes Pinky, Do-more will control the world!
Re: SS memory and pointers
« Reply #58 on: December 22, 2017, 08:51:35 PM »
I have a hunch. I'm not excited about it.

What may be happening is that the remote server is sending the data and immediately disconnecting. The problem is that when the socket is forceably closed by the remote connection, the TCP device grabs the data in the socket FIFO and stashes it into his buffer...but...his buffer is only 1024 bytes. Any additional data in the socket is getting tossed on the floor. The difference between how this is working in the Sim vs PLC is just speed.
"It has recently come to our attention that users spend 95% of their time using 5% of the available features. That might be relevant." -BobO

plcnut

  • Hero Member
  • *****
  • Posts: 801
    • premiersi.com
Re: SS memory and pointers
« Reply #59 on: December 22, 2017, 09:51:53 PM »
I have reliably moved a LOT of data into a Do-more from local as well as remote servers over the past several years (ever since Do-more launched). I think that the way the timer rungs are structured are causing the issues. I *may* have time to do some testing some time next week.
Circumstances don't determine who we are, they only reveal it.

~Jason Wolthuis
Premier Systems Integration, LLC
http://premiersi.com