FILEREAD as Lookup Table

[franji1]

If it's just a "batch import", you can definitely stick with the original plan.  Sorry for the confusion.

Changing your .TimeSlice from 100 to 200 microseconds will definitely help.  It just adds 0.2ms of PLC scan time.  You mentioned you are around 5ms, so 5.2ms ain't bad.  So, by doubling your work time per PLC scan, you cut the calendar time IN HALF.  That's huge.

Bring up a Data View and stick some "profile" logic in there to see how much of an impact tweaking the .TimeSlice value.  100uSec is minimal PLC scan hit.  The "sweet spot", depending upon what you are doing, is usually around 200-400uSec.  Anything more doesn't help much (e.g. changing 400uSec to 500uSec only gives a 25% improvement in calendar time, while the same 100uSec hit from 100 to 200 gives you 100% improvement in calendar time)

[Controls Guy]

But how do I actually jump to the 5,000th row?  Would I need to fix my CSV's row length so simple math can be used to determine the starting offset?  I'm trying to keep this as flexible as possible.  Also, the 10,000 number was arbitrary, it will change over time, 6,000, 8,000, etc.

I think there might be several options.

Since the tables are coming to you electronically rather than somebody walking over and plugging in an SD card, could you have it written directly to memory rather than to file?

If not, could you move the data from file to memory after it gets sent to you?

If not, yes, fixing the line lengths for the CSV file should work.

If not that, then if you have the total file size in bytes, you could do the 50%-75%-25% thing byte-wise instead of record-wise, then from that byte position search forward and back for $0D$0A to find the nearest record boundary.    Then do the inequality check on the record data.

Sorry for being obtuse, I had the database-in-memory idea stuck in my head from the earlier part of the thread.

[franji1]

Every time you get a new file, you need to parse the file, keeping track of the FILE OFFSET (it is in the File Handle structure).

Are all the error codes between 0 and 9999?  Basically, create a Signed DWORD array 10,000 long, these will store the FILE OFFSET at the INDEX of that block, where the INDEX equals the ERROR CODE.

Say the file is as follows
1, Bad Stuff Happened
9002, Worse Stuff Happened
3, Even Worst Stuff Happened

I am guessing, but assuming CR/LF, the error text for error codes 1, 9002, and 3 would be at index
3 // B Bad...
27  // W Worse...
52  // E Even...
ErrorOffset would be a block, where index of 1, 9002, and 3 would have those values:

ErrorOffset0 0
ErrorOffset1 3
ErrorOffset2 0
ErrorOffset3 52
ErrorOffset4 0
ErrorOffset5 0
ErrorOffset6 0
ErrorOffset7 0
ErrorOffset8 0
ErrorOffset9 0
ErrorOffset10 0
...
ErrorOffset8999 0
ErrorOffset9000 0
ErrorOffset9001 0
ErrorOffset9002 27
ErrorOffset9003 0
...
ErrorOffset9999 0

Then, whenever you get error code, you would do
FILEOPEN
FILESEEK to position ErrorOffset[ErrorCode]
FILEREAD to CR/LF into SL0
FILECLOSE

This is how databases work.  They have "index" tables to get to the actual file position quickly.

[Bolt]

Since the tables are coming to you electronically rather than somebody walking over and plugging in an SD card, could you have it written directly to memory rather than to file?

If not, could you move the data from file to memory after it gets sent to you?

By memory do you mean RAM or into a Memory Block?
Currently, for testing, I'm copying the CSV to SD Card, then placing it in RAM before doing lookups.

If not that, then if you have the total file size in bytes, you could do the 50%-75%-25% thing byte-wise instead of record-wise, then from that byte position search forward and back for $0D$0A to find the nearest record boundary.    Then do the inequality check on the record data.

This would be the most future proof, but also the most work to code.  Less restrictions in the incoming data.

Sorry for being obtuse, I had the database-in-memory idea stuck in my head from the earlier part of the thread.

No worries, I really should have started a new thread...  To me it dovetailed right into the old project.

[Bolt]

Are all the error codes between 0 and 9999?  Basically, create a Signed DWORD array 10,000 long, these will store the FILE OFFSET at the INDEX of that block, where the INDEX equals the ERROR CODE.

No, they are not sequential, and 15 digits, so that makes this tricky.

I am guessing, but assuming CR/LF, the error text for error codes 1, 9002, and 3 would be at index

Yes, they are CR/LF separated, I follow your logic, but afraid it will not work at the scale I'm dealing with.  Thanks though, please keep the ideas coming!

[Controls Guy]

By memory do you mean RAM or into a Memory Block?
Currently, for testing, I'm copying the CSV to SD Card, then placing it in RAM before doing lookups.

OK, I guess when you're saying RAM, you mean a file system in RAM.   I meant to memory blocks in ram with defined record delineation.

This would be the most future proof, but also the most work to code.  Less restrictions in the incoming data.

I don't know, that seems pretty easily doable.

[Bolt]

OK, I guess when you're saying RAM, you mean a file system in RAM.   I meant to memory blocks in ram with defined record delineation.

Yes, I meant as a CSV file in the RAM.  Please tell me more, how do I create a memory block in RAM?

[Controls Guy]

I think any memory block you create is in RAM.

[Bolt]

I think any memory block you create is in RAM.

I don't think so, as Memory Blocks can be retentive, and they don't seem to take away from the 1024 KB the file manager shows the RAM's capacity to be.

[Controls Guy]

Well, I'm sure the memory blocks aren't in flash, but battery backed RAM.  If there's some value for 'RAM' somewhere, I think it's referring to RAM for the RAM file system.

[franji1]

MQTT broker where the topic name is the ID and the data is the error message.

Write a PC program to publish the topic ID key/message text (have it remove the old ones first)
Have the PLC subscribe to the topic.

[BobO]

The RAM disk is located in a 1MB chunk of SDRAM. The image register (blocks, heap) is located in a different area of SDRAM that is mirrored to a small battery backed SRAM. Neither exist in flash.

[Bolt]

I have this working to my satisfaction.  A BRX with a ~5ms scan time can find any one numerical string out of 16K in under 1/4 second.  Once it's all said and done, it's really not that complicated...    Thanks for the help!

[franji1]

I have this working to my satisfaction.  A BRX with a ~5ms scan time can find any one numerical string out of 16K in under 1/4 second.  Once it's all said and done, it's really not that complicated...    Thanks for the help!

AWESOME!  What are your scan times during the "lookup", and related - what .TimeSlice value did you end up using?

[Bolt]

AWESOME!  What are your scan times during the "lookup", and related - what .TimeSlice value did you end up using?

You really have to beat on the thing (ask it to find a value out of the bounds of the table, and let it keep trying after exhausting all possible searches) to even get the avg scan time to budge.  And then worst case scenario it's 5.2 vs 4.8 ms.
TimeSlice makes virtually no difference, quite similar performance with 0, 100, 200 or 2000 as a value.  They all take about 15ms per search iteration.
Keep in mind this is on a BRX that's actively running ~20 other programs at any given time, lots of communications, etc, so once on a more dedicated PLC it should have no problems at all.  Quite the impressive little box of rocks!