Host Engineering Forum
General Category => Do-more CPUs and Do-more Designer Software => Topic started by: Bolt on August 14, 2020, 08:50:43 AM
-
I have a device that sends ASCII error codes to BRX via Modbus TCP. Then I have a CSV file that references said error code to a error description. Currently, I am hosting this CSV on a webserver, sending a HTTPCMD containing error code, cross referencing this to error description, and returning that to the BRX. Works great.
However, due to changing network architecture, I would like to do this all on the BRX, have the lookup table on the SDCard, and use FILEREAD to do the looking up. How would I go about doing this?
-
I have a device that sends ASCII error codes to BRX via Modbus TCP. Then I have a CSV file that references said error code to a error description. Currently, I am hosting this CSV on a webserver, sending a HTTPCMD containing error code, cross referencing this to error description, and returning that to the BRX. Works great.
However, due to changing network architecture, I would like to do this all on the BRX, have the lookup table on the SDCard, and use FILEREAD to do the looking up. How would I go about doing this?
How large is this database? Is it just a few hundred error codes and 80 characters per error message? You can do that easily with just 2 data blocks, one with an error code number and one with strings, up to 80 characters long, where the index of each correspond to each other. No need for the SD Card/File system (unless you need it to LOAD the data on first scan). If it is static, stick it in code, don't bother with the SD card. Initialize the 2 blocks in $tFirstScan via ladder logic, definitely no SD card needed there.
To look up the error code text, use FOR/NEXT loop to find error code in the error code block, get that V index, and use that index in the string error text table.
STRPRINT ErrorText "Unknown Error:" ErrorCode // assume ErrorCode is not in our table
// loop through ErrorCodeTable, looking for ErrorCode
FOR V42 0 199
STRE ErrorCode ErrCodeTable[V42] // found it?
STRCOPY ErrorTable[V42] ErrorText // copy it to ErrorText
BREAK // done! break the FOR/NEXT loop
NEXT
// ErrorText will either be assigned the found error code, or it will be "Unknown Error
RETURN
If the error codes are sequential, say 0..199 (i.e. they are not sparse), you do not even need the error code table, the error code IS the string table V array index.
However, if there are HUGE number of error codes (more than 500) with HUGE error messages (some with more than 500 characters), then the SD Card mechanism will be necessary, parsing the text. But there are ways to implement a database lookup function in a PLC (e.g. have an Error code table with a corresponding FILE OFFSET DWORD table, where the lookup doesn't return a string, it returns the INDEX INTO THE FILE WHERE THAT ERROR TEXT EXISTS. That's sort of what databases do on Server (table "keys" => corresponding data locations).
-
This is an interesting angle, and I could likely make it all fit by condensing the text strings (there are ~240 codes, and many have a lengthy description), limiting the oddball errors that will likely not occur, etc, but .
What if I did a hybrid approach? Error codes in PLC data block (4 or 5 alpha numeric characters), lookup a numeric value, then reference a SD Card file to get the full description? Then it would be a matter of which line the result is on? How would I do that (the FILEREAD line number part)?
-
I would not bother with hybrid - you run into issues of inconsistency between the HARDCODED error codes and the error codes in the FILE. You could do a hybrid of the hybrid (ha!)
Store the short and the long in the file, but when we are parsing for the long text file offset, we can build up the table of strings of the short text:
error code 1; short text 1; long text 1
error code 2; short text 2; long text 2
e.g.
21; RC; Right fielder, Pittsburgh Pirates
42; JR; Secondbase, Brooklyn Dodgers
Note I am using semicolons as delimiters. I did not use commas in case you need to use commas in your error message text - so do NOT use semicolons in your short/long descriptions!!
On first scan, you would need to generate 3 tables
UNSIGNED WORD ErrorCodes block 256 long
STRING MAX 8 ShortErrorText block 256 long
SIGNED DWORD LongFileOffset block 256 long
1. Open the file
2. Read a line (CR/LF delimiter)
3. tokenize the line (semicolon delimits tokens), getting the 3 pieces
4. parse the 3 pieces and put into the 3 blocks
5. go back to step 2
It's not hard. What's great is that you can do all this with the Simulator and make sure the logic all works before dealing with the physical SD Card in a BRX
Do step 1, 2, and 5 for now. Try to get that working using FILEOPEN, with FILEREAD inside FOR/NEXT loop, and FILECLOSE after the NEXT.
You also have some possible issues you have to deal with. Just think about them, we can keep these in mind
1. What happens if the SD card is not there, or the desired file does not exist? Missing SD card is a definite point of failure; I say you just set a flag and when you look up error codes, you just get the error number as text in the short description (maybe with a question mark, e.g. "42?") and "Unknown Error: 42" for the long description???
2. What happens if some of the file content is invalid; set a flag, but just keep parsing the file (unknown error handling from 1 can be utilized here also)
-
That will get me started. I don't have much time to work on it today, but will report back. Thanks for getting me in the right direction!
-
I would not bother with hybrid - you run into issues of inconsistency between the HARDCODED error codes and the error codes in the FILE. You could do a hybrid of the hybrid (ha!)
Store the short and the long in the file, but when we are parsing for the long text file offset, we can build up the table of strings of the short text:
error code 1; short text 1; long text 1
error code 2; short text 2; long text 2
e.g.
21; RC; Right fielder, Pittsburgh Pirates
42; JR; Secondbase, Brooklyn Dodgers
Note I am using semicolons as delimiters. I did not use commas in case you need to use commas in your error message text - so do NOT use semicolons in your short/long descriptions!!
On first scan, you would need to generate 3 tables
UNSIGNED WORD ErrorCodes block 256 long
STRING MAX 8 ShortErrorText block 256 long
SIGNED DWORD LongFileOffset block 256 long
1. Open the file
2. Read a line (CR/LF delimiter)
3. tokenize the line (semicolon delimits tokens), getting the 3 pieces
4. parse the 3 pieces and put into the 3 blocks
5. go back to step 2
It's not hard. What's great is that you can do all this with the Simulator and make sure the logic all works before dealing with the physical SD Card in a BRX
Do step 1, 2, and 5 for now. Try to get that working using FILEOPEN, with FILEREAD inside FOR/NEXT loop, and FILECLOSE after the NEXT.
You also have some possible issues you have to deal with. Just think about them, we can keep these in mind
1. What happens if the SD card is not there, or the desired file does not exist? Missing SD card is a definite point of failure; I say you just set a flag and when you look up error codes, you just get the error number as text in the short description (maybe with a question mark, e.g. "42?") and "Unknown Error: 42" for the long description???
2. What happens if some of the file content is invalid; set a flag, but just keep parsing the file (unknown error handling from 1 can be utilized here also)
Okay, I have 1-4 working, and can get it to return to step 2, but how do I have it read the next line of the file, and not re-read the first line over and over again?
-
I got it to work, but I could not get it to run successfully in a FOR/NEXT loop arrangement. It is almost like it ran too fast, and would never calculate FILESEEK position quick enough to read the correct line in the file. Now I have it running as a Program with Stages, and manually incrementing the pointer after every line, and it works slick.
However, now I'm wondering what that advantage is to create these tables on the PLC. Could I not run through this logic, and instead of writing to ErrorCodes[V0] (and LongFileOffset[V0], find the semicolon, do a STRCMP with the error code received via Modbus TCP, if matches, calculate offset and FILESEEK/FILEREAD the value, if no match, increment count and try again? Is this simply too scan time intensive?
-
I don't think it would be scan time intensive, I think It'd be slow.
Flash is faster than spinning media, but it's still not RAM. Plus there's gonna be some software and hardware overhead/bottleneck getting the data off the flash.
Easier to move the whole table, or at least the index, to PLC memory.
-
Sorry, FOR/NEXT won't work. I was thinking table lookup, not reading a file. Oops.
A looping Stage program works
Program LoadErrorTables
SG S0
FILEOPEN OnSuccess JMP S1
SG S1
FILEREAD OnSuccess JMP S2 OnError JMP S99
SG S2
parse out pieces/process data
JMP S1 // loop back to FILEREAD stage
SG S99
FILECLOSE OnSuccess JMP S100
SG S100
EXIT
-
I have this working to my satisfaction now. Thanks for the help.
In Simulator:
Populating the data (298 rows) into some lookup tables takes ~150 milliseconds
Looking up the code in the tables and reading corresponding line from file takes ~2 to 75 milliseconds (first and last entries)
Looking up the code directly from the SD card without making lookup tables takes ~2 to 150 milliseconds
In BRX:
Looking up the code directly from the SD card without making lookup tables takes ~2 to 5700 milliseconds (with no impact to scan time)
Still fast enough for my purposes.
-
In BRX:
Looking up the code directly from the SD card without making lookup tables takes ~2 to 5700 milliseconds (with no impact to scan time)
Still fast enough for my purposes.
GREAT!! You're now a BRX expert!
-
BobO mentioned to speed it up, you could see if it does not exist in the RAM drive, do a FILECOPY if it doesn't exist there, then always do the error lookup from the RAMDRIVE copy.
-
Ooh, I like that idea. Plus, if the codes are in order, or you can put them in order, at most 9 queries should find the record if random access is possible.
-
BobO mentioned to speed it up, you could see if it does not exist in the RAM drive, do a FILECOPY if it doesn't exist there, then always do the error lookup from the RAMDRIVE copy.
Cool idea. Shaves about 1 second off the first attempt (having to copy file from SD Card to RAM), and subsequent attempts take almost 2 seconds less than reading the whole file from the SD Card everytime.
Ooh, I like that idea. Plus, if the codes are in order, or you can put them in order, at most 9 queries should find the record if random access is possible.
The error codes are ASCII, not numeric. along the lines of AB-12 or XY-789, so that would be a little too tricky.
Now, my question is, what is the cleanest way to set my loop max index (in event of error code not found)? It should be number of rows in file. Using FILEQUERY, I don't think there's a way to see this, only bytes, but each line has varying byte count. Would I have to loop through the file and count the number of <CF><LF> occurances?
-
Basically, if you reach the end of the file and have not found it, it's not found.
Not sure how you've written your code, but once you find the code, you should JMP out of the "loop" (so typically you should never reach the end of the file).
If the FileHandle structure's .EndOfFile bit is set, then you've reached the end of the file before finding the code. You should be able to tell the difference.
-
Yes, once code found, it JMPs to FILECLOSE et al.
Structure.EndOfFile is what I was looking for in Code not found scenarios.
I suppose .FilePointer = .FileSize would have done the trick as well.
-
I am trying to implement this concept of looking up in a very large table 10K+ lines, and it (obviously) takes a while to find data in the later rows.
This time, the lookup "column" data is all numeric, but much larger than a DWord can hold, like UPC codes. The numbers will not be consecutive. What are some tricks I can use to speed it up? Sort by first column, create indexed steps, etc? How do I do such things on a BRX?
-
...if the codes are in order, or you can put them in order, at most 9 queries should find the record if random access is possible.
How would you do this with 9 queries?
-
How would you do this with 9 queries?
I understood your reply #9 to say there were 289 rows, and 2^9 = 512 > 289.
So, assuming the records have been sorted in order, you check the 50% record, or in this case, either 256 or 145. if the value you're looking for is less than the retrieved value, then go to 25%, or 72. If >, then go to 75%, or 217. Based on that comparison go half way to the endpoint or the previous test point, and so on. Since 2^9 > 289, it is guaranteed that you'll find the desired register in at most 9 attempts. Now since you're saying 10,000 records, it might take as many as 14, if they're sorted.
-
Gotcha. I am now re-applying the concept to a much larger project, and need to optimize. I follow your split in half, check if high or low, I just struggle to apply it to ladder logic.
How do I count the number of rows the file contains to do the divisions? Do I need to do that with how many Bytes the file is large? That would kind of require each row to be of the same length, a restriction I'd rather not put on the data source at this time.
-
Is the data static or is it always changing?
It may not be practical to physically sort a table that large quickly enough if it's constantly changing, and a sorted table, premised on the idea that the tabular data is cast in stone, or at least changing only infrequently, is necessary for my 2^n query bound.
Another approach might be to create an index of pointers to the relative rank of the table records.
-
The data is changing on an interval, daily at a minimum, 2 or 3 time daily at most. When the data is changed, it would be possible to take some time to (re)generate an index, subdivisions, etc.
-
And what's the record format again? Very large numbers stored as text?
What are you doing with the data once you retrieve it? Do you have to do math on it? Display on HMI? Communicate it via comms?
Would 32-pt float precision be adequate? Are the numbers signed or unsigned?
Generating a sorted index on 10K strings or floats (or DIY 48 or 64-bit integers) should be quite feasible in a BRX.
You could even do RANK() in Excel and make the index part of what you import, depending on how the data comes to you. Or even presort the data outside the PLC and go back to using the 2^n search.
-
And what's the record format again? Very large numbers stored as text?
15 digit numbers, followed by a (alphanumeric) code, in a CSV file
What are you doing with the data once you retrieve it? Do you have to do math on it? Display on HMI? Communicate it via comms?
Enable output(s) based on looked-up code
Would 32-pt float precision be adequate? Are the numbers signed or unsigned?
Unsigned, float not accurate enough
Generating a sorted index on 10K strings or floats (or DIY 48 or 64-bit integers) should be quite feasible in a BRX.
I'm all ears (eyes). 48 bit would not be enough, 64 would. This would be all this BRX needs to do, so space shouldn't be an issue (famous last words...)
You could even do RANK() in Excel and make the index part of what you import, depending on how the data comes to you. Or even presort the data outside the PLC and go back to using the 2^n search.
It would be all automated, no human interaction, let me dig into what the data source can pre sort/filter. The data can be sorted by column, so should not be a problem there. The 2^14 search is fast enough for me, just not sure how to implement it.
-
If you actually need to MODIFY the "database", random CSV will not work. I thought the database was static (error lookup).
CSV only works for complete import/export of the entire database. Files are like RAM, so you just cannot just insert a record 28 characters long, or replace one of 20 characters with one that is 22 characters. Flash ROM/file systems don't work that way. This is why databases are fast and $$$ and LOTs of code.
The only way this could work is if you had a FIXED record format. Think COBOL 1968. Every record is a FIXED 40 characters (64, whatever). Hence, record at 40, 80, 120, etc. You would need to maintain a list of DELETED records in a separate file. And a list of KEY->INDEX values in a third file. You could maintain the KEY->INDEX in the RAM drive, or even a UDT block, that you can easily regenerate on a moment's notice (but the MAIN TABLE will NOT need to be rewritten). Deleted records would also need to be flagged in the primary file (think FAT file system), this way you could re-generate the DELETED records quickly by looking at that flag one time at startup (the same time you re-generate the Key->index map).
it's do-able. What's cool about Do-more is you can tweak the .TimeSlice to trade off calendar time with scan time. Higher .TimeSlice (in microseconds), the more database processing you get done per PLC scan, at the price of bumping the PLC scan. A lower .TimeSlice means less time doing database work every scan, so your PLC scans are faster, but your database operations take longer calendar-time-wise.
-
Sounds like this is a new project, if I'm reading his post correctly.
-
It would be all automated, no human interaction, let me dig into what the data source can pre sort/filter. The data can be sorted by column, so should not be a problem there. The 2^14 search is fast enough for me, just not sure how to implement it.
Start at 5000. Depending on too high / too low, then check 2500 or 7500, then 1250, 3750, 6250, 8750, etc. Just keep dividing the range in half and the half with your value becomes the new range. Rinse and repeat till you test and hit the target value.
-
If you actually need to MODIFY the "database", random CSV will not work. I thought the database was static (error lookup).
Sorry, I'm starting a new project with the old code from last year. That works great to this day, looking up error codes. Perhaps I should have started a new post...
CSV only works for complete import/export of the entire database. Files are like RAM, so you just cannot just insert a record 28 characters long, or replace one of 20 characters with one that is 22 characters. Flash ROM/file systems don't work that way. This is why databases are fast and $$$ and LOTs of code.
The only way this could work is if you had a FIXED record format. Think COBOL 1968. Every record is a FIXED 40 characters (64, whatever). Hence, record at 40, 80, 120, etc. You would need to maintain a list of DELETED records in a separate file. And a list of KEY->INDEX values in a third file. You could maintain the KEY->INDEX in the RAM drive, or even a UDT block, that you can easily regenerate on a moment's notice (but the MAIN TABLE will NOT need to be rewritten). Deleted records would also need to be flagged in the primary file (think FAT file system), this way you could re-generate the DELETED records quickly by looking at that flag one time at startup (the same time you re-generate the Key->index map).
By "changing" the data, I mean at a certain interval, a new CSV file will be pushed to the BRX. It will overwrite the previous one, containing the new sorting allocations.
it's do-able. What's cool about Do-more is you can tweak the .TimeSlice to trade off calendar time with scan time. Higher .TimeSlice (in microseconds), the more database processing you get done per PLC scan, at the price of bumping the PLC scan. A lower .TimeSlice means less time doing database work every scan, so your PLC scans are faster, but your database operations take longer calendar-time-wise.
Yep, that's part of it. I used the code in DmSim, and it found the 16,000th random record in well under 2 seconds. Slapped the same code into a BRX running many other things, and it took over 4 minutes. I didn't tweak the slice there, as I didn't want to mess with the running processes. 100 us out of 5 ms is not much processing time I think much of my Sim success was also related to the PC's greater processing capacity?
-
Start at 5000. Depending on too high / too low, then check 2500 or 7500, then 1250, 3750, 6250, 8750, etc. Just keep dividing the range in half and the half with your value becomes the new range. Rinse and repeat till you test and hit the target value.
But how do I actually jump to the 5,000th row? Would I need to fix my CSV's row length so simple math can be used to determine the starting offset? I'm trying to keep this as flexible as possible. Also, the 10,000 number was arbitrary, it will change over time, 6,000, 8,000, etc.
-
Sorry, forgot yoy were doing file ops as opposed to having the whole table in BRX memory. More suggestions tomorrow.
-
If it's just a "batch import", you can definitely stick with the original plan. Sorry for the confusion.
Changing your .TimeSlice from 100 to 200 microseconds will definitely help. It just adds 0.2ms of PLC scan time. You mentioned you are around 5ms, so 5.2ms ain't bad. So, by doubling your work time per PLC scan, you cut the calendar time IN HALF. That's huge.
Bring up a Data View and stick some "profile" logic in there to see how much of an impact tweaking the .TimeSlice value. 100uSec is minimal PLC scan hit. The "sweet spot", depending upon what you are doing, is usually around 200-400uSec. Anything more doesn't help much (e.g. changing 400uSec to 500uSec only gives a 25% improvement in calendar time, while the same 100uSec hit from 100 to 200 gives you 100% improvement in calendar time)
-
But how do I actually jump to the 5,000th row? Would I need to fix my CSV's row length so simple math can be used to determine the starting offset? I'm trying to keep this as flexible as possible. Also, the 10,000 number was arbitrary, it will change over time, 6,000, 8,000, etc.
I think there might be several options.
Since the tables are coming to you electronically rather than somebody walking over and plugging in an SD card, could you have it written directly to memory rather than to file?
If not, could you move the data from file to memory after it gets sent to you?
If not, yes, fixing the line lengths for the CSV file should work.
If not that, then if you have the total file size in bytes, you could do the 50%-75%-25% thing byte-wise instead of record-wise, then from that byte position search forward and back for $0D$0A to find the nearest record boundary. Then do the inequality check on the record data.
Sorry for being obtuse, I had the database-in-memory idea stuck in my head from the earlier part of the thread.
-
Every time you get a new file, you need to parse the file, keeping track of the FILE OFFSET (it is in the File Handle structure).
Are all the error codes between 0 and 9999? Basically, create a Signed DWORD array 10,000 long, these will store the FILE OFFSET at the INDEX of that block, where the INDEX equals the ERROR CODE.
Say the file is as follows
1, Bad Stuff Happened
9002, Worse Stuff Happened
3, Even Worst Stuff Happened
I am guessing, but assuming CR/LF, the error text for error codes 1, 9002, and 3 would be at index
3 // B Bad...
27 // W Worse...
52 // E Even...
ErrorOffset would be a block, where index of 1, 9002, and 3 would have those values:
ErrorOffset0 0
ErrorOffset1 3
ErrorOffset2 0
ErrorOffset3 52
ErrorOffset4 0
ErrorOffset5 0
ErrorOffset6 0
ErrorOffset7 0
ErrorOffset8 0
ErrorOffset9 0
ErrorOffset10 0
...
ErrorOffset8999 0
ErrorOffset9000 0
ErrorOffset9001 0
ErrorOffset9002 27
ErrorOffset9003 0
...
ErrorOffset9999 0
Then, whenever you get error code, you would do
FILEOPEN
FILESEEK to position ErrorOffset[ErrorCode]
FILEREAD to CR/LF into SL0
FILECLOSE
This is how databases work. They have "index" tables to get to the actual file position quickly.
-
Since the tables are coming to you electronically rather than somebody walking over and plugging in an SD card, could you have it written directly to memory rather than to file?
If not, could you move the data from file to memory after it gets sent to you?
By memory do you mean RAM or into a Memory Block?
Currently, for testing, I'm copying the CSV to SD Card, then placing it in RAM before doing lookups.
If not that, then if you have the total file size in bytes, you could do the 50%-75%-25% thing byte-wise instead of record-wise, then from that byte position search forward and back for $0D$0A to find the nearest record boundary. Then do the inequality check on the record data.
This would be the most future proof, but also the most work to code. Less restrictions in the incoming data.
Sorry for being obtuse, I had the database-in-memory idea stuck in my head from the earlier part of the thread.
No worries, I really should have started a new thread... To me it dovetailed right into the old project.
-
Are all the error codes between 0 and 9999? Basically, create a Signed DWORD array 10,000 long, these will store the FILE OFFSET at the INDEX of that block, where the INDEX equals the ERROR CODE.
No, they are not sequential, and 15 digits, so that makes this tricky.
I am guessing, but assuming CR/LF, the error text for error codes 1, 9002, and 3 would be at index
Yes, they are CR/LF separated, I follow your logic, but afraid it will not work at the scale I'm dealing with. Thanks though, please keep the ideas coming!
-
By memory do you mean RAM or into a Memory Block?
Currently, for testing, I'm copying the CSV to SD Card, then placing it in RAM before doing lookups.
OK, I guess when you're saying RAM, you mean a file system in RAM. I meant to memory blocks in ram with defined record delineation.
This would be the most future proof, but also the most work to code. Less restrictions in the incoming data.
I don't know, that seems pretty easily doable.
-
OK, I guess when you're saying RAM, you mean a file system in RAM. I meant to memory blocks in ram with defined record delineation.
Yes, I meant as a CSV file in the RAM. Please tell me more, how do I create a memory block in RAM?
-
I think any memory block you create is in RAM.
-
I think any memory block you create is in RAM.
I don't think so, as Memory Blocks can be retentive, and they don't seem to take away from the 1024 KB the file manager shows the RAM's capacity to be.
-
Well, I'm sure the memory blocks aren't in flash, but battery backed RAM. If there's some value for 'RAM' somewhere, I think it's referring to RAM for the RAM file system.
-
MQTT broker where the topic name is the ID and the data is the error message.
Write a PC program to publish the topic ID key/message text (have it remove the old ones first)
Have the PLC subscribe to the topic.
-
The RAM disk is located in a 1MB chunk of SDRAM. The image register (blocks, heap) is located in a different area of SDRAM that is mirrored to a small battery backed SRAM. Neither exist in flash.
-
I have this working to my satisfaction. A BRX with a ~5ms scan time can find any one numerical string out of 16K in under 1/4 second. Once it's all said and done, it's really not that complicated... :P Thanks for the help!
-
I have this working to my satisfaction. A BRX with a ~5ms scan time can find any one numerical string out of 16K in under 1/4 second. Once it's all said and done, it's really not that complicated... :P Thanks for the help!
AWESOME! What are your scan times during the "lookup", and related - what .TimeSlice value did you end up using?
-
AWESOME! What are your scan times during the "lookup", and related - what .TimeSlice value did you end up using?
You really have to beat on the thing (ask it to find a value out of the bounds of the table, and let it keep trying after exhausting all possible searches) to even get the avg scan time to budge. And then worst case scenario it's 5.2 vs 4.8 ms.
TimeSlice makes virtually no difference, quite similar performance with 0, 100, 200 or 2000 as a value. They all take about 15ms per search iteration.
Keep in mind this is on a BRX that's actively running ~20 other programs at any given time, lots of communications, etc, so once on a more dedicated PLC it should have no problems at all. Quite the impressive little box of rocks!