Read/Write JSON to File System

[etElMadr]

Using JSONBuild to create a JSON Object for each record in a UDT, I'd like to save those objects in a file on the file system. Eventually, I may need to read those objects back into a UDT. The UDT may have the same fields, but not necessarly the same structure.

In short, its possible the UDT structure may change as my program improves but I don't want to lose that data. By saving it as JSON Objects, I'm hoping I can restore that data to the same UDT fields even if the UDT structure changes. Unless I misunderstand, I don't think a memory image will work in this case.

Rather than spinning my wheels experimenting, can you provide some direction on how to accomplish this using File System commands?

[Bolt]

If by "restoring that data to the same UDT" you mean open file, start a FOR/NEXT loop, do a FILEREAD, and JSONPARSE in the loop, yes it should work.  It will take an initial manual assignment of the corresponding fields if you change the UDT.

[franji1]

Yes, stick the looping algorithm it in its own PROGRAM code block with a .TimeSlice of 0 and your PLC scan shouldn't take much of a hit.

Reading large amounts of text and parsing it is VERY time consuming (relative to a PLC's I/O scan), hence the reasoning behind a .TimeSlice of 0.

Will you be attempting to find 1 specific record (i.e. you have a "key" for each JSON record), or are you looking to fill up a block of your UDT with large amounts of data stored on disk, or possibly a query, e.g. all the records with a "Date" JSON field value of "12/25/2021", for example (i.e. MULTIPLE UDTs need to be read back into memory)?

EDIT: ALL of this is VERY possible with Do-more BTW.

[etElMadr]

What I'm envisioning is backing up the data to the uSD card in the form of JSON Objects, the possible restoring it to the same or modifed version of the UDT. Each object has a unique key/id.

The current UDT might have a structure of say:
|dev|type|state|
from which we would JSONBuild each record to say {uniqueID:1, type 1, state:0}, {uniqueID:2, type 1, state:1}

Some future UDT might have a structure of say:
|dev|addr|type|level|

So, I'd want to restore the data from the JSON file to the future UDT, mapping the stored data to the corropsonding fields in the future UDT. Its more than likely this would be a batch operation, not querying for specific individual records.


I get how to read the UDT to JSONBuild, JSONParse to write the UDT. I'm actually doing quite a bit of that already. I haven't done any File operations (other than Backup) as of yet, so I'm trying to figure-out where to start. Its not clear to me yet how to write the file. For example, say we've already written the UDT to uSD in the form of comma-delimitted JSON Objects, but the data in the UDT for ID/Key 5 changes. I'll need to update that record in the uSD File also. Do I need to read the entire file into memory, update the data for ID 5, then rewrite the file? 

[franji1]

I would use JSON format, not CSV, ESPECIALLY since you will be adding fields in the future.  JSON lets you easily specify any fields you want (or not want) and then then JSONPARSE would either find the field or not.  CSV gets messy in that you have to process missing fields by hand (e.g. empty comma separators? value1,,,value4,,
You also must handle the numeric/text parsing for each field by hand.  JSONBUILD and JSONPARSE formats it as part of the instruction - you say whether it's numeric vs. text, and it does all the work.

Read/Modify/Write in text form gets a little hairy.  You are better off using 2 files, the original one you use as the source to the the "edited" one.  When you're done, copy the edited one back on top of the original source.

Code: [Select]

FILEOPEN Original for reading
FILEOPEN Edited for creation // do this on the RAMDRIVE???
FOR
FILEREAD Original to string (use CR/LF as end of JSON record)
JSONPARSE string all fields to corresponding temp values
see if need to tweak any key/value pair, if so, modify temp value(s)
JASONBUILD all temp values back to string
append CR/LF to string (STRPRINT append "$0D$0A", no whitespace separator)
FILEWRITE string to Edited
NEXT
FILECLOSE Original
FILECLOSE Edited
FILECOPY Edited to Original


[etElMadr]

OK. Reading to ramdisk makes sense I understand what you're saying about using pure JSON, but a JSON Array of the JSON Objects would be HUGE! I might have, as example, an Array of 512 Objects with 8 key-pairs. So, I don't think we can put that in memory and JSONParse. I was looking at the FileRead instruction with "Complete when" set with a comma or tab delimiter, but I guess 0x0D 0x0A would work better.

I assume when before I FileWrite, I'dd need to STRPRT the JSON OBject with 0x0D 0x0A

[franji1]

OK. Reading to ramdisk makes sense I understand what you're saying about using pure JSON, but a JSON Array of the JSON Objects would be HUGE! I might have, as example, an Array of 512 Objects with 8 key-pairs. So, I don't think we can put that in memory and JSONParse. I was looking at the FileRead instruction with "Complete when" set with a comma or tab delimiter, but I guess 0x0D 0x0A would work better.

I assume when before I FileWrite, I'dd need to STRPRT the JSON OBject with 0x0D 0x0A

You will be WRITING to RAM Disk (highly volatile).  READING from the SD card (long term flash), but in the opposite order

Your UDT block in the PLC is NOT JSON block.  JSON is just the "file storage" format (human readable also!).  The rest of your program should be dealing with your block of UDTs, NOT JSON (only this ONE code block deals with JSON, nothing else).

I think I need to get a better understanding of the "big picture" - what I described is a batch read everything one at a time, update as you read each one, then write each one back out, but do it for everything.

I am unsure when you want to READ all the records from the flash file stick it in the UDT block?  Is the media being edited on a PC then needing to be reloaded in your PLC?  What's the purpose of the flash drive copy (the UDT block can be retentive in battery backed RAM).

Likewise, I am not sure when you want to UPDATE all the records from your UDT block?  Once a day?

Not sure when you want to write your UDT back to the flash?  Continuously?  Once a day?

OR are there thousands of records and you don't have enough memory to store them all, and you need a large database and want to tweak them one at a time as they change???  This is a completely different problem.

[etElMadr]

You will be WRITING to RAM Disk (highly volatile)

Yes, I meant Reading [from uSD and writing] to RAMDrive.

Your UDT block in the PLC is NOT JSON block

If we have a large JSON Array of JSON Objects, then doesn't the FILEread need to put that Array into a memory location to JSONparse? Would this not be a huge string, too large for a typical string memory block? Thats why I'm thinking that FILEread is reading one JSON Object at a time (not the whole Array).

This is a data back-up, as opposed to an image backup. Its possible that the UDT this data normally lives in to be processed by the ladder logic may be [re]structured as the program is further developed. However, much of that data will still be "good". Storing a backup of that data in a key-pair format (JSON Object) will allow me to restore that good data to some future restrucuted UDT that uses the same [or] simlar data fields.

These backups should happen soon after the data in the UDT chages. This may or may not happen often, but will likely normally happen in batches... a few or many individual UDT records will be updated by a human over a few minutes or hours, one record at a time, then not touched for possibly months. I've already got a mechanism in place to track when these changes occur, which I'm using for other reasons. Ideally, the changes are written to uSD immediatly as each UDT record is modifed (this is not a requirement though; we could batch write the changes. Batch writting will require a new mechanism to track changes, thats not a big deal and I know how I would do that).

Restoration will happen in a single batch, very rarely. This is a back-up, which ideally we'll never need. I suppose the data could be modifed on  PC then restored, though thats not what I had in mind. The Flash Drive copy is a back-up, in the case the UDT structure is modified and therefore the data is lost (or, maybe the battery on the PLC fails and loses retention, or whatever unexpected thing might happen.)

Right now, I don't foresee any memory shortage issues with the PLC necessating the data be stored in the uSD. If anything, I have scantime issues with processing the UDT data, but thats why I'm thinking the UDTs may need to be restructured in the future.

Sorry for the lack of clarity.

[franji1]

This definitely makes sense.

What you call a JSON Object is really just a UDT with a bunch of numeric fields, no strings or other structures (although we are adding nested structures to the nest version due out soon).

So, all you really need are 2 PROGRAM code blocks
LoadFromDisk
WriteToDisk
with the UDT block always being the intermediary (i.e. it is the Output of the Load... and the Input of the Write...)?

These code blocks are edge triggered from $Main (or some other PROGRAM) based on some event.

As an aside, if you look at the JSONBUILD instruction editor, it lets you provide a UDT as a template and it then fills out the fields of the JSONBUILD based on that UDT (see the attached JSONBUILD example generating JSON object of the "most useful" fields of $Now (SDT0).

[etElMadr]

LOL, I'm glad you can make sense of what I'm writing!  I just re-edited a few typos in my last post.

The PLC is processing data in a UDT. I'm trying to back up that data to in JSON format to the uSD card, that can be read back into the UDT or some restructed future version of it. I don't think I can store it in pure JSON (an array of objects), but instead delimited lines of JSON Objects. Anyway, I think I have enough guidiance to figure out a reasonable approach to this now. 

Thank You

[franji1]

I don't think I can store it in pure JSON (an array of objects), but instead delimited lines of JSON Objects.

YES!

If you were interfacing with a Server, you may consider that, but internally as a text file in a PLC that doesn't have 64 Gigabytes of memory like a server, K.I.S.S. principle applies, and make it cr/lf delimited lines of individual JSON objects (vs. one big JSON array object that you would have to read in COMPLETELY).