BRX File System Commands

[JeffS]

What I have been doing is logging some data to the SD card using the file system commands.  My system is structured so that I log data to NotSent/1/1.txt then increment my file number until I get to file number 1400, then I increment the folder and start back at file number 1.  I also send this data to a server and once successful I move the logged file from my Notsent folder to the Sent folder.  I only plan to log for 100 days, so no need to go another folder deep, so I will stop logging at folder 1400.

What I have noticed recently is that the time it takes to generate and move these files is taking longer and longer.  One of my locations has data logging every 7 seconds or so and recently logging the data can't keep up with the data being generated.  I started noticing the slow down once I got up to 150 folders.  Is there a better file structure I could be using that wouldn't experience the slowdown I am seeing?  Is the slowdown to be expected?  Is there a way to estimate the slowdown at 1000 folders?

[Greg]

I don't know if "the slowdown [is] to be expected," or if "there is a way to estimate the slowdown at 1000 folders," but is there a reason you have to have so many files and folders? If you are using the FILELOG instruction, you can log tons of log entries in a single file. You don't have to just enter one log entry per file (which sounds like what you might be saying). Each execution of the FILELOG instruction will just simply add another line of data to the already-existing file, and this file can be huge by itself. So, unless there is some reason why you need so many files and so many folders, this would be "a better file structure" plan than having so many files and so many folders. You have to remember that you are using a PLC, and PLCs have PLC stuff to do... that is, resolve logic and update I/O. The PLC is not a PC where file handling is a primary function. The PLC's primary function is to execute ladder code in a very efficient manner.

[BobO]

There is a limit to the number of files/folders per directory. Don't remember what it is, I'd have to jump into the code, but the limit was added specifically to prevent the slowdown from becoming unbearable.

[JeffS]

The reason I went with individual files was because my logged data can vary greatly in length, and rather than keeping track of where I was in a 500mb file I just have to open and read out the contents of a file that I send to my server.    If I was to use this method of logging I would be adding to the end of the file as I log new data, but I am also reading from the end of the file to send to my server and then moving forward in the file based on each the length of each log instance that I would have to also log per instance.  This is doable, but would require rewriting how I handle things currently.

So would opening and closing a 500mb file be better than having my nested 500,000 - 1kb files?

[JeffS]

Possibly related, on the PLC where I have the large file numbers I noticed that sometimes an index value that I am logging (Incremented after each log entry) was sometimes recorded as zero, but then later on would record correctly.  The index value is not being set to zero anywhere in the program, and the count picks back up where it should so it is still incrementing correctly even though it is being logged as a zero.

This has me worried as it is intermittent and sometimes log correctly after having logged as zeros.   Part of the same instruction has a timestamp and those do update correctly with each record.  Any idea why this would be happening? 

I find it more than a little alarming as it means I can't trust the data being logged.  Second pic is of the data sent to the server, the red lines show where the data had been coming in with a 0 index (first column) and so aren't shown.

[BobO]

Possibly related, on the PLC where I have the large file numbers I noticed that sometimes an index value that I am logging (Incremented after each log entry) was sometimes recorded as zero, but then later on would record correctly.  The index value is not being set to zero anywhere in the program, and the count picks back up where it should so it is still incrementing correctly even though it is being logged as a zero.

This has me worried as it is intermittent and sometimes log correctly after having logged as zeros.   Part of the same instruction has a timestamp and those do update correctly with each record.  Any idea why this would be happening? 

I find it more than a little alarming as it means I can't trust the data being logged.  Second pic is of the data sent to the server, the red lines show where the data had been coming in with a 0 index (first column) and so aren't shown.

We will do a bit of stress testing, but the large number of files is well outside the intent of this feature. These PLCs are small memory constrained devices, and despite our best efforts to cram more and more in them, there is a limit to how much work they can do and still efficiently run your machine. When testing the SDCard file system, I did notice a significant slowdown on creating new files, and determined that the length of an individual directory was where the issue was. I was thinking the limit was 1000 files, but looking at the code, the comment says 1000 but the code says 3000. Either way, that's there because there was a big slowdown.

It should be more efficient to use less/bigger files, and it isn't necessary to open and close every time. The log instruction does that to simplify the implementation and keep it cleaner for the user, but if you plan to log something continuously, you can keep the file open. This will require that you write your own logging code, but that's just STRPRINT and FILEWRITE.

As I said, we'll do some stress testing and see what we can learn, but please understand, this isn't a PC or a cell phone with virtually unlimited RAM and processor performance, there are definitely some compromises there.

[JeffS]

So what I do have is a custom logging logic using STRPRINT and FILEWRITE commands.  In my code I only have the file open while I am writing the file or reading the file.  What are the downsides to having the log file open and the PLC losing power?

[BobO]

So what I do have is a custom logging logic using STRPRINT and FILEWRITE commands.  In my code I only have the file open while I am writing the file or reading the file.  What are the downsides to having the log file open and the PLC losing power?

Like any file system,  pending writes can be lost. I didn't write the file system, so I'm honestly not sure how much is being cached. I doubt it's very much, but it's probably something. Closing the file will definitely force a flush.

With that said, I threw together a small test:
SG1 open file for append
SG2 write to file
SG3 close file
SG4 JMP to SG1

Running flat out, it takes anywhere from 20ms to 200ms per pass to complete. I got the file to about 2M, and saw no increase in average time per pass. I also saw no gaps in the data.

So...
1) use bigger/fewer files
2) be sure that the data you are writing to the file system isn't changing until after the write completes
3) (you know this, I'm sure) be sure that none of the file operations if being interrupted before completion...i.e. uses stages to sequence the file ops.

[BobO]

Log is nearing 10MB without any increase in access time...

[BobO]

Ran over night. Log is up to 80MB. No slowdown and the log appears to have to no gaps.

[JeffS]

I finally had a chance to try this method out and let it run for awhile.  I let it run overnight and when I started the file was writing in about 0.5 seconds logging 3200 bytes of data each write.  It was still basically this speed at 5MB file size, but this morning it is a 109MB file and takes 2 seconds to write 3200 bytes. 

In your post you mentioned not seeing any slowdown at 80MB so figured I might have some issue since I am seeing a significant increase in write time.  Since this was overnight I don't know if it was a gradual increase or a sudden increase in write time.  Any thoughts as to what I should look for?

Edit: I am using the FILEWRITE command from a byte buffer as pictured.  I am timing while my write program is active which really just includes some logic to index to a new file number/folder when file size or number limits are reached.  I was going for a 200MB file before indexing to a new file.