Trend History Shorter Than Desired

[Controls Guy]

I have a trend with one analog and one discrete pane with the history depth set to 20MB/element and 1 second logging.    The analog pane has one R register and is discarding old data sooner than I want.    I just checked and I currently have about 9:09 of data.   At one second, that's 549 minutes * 60 sec/min * 4 bytes per record, or 131,760 bytes, assuming no overhead per record.    If I click for a snapshot, the stated time resolution is ms, and it would make sense that for the graphic you'd want finer than 1 second resolution, so I assume the frequency setting is only for external logging.   But if that's the case, how should I estimate required storage volume for a desired trend history length?    I don't remember ever having it come up short, but I have no idea what I had the depth/element set to in previous trends.

[BobO]

I have a trend with one analog and one discrete pane with the history depth set to 20MB/element and 1 second logging.    The analog pane has one R register and is discarding old data sooner than I want.    I just checked and I currently have about 9:09 of data.   At one second, that's 549 minutes * 60 sec/min * 4 bytes per record, or 131,760 bytes, assuming no overhead per record.    If I click for a snapshot, the stated time resolution is ms, and it would make sense that for the graphic you'd want finer than 1 second resolution, so I assume the frequency setting is only for external logging.   But if that's the case, how should I estimate required storage volume for a desired trend history length?    I don't remember ever having it come up short, but I have no idea what I had the depth/element set to in previous trends.

It stores on change, not frequency, but higher frequency opens the door to more changes.

[franji1]

Your assumption of 4 bytes per record is not correct.  It's our record, not simply bits of data (i.e. we store high resolution time stamp along with the value since we only log deltas, 2 QuadWords, so 2*4*2 or 16 bytes, not 4 bytes per delta).

I recommend you set it to 100MB to get what you want time-wise.

[Controls Guy]

Your assumption of 4 bytes per record is not correct.  It's our record, not simply bits of data (i.e. we store high resolution time stamp along with the value since we only log deltas, 2 QuadWords, so 2*4*2 or 16 bytes, not 4 bytes per delta).

Right, I assumed there was some overhead but didn't know how much, so I was calculating the ratio between the 131.7KB and 20MB as a sanity check.  In this case, assuming the overhead wasn't 160:1 over the data, which led me to believe the 1 sec interval was only for external logging.

I recommend you set it to 100MB to get what you want time-wise.

OK will do!

[Controls Guy]

It stores on change, not frequency, but higher frequency opens the door to more changes.

Guess I had to chew on that one a bit.

So if I understand you, you're saying that the logging frequency setting isn't wholly deterministic to history storage but they are related.   Like for example, maybe the trend only checks 1 (or x, like maybe 5 or 10) times per frequency interval for changes in the value.

The answer to that may also be the answer to: How do you determine when a 32-bit floating point value changes?   Depending on the source of the number it might be a bit quantized (like an RXn value scaled from a 15-16 bit integer WXn), but some reals are going to look like they changed every time you look at them so you'd have to have a look interval or a change threshold.  How do you guys work that?

[franji1]

Guess I had to chew on that one a bit.

So if I understand you, you're saying that the logging frequency setting isn't wholly deterministic to history storage but they are related.   Like for example, maybe the trend only checks 1 (or x, like maybe 5 or 10) times per frequency interval for changes in the value.

The answer to that may also be the answer to: How do you determine when a 32-bit floating point value changes?   Depending on the source of the number it might be a bit quantized (like an RXn value scaled from a 15-16 bit integer WXn), but some reals are going to look like they changed every time you look at them so you'd have to have a look interval or a change threshold.  How do you guys work that?

Think MQTT Broker.  The Comm Server is working very hard polling the PLC based on a View's set of Recurrent Link Requests at some View specific interval (e.g. 20ms for Ladder/Trend or 50ms for Data View or whatever).  The View doesn't get any notifications EXCEPT when the Comm Server sees a change in value - actually just a change in the bits (no scaling/no real compare - just data bits changed), at which time it sends a Notification to the View that the value for that specific Link Request is different.  The CommServer does not know every scan's value change, but at a poll rate that is good enough.  All instances of Designer share the same Comm Server - so you could have 3 Designer sessions talking to the SAME PLC using 3 different Comm Links and it works like a hose.

What the View does with it is View specific.  For Trend View, it literally just pushes the data to a large linked list of quantized sets of 512 datapoints of 16 BYTEs, and throws out (deletes) the "end" group of 512 once it reaches your specified limit (20MB, 100MB, 1MB).  Very streamlined - no plotting - no graphics - just add the data point to the current preallocated chunk of 512 points.  Each data point stores the current high precision timestamp and the data value.  That's all Trend View does with its notification.  The DRAWING of that is completely asynchronous to the data it's collecting - it refreshens with the view ports' time interval since the last refresh - when it reaches the right end, it redefines the left edge time to half way, and redraws the whole thing.

[Controls Guy]

So a real value will generate 16B of history every time there's any change, no matter how small?   So a real that actively uses all 24 bits of significand/sign could easily generate 800B/sec of storage (because the LSBs are pretty small in value).   That helps me estimate storage requirements for a given length of history.

But then what does BobO mean when he says that a shorter interval (I assumed he meant the logging interval) opens the door to more value changes that would get recorded?   In your explanation is sounds as if the history bandwidth is completely independent of the logging frequency setpoint.

[BobO]

So a real value will generate 16B of history every time there's any change, no matter how small?   So a real that actively uses all 24 bits of significand/sign could easily generate 800B/sec of storage (because the LSBs are pretty small in value).   That helps me estimate storage requirements for a given length of history.

But then what does BobO mean when he says that a shorter interval (I assumed he meant the logging interval) opens the door to more value changes that would get recorded?   In your explanation is sounds as if the history bandwidth is completely independent of the logging frequency setpoint.

You're overthinking this.

We look for changes at the poll frequency. We can only find changes when we poll. A change is defined as LastVal != CurVal.

Sample 1 (0 ms): Val = 1.2345 (Stored)
Sample 2 (+50 ms): Val = 1.12346 (Stored)
Sample 3 (+100 ms): Val = 1.2346 (Not Stored)
Sample 4 (+150 ms): Val = 1.2347 (Stored)

If the data is changing frequently LastVal will usually be different than CurVal, so buffer gets burnt fast.

[BobO]

And as an aside, the new MROUND would make it very easy to reduce the precision, thus increasing the likelihood that LastVal == CurVal.

[franji1]

So a real value will generate 16B of history every time there's any change, no matter how small?   So a real that actively uses all 24 bits of significand/sign could easily generate 800B/sec of storage (because the LSBs are pretty small in value).   That helps me estimate storage requirements for a given length of history.

When you say "storage", are you talking about RECORDING/EXPORT?  Or just normal Trend View session "storage"?

But then what does BobO mean when he says that a shorter interval (I assumed he meant the logging interval) opens the door to more value changes that would get recorded?   In your explanation is sounds as if the history bandwidth is completely independent of the logging frequency setpoint.

Polling Interval (20ms vs 50ms vs. whatever).  This is configurable only via the INI file.

What do YOU mean by History Bandwidth?  If you are truly LOGGING to a FILE - it's like 1Gigabyte (Trend Archive)?

Please answer the first question so I understand what you mean by logging (RECORD/EXPORT to file, or just the View "log", i.e. 20MB vs. 1MB vs. 100MB)?

[Controls Guy]

When you say "storage", are you talking about RECORDING/EXPORT?  Or just normal Trend View session "storage"?

Polling Interval (20ms vs 50ms vs. whatever).  This is configurable only via the INI file.

Oh OK, I think you answered everything there.   I've tried to use "logging" to indicate non-volatile storage to disk, which is what I believe you guys refer to as "logging".    Storage for the trend itself that will disappear when DMD or the trend is closed I'm trying consistently to call "history".    That's what was short for me, and what I was looking for a means to estimate, and some of the answers had me thinking the potential log rate for changes in that internal log might be affected by what the dialog calls the logging interval.

I see from your comment about the INI file that the Comm Server poll rate per window type is settable but fixed in real time, and variations due to changing THAT setting are what BobO's talking about, rather than some interaction with the disk file logging rate in the Trend View options dialog.

[franji1]

Good - now I understand what you mean.

The (VCR) RECORD mechanism for CSV format updates is fixed at 1 record per second.  It writes ALL the values out in a single CSV row.
The RECORD mechanism for TREND ARCHIVE format (.trarc) writes ALL the Trend View entries AS THEY HAPPEN - although it caches up some before writing the bunch to disk.
The RECORD mechanism has "no" limitation based on the Trend View Point storage (e.g. 20MB or 1MB or 100MB) - it WRITES AS IT GOES.  There is a 1G? limit on the IMPORT side - but on the RECORD side - it just keeps appending, so disk space? is the limit?

The EXPORT mechanism can only export what the VIEW has STORED - so ITs data set is limited by the current data set range.
The EXPORT mechanism for CSV format lets you specify the PERIOD of the row of data in the CSV file (e.g. a data row once every 1 second, every 1 minute, every 42 milliseconds, whatever)
The EXPORT mechanism for TREND ARCHIVE format (.trarc) writes all VIEW data points.

I hope this helps.

[Controls Guy]

Yup, I think I have it clear now.  Plus, I didn't know I could adjust the poll rates in the INI file, that'll be helpful.

[franji1]

Yup, I think I have it clear now.  Plus, I didn't know I could adjust the poll rates in the INI file, that'll be helpful.

Just sent you an email.  Good luck!

[franji1]

And as an aside, the new MROUND would make it very easy to reduce the precision, thus increasing the likelihood that LastVal == CurVal.

Here's what BobO was talking about.
https://community.automationdirect.com/s/question/0D53u00007LvDUyCAN/request-add-floor-ceiling-and-mround-to-the-math-instruction

You can emulate it in 2.9 by doing the following
MROUND(number, multiple), where R1 is number and R2 is multiple:

(ROUND(R1 / R2) * R2)

MATH R42 "(ROUND(RX0 / 0.01) * 0.01)"
will give you the value of RX0 to the nearest hundredths in R42.