Cheshire Cat Computing

The round robin database does not grow with time. It holds a fixed time window, and automatically expires data as it passes out of this. It rolls up data into summary archives as it goes -- hence, detailed data (5 min intervals) are only kept for 400 intervals (about 1 day), the next summary archive is 30-min intervals (kept for about 1 week) and so on. This is why the grahps are daily (interval=1 data point = 5min), weekly (6dp=30min), monthly (24dp=2hr), yearly(1day). After 2 days, you no longer have the 5-min detailed data stored, unless you have previously copied the .rrd file.

The rrd-archive function works by copying the current .rrd files to the archive directories. It keeps a snapshot of them as of that particular day -- the instructions warn you of the required disk space for this archiving!

You can tune how long to keep the individual archives for -- default is 30 days, but can be set to 90 if you want. Monthly archives (defined as any archive made on the 1st of the month) care kept for a defined number of months (default is 12). Yearly archives (1/1/xxxx) are kept forever.

If you want to keep only weekly archives, then only schedule rrd-archive to run on a Sunday night, not nightly. Set your expiredaily to be 90 to keep them for 3 months, or even more.

When rr-archive runs, it will delete any archives it finds that are past their expiry age.

If you extend the rrd archive to double then you can see last year's data. See the 'rrd-extend.pl' script in the routers.cgi extras directory, which does this.

However... what this does is extend the summary archives to double length. The 'daily' graph (5 min intervals) is extended to 2 days at this resultion, the 'yearly' (1 day interval) is extended to 2 years. It allows you to see 1 year ago today, but only at this resolution.

If you want to compare the daily graph of 1 year ago with the daily graph of today, then only way to do it is to either save the graph (the Archive function), save the rrd file (the rrd-archive function) or extend the rrd file to 365 times its normal length (which is not supported by routers.cgi).

The way the rrd database works is very handy in some ways (it takes care of summaries, and expiration of data automatically) but it can sometimes mislead people into believing that there is more data stored than in fact is the case. There is no way to store 1 year of data without either summarising and throwing away the detail, or having a VERY large file. Reading up on the copious documentation for RRDTool to understand more about how it works.

Your example above increases the 1stepAVG RRA to be long enough to cover a whole year. While this is in theory going to give what you want, in practice it causes a few more problems.

It will not allow the weekly or monthly resolutions to continue this far, as these are still their original size. More importantly, routers.cgi will get a bit confused in that it will think (if extendedtime = test has been set) that it is a double-length RRD and will try to provide 'last week' graphs (which may not work correctly, depending on how good RRDTool is at working out which RRAs to approximate with). Also, routers.cgi is not able to access this earlier data as it is not coded to cope with a sliding window (all graphs should be anchored to 'now' at the end)

It would take a lot of changes to make routers.cgi correctly and flexibly support this -- that is why I chose to go the route of using an rrd-archiver instead.

I'm trying to accurately measure the 95th percentile of traffic over a one month period for billing bandwidth usage. Any way to do this or do I just allow the automatic summarization to give me the 95th percentile?

How accurate is the standard MRTG readings over a month compared to using all 5 minute data to calculate 95th percentile?

Thanks,
Chris

In the README file there is an explanation of the 95th percentile problems. Basically, the more you (mean) average the data -- by going from daily, to weekly, to monthly etc -- the lower your perceived 95th percentile will appear, since the peaks of data have been smoothed out. The amount of inaccuracy depends on your data, how many brief unusual spikes it contained.
For example, if there were a couple of 5-min spikes every 2 hours, then you would have a big difference between monthly and daily. If you have a more constant pattern, then there would be little if any difference. You need some statistical analysis knowledge to understand the underlying reasons, and I tried to give that up 10 years ago

One possibility is to make big extensions to your MRTG .rrd file. If you extend the daily RRA to a whole month, then the 95th percentile calculations *should* spot this and pull out the data in the best resolution possible. This will take longer to calculate, but will be much more accurate. I will check the code to see if this will indeed be the case.

Steve

I thought there would be some issues with bursty traffic and the standard rrd db setup with only one day's time being kept for a shorter period than one month (the timeframe I need to look at).

I've resized my rrds for the interfaces I'm concerned with to extend to one year, it nearly multiplied my rrd file size by 10x.

I'm not worried about (too) much processor or disk space usage, but more about the accuracy of the measurement.

Thanks for checking into this.

Checked the code.

routers2.cgi will use the default, ie the highest resolution available for the time window specified. Therefore, if you have an RRA for the data that has cf=AVG, step=300 (1 data point) and a length of 1 year (400*24*12 data points) then it will use this for your yearly percentile (and total usage) calculations. This will be more accurate *but slower*. Naturally, since you are holding a years detail for the target rather than just a day, your .rrd file will go up in size considerably.

This will be the most accurate 95% percentile available, unless you decrease the Interval: on your MRTG collection to 1 minute (and then increase your RRA again, it will go up in size by a factor of 5 as step will now be 60 and the RRA length will be 5 times more)

The 95th percentile will be more accurate the smaller your interval is. Default MRTG displays yearly with a 1 day interval. By extending the RRA as you have, you get a 5-min interval. The smallest possible with MRTG is a 1-min interval. The payoff for this is in Disk and CPU.