Discovering similar files in folder to see if they are still being renewed
Introduction
For some of my clients I use syncthing so they will have a duplicate version off premises on my server.
Typically those are nightly back-ups.
I can turn versioning on in syncthing and it will automatically create a copy whenever the file is replaced on the client.
This is much more elegant way of doing it than writing scripts on each client that keeps older versions.
All this has not much to do with Zabbix, but this does.
If these nightly back-ups fail it may go unnoticed for sometimes months...
Of course you can let that client script report if anything goes wrong, but if the script itself is the culprit?
Anyway... those reports can be easily overlooked.
A long time ago I created a script, items and triggers that would return the age of the youngest file in a given directory conforming a minimal size and regular expression. A tandem script would report its name. These scripts are called "latestfileage" and "latestfilename".
This is working for years...
It is a bit of a hassle though to create the items and triggers each time that I create a new sync. Sometimes I don't even do it because of that because I deem that back-up not important enough.
Although I do that sometimes, I do not think it's correct.
You do need to know that I'm monitoring things for free a lot. No other firm this small with clients this small is doing this so extensively.
Using Discovery to create the items automatically
But back to the subject.
I thought of having these items automatically discovered and I succeeded.
In the case of syncthing the discovery process only needs 2 parameters.
I will show you the redacted output of it at the bottom
the script "fileagediscover" is give "/var/syncthing" and ".stversions" and it will parse all the .stversions folders withing /var/syncthing and look for files that have the same first 4 letters. It will even go 1 folder deep.
I can give a 3rd numerical parameter, so it will override the script's minimal size. Maybe I expand it one day so it will optionally take a 4th parameter so it looks deeper than 1 level into that folder.
It returns a complete folder, a regular expresion for the file filter and the minimal size it needs to be.
Regarding this subject I have all the answers,but I also have 1 question !
This question is about the minimal size.
I'm feeding a minimal size to my latestfileage and latestfilename scripts, so it will also triggers if the size is significantly smaller.
A botched back-up.
The size the discovery is giving is a rounded down value of the smallest file this far that meets the regular expression.
So far, so good.
Although back-ups tend to get only bigger and bigger this is most of the time a good thing.
I'm rounding it down, so it has a bit of play as well.
I only write a discovery script once a year and I only investigate its possibilities as far as I need them.
In the future the discovery script will return the same folder and same regular expression, but a different size and will create a new item with triggers.
How can I prevent this?
How can I prevent the creation when param#1 and param#2 stays the same, but param#3 is different?
If anyone is interested in this template and the accompanying scripts, I will post them after I have solved this.
Maybe it's far easier than what I did thus far, but for now I don't see it.
# fileagediscover /var/syncthing .stversions
For some of my clients I use syncthing so they will have a duplicate version off premises on my server.
Typically those are nightly back-ups.
I can turn versioning on in syncthing and it will automatically create a copy whenever the file is replaced on the client.
This is much more elegant way of doing it than writing scripts on each client that keeps older versions.
All this has not much to do with Zabbix, but this does.
If these nightly back-ups fail it may go unnoticed for sometimes months...
Of course you can let that client script report if anything goes wrong, but if the script itself is the culprit?
Anyway... those reports can be easily overlooked.
A long time ago I created a script, items and triggers that would return the age of the youngest file in a given directory conforming a minimal size and regular expression. A tandem script would report its name. These scripts are called "latestfileage" and "latestfilename".
This is working for years...
It is a bit of a hassle though to create the items and triggers each time that I create a new sync. Sometimes I don't even do it because of that because I deem that back-up not important enough.
Although I do that sometimes, I do not think it's correct.
You do need to know that I'm monitoring things for free a lot. No other firm this small with clients this small is doing this so extensively.
Using Discovery to create the items automatically
But back to the subject.
I thought of having these items automatically discovered and I succeeded.
In the case of syncthing the discovery process only needs 2 parameters.
I will show you the redacted output of it at the bottom
the script "fileagediscover" is give "/var/syncthing" and ".stversions" and it will parse all the .stversions folders withing /var/syncthing and look for files that have the same first 4 letters. It will even go 1 folder deep.
I can give a 3rd numerical parameter, so it will override the script's minimal size. Maybe I expand it one day so it will optionally take a 4th parameter so it looks deeper than 1 level into that folder.
It returns a complete folder, a regular expresion for the file filter and the minimal size it needs to be.
Regarding this subject I have all the answers,but I also have 1 question !
This question is about the minimal size.
I'm feeding a minimal size to my latestfileage and latestfilename scripts, so it will also triggers if the size is significantly smaller.
A botched back-up.
The size the discovery is giving is a rounded down value of the smallest file this far that meets the regular expression.
So far, so good.
Although back-ups tend to get only bigger and bigger this is most of the time a good thing.
I'm rounding it down, so it has a bit of play as well.
I only write a discovery script once a year and I only investigate its possibilities as far as I need them.
In the future the discovery script will return the same folder and same regular expression, but a different size and will create a new item with triggers.
How can I prevent this?
How can I prevent the creation when param#1 and param#2 stays the same, but param#3 is different?
If anyone is interested in this template and the accompanying scripts, I will post them after I have solved this.
Maybe it's far easier than what I did thus far, but for now I don't see it.
# fileagediscover /var/syncthing .stversions
Code:
{
"data": [
{
"{#PATH}":"/var/syncthing/Neeskens/Neeskens MicroHIS_Backup/.stversions/Huisartsenpraktijk",
"{#FILTER}":"AGD_.*",
"{#SIZE}":"103450"
},
{
"{#PATH}":"/var/syncthing/Neeskens/Neeskens MicroHIS_Backup/.stversions/Huisartsenpraktijk",
"{#FILTER}":"LOG_.*",
"{#SIZE}":"76830"
},
{
"{#PATH}":"/var/syncthing/Neeskens/Neeskens MicroHIS_Backup/.stversions/Huisartsenpraktijk",
"{#FILTER}":"MHX_.*",
"{#SIZE}":"1647690"
},
{
"{#PATH}":"/var/syncthing/Cruijf/MicroHIS-Backup/.stversions/Huisartsenpraktijk-Cruijf",
"{#FILTER}":"AGD_.*",
"{#SIZE}":"97310"
},
{
"{#PATH}":"/var/syncthing/Cruijf/MicroHIS-Backup/.stversions/Huisartsenpraktijk-Cruijf",
"{#FILTER}":"LOG_.*",
"{#SIZE}":"39960"
},
{
"{#PATH}":"/var/syncthing/Cruijf/MicroHIS-Backup/.stversions/Huisartsenpraktijk-Cruijf",
"{#FILTER}":"MHX_.*",
"{#SIZE}":"2881020"
}
]
}
Comment