Discovering a directory tree to monitor latest backup files

[Glide]

I have a tree structure containing backups for dozens of databases and other services, laid out like:

mnt
  backup
    host1
      latest
        2021-12-20_06h32m.monday.sql.gz   <- File age must not exceed 1 day
      daily
        2021-12-20_06h32m.monday.sql.gz
        2021-12-19_06h33m.sunday.sql.gz
        2021-12-17_06h43m.saturday.sql.gz
      weekly
        ...
    host2
      db_backup_20211219230000.tgz        <- File age must not exceed 1 day
      db_backup_20211218000000.tgz
      db_backup_20211217000000.tgz
    host3
      db
        latest
          2021-12-20_04h42m.monday.sql.gz <- File age must not exceed 1 hour
        daily
          2021-12-20_04h42m.monday.sql.gz
          2021-12-19_04h43m.sunday.sql.gz
      upload
        latest
          2021-12-20_03h52m.monday.sql.gz <- File age must not exceed 2 hours
    ...

As you can see, this is an heterogeneous structure
I need to monitor the age of the latest backups in specific directories, for example:

• /mnt/backup/host1/latest/ Here latest file age must not exceed 1d
• /mnt/backup/host2/ Here latest file age must not exceed 1d
• /mnt/backup/host3/db/latest/ Here latest file age must not exceed 1h
• /mnt/backup/host3/upload/latest/ Here latest file age must not exceed 2h

At first I manually created all required items, triggers & graphs for them one by one, but as you can imagine it's quite heavy to maintain.

So I though about LLD to discover all backup files with their respective age/timestamp, and context macros to specify the max age for each target directories, like:

• {$BACKUP_MAX_AGE:/mnt/backup/host1/latest/}: 1440 (1 day in minutes)
• {$BACKUP_MAX_AGE:/mnt/backup/host2/}: 1440
• {$BACKUP_MAX_AGE:/mnt/backup/host3/db/latest/}: 120
• {$BACKUP_MAX_AGE:/mnt/backup/host3/upload/latest/}: 60

Questions now:

• Do you think I'm going on the correct direction ?
• Should I use a system.run[find /mnt/backup] Key to find the files in my LDD ?
• Should/can I use context macros to filter LDD results ?

[Glencoe]

I think you are heading in the right direction. Just FYI, Zabbix 6.0 will make such LLD discoveries much easier with the new item vfs.dir.get: https://www.zabbix.com/documentation...s/zabbix_agent

[Glide]

Thanks Glencoe that's interesting, I'll take a closer look

[Glide]

Here is how I did:

• On the host, create a directory with symlinks to the latest backups dirs
  

  mkdir /mnt/backup/latest
  ln -n /mnt/backup/host1/latest /mnt/backup/latest/host1-db
  ln -n /mnt/backup/host2 /mnt/backup/latest/host2-db
  ln -n /mnt/backup/host3/db/latest /mnt/backup/latest/host3-db
  ln -n /mnt/backup/host3/upload/latest /mnt/backup/latest/host3-upload

  That'll be the discovered list
• On Zabbix, create a Discovery rule for the host, Key:
  

  system.run[ls -d /mnt/backup/latest/*]

• Preprocessing with a Javascript step:
  

  const lld = [];
  const lines = value.split("\n");
  for (i = 0; i < lines.length; i++) {
    const row = {};
    row["{#SERVICE}"] = lines[i].split('/').pop();
    row["{#DIR}"] = lines[i];
    lld.push(row);
  }
  return JSON.stringify(lld);

  This will store the name of the symlink into {#SERVICE} and the absolute path on {#DIR} for each directory found
• Add an Item prototype
  

  Name: Backup age {#SERVICE}
  Key: system.run[echo $(($(date +%s) - $(date +%s -r `ls -td {#DIR}/* | head -n1`)))]
  Unit: s

  "Unit: s" will make sure the numbers are properly formatted
  This will get the age (in seconds) of the latest backup file for each "service" discovered
• Add a Trigger prototype
  

  Name: Backup too old {#SERVICE}
  Expression: last(/<host>/system.run[echo $(($(date +%s) - $(date +%s -r `ls -td {#DIR}/* | head -n1`)))])>({$BACKUP_MAX_AGE:"{#SERVICE}"}*60)

  Replace <host> with the name of the host (The usual //item doesn't seem to work here)
  This triggers when a backup is considered too old
• On the host configure the default Macro
  

  Macro: {$BACKUP_MAX_AGE}
  Value: 1500

  This macro defines the maximum backup age in minutes (Here ~1 day)
• If needed for specific services, context macros can be defined
  

  Macro: {$BACKUP_MAX_AGE:host3-db}
  Value: 140
  
  Macro: {$BACKUP_MAX_AGE:host3-upload}
  Value: 80

  These services will need to have backups not older the 140 & 80 minutes respectively
• Eventually it can be interesting to monitor all the backup ages with a graph, so just throw all the discovered items in a normal graph, putting short frequency backups on the right axis can be interesting too