I recently ran into a bit of a wall with how Zabbix is architected and it seems what I am trying to do is impossible (or at least so custom that it doesn't make sense to implement myself) with the current tools.
I monitor various 'files' in /proc for changes, and this data is not exactly useful for identifying what caused the entry to appear on its own.
For example, in /proc/net/packet, if someone were to run tcpdump, there would be an entry in that file that references the inode value of that socket.
I can make a zabbix item to pull the contents of /proc/net/packet and alert when it changes as well as record the contents themselves. But when it comes to triaging the reason, its very much manual, and even if I used scripts with remote commands, the automated results wouldn't be stored with zabbix.
Possible Solution:
I think it would be very useful to have some sort of mechanism to execute several zabbix agent items and store the result into the database for that host and link it to the trigger that activated it.
That way, when a trigger activates, there can be a script/series of items to do some automatic triage and when viewing the trigger afterwards, I can see the enriched data associated with it. (for the tcpdump example, I could then see what program name opened the socket and what user ran the program)
Example workflow:
If the contents of /proc/net/packet changes, then
execute the 1st agent item, iterating over the rows in the changed file contents, filter out the inodes in each row, then pass the result to the second agent item
execute the 2nd agent item, iterating over the inode list find the corresponding PIDs associated with the socket inodes in /proc/*/fd/ and pass to the third agent item
execute the 3rd agent item, iterating over the PID list, get the relevant Process name, command line, exe location, ect and then store the resultant data under the original trigger.
A few (disorganized) notes:
- I do realize that I can already do the "chaining of items" using a script run through the remote commands, but that doesn't store the output data with the trigger (or at all in the zabbix database as far as I am aware).
- The reason I wanted to use agent items only is because much of the processing for my own use-case can be done with the built-in agent utilities OR user parameters. If using just built-in agent items, the deployment is much easier without the need to deploy scripts to all hosts. With user parameters, its a more secure locked-down option as compared with remote commands.
- For certain operations, it may require running of scripts on the remote host outside of the agent for superuser privileges. So I don't want to limit this to just built-in agent items. (But the storage of those script outputs should be able to be stored).
- I also realize that trigger actions can run the commands necessary, but the output data is not returned to and stored with the trigger.
- Some ideas for other use cases:
- File Checksums - Storing a single copy of the file, when trigger activates, alert on checksum change, pull new file, compare changes and store changed lines with the trigger
- High utilization - Finding offending processes/connections that caused high cpu/ram/disk/io utilization on host
- Package versions - When package version changes, check CVE Database for matching version+name and put CVE score and description into trigger (could all be done locally on the zabbix server - offline)
- It seems the zabbix trigger actions / remote commands were more intended for administrative functions (like restarting services) than monitoring functions. It would be cool to see these functions have additional uses for security monitoring/incident response!
I'm sure this idea could use a lot of refining, let me know what you think!
I monitor various 'files' in /proc for changes, and this data is not exactly useful for identifying what caused the entry to appear on its own.
For example, in /proc/net/packet, if someone were to run tcpdump, there would be an entry in that file that references the inode value of that socket.
I can make a zabbix item to pull the contents of /proc/net/packet and alert when it changes as well as record the contents themselves. But when it comes to triaging the reason, its very much manual, and even if I used scripts with remote commands, the automated results wouldn't be stored with zabbix.
Possible Solution:
I think it would be very useful to have some sort of mechanism to execute several zabbix agent items and store the result into the database for that host and link it to the trigger that activated it.
That way, when a trigger activates, there can be a script/series of items to do some automatic triage and when viewing the trigger afterwards, I can see the enriched data associated with it. (for the tcpdump example, I could then see what program name opened the socket and what user ran the program)
Example workflow:
If the contents of /proc/net/packet changes, then
execute the 1st agent item, iterating over the rows in the changed file contents, filter out the inodes in each row, then pass the result to the second agent item
execute the 2nd agent item, iterating over the inode list find the corresponding PIDs associated with the socket inodes in /proc/*/fd/ and pass to the third agent item
execute the 3rd agent item, iterating over the PID list, get the relevant Process name, command line, exe location, ect and then store the resultant data under the original trigger.
A few (disorganized) notes:
- I do realize that I can already do the "chaining of items" using a script run through the remote commands, but that doesn't store the output data with the trigger (or at all in the zabbix database as far as I am aware).
- The reason I wanted to use agent items only is because much of the processing for my own use-case can be done with the built-in agent utilities OR user parameters. If using just built-in agent items, the deployment is much easier without the need to deploy scripts to all hosts. With user parameters, its a more secure locked-down option as compared with remote commands.
- For certain operations, it may require running of scripts on the remote host outside of the agent for superuser privileges. So I don't want to limit this to just built-in agent items. (But the storage of those script outputs should be able to be stored).
- I also realize that trigger actions can run the commands necessary, but the output data is not returned to and stored with the trigger.
- Some ideas for other use cases:
- File Checksums - Storing a single copy of the file, when trigger activates, alert on checksum change, pull new file, compare changes and store changed lines with the trigger
- High utilization - Finding offending processes/connections that caused high cpu/ram/disk/io utilization on host
- Package versions - When package version changes, check CVE Database for matching version+name and put CVE score and description into trigger (could all be done locally on the zabbix server - offline)
- It seems the zabbix trigger actions / remote commands were more intended for administrative functions (like restarting services) than monitoring functions. It would be cool to see these functions have additional uses for security monitoring/incident response!
I'm sure this idea could use a lot of refining, let me know what you think!
Comment