We had some strange alerts go off today and it looks like we have some orphaned items/triggers from LLD. As seen in the attached screen shots, the triggers we're seeing here don't match the actual template. Additionally, it's mixing up data points and hosts; and triggering on things that shouldn't even be associated to one another. It looks like they're probably just orphaned LLD records from changes that were made to hosts or templates over the years; and we can probably just delete the offending items/triggers.
We are currently running zabbix-server-mysql-3.0.31-1.el7.x86_64, with table partitioning for managing the large table sizes. Our server has over 300k items, collected at around 5000 NVPS, and we keep a full 30 days of history plus 2 years of trend data. Partitions are created daily to keep disk usage sane; and then cleaned up through MySQL events and stored procedures.
I think we just need to run something like the orphaned record cleanup script, but want to get a second opinion before we run something like this on our large database. https://github.com/mattiasgeniar/zab...d-data-cleanup
Any thoughts on running this cleanup script and/or other things we should be doing here? Maybe we can get away with only cleaning specific table(s)? Or, do we need to run the entire cleanup script to be safe?
In the below screen shots I have marked out the host names, as these contain sensitive information about our customers. I have marked "HOST1" in white, and "HOST2" in grey, so you can see how it's mixing up host information. One of the screen shots is the orphaned triggers, the other screen shot is the actual LLD template.
We are currently running zabbix-server-mysql-3.0.31-1.el7.x86_64, with table partitioning for managing the large table sizes. Our server has over 300k items, collected at around 5000 NVPS, and we keep a full 30 days of history plus 2 years of trend data. Partitions are created daily to keep disk usage sane; and then cleaned up through MySQL events and stored procedures.
I think we just need to run something like the orphaned record cleanup script, but want to get a second opinion before we run something like this on our large database. https://github.com/mattiasgeniar/zab...d-data-cleanup
Any thoughts on running this cleanup script and/or other things we should be doing here? Maybe we can get away with only cleaning specific table(s)? Or, do we need to run the entire cleanup script to be safe?
In the below screen shots I have marked out the host names, as these contain sensitive information about our customers. I have marked "HOST1" in white, and "HOST2" in grey, so you can see how it's mixing up host information. One of the screen shots is the orphaned triggers, the other screen shot is the actual LLD template.
Attached Files