Hello Experts,
We had an issue in our environment where a server which was getting decommissioned started throwing thousands of errors per second to its /var/adm/messages. Since we monitor the log file for the error it was all getting received at Zabbix server. Since these entries were matching the trigger condition and we have enabled "multiple alerting" almost all of them became problems and related actions/escalations were engaged. Escalation and housekeeper were running very hot for few days to churn the data.
What should be best practice followed to avoid such a kill? We looked at number of lines per second but it queued the read and sent after some time which was not much help in such scenario,
Thank for your suggestions
Pradeep
We had an issue in our environment where a server which was getting decommissioned started throwing thousands of errors per second to its /var/adm/messages. Since we monitor the log file for the error it was all getting received at Zabbix server. Since these entries were matching the trigger condition and we have enabled "multiple alerting" almost all of them became problems and related actions/escalations were engaged. Escalation and housekeeper were running very hot for few days to churn the data.
What should be best practice followed to avoid such a kill? We looked at number of lines per second but it queued the read and sent after some time which was not much help in such scenario,
Thank for your suggestions
Pradeep
Comment