Random trigger issue on Zabbix 6.4

[poloniumx]

Hi everyone!

I'm currently facing an issue with Zabbix version 6.4 regarding the detection of errors in the SMTP logs of my Linux servers.
I have set up a template that includes items and triggers to monitor SMTP logs on my Linux servers(hosts), but I’m noticing random behavior in the trigger activation (irregular detection and reporting).

Sometimes, Zabbix detects the errors within the first second, but at other times, it takes several long minutes before the SMTP error log is recognized by Zabbix and reported as a "problem" in the monitoring dashboard. This random delay makes real-time tracking unreliable.

I’ve already tried several adjustments, including duplicating and modifying items to optimize detection, as well as changing the Zabbix agent’s check frequency, but with no success.

Do you have any suggestions for optimization or any ideas to improve faster detection?

Thank you in advance for your help

[cyber]

and you forgot to add your current item config... how can we suggest anything, if we only know that you are trying to read SMTP error logs...

[poloniumx]

Hello,
There is a Zabbix agent activated on each of my hosts (SMTP servers).

Below are 2 examples of SMTP error logs I'm looking for :

- to=<[email protected]>, relay=gmail-smtp-in.l.google.com[64.233.166.27]:25, delay=69272, delays=67519/1752/0.11/0.35, dsn=5.7.1, status=bounced (host gmail-smtp-in.l.google.com[64.233.166.27] said: 550-5.7.1 [62.4.31.162 19] Gmail has detected that this message is likely 550-5.7.1 suspicious due to the very low reputation of the sending domain. To 550-5.7.1 best protect our users from spam, the message has been blocked. For 550-5.7.1 more information, go to 550 5.7.1 https://support.google.com/mail/answer/188131 ffacd0b85a97d-378956cffe3si5957601f8f.912 - gsmtp (in reply to end of DATA command))

- to=<[email protected]>, relay=eur.olc.protection.outlook.com[52.101.68.1]:25, delay=1173, delays=838/335/0.16/0.04, dsn=5.7.1, status=bounced (host eur.olc.protection.outlook.com[52.101.68.1] said: 550 5.7.1 Unfortunately, messages from [IP] weren't sent. Please contact your Internet service provider since part of their network is on our block list (S3150). You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. [Name=Protocol Filter Agent][AGT=PFA][MxId=11B9A2E50865927E] [DU6PEPF00009525.eurprd02.prod.outlook.com 2024-09-13T12:04:33.166Z 08DCD17289A171E5] (in reply to MAIL FROM command))


Sometimes the detection is immediate, but at other times it takes several minutes or even several hours.
I even tried creating multiple items to detect the same error because one item alone is not enough, but even 3 items don't work.

For Microsoft:
Item01:
log.count[/var/log/maillog,(?i)(?=.*\s*Unfortunately\s*)(?=.*\s*block list\s*)(?=.*\s*S3150\s*)]
Item02:
log.count[/var/log/maillog,(?i)unfortunately.*part of their network is on our block list,skip]
Item03:
log.count[/var/log/maillog,(?i)unfortunately.*block list.*S3150,skip]

For Gmail:
Item01:
log.count[/var/log/maillog,(?=.*550-5\.7\.1)(?=.*Gmail)(?=.*suspicious)(?=.*very low reputation)(?=.*spam)(?=.*blocked)]
Item02:
log.count[/var/log/maillog,(?i)(?=.*\s*low reputation\s*)(?=.*\s*domain\s*)(?=.*\s*gsmtp\s*)]


Each item is configured like this:
Item configuration :
Type: Zabbix agent (active)
Type of information: Numeric (unsigned)
Update interval: 5s

About the trigger expression:
Problem expression trigger for Microsoft :
last(/myExample/log.count[/var/log/maillog,(?i)unfortunately.*block list.*S3150,skip])>0
last(/myExample/log.count[/var/log/maillog,(?i)(?=.*\s*unfortunately\s*)(?=.*\s*block list\s*)(?=.*\s*S3150\s*)],#1)=1

Problem expression trigger for Gmail :
last(/WarmupV01_Mode/log.count[/var/log/maillog,(?=.*550-5\.7\.1)(?=.*Gmail)(?=.*suspicious)(?=.*very low reputation)(?=.*spam)(?=.*blocked)],#1)=1

I have tried several regular expressions for items and triggers to detect the error quickly.
I also adjusted configuration file of each Zabbix Agent on each host.


Many thanks for your help !

[cyber]

Quick glance over this does not show me anything suspicious about items or triggers.
I have seen couple of cases, where logfile was just growing so rapidly, that agent was never going to keep up with it... even if we increase maxlines parameter in item to maximum.
Also, is that log rotated (often)? With log rotation it is easy to miss ends of files, if there is a lot of lines and quick rotation...