duplicated lines captured in log monitoring

[lewis lee]

Hi All,

I encountered a problem in log monitoring. I have item configuration to use logrt() function, that will monitor the application log. If the line of application log contain the key word "WARN", the log should be returned. Then I have defined some triggers to send alert notification.

However I received multiple notification for a single problem event. Finally I disabled all triggers on this item, and monitor the latest date of it. I have noticed that the last few lines of problem event will be captured several times.

Could anyone tell me what wrong in my item configuration? How to avoid this problem? Thanks in advance.


Zabbix server version: Zabbix 4.4.6
Zabbix agent version: zabbix-agent-4.4.6-1.el7.x86_64
OS : Red Hat Enterprise Linux Server release 7.7 (Maipo)

item key: logrt["/k8s_log/smx_stub_smpp_client_1/log/SYSTEM.LOG","WARN",,,skip,,,]
Result: Latest Data

[gofree]

can you paste the log if possible ...

sometimes it happens that if you add skip aparameter to existing item key its not working - see documentation - eg. the skip paramter is ignored and the log is being reread from beginning every time

[lewis lee]

Hi gofree,

Thank you for your response. Here is the log and latest data screen capture.


[projadm@itech-app04 /k8s_log/smx_stub_smpp_client_1/log] tail -50f SYSTEM.LOG |grep WARN
20200324 16:53:05.800 [WARN ] [d74b357a-78c3-4f43-9a74-98a29db55af2] [TOGUS_STX0thread] WARNING smppClnt1 203 Connection in stub TOGUS_STX0 disconnected!
20200324 16:54:06.098 [WARN ] [1d039e31-391f-4a60-b8ba-7625b1d90105] [pool-24-thread-1] WARNING smppClnt1 309 Connect to SMSC Orange_STRX0 Fail!
20200324 16:54:11.730 [WARN ] [d74b357a-78c3-4f43-9a74-98a29db55af2] [pool-24-thread-2] WARNING smppClnt1 309 Connect to SMSC CMI_S2TRX0 Fail!
20200324 16:56:13.331 [WARN ] [1d039e31-391f-4a60-b8ba-7625b1d90105] [pool-24-thread-1] WARNING smppClnt1 309 Connect to SMSC Tcel_STRX0 Fail!
20200324 16:56:13.520 [WARN ] [1d039e31-391f-4a60-b8ba-7625b1d90105] [pool-24-thread-1] WARNING smppClnt1 309 Connect to SMSC Mitto_S2TRX0 Fail!
20200324 16:56:13.529 [WARN ] [1d039e31-391f-4a60-b8ba-7625b1d90105] [pool-24-thread-1] WARNING smppClnt1 309 Connect to SMSC TOGUS_S2TRX0 Fail!
20200324 16:56:13.691 [WARN ] [1d039e31-391f-4a60-b8ba-7625b1d90105] [Restcomm_STRX0thread] WARNING smppClnt1 203 Connection in stub Restcomm_STRX0 disconnected!
20200324 16:56:18.963 [WARN ] [d74b357a-78c3-4f43-9a74-98a29db55af2] [pool-24-thread-2] WARNING smppClnt1 309 Connect to SMSC SynSmsGw2TRX0 Fail!
20200324 16:56:19.162 [WARN ] [d74b357a-78c3-4f43-9a74-98a29db55af2] [pool-24-thread-2] WARNING smppClnt1 309 Connect to SMSC Mitto_S1TRX0 Fail!
20200324 16:58:20.818 [WARN ] [1d039e31-391f-4a60-b8ba-7625b1d90105] [pool-24-thread-1] WARNING smppClnt1 309 Connect to SMSC NTT_STRX0 Fail!
20200324 16:58:26.451 [WARN ] [d74b357a-78c3-4f43-9a74-98a29db55af2] [pool-24-thread-2] WARNING smppClnt1 309 Connect to SMSC Axiata_STRX0 Fail!
20200324 16:59:59.416 [WARN ] [1d039e31-391f-4a60-b8ba-7625b1d90105] [TOGUS_STX0thread] WARNING smppClnt1 203 Connection in stub TOGUS_STX0 disconnected!
20200324 17:00:28.050 [WARN ] [1d039e31-391f-4a60-b8ba-7625b1d90105] [pool-24-thread-1] WARNING smppClnt1 309 Connect to SMSC Orange_STRX0 Fail!
20200324 17:00:33.683 [WARN ] [d74b357a-78c3-4f43-9a74-98a29db55af2] [pool-24-thread-2] WARNING smppClnt1 309 Connect to SMSC CMI_S2TRX0 Fail!
20200324 17:00:33.875 [WARN ] [d74b357a-78c3-4f43-9a74-98a29db55af2] [pool-24-thread-2] WARNING smppClnt1 309 Connect to SMSC Mitto_S2TRX0 Fail!

[gofree]

basically theye nor the same events in the log - so zabbix treats them as different events - they differ in the time section just before [WARN] string - in my opinion works as designed - zabbix is not priamrilly intended for logs if youd like to work with serious log monitoring go with ELK. In your case it seems that you have too frequent outages and zabbix cathes them all exatly as item and trigger have been configured.

[lewis lee]

Hi gofree,


Finally, I found the problem in zabbix_agent.conf. It is the integration problem with K8s. In my deployment, the zabbix-server and zabbix-web are running as container in K8s. the zabbix-agent is running in host level.

Previously, I have put all K8s master node's IP in agent config. Such zabbix-server received data from zabbix-agent 4 times.

Now only one IP address is assigned to "ServerActive", the duplicated log event problem is solved.

Thanks a lot!


#ServerActive=10.10.50.11:33011,10.10.50.12:33011, 10.10.50.13:33011,10.10.50.14:33011
ServerActive=10.10.50.11:33011

[gofree]

just from curiosity would you be willing to share your deployment files in k8s - how do you deploy zabbix as a deployment, statefullset? how do you handle secrets ?

here is mine work in progress - cant get to work the secrets properly

https://github.com/posuch/zabbix_k8_deployment/blob/master/deployment.yml

[lewis lee]

Hi gofree

We just used zabbix-appliance image to deploy zabbix as deployment. Attached the yaml file for your reference