send data to ZABBIX using KAFKA (or an other MOM)

[slhotellier]

Hello
I'm working on a POC to integrate messages form ours applications to ZABBIX (push mode).
In order to prevent use from loosing messages we will use a message broker (probably KAFKA)
Integration in ZABBIX is done by zabbix_sender using a pipe ex :
kafka-console-consumer.sh --bootstrap-server X.X.X.X:9092 --topic test --skip-message-on-error | zabbix_sender -z localhost -r -vv -i -

But zabbix sender crash when messages are not well formated, for expemple blank messages.

Is there a way to ignore bad messages, or an other way to have push mode monitoring ?

Best regards

Using Zabbix 4.0.3 Centos7

[slhotellier]

Hello

I will try this :
kafka-console-consumer.sh --bootstrap-server X.X.X.X:9092 --topic test --skip-message-on-error | sed 's/^\ *//' | grep ".* .* [0-9]* .*" | cut -f1-4 -d " " | zabbix_sender -z localhost -r -vv -i -

sed 's/^\ *//' -> delete spaces at the line's begening
grep ".* .* [1-9]* .*" -> delete line with less than 4 elements, the third is only numeric for timestamp
cut -f1-4 -d " " -> delete all elements after the fourth

If some one have better idea to prevent loosing messages in push mode, please make a comment

Best regards

Using Zabbix 4.0.3 Centos7