MQTT plugin in Zabbix Agent 2 - subscribe was broken by timeout

[vchrizz]

Strange is, after the Upgrade to 6.4 and after some restarts/reboots sometimes data is received and sometimes not.
In following screenshot you see the widgets from the items when data was received and when not:

[Markku]

Based on your earlier packet capture and the analysis in message number 4 in this thread, the agent disconnects the TCP session earlier than it should. Thus it is natural that data is not collected properly.

If I were you I would try to find the make-or-break level: how many items you need to have to get the problem reliably reproduced (if you think it is related to the number of MQTT items), or whatever information is needed so that you can open a ticket on support.zabbix.com due to a software bug. Use packet analysis as needed.

Markku

[dsteinkopf]

It seems that I also have the same problem after I upgraded from 6.2. to 6.4.
My temporary workaround was to downgrade (only) the zabbix-agent2 (running as docker-container) back to 6.2.

[[email protected]]

If you use mosquitto as mqtt broker, then please disable support for RETAINED messages in mosquito configuration.

[rpav_ia]

MQTT Plugin which comes with Zabbix Agent 2 is very unstable and unreliable. I'm monitoring over 100 metrics and I've been getting either 'Timeout' or 'Incorrect update interval' errors. The second one is most likely caused by incorrect processing algorithm of slowly changing values at MQTT broker.

So, I switched to use MQTT-Zabbix (https://github.com/rpavlyuk/mqtt-zabbix) which acts as the proxy between Zabbix and MQTT. It solved my issues completely. The only downside is that you have to run one additional service or container.

[xtvdata]

MQTT Plugin which comes with Zabbix Agent 2 is very unstable and unreliable. I'm monitoring over 100 metrics and I've been getting either 'Timeout' or 'Incorrect update interval' errors. The second one is most likely caused by incorrect processing algorithm of slowly changing values at MQTT broker.

So, I switched to use MQTT-Zabbix (https://github.com/rpavlyuk/mqtt-zabbix) which acts as the proxy between Zabbix and MQTT. It solved my issues completely. The only downside is that you have to run one additional service or container.

Indeed Zabbix Agent2's MQTT plugin has issues (e.g.: from the logs it looks like Go MQTT library is not completely configured - at least for resume subscriptions).
In my case it starts to happen when a single template subscribes 2 or more topics (with or without wildcards).
I guess there is something to be optimized in the clients opened any the agent; looking at the logs, possibly, it could be a matter of using a single client per subscription with the same client id, which might confuse the broker and reset the connections.

Anyway, indeed having to run an additional service is inconvenient, but unfortunately that Python script also has issues:
- I don't like ditching an high-performance Go program for a Python script (I have already rewritten a couple of data collectors from Python to Go, and result is significantly lower CPU usage and less memory used);
- I'm not sure if the Python script supports TLS on MQTT (with certificates from a custom CA) - the library does but I can't see any related info in the script config;
- I'm not sure if it manages authentication on Zabbix with TLS certs/PSK;
- I need a collector that collects on behalf of multiple hosts (e.g.: one agent collects on behalf of multiple Zabbix Hosts). This means that it would be not an extra service, but a number of extra services.

[dsteinkopf]

Does anyone have any recent experience with the latest zabbix agent? Does 6.4.x now work (more) properly? (I am still using zabbix-agent 6.2 because of this...)

[R1MIDALV]

We're facing the same Issue with 6.4.12 and 6.4.16. A downgrade to 6.2.9 does not resolve it.
A discovery rule and an Item prototype generate 94 Items, each is a mqtt.get on different topics.

The Agent logs are always the same:
2024/07/04 10:20:55.021671 [MQTT] cannot subscribe topic 'mytopic/devicevalue' to [tcp://rabbitmq:1883]: not currently connected and ResumeSubs not set
2024/07/04 10:21:46.059838 [MQTT] connection lost to [tcp://rabbitmq:1883]: pingresp not received, disconnecting
2024/07/04 10:21:49.098338 [MQTT] cannot subscribe topic 'mytopic/devicevalue' to [tcp://rabbitmq:1883]: timed out while subscribing

2024-07-04 10:23:27.111939+00:00 [info] <0.2065.0> MQTT connection <<"172.23.0.4:57872 -> 172.23.0.5:1883">> will terminate because peer closed TCP connection
2024-07-04 10:23:28.171362+00:00 [info] <0.2315.0> Accepted MQTT connection 172.23.0.4:56426 -> 172.23.0.5:1883 for client ID Zabbix agent 2 6.4.16 44746a91-fa52-395e-91c8-c8190c3a540b
2024-07-04 10:24:18.172231+00:00 [info] <0.2315.0> MQTT connection <<"172.23.0.4:56426 -> 172.23.0.5:1883">> will terminate because peer closed TCP connection
2024-07-04 10:24:19.244060+00:00 [info] <0.2568.0> Accepted MQTT connection 172.23.0.4:42540 -> 172.23.0.5:1883 for client ID Zabbix agent 2 6.4.16 44746a91-fa52-395e-91c8-c8190c3a540b

It worked properly for multiple months after first configuration. After recreating the items, it stopped.

After deleting and rediscover, some values are updated once, but not permanently. It seems at the first time the connection is established it works, but only for one topic of 94 and only once.

[R1MIDALV]

A small correction to my previous posting.
6.2.9 is working properly.
We had an issue in the mqtt.conf, which caused the container to crash periodically. It has now been resolved, and all topics/items are delivering values.

[R1MIDALV]

We faced the same problem again after upgrading to zabbix/zabbix-agent2:7.0.3-alpine. The host was a x86 System

But: After installing same under ARM, the Problem is gone and dies not occurred so far. So we moved all Container to an ARM Server.

[R1MIDALV]

ARM is not a solution. Same issue accrued again.

[klandos]

Just a bump on this, I ran into the same issue with 7.0.4 and 7.0.5 on arm on several Debian 11 installations
Made a downgrade to 6.2.9 and it seems to be working now.

Hopefully there will be a fix as mqtt is mandatory in my eyes.

[roteki]

i have found this file /etc/zabbix/zabbix_agent2.d/plugins.d/mqtt.conf in which i have configured the options, then idk how to continue the configuration, I have read https://www.zabbix.com/fr/integrations/mqtt#mqtt but I couldn't understand quite well what must I do next
if you can tell me please in details how to configure the plugin
I'm using zabbix 7.0.5 btw

[dsteinkopf]

I upgraded my problematik agent from 6.2 to 7.0. And: zabbix agent 7.0.10 seems to work better. Hope that is not only an early impression.

See also: https://support.zabbix.com/browse/ZBX-25236

[R1MIDALV]

I upgraded my problematik agent from 6.2 to 7.0. And: zabbix agent 7.0.10 seems to work better. Hope that is not only an early impression.

See also: https://support.zabbix.com/browse/ZBX-25236

Any updates so far?
The problem always seemed to be solved when we updated the version, but it always reappears some time later.

[dsteinkopf]

Any updates so far?
The problem always seemed to be solved when we updated the version, but it always reappears some time later.

Yes, that's correct. But I hope not to promise too much, when I say that at least on my site, I did not have to revert to the old version this time. The agent's MQTT even seems to "survive" short hickups, which always forced me to restart the agent to trigger an mqtt reconnect. This has not happened here any more. Nice! I am running then docker version "zabbix/zabbix-proxy-sqlite3:ubuntu-7.0-latest" at the moment. (BTW: But still, I'm not sure if I'd call the MQTT client impementation in the agent stable...)