Setting up MQTT monitoring with Zabbix 7.2.6 and agent2...

[BradKnowles]

Folks,

We've been using Zabbix for a while to monitor a bunch of systems in our network, but this is the first time we've ever needed to do an integration with MQTT, in this case for supporting Ignition SCADA.

I found the page at https://www.zabbix.com/integrations/mqtt for the official integration, but that doesn't seem to include any template files. I'm looking at the various third-party solutions shown at https://www.zabbix.com/integrations/mqtt#3rd_party but so far it doesn't look like any of them are designed for use with Zabbix 7, and they all seem to be pretty old. I'm digging through them anyway, to see if I can find anything of value.

I also looked here on the forums https://www.zabbix.com/forum/search?...A%22mqtt%22%7D and found some hits, but so far I'm not finding anything with any template files. I did find a couple of links on how to set up Zabbix via the web GUI to enable certain things (https://blog.zabbix.com/let-me-subsc...-topics/12710/ and https://medium.com/@flaviofranca/usi...r-6df748b8f93e), but I have not yet found anything with any template files.

I have updated the local agent.conf with things like this (example from https://medium.com/@flaviofranca/usi...6df748b8f93e):

Hostname=<my-host>
Server= <IP Zabbix Server>
ServerActive= <IP Zabbix Server>

##### PLUGINS ####
Plugins.MQTTSubscribe.Timeout=5

However, I have not yet put any additional items into that configuration file.

It is not clear to me where we need to define things so that the local agent2 picks up the MQTT topics that we care about and makes them available to the Zabbix server, or how to surface those items in the Zabbix web GUI.

Surely there is some documentation or web page examples that show where and how to define these things from templat file, CLI commands, or via an API, and not just from the web GUI, right? But if so, then where do I find that? It's kind of hard to automate the management of the Zabbix system if you have to hand-jam in a bunch of stuff via the web GUI, so we try to do everything with predefined template files, CLI commands, API calls, etc.... I just can't figure out where I can find guidance like that for MQTT integration with Zabbix.

Any suggestions you may have would be greatly appreciated!

[BradKnowles]

Okay, so now it turns out that once I got the exact string I was looking for, I was able to easily turn that into a number. Here's my pre-processing steps now:


I also updated the jSONPath slightly, to account for the possibility that I might get multiple updates in the same MQTT message. We might lose a bit of data here, but as quickly as we're getting the data from MQTT, those should happen within the same second and that shouldn't be an issue:

$['spBv1.0/Zabbix/DDATA/System/Gateway'].metrics[?(@.name == "UptimeSeconds")].value.first()

So, I consider this to be my final answer for getting a single data element from Ignition SCADA via MQTT and getting that into Zabbix server in a format that can be graphed. Here is the very bland looking graph that results: Thanks!

[ysus]

Have you seen this one? It describes basic setup but you can try to get in touch with author directly.

https://assets.zabbix.com/files/events//2025/conference_benelux_2025/Ivo_Schooneman_Feeding_Zabbix_with_MQTT_data.pdf

Also, as far as I understand, Zabbix agent supports only “classic” payload in JSON format. In case Ignition SCADA uses Sparkplug B you probably would not be able to achieve your goal.

[BradKnowles]

ysus -- I had not seen that PDF, thanks! It looks like slide #16 from that presentation shows an example mqtt.conf file that needs to go on the machine that is running agent2, which we are doing on the Core and Edge Ignition SCADA monitoring servers.

We need to do basic host level monitoring for those boxes anyway, so it makes sense to use the local agent2 already running on those systems to also bring in the MQTT data as well.

Then we get to try to figure out how to surface that data in the Zabbix web UI, build dashboards around that data, etc.... We've done plenty of this in the past, but all the examples I've been able to find so far focus on how to do that from the GUI as opposed to templates or local config files, and doing everything through the GUI is hard to automated across a bunch of servers distributed across multiple datacenters. We want to automate as much of this process as possible, and the GUI does not lend itself to automation.

I had heard they used SparkPlug format instead of JSON, so I figured there might be some additional things we need to do to process the data before it can be brought into Zabbix. But we need to get the basic connectivity stuff figured out first.

Thanks again!

[ysus]

Hi BradKnowles,
We also have couple of PLCs which can report diagnostic data over MQTT and I was thinking to use Agent2. The only problem was Sparkplug B format because those PLCs do not support classic JSON so I gave up.

For basic connectivity test I would suggest to install free mosquitto broker on test machine and verify you can see data in MQTT explorer on the same machine. And then try to read sent/received bytes into Zabbix.

Good luck!

[BradKnowles]

Hi BradKnowles,
We also have couple of PLCs which can report diagnostic data over MQTT and I was thinking to use Agent2. The only problem was Sparkplug B format because those PLCs do not support classic JSON so I gave up.

With Ignition, it turns out that you can have it return JSON format instead of Sparkplug. We have a main Ignition admin that I've been working with on this process, and he was able to configure that. We still get a weird top level element in the path to the topic, but the data we get is definitely JSON formatted.

For basic connectivity test I would suggest to install free mosquitto broker on test machine and verify you can see data in MQTT explorer on the same machine. And then try to read sent/received bytes into Zabbix.

I found the CLI tool known as mqttx and that has been very useful for debugging the data we're getting back from the Ignition server. We now have data flowing from the Ignition server to the Zabbix agent2 on the box, and our biggest challenge is now getting that data from the Zabbix agent2 on the box back to the Zabbix server. We can confirm this with debug level logging on the Zabbix agent2 process, as well as lots of tcpdump sessions.

I've followed the old example from https://blog.zabbix.com/let-me-subsc...-topics/12710/ as well as the more recent example provided here at https://assets.zabbix.com/files/even..._MQTT_data.pdf and the Master Item has been configured and appears to be working. However, the Dependent Item is not working, even though we seem to have worked out the JSONPath parsing that is necessary to pull the unsigned integer we want out of the array that is returned inside of the JSON data.

In our JSON response, we get an array called "metrics" returned to us, with multiple metrics inside that array. The first metric we're trying to pull out is called "UptimeSeconds", but it gets returned as an array of one item. So we have to do JSONpath parsing like this to pull it out, and hopefully turn it into an unsigned integer:

$.metrics[?(@.name == 'UptimeSeconds')].value.sum()

Doing a plain ".value" gives us the array with a single element, which is not a value that Zabbix knows how to deal with. So, we have to take the ".sum()" of that value to strip the square brackets and turn it into something that looks like a number. Which Zabbix should be able to deal with.

Unfortunately, the Dependent Item never gets set to an "Enabled" status, and remains "Not Supported". But the Master Item is now finally "Enabled".

[BradKnowles]

The JSON we're trying to extract from looks like this:

{"timestamp":1750455084054,"metrics":[{"name":"Performance/Memory Usage","timestamp":1750455083554,"dataType":"Int32","value":386320320},{"name":"CurrentDateTime","timestamp":1750455083594,"dataType":"DateTime","value":1750455083594},{"name":"UptimeSeconds","timestamp":1750455083594,"dataType":"Int64","value":225093}],"seq":134}

If you un-wrap that with jq, it looks like this:

echo '{"timestamp":1750455084054,"metrics":[{"name":"Performance/Memory Usage","timestamp":1750455083554,"dataType":"Int32 ","value":386320320},{"name":"CurrentDateTime","ti mestamp":1750455083594,"dataType":"DateTime","valu e":1750455083594},{"name":"UptimeSeconds","timesta mp":1750455083594,"dataType":"Int64","value":22509 3}],"seq":134}' | jq -S
{
"metrics": [
{
"dataType": "Int32",
"name": "Performance/Memory Usage",
"timestamp": 1750455083554,
"value": 386320320
},
{
"dataType": "DateTime",
"name": "CurrentDateTime",
"timestamp": 1750455083594,
"value": 1750455083594
},
{
"dataType": "Int64",
"name": "UptimeSeconds",
"timestamp": 1750455083594,
"value": 225093
}
],
"seq": 134,
"timestamp": 1750455084054
}

Because we can't guarantee that "UptimeSeconds" will always be in the same place in the array, we can't just assume it will always be:

.metrics[2].name = "UptimeSeconds";

It might end up as a different index at other times.

Thus, we have to do the more complex JSONPath parsing.

[BradKnowles]

So, here's a perfect example of what I can see with mqttx:

$ mqttx sub -h '127.0.0.1' -p 1883 -V 3.1.1 -u 'nottherealuser' -P 'nottherealpassword' -t 'spBv1.0/Zabbix/DDATA/#'
✔ Connected
✔ Subscribed to spBv1.0/Zabbix/DDATA/#
topic: spBv1.0/Zabbix/DDATA/System/Gateway, qos: 0
{"timestamp":1750697657026,"metrics":[{"name":"CurrentDateTime","timestamp":1750697656525,"dataType":"DateTime","value":1750697656525},{"name":"UptimeSeconds","timestamp":1750697656525,"dataType":"Int64","value":467666}],"seq":152}

topic: spBv1.0/Zabbix/DDATA/System/Gateway, qos: 0
{"timestamp":1750697658026,"metrics":[{"name":"CurrentDateTime","timestamp":1750697657525,"dataType":"DateTime","value":1750697657525},{"name":"UptimeSeconds","timestamp":1750697657525,"dataType":"Int64","value":467667}],"seq":153}

topic: spBv1.0/Zabbix/DDATA/System/Gateway, qos: 0
{"timestamp":1750697658919,"metrics":[{"name":"Performance/Memory Usage","timestamp":1750697658419,"dataType":"Int32","value":397703952},{"name":"CurrentDateTime","timestamp":1750697658526,"dataType":"DateTime","value":1750697658526},{"name":"UptimeSeconds","timestamp":1750697658526,"dataType":"Int64","value":467668}],"seq":154}

topic: spBv1.0/Zabbix/DDATA/System/Gateway, qos: 0
{"timestamp":1750697660026,"metrics":[{"name":"CurrentDateTime","timestamp":1750697659526,"dataType":"DateTime","value":1750697659526},{"name":"UptimeSeconds","timestamp":1750697659526,"dataType":"Int64","value":467669},{"name":"Alarming/Clear and Unacked","timestamp":1750697659754,"dataType":"Int32","value":4},{"name":"Alarming/Clear and Acked","timestamp":1750697659754,"dataType":"Int32","value":1},{"name":"Alarming/Clear and Acked","timestamp":1750697659754,"dataType":"Int32","value":0},{"name":"Alarming/Active and Unacked","timestamp":1750697659754,"dataType":"Int32","value":1}],"seq":155}

topic: spBv1.0/Zabbix/DDATA/System/Gateway, qos: 0
{"timestamp":1750697661026,"metrics":[{"name":"CurrentDateTime","timestamp":1750697660526,"dataType":"DateTime","value":1750697660526},{"name":"UptimeSeconds","timestamp":1750697660526,"dataType":"Int64","value":467670}],"seq":156}

Note that UptimeSeconds doesn't always appear in the same position in the metrics[] array that gets returned. So, we can't use a simple numerical index. We have to match on the name instead.

On the Zabbix server, looking in the Docker logs, we see lines like this:

2025-06-23T16:34:54.445310658Z 55:20250623:163454.445 error reason for "dev-server:UptimeSeconds" changed: Preprocessing failed for: {"spBv1.0/Zabbix/DDATA/System/Gateway":"{\"timestamp\":1750696492842,\"metrics\" :[{\"name\":\"Cur...
2025-06-23T16:34:54.445313884Z 1. Failed: cannot extract value from json by path "$.metrics[?(@.name == 'UptimeSeconds')].value.sum()": cannot apply aggregation function to empty array
2025-06-23T16:34:54.446780009Z 55:20250623:163454.446 error reason for "dev-server:UptimeSeconds" changed: Preprocessing failed for: {"spBv1.0/Zabbix/DDATA/System/Gateway":"{\"timestamp\":1750696489842,\"metrics\" :[{\"name\":\"Cur...
2025-06-23T16:34:54.446782804Z 1. Failed: cannot extract value from json by path "$.metrics[?(@.name == 'UptimeSeconds')].value.sum()": cannot apply aggregation function to empty array
2025-06-23T16:35:00.261411898Z 54:20250623:163500.261 error reason for "dev-server:UptimeSeconds" changed: Preprocessing failed for: {"spBv1.0/Zabbix/DDATA/System/Gateway":"{\"timestamp\":1750696494843,\"metrics\" :[{\"name\":\"Cur...
2025-06-23T16:35:00.261440994Z 1. Failed: cannot extract value from json by path "$.metrics[?(@.name == 'UptimeSeconds')].value.sum()": cannot apply aggregation function to empty array
2025-06-23T16:35:00.296531830Z 54:20250623:163500.296 error reason for "dev-server:UptimeSeconds" changed: Preprocessing failed for: {"spBv1.0/Zabbix/DDATA/System/Gateway":"{\"timestamp\":1750696495842,\"metrics\" :[{\"name\":\"Cur...
2025-06-23T16:35:00.296544046Z 1. Failed: cannot extract value from json by path "$.metrics[?(@.name == 'UptimeSeconds')].value.sum()": cannot apply aggregation function to empty array
2025-06-23T16:35:00.298590721Z 54:20250623:163500.298 error reason for "dev-server:UptimeSeconds" changed: Preprocessing failed for: {"spBv1.0/Zabbix/DDATA/System/Gateway":"{\"timestamp\":1750696496843,\"metrics\" :[{\"name\":\"Cur...

Okay, so I drop the "sum()" at the end of the JSONPath parsing, and instead I see this:

2025-06-23T16:43:04.988992986Z 1. Failed: cannot extract value from json by path "$.metrics[?(@.name == 'UptimeSeconds')].value": no data matches the specified path
2025-06-23T16:43:05.013912967Z     54:20250623:164305.013 error reason for "dev-server:UptimeSeconds" changed: Preprocessing failed for: {"spBv1.0/Zabbix/DDATA/System/Gateway":"{\"timestamp\":1750696980918,\"metrics\":[{\"name\":\"Cur...
2025-06-23T16:43:05.013938987Z 1. Failed: cannot extract value from json by path "$.metrics[?(@.name == 'UptimeSeconds')].value": no data matches the specified path
2025-06-23T16:43:05.016339502Z     54:20250623:164305.016 error reason for "dev-server:UptimeSeconds" changed: Preprocessing failed for: {"spBv1.0/Zabbix/DDATA/System/Gateway":"{\"timestamp\":1750696981918,\"metrics\":[{\"name\":\"Cur...
2025-06-23T16:43:05.016379870Z 1. Failed: cannot extract value from json by path "$.metrics[?(@.name == 'UptimeSeconds')].value": no data matches the specified path
2025-06-23T16:43:05.017738871Z     54:20250623:164305.017 error reason for "dev-server:UptimeSeconds" changed: Preprocessing failed for: {"spBv1.0/Zabbix/DDATA/System/Gateway":"{\"timestamp\":1750696982919,\"metrics\":[{\"name\":\"Cur...
2025-06-23T16:43:05.017745076Z 1. Failed: cannot extract value from json by path "$.metrics[?(@.name == 'UptimeSeconds')].value": no data matches the specified path
2025-06-23T16:43:05.019273531Z     54:20250623:164305.019 error reason for "dev-server:UptimeSeconds" changed: Preprocessing failed for: {"spBv1.0/Zabbix/DDATA/System/Gateway":"{\"timestamp\":1750696983751,\"metrics\":[{\"name\":\"Per...

When I do the JSONPath parsing tests in the Zabbix server web GUI, I get exactly the value we're looking for when we use:

$.metrics[?(@.name == 'UptimeSeconds')].value.sum()

But it's not actually working on the live data coming in.

I'm stumped.

[BradKnowles]

So, one thing I've discovered is that the JSON being sent to the Zabbix server is being munged in the process. It's coming through with backslashes and double-quote characters. If you try to verify that the JSON is valid as your step #1, you get an error from JSONPath. If you use regular expressions to strip all backslashes and all double quote characters, now the JSON is valid.

Here's an example of an invalid JSON string:

{"spBv1.0/Zabbix/DDATA/System/Gateway":"{\"timestamp\":1750705505820,\"metrics\" :[{\"name\":\"Alarming/Clear and Unacked\",\"timestamp\":1750705505320,\"dataType\" :\"Int32\",\"value\":4},{\"name\":\"Alarming/Clear and Acked\",\"timestamp\":1750705505320,\"dataType\":\ "Int32\",\"value\":1},{\"name\":\"Alarming/Clear and Acked\",\"timestamp\":1750705505320,\"dataType\":\ "Int32\",\"value\":0},{\"name\":\"Alarming/Active and Unacked\",\"timestamp\":1750705505320,\"dataType\" :\"Int32\",\"value\":1},{\"name\":\"CurrentDateTim e\",\"timestamp\":1750705505810,\"dataType\":\"Dat eTime\",\"value\":1750705505810},{\"name\":\"Uptim eSeconds\",\"timestamp\":1750705505810,\"dataType\ ":\"Int64\",\"value\":475515}],\"seq\":164}"}

And here's an example of a valid JSON string:

{spBv1.0/Zabbix/DDATA/System/Gateway:{timestamp:1750708108759,metrics:[{name:CurrentDateTime,timestamp:1750708108258,dataType:DateTime,value:1750708108258},{name:UptimeSeconds,timestamp:1750708108258,dataType:Int64,value:478118}],seq:50}}

See the attached screenshot showing what my pre-processing steps are now that validate the JSON and at least capture the whole JSON string and bring it into Zabbix server:

This works to at least let me see the data in the Zabbix server web GUI, and I can go back and look at the history of previous values.

I'm still stumped on the part about properly parsing the JSON to pull out the specific value we want.

EDIT: But at least now we know we have valid JSON.

[BradKnowles]

Oh, boy. This is even better. You can't trust the internal JSON validator. Because stuff it validates successfully will still cause JSONPath to barf. Also, you can't remove all double quote characters, just the ones that are incorrectly placed around the outermost object braces. Testing with https://jsonformatter.org/json-parser to get the JSON correctly formatted has really helped.

So, you do still need to strip out the backslashes. Starting with this invalid JSON string with the backslashes removed:

{"spBv1.0/Zabbix/DDATA/System/Gateway":"{"timestamp":1750713947678,"metrics":[{"name":"CurrentDateTime","timestamp":1750713947178,"dataType":"DateTime","value":1750713947178},{"name":"UptimeSeconds","timestamp":1750713947178,"dataType":"Int64","value":483957},{"name":"Performance/Memory Usage","timestamp":1750713947257,"dataType":"Int32","value":722997864}],"seq":27}"}

The corrected JSON string is:

{"spBv1.0/Zabbix/DDATA/System/Gateway":{"timestamp":1750713947678,"metrics":[{"name":"CurrentDateTime","timestamp":1750713947178,"dataType":"DateTime","value":1750713947178},{"name":"UptimeSeconds","timestamp":1750713947178,"dataType":"Int64","value":483957},{"name":"Performance/Memory Usage","timestamp":1750713947257,"dataType":"Int32","value":722997864}],"seq":27}}

This latter string will validate at https://jsonformatter.org/json-parser but the earlier string will not. Unfortunately, JSON parsers aren't very good at telling you what is really wrong with your JSON.

Now, you need to strip out the double quote character before the brace enclosing the first "timestamp", and you need to strip out the last double quote character before the final closing brace. Here's what that looks like:

Now, the JSONPath string actually came from a conversation I had with Claude. I did have to tell Claude where the web page is that defines how Zabbix works with JSONPath. But when I did that, it gave me the following JSONPath to use:

$['spBv1.0/Zabbix/DDATA/System/Gateway'].metrics[?(@.name == "UptimeSeconds")].value

However, this is not quite correct. This still gives me a value as an array of one object, with square brackets around it. Unfortunately, the https://jsonpath.com/ validator did not like me adding ".sum()" to the end of that JSONPath, so I didn't think it would work. However, the JSONPath interpreter in Zabbix did take this, and gave me exactly the value I wanted.

However, that number is still being interpreted as a text field. Now I have to figure out how to turn that back into a number that can actually be graphed.

[BradKnowles]

Okay, so now it turns out that once I got the exact string I was looking for, I was able to easily turn that into a number. Here's my pre-processing steps now:


I also updated the jSONPath slightly, to account for the possibility that I might get multiple updates in the same MQTT message. We might lose a bit of data here, but as quickly as we're getting the data from MQTT, those should happen within the same second and that shouldn't be an issue:

$['spBv1.0/Zabbix/DDATA/System/Gateway'].metrics[?(@.name == "UptimeSeconds")].value.first()

So, I consider this to be my final answer for getting a single data element from Ignition SCADA via MQTT and getting that into Zabbix server in a format that can be graphed. Here is the very bland looking graph that results: Thanks!

[ysus]

Hi BradKnowles,

Looks like you need to talk to your Ignition admin once again because he did only part of the job. He removed "protobuf" digital encoding but still sending you payload in Sparkplug B format because of spBv1.0/ prefix. This is not a valid JSON which could be easily parsed by Zabbix because it has "dataType", "timestamp" and other fields you do not need.

Your topic must be "Zabbix/System/Gateway/#" and not "spBv1.0/Zabbix/DDATA/#"

And you shoud receive something like {"UptimeSeconds","478118"}

Also I would recommend native "mosquitto_sub.exe" for subscription which is part of Mosquitto broker installation.





Don't mess up with preprocessing - Sparkplug payload could have much more complex structure . See examples below.

[BradKnowles]

Hi BradKnowles,

Looks like you need to talk to your Ignition admin once again because he did only part of the job. He removed "protobuf" digital encoding but still sending you payload in Sparkplug B format because of spBv1.0/ prefix. This is not a valid JSON which could be easily parsed by Zabbix because it has "dataType", "timestamp" and other fields you do not need.

Yeah, I'm fine with ignoring those other values. I'll pull out only the ones I need, and the rest just won't be used.

Your topic must be "Zabbix/System/Gateway/#" and not "spBv1.0/Zabbix/DDATA/#"

And you shoud receive something like {"UptimeSeconds","478118"}

That would be easier to parse, yes. But I've already gone through all this pain to get to where we are now.

At this point, it shouldn't be too painful to add more master items for more topics and more dependent items for each datapoint type we want to track.

Also I would recommend native "mosquitto_sub.exe" for subscription which is part of Mosquitto broker installation.

I believe that he only has the proprietary Ignition MQTT data broker on the system. I'm pretty sure that he doesn't have Mosquitto. More importantly, this is on a Linux box and not Windows.

Don't mess up with preprocessing - Sparkplug payload could have much more complex structure . See examples below.

Yup, I've seen some of the more complex structure. We're doing other things to help ensure that won't be a problem.

[BradKnowles]

In fact, in just a few minutes, I have already added a second datapoint type that we want to track, as another dependent item under the current master item. The graph is already showing up.

I'll check to see if there are any other datapoint types we want to track under this master item, and then I will proceed to create a new master item for the next topic we want to subscribe to.

[BradKnowles]

And now I have five data elements that I'm pulling out from that one topic, and I've set up a prototype dashboard to collect all that data into one place.

I think I'm ready to add the next master item for the next topic.

[ysus]

But I've already gone through all this pain to get to where we are now

And you did really a great job!

I believe that he only has the proprietary Ignition MQTT data broker on the system

Sorry, my fault. I was under impression Ignition publishes data to 3rd party external broker like Mosquitto which could run on both Windows and Linux. Have no previous experience with Ignition and was not aware it has it's own broker.

Once again thanks fo sharing your experience. I'm sure it will be helpful for other forum users as well.