UserParameter: "Terminate called after throwing an instance of 'std::logic_error' "

[ITOMDave]

Hi all,
I'm using the 4.4 appliance to test out a UserParameter. I've downloaded and installed the "speedtest" executable from OOKLA (https://www.speedtest.net/apps/cli) onto the Zabbix server, which incidentally is running in VMWare Workstation 15.

Here's the entry I've created in my zabbix_agentd.conf:

UserParameter=internet.speedtest,speedtest -f json

I then tested the UserParamer using

zabbix_agentd -t internet.speedtest

Which worked well, indicating to me that the key and script in the UserParameter were working fine.

The next step was to to setup an item. As you might guess from the actual UserParameter, the output I get is in JSON format so I'm going to do some preprocessing to get real values.....




So nothing too contentious there.
As far as preprocessing goes, I'm trying to keep it simple;



If I then try and test by using zabbix_get (which I understand uses the item definition rather than just running the UserParameter script), I get the below message;

root@zabbix:/etc/zabbix# zabbix_get -s 127.0.0.1 -k internet.speedtest
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
Aborted
root@zabbix:/etc/zabbix#

Can anyone help me diagnose the cause of this message? I get the following message in the server log file as well;

root@zabbix:/var/log/zabbix# tail -f zabbix_server.log
 16197:20191202:225017.194 server #39 started [preprocessing worker #3]
 16179:20191202:225034.306 enabling Zabbix agent checks on host "Zabbix server": host became available
 16153:20191202:225722.646 forced reloading of the configuration cache
 16161:20191202:230141.185 error reason for "Zabbix server:internet.speedtest" changed: Preprocessing failed for: terminate called after throwing an instance of 'std::logic_error'.  what():  basic_string::_M_con...
1. Failed: cannot extract value from json by path "$.ping.latency": cannot parse as a valid JSON object: invalid object format, expected opening character '{' or '[' at: 'terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
Aborted'

I'm pretty sure that the error message in the server log is down to Zabbix trying to preprocess the error message rather than the output I'd expected.

As always, all advice and guidance is gratefully received.

Dave

[Atsushi]

A timeout may have occurred during execution of the speedtest command, and the execution result of the speedtest command may not have been received correctly.
How many seconds is the timeout setting for Zabbix server and Zabbix agent? Is the time specified longer than the speedtest command execution time?
If this is not the case, you should output the execution result of the speedtest command to a file and refer to that file from Zabbix.

[ITOMDave]

The "zabbix_agentd -t internet.speedtest" works fine. I had already increased the timeout to a larger value. It's the error message from the zabbix_get command that's confusing me....
terminate called after throwing an instance of 'std::logic_error' what(): basic_string::_M_construct null not valid Aborted Thanks for the suggestions though. I could, as you say, send the output to a file but I'd rather keep things a little more elegant.

[Hamardaban]

The error message says that the JSON format is incorrect. Show what json is created

[ITOMDave]

Hi Hamardaban,

Yes, the message in the server log does say that the JSON is incorrect. As I said in my original post, that's because the Zabbix item is trying to process the error message generated when the userparameter executes rather than the JSON that the userparameter DOES produce when it executes properly.

Maybe the output below clarifies it a bit more :

root@zabbix:/home/appliance# zabbix_agentd -t internet.speedtest
internet.speedtest [t|{"type":"result","timestamp":"2019-12-04T12:18:53Z","ping":{"jitter":0.53800000000000003 ,"latency":18.855},"download":{"bandwidth":8464525 ,"bytes":67459920,"elapsed":8007},"upload":{"bandw idth":2331905,"bytes":15618572,"elapsed":6703},"pa cketLoss":0,"isp":"BT","interface":{"internalIp":" 192.168.146.128","name":"ens33","macAddr":"00:0C:2 9:7D1:37","isVpn":false,"externalIp":"31.48.90.99"},"s erver":{"id":17629,"name":"Exascale Limited","location":"Manchester","country":"United Kingdom","host":"speedtest.man0.uk.as61049.net","p ort":8080,"ip":"185.195.119.42"},"result":{"id":"9 0fa34ee-1512-4ce4-b499-983324c6918a","url":"https://www.speedtest.net/result/c/90fa34ee-1512-4ce4-b499-983324c6918a"}}]
root@zabbix:/home/appliance#
root@zabbix:/home/appliance#
root@zabbix:/home/appliance# zabbix_get -s 127.0.0.1 -k internet.speedtest
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_M_construct null not valid
Aborted
root@zabbix:/home/appliance#


As you can see, when I run zabbix_agentd -t internet.speedtest the output is fine, producing valid JSON.

When I run zabbix_get -s 127.0.0.1 -k internet.speedtest I get the "terminate called after throwing an instance of 'std::logic_error' what(): basic_string::_M_construct null not valid ...." message.

The "Terminate called......" string is the error that I'm trying to fix.

Does anyone know why the UserParameter executes fine when I invoke via zabbix_agentd -t, but doesn't execute when invoked via zabbix_get -s

Dave

[dimir]

Looks like your application speedtest only works when run as root. I assume the agent is run as zabbix. Try

sudo -u zabbix speedtest -f json

If you get the same error, here are few options:
- start agent as root (not recommended)
- add sudo to UserParameter and set up sudo to be passwordless for user zabbix when executing binary speedtest
- set SETUID bit on the speedtest binary and chown it root
I think the latter will be simplest, but it depends on the security level you expect from the system.

[Hamardaban]

I agree-of course you need to check the work of the user zabbix!
If it doesn't work - show the output of the zabbix_get -V command. And you would update or rebuild zabbix_get

[ITOMDave]

Hi Dimir,
Thanks for the pointer. When I ran the command

sudo -u zabbix speedtest -f json

it works fine. The permissions on the executable are 755, so no SetUID bit.

I then took your second suggestion - adding sudo to the UserParameter and restested using

zabbix_agentd -t internet.speedtest

and while it worked, I did get some very useful additional error messages relating to directory permissions.

To cut a long story short, the very first execution of the speedtest binary asks the user to accept some licensing terms. The response to this acceptance is stored in a directory below the home directory of the executing user.

I'm now getting "permission denied" messages on this directory.

I've got a lot of information I can now use to fix this.

I'll post a full solution when I've completed it all.

Many thanks to all who contributed to this thread, especially Dimir who gave me a pointer to better diagnosis.

Dave.

[ITOMDave]

OK, I *think* I've fixed my problem thanks to previous contributors.......

The "speedtest" I'm using is taken from https://www.speedtest.net/apps/cli . There is a python script available on Github which - as I understand it, uses the same API. There's nothing wrong with this, it does much the same thing as the executable I want to use.

The issue was made harder to diagnose by my not fully appreciating the difference between

zabbix_agentd -t

and

zabbix_get -k

Anyway, the core of the problem is that in order to run correctly, the first time that the speedtest executable is run it prompts to accept licensing and GDPR terms then writes a file to [CODE]~<currentuser>/.config/ookla/speedtest-cli.json[CODE]

For the user zabbix, this doesn't exist and zabbix isn't a user that you can login as (/bin/false is the shell for this user in /etc/passwd). What does exist though is that the home directory of the user is /var/lib/zabbix. However, the directory DOESN'T actually exist in the Zabbix 4.4 appliance image (which is understandable).

So, the instructions to fix this :

1. Add the user zabbix to the sudoer's file :

   sudo adduser zabbix sudo

2. Change the UserParameter definition to :

   UserParameter=internet.speedtest,sudo -u zabbix speedtest -f json --accept-license --accept-gdpr

3. Create the directory structure zabbix/.config/ookla beneath /var/lib and ensure they are owned by zabbix:zabbix so it looks like this :
   

   root@zabbix:~# [B]cd /var/lib/[/B]
   	root@zabbix:/var/lib# [B]ls -alR zabbix[/B]
   	zabbix:
   	total 12
   [B]drwxr-xr-x  3 zabbix zabbix 4096 Dec  4 19:44 .[/B]
   	drwxr-xr-x 34 root   root   4096 Dec  4 19:42 ..
   [B]drwxr-xr-x  3 zabbix zabbix 4096 Dec  4 19:43 .config[/B]
   	-rw-r--r--  1 zabbix zabbix    0 Dec  4 19:44 .sudo_as_admin_successful
   
   	zabbix/.config:
   	total 12
   	drwxr-xr-x 3 zabbix zabbix 4096 Dec  4 19:43 .
   	drwxr-xr-x 3 zabbix zabbix 4096 Dec  4 19:44 ..
   	drwxr-xr-x 2 zabbix zabbix 4096 Dec  4 19:44 ookla
   
   [B]zabbix/.config/ookla:[/B]
   	total 12
   [B]drwxr-xr-x 2 zabbix zabbix 4096 Dec  4 19:44 .[/B]
   	drwxr-xr-x 3 zabbix zabbix 4096 Dec  4 19:43 ..
   	-rw-r--r-- 1 zabbix zabbix  159 Dec  4 19:44 speedtest-cli.json

4. Restart the Zabbix agent

That now seems to work with both zabbix_agentd and zabbix_get

I'm not totally comfortable with it as I think it weakens security so if anyone wants to offer a revised solution, then I'm happy to take it on board.

Thanks again to all contributors.

Dave

[dimir]

Since zabbix agent is already run by user zabbix you don't need sudo in userparameter. So you don't need to add zabbix to sudoers. And you can even remove it's home directory, the speedtest will each time produce the error message but you can ignore it. This user parameter will work without sudo and home directory of zabbix user:

UserParameter=internet.speedtest,speedtest -f json --accept-license --accept-gdpr 2>&1 | grep bandwidth

[ITOMDave]

Thanks Dimir,
I'll test that out, although I'm pretty sure that unless I have zabbix in sudoers the speedtest executable throws an error. I didn't document that carefully at the time, so I'll retest. I'm prepared to be proved wrong

Thanks again.

Dave

[Kos]

It seems that the speedtest utility tries to find a home directory of the current user using an environment variable ($HOME).
So, a bit strange result (successful working via sudo with the same user) could be explained by setting this variable via sudo.
Therefore, a bit simpler solution could be:

UserParameter=internet.speedtest,HOME=/var/lib/zabbix speedtest -f json --accept-license --accept-gdpr

Of course, a real home directory should exist; however, there is no necessity of sudo, setEUID bit or something similar.

(link onto Russian-language discussion)