Communicate between Zabbix Client and Zabbix Server are having an issue with PSK

[tienpt]

Hello,

First of all, I am having an issue with communication PSK. Hope you help me understand why? and how to fix it?

1. Topology
Zabbix Client: 10.30.3.115
Zabbix Server: 10.30.3.116
OS: Ubuntu 16.04 LTS
Zabbix Server

sky@zabbix-srv-01:~$ dpkg -l | grep zabbix
ii zabbix-agent 1:3.2.11-1+xenial amd64 Zabbix network monitoring solution - agent
ii zabbix-frontend-php 1:3.2.11-1+xenial all Zabbix network monitoring solution - PHP front-end
ii zabbix-get 1:3.2.11-1+xenial amd64 Zabbix network monitoring solution - get
ii zabbix-release 3.2-1+xenial all Zabbix official repository configuration
ii zabbix-server-mysql 1:3.2.11-1+xenial amd64 Zabbix network monitoring solution - server (MySQL)

Zabbix Client

sky@graylog-srv-01:~$ dpkg -l | grep zabbix
ii zabbix-agent 1:3.2.11-1+xenial amd64 Zabbix network monitoring solution - agent
ii zabbix-get 1:3.2.11-1+xenial amd64 Zabbix network monitoring solution - get
ii zabbix-release 3.2-1+xenial all Zabbix official repository configuration

sky@graylog-srv-01:~$ ls -lrt /etc/zabbix/
total 32
drwxr-xr-x 2 root root 4096 Feb 12 02:20 zabbix_agentd.d
-rw-r--r-- 1 root root 65 Feb 12 02:20 zabbix_agentd.psk
-rw-r--r-- 1 root root 10347 Feb 12 02:20 zabbix_agentd.conf.orig
-rw-r--r-- 1 root root 10364 Feb 13 02:38 zabbix_agentd.conf

Zabbix Client communicate directly with Zabbix Server without Zabbix Proxy. I installed Zabbix package from repository for both of them.

2. What did I implement?
On Zabbix Client, I wrote the agent configuration as below:

~$ sudo vim /etc/zabbix/zabbix_agentd.conf

...

Server=10.30.3.116 # Zabbix Server
ServerActive=10.30.3.116
TLSConnect=psk
TLSAccept=psk
TLSPSKIdentity=PSK 007
TSLPSKFile=/etc/zabbix/zabbix_agentd.psk

Zabbix agent is running on port 10050

sky@graylog-srv-01:~$ netstat -tapn | grep 10050
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN -
tcp6 0 0 :::10050 :::* LISTEN -
sky@graylog-srv-01:~$

and Zabbix server is running on port 10051

sky@zabbix-srv-01:~$ netstat -tapln | grep 10051
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:10051 0.0.0.0:* LISTEN -
tcp6 0 0 :::10051 :::* LISTEN -
tcp6 0 0 ::1:41360 ::1:10051 TIME_WAIT -
tcp6 0 0 ::1:41344 ::1:10051 TIME_WAIT -
tcp6 0 0 ::1:41460 ::1:10051 TIME_WAIT -
tcp6 0 0 ::1:41396 ::1:10051 TIME_WAIT -
tcp6 0 0 ::1:41432 ::1:10051 TIME_WAIT -

Then I added successfully Zabbix Client to Zabbix Server, also enabled PSK at "connections to host" and "connection from host" of Encryption tab in Configuration/ Host. Screen of Zabbix Client appears well, I can create some screens about CPU, network and memory.



After that, I need to create a custom monitoring for Graylog process. Now I need to use zabbix-get, and, from here, I found an issue with PSK.

3. What happened?
Zabbix Client: 10.30.3.115
Zabbix Server: 10.30.3.116

I declared the IP of Zabbix Agent to Zabbix Agent configuration on Zabbix Server.

sky@zabbix-srv-01:~$ sudo vim /etc/zabbix/zabbix_agentd.conf
Server=127.0.0.1,10.30.3.115,10.30.3.116

And zabbix-get works on Zabbix Server

sky@graylog-srv-01:~$ zabbix_get -s 10.30.3.116 -k 'proc.num[zabbix_server,zabbix]'
28
sky@graylog-srv-01:~$

But, it is not working with Zabbix Client

sky@graylog-srv-01:~$ zabbix_get -s 10.30.3.115 -k 'proc.num[zabbix_server,zabbix]'
zabbix_get [8674]: Check access restrictions in Zabbix agent configuration

Although, I already declared IP of Zabbix Client at Server directive at /etc/zabbix/zabbix_agentd.conf. So I traced the log file and found the issue with PSK.

Zabbix server log

sky@zabbix-srv-01:~$ tail -f /var/log/zabbix/zabbix_server.log
31582:20190213:023538.060 cannot send list of active checks to "10.30.3.115": connection of type "TLS with PSK" is not allowed for host "Zabbix server"
31581:20190213:023625.636 cannot send list of active checks to "10.30.3.115": connection of type "TLS with PSK" is not allowed for host "Zabbix server"
31581:20190213:023734.472 cannot send list of active checks to "10.30.3.115": connection of type "TLS with PSK" is not allowed for host "Zabbix server"
31581:20190213:023808.172 cannot send list of active checks to "10.30.3.115": connection of type "TLS with PSK" is not allowed for host "Zabbix server"
31581:20190213:023900.675 cannot send list of active checks to "10.30.3.115": connection of type "TLS with PSK" is not allowed for host "Zabbix server"
31581:20190213:024100.690 cannot send list of active checks to "10.30.3.115": connection of type "TLS with PSK" is not allowed for host "Zabbix server"
31583:20190213:024300.703 cannot send list of active checks to "10.30.3.115": connection of type "TLS with PSK" is not allowed for host "Zabbix server"
31582:20190213:024500.717 cannot send list of active checks to "10.30.3.115": connection of type "TLS with PSK" is not allowed for host "Zabbix server"
31581:20190213:024700.732 cannot send list of active checks to "10.30.3.115": connection of type "TLS with PSK" is not allowed for host "Zabbix server"
31581:20190213:024809.099 cannot send list of active checks to "10.30.3.115": connection of type "TLS with PSK" is not allowed for host "Zabbix server"

Zabbix client log

sky@graylog-srv-01:~$ zabbix_get -s 127.0.0.1 -k 'proc.num[zabbix_server,zabbix]'
zabbix_get [29503]: Check access restrictions in Zabbix agent configuration
29495:20190213:031609.353 no active checks on server [10.30.3.116:10051]: connection of type "TLS with PSK" is not allowed for host "Zabbix server"
29494:20190213:024812.549 failed to accept an incoming connection: from 127.0.0.1: unencrypted connections are not allowed

As I see, the PSK is having an issue. So now, my question are:
- How to fix an issue "zabbix_get [8674]: Check access restrictions in Zabbix agent configuration"
- How to fix an issue with PSK?
- What did I misconfigured?

Thank you so much!!!

[ingus.vilnis]

Hi,

Awesome level of detail in your description.

Let's brake it down into two issues.
Zabbix Get first.

You can't use the zabbix_get in the unencrypted way anymore as you used to do before. Add the additional PSK info so the command looks something like this:

zabbix_get -s 10.30.3.115 -k 'proc.num[zabbix_server,zabbix]'  --tls-connect=psk --tls-psk-identity="PSK 007" --tls-psk-file=/etc/zabbix/zabbix_agentd.psk

Check the docs here: https://www.zabbix.com/documentation...cation_example

Another issue is that you get misleading messages in the logs. And they are because the Hostname setting in your zabbix_agentd.conf file is left to default "Zabbix server" whereas you need it to match the name which you defined in the web interface, in your case Hostname=graylog-srv-01

Restart the agent to apply and check the logs again.

Hope this helps!

[tienpt]

Thank you so much @ingus.vilnis

Another issue was solved. Zabbix Get still didn't solve. I used to run the below command

skylab@graylog-srv-01:~$ zabbix_get -s 10.30.3.115 -k 'proc.num[zabbix_agentd,zabbix]' --tls-connect=psk --tls-psk-identity="PSK 007" --tls-psk-file=/etc/zabbix/zabbix_agentd.psk
zabbix_get [11511]: Warning: SSL_shutdown() with 10.30.3.115 returned error code 5: TLS read warning alert "close notify"
zabbix_get [11511]: Get value error: connection closed during read
zabbix_get [11511]: Check access restrictions in Zabbix agent configuration

Then the log generated

11445:20190213:225423.072 failed to accept an incoming connection: connection from "10.30.3.115" rejected, allowed hosts: "10.30.3.116"
11444:20190213:225429.137 failed to accept an incoming connection: connection from "10.30.3.115" rejected, allowed hosts: "10.30.3.116"
11444:20190213:225640.934 failed to accept an incoming connection: connection from "10.30.3.115" rejected, allowed hosts: "10.30.3.116"

[ingus.vilnis]

You can't run zabbix_get on the server which you are actually monitoring. Do this on the Zabbix server.

If you still need to tun zabbix_get on the graylog-srv-01 then add 127.0.0.1 to the Server setting in zabbix_agentd.conf. But I am now not sure why you would want to do this in the first place. zabbix_get is great for testing communication typically between server and agent.

[tienpt]

Firstly, the command still didn't succeed.

skylab@graylog-srv-01:~$ zabbix_get -s 10.30.3.115 -k 'proc.num[zabbix_agentd,zabbix]' --tls-connect=psk --tls-psk-identity="PSK 007" --tls-psk-file=/etc/zabbix/zabbix_agentd.psk
zabbix_get [11701]: Get value error: connection closed during read
zabbix_get [11701]: Check access restrictions in Zabbix agent configuration

and log

11698:20190213:230858.612 failed to accept an incoming connection: connection from "10.30.3.115" rejected, allowed hosts: "127.0.0.1,10.30.3.116"

However, the Warning: SSL_shutdown() with 10.30.3.115 returned error code 5: TLS read warning alert "close notify" have gone :-)

Another point, I got your recommendation about zabbix get. It means that I need to create a library of PSK on the Zabbix Server side? to store all of PSK of Zabbix Client. As you said before: "You can't use the zabbix_get in the unencrypted way anymore as you used to do before. Add the additional PSK info so the command looks something like this:"

Then it should look like

skylab@zabbix-srv-01:~$ zabbix_get -s 10.30.3.115 -k 'proc.num[zabbix_agentd,zabbix]' --tls-connect=psk --tls-psk-identity="PSK 007" --tls-psk-file=/etc/zabbix/zabbix_agentd.psk

[tienpt]

You are right! I have just copied the PSK of Zabbix Client to Zabbix Server, then it worked as well

skylab@graylog-srv-01:~$ scp /etc/zabbix/zabbix_agentd.psk [email protected]:/tmp

and

skylab@zabbix-srv-01:~$ zabbix_get -s 10.30.3.115 -k 'proc.num[zabbix_agentd,zabbix]' --tls-connect=psk --tls-psk-identity="PSK 007" --tls-psk-file=/tmp/zabbix_agentd.psk
7

[ingus.vilnis]

Hold on.

Don't run zabbix_get on graylog-srv-01 for now. Run it from zabbix-srv-01 !

And don't mess with PSK on Zabbix server. That was not what I meant with that comment.

[ingus.vilnis]

Ok, now you got this running. Nice.

[tienpt]

skylab@zabbix-srv-01:~$ zabbix_get -s 10.30.3.115 -k 'proc.num[graylog-server,graylog]' --tls-connect=psk --tls-psk-identity="PSK 007" --tls-psk-file=/tmp/zabbix_agentd.psk
1

Here is what I need to do. I am going to create an script or something like that to monitor Graylog process at graylog-srv-01. On this topic, I learned from you about best practice for using Zabbix Get. I am a newbie, then my knowledge is breaking by google. [I need to read the book to structure my knowledge]

So now, of course, I don't need to run Zabbix Get on Client [graylog-srv-01]. Thus I need to organize the PSK's Client on Server.

I see that's what you want to let me know.

[ingus.vilnis]

Ok, that's fine.

But still why you need Zabbix Get if (what I think) you really need this configured in Zabbix from web interface?

[tienpt]

And don't mess with PSK on Zabbix server. That was not what I meant with that comment.

If I don't mess with PSK on Zabbix Server, Zabbix Get could to succeed.

skylab@zabbix-srv-01:~$ zabbix_get -s 10.30.3.115 -k 'proc.num[graylog-server,graylog]' --tls-connect=psk --tls-psk-identity="PSK 007" --tls-psk-file=/tmp/zabbix_agentd.psk
1

skylab@zabbix-srv-01:~$ zabbix_get -s 10.30.3.115 -k 'proc.num[graylog-server,graylog]' --tls-connect=psk --tls-psk-identity="PSK 007"
zabbix_get [27289]: ERROR: parameter "--tls-psk-identity" is defined, but "--tls-psk-file" is not defined

skylab@zabbix-srv-01:~$ zabbix_get -s 10.30.3.115 -k 'proc.num[graylog-server,graylog]' --tls-connect=psk
zabbix_get [27293]: ERROR: parameter "--tls-connect" value requires "--tls-psk-file", but it is not defined

skylab@zabbix-srv-01:~$ zabbix_get -s 10.30.3.115 -k 'proc.num[graylog-server,graylog]'
zabbix_get [27296]: Check access restrictions in Zabbix agent configuration

And log on Client

11698:20190213:232251.765 failed to accept an incoming connection: from 10.30.3.116: unencrypted connections are not allowed

[tienpt]

Ah I need Zabbix Get because some guys [from result's Google] let me should use Zabbix Get to create an script to monitor Graylog process or another Linux process. Could you let me know what is best practice to monitor Linux process on Client with Zabbix?

[ingus.vilnis]

If you just need to list the process count then use the same "proc.num[graylog-server,graylog]" item and configure it in a Zabbix Template. Are you familiar with that?

If you need something more then need to know what it is.

[tienpt]

I don't familiar with that. But, I can read Zabbix documentation and try it. Do you have any advice for me?

[tienpt]

My purpose is monitoring completely Linux process to keep the monitoring 24/7. Anytime, when the Linux process have any issue, then stop, I will know as soon as possible and go to fix.