Data send processes on proxy always busy

[Toms]

Hello everyone,

I've some difficulties to troubleshoot my proxy installation.

I've done a fresh install with a proxy and a server in a 4.4 version. The server is located in a cloud and the proxy in our local infrastructure.

It worked well the first couple of days, but then, I added a couple of hosts (15 in total with 1320 item and a vps of 14.58) and it started to slow done.

The data were taking a more and more delay to appear on the interface and the data send processes on the proxy was always 100%.

I used the default proxy configuration.

My proxy is running on a VmWare hypervisor with the following settings:
2 CPU
4 GB ram
The db disk is on another partition than the system and they are both on a SSD datastore.
Mysql 8

I've increased the log verbosity, and I can see those messages about the data send process:
2184:20200303:174840.687 zbx_setproctitle() title:'data sender [sent 537 values in 60.538567 sec, idle 0 sec]'
2184:20200303:174840.687 zbx_setproctitle() title:'data sender [sent 537 values in 60.538567 sec, sending data]'

I've tried some tweaks, but nothing improved:
- Moved to a faster vmware datastore (ssd)
- Optimize mysql configuration (https://www.percona.com/blog/2014/11...-mysql-zabbix/)
- Changing DataSenderFrequency setting in the proxy configuration (from 1 to 10)
- Recreate a new empty proxy database
- Changing the ip address of the zabbix server from public IP to VPN.

I've ran an iperf test between the proxy and the server which seems pretty good:
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 7.42 MBytes 62.2 Mbits/sec 0 320 KBytes
[ 4] 1.00-2.00 sec 3.79 MBytes 31.8 Mbits/sec 41 100 KBytes
[ 4] 2.00-3.00 sec 3.42 MBytes 28.7 Mbits/sec 0 110 KBytes
[ 4] 3.00-4.00 sec 3.20 MBytes 26.8 Mbits/sec 1 93.7 KBytes

Is it possible to increase the number of sending process ? Maybe there is another parameter to change ?

Thank's by advance

[Toms]

I've done more tests to isolate the possible causes:
- I've created a new server in the same network than the proxy, so I'm sure that I didn't have network/firewall issues. But even then, I got the same performance in the data sender process (about 500/600 values send in 60sec).

- I've create a new proxy with a sqlite backend. And it seems to perform much, much better:
5546:20200305:111527.028 zbx_setproctitle() title:'data sender [sent 15 values in 0.117777 sec, sending data]'
5546:20200305:111527.147 zbx_setproctitle() title:'data sender [sent 40 values in 0.119781 sec, idle 1 sec]'
5546:20200305:111528.147 zbx_setproctitle() title:'data sender [sent 40 values in 0.119781 sec, sending data]'
5546:20200305:111528.262 zbx_setproctitle() title:'data sender [sent 13 values in 0.114940 sec, idle 1 sec]'

I don't have delay anymore and the items queue is normal.

I have then changed the proxy configuration to send the data to the cloud server and it work in the same way.

So now I assume it's a mysql misconfiguration. I've installed the proxy with a mysql 8 version, can it be the cause ?

[Toms]

I've resolved my problem !

As I thought, it was a mysql 8 misconfiguration.

I had used a mysql 5.7 config file used on our others servers, it was obviously not adapted for v8 and zabbix.

So now I use the default mysql configuration ahd the parameters provided in https://www.percona.com/blog/2014/11...-mysql-zabbix/.

[tim.mooney]

Nice find, I had not found that Percona post before. Something specifically for Zabbix from the Percona people is very useful!

[ricofranco]

Did you reach the same performance instead the sqlite ?