zabbix sender is slow on some of servers

[BlackCrystal]

I have a bash script that calculates 3 database parameters then send them to zabbix via zabbix_sender. here are the zabbix sender commands at the end of script:

/opt/zabbix_agent/bin/zabbix_sender -z 153.32.111.26 -p 10700 -s hostVIP -k "avgResponseTime" -o $avg
/opt/zabbix_agent/bin/zabbix_sender -z 153.32.111.26 -p 10700 -s hostVIP -k "99percentResponseTime" -o $PERC99
/opt/zabbix_agent/bin/zabbix_sender -z 153.32.111.26 -p 10700 -s hostVIP -k "95percentResponseTime" -o $PERC95

this script is on like 120 servers which it run every 2 minutes without any problems. but there are 2 servers that zabbix sender is too slow on them. i ran the script with `bash -x` to debug and i saw that `avg` is send on them as fast as other servers and `PERC99` and `PERC95` are the slow ones. it took like 1 minutes for each to send their values.

/opt/zabbix_agent/bin/zabbix_sender -z 153.32.111.26 -p 10700 -s hostVIP -k "99percentResponseTime" -o 20
/opt/zabbix_agent/bin/zabbix_sender -z 153.32.111.26 -p 10700 -s hostVIP -k "95percentResponseTime" -o 25

i also get this error: with -vv option:
zabbix_sender [118696]: Warning: timeout while executing operation
sent: 1; skipped: 0; total: 1

it is confusing that only those 2 commands are slow and first command has no problem. in zabbix web these 2 items are show as [no data]. anyone has any suggestion to how to debug and solve this?

zabbix version is 5.

[Markku]

You can use tcpdump (on the clients and on the Zabbix server) to see what is happening differently with the problem hosts/items.

A relevant question is: what is different in/for those two problem hosts?

Markku