Problems with script running on remote host

[marco-shagrat]

Hi,
I have a strange problem on some monitored hosts.

I have a zabbix server monitoring 100+ centos7 linux servers.
All servers are installed exactly the same way, same packages, same configuration files same zabbix agent.

I created a new custom item of type "zabbix agent" that runs a custom script on the servers.
On about 80 out of 100+ servers all runs fine and the script returns the expected json result, on the remaining servers I get 7 times out of 10 checks "/usr/bin/tac: write error: Broken pipe" followed by the correct Json result.

I did many tests manually from:

• zabbix server web interface "test button": same result (7 times out of 10 broken pipe error)
• zabbix server command line:
  zabbix_get -s x.x.x.x -p yyyy --tls-connect psk --tls-psk-identity PSKXXXXX --tls-psk-file /tmp/xxxx.psk -k zabbix_backup_remote_sync_check["<parameter>"] -> same result (7 times out of 10 broken pipe error)
• monitored server command line root user:
  zabbix_agentd -t zabbix_backup_remote_sync_check["<parameter>"] -> 100% success
• monitored server command line zabbix user:
  sudo -H -u zabbix bash -c 'zabbix_agentd -t zabbix_backup_remote_sync_check["<parameter>"]' -> 100% success

I tried changing zabbix_agent.conf Timeout setting on the monitored server (from 3 to 10) nothing changed (by the way the script runs in milliseconds)
I tried changing zabbix_server.conf Timeout setting on the zabbix server (from 3 to 10) nothing changed
I tried to write full paths into scripts command (grep -> /usr/bin/grep and so on) nothing changed

I don't know what to do more.

Thanks a lot for any advice.

[marco-shagrat]

Just one more info that i realized i didn't specify, all monitored servers are remote servers.

Zabbix agent ports are open only from the zabbix server public ip address.

All other monitoring checks work fine.

Thanks