Proxy Stops Executing External Scripts on Process Restart

[Massimiliano]

Hi,

I'm experiencing an odd behavior with the restart of Zabbix Proxy 2.4.x

Below the action sequence:
1) stop the zabbix proxy
2) delete the sqlite DB file
3) start again the zabbix proxy

The zabbix proxy process results up&running in a few seconds but items related to external scripts execution are working again after about 15 minutes.
I know the item configuration must be downloaded from the zbx server again, in fact I configured this within the proxy conf file:
ConfigFrequency=30

The is issue seems to disappear if I avoid removing the sqlite DB file.

- Is this an expected beaviour?
- Is there any parameter I can tune to reduce the 15 minutes delay?

Thanks,

[kloczek]

ConfigFrequency it is parameter used on active proxy (is you proxy is registered on zabbix server as active one?) which regulates how frequently active proxy is asking/querying zabbix srv for monitoring configuration of the hosts with items monitored over this proxy.
If your proxy is passive one
If you ave delays in delivery data from proxy to srv it may mean that for example:
- you have a to transfer from proxy to server
- you have database performance issues
- you external scripts are stuck on sampling monitoring data
- on server side you have saturated trapper processes (in case of using active proxy or pollers (in case of using passive proxy)

Above are not only possible causes.

Are you monitoring you proxy and server using dist zabbix proxy sand server templates?
Items grouped in those templates shows state of the proxies and server. Usually over those data is possible to say more what possibly goes wrong.

[Massimiliano]

I can confirm my proxy is configured as an active one.

I'm wondering if there is any specific log trace can highlight the fact the configuration is pulled from the zabbix server with a long delay.

Thanks,

[kloczek]

IMO your symptoms looks like some prx DB backend bottleneck.

Do you have standard OS level monitoring on the host where is running prx?
Do you have prx monitoring (using standard zabbix proxy template)?
Did you check IOs on the storage used by DB backend?
Which one DB backen is using your proxy?

Every received batch of cfg data is logged and looking on time stamps of those entries you can check are they in logs every ConfigFrequency seconds.

[Massimiliano]

The proxy backend is a sqlite DB and the I/O is on SD card since I'm running the proxy on a Rpi 2.

Actually I'm not experiencing a device load could justify the 15 minute delay.

I'll look at the log traces you said are relevant for this kind of investigation.

[kloczek]

The proxy backend is a sqlite DB and the I/O is on SD card since I'm running the proxy on a Rpi 2.

Actually I'm not experiencing a device load could justify the 15 minute delay.

I'll look at the log traces you said are relevant for this kind of investigation.

What I suppose is not related at all to CPU load
It is about number of IOs which you are doing on this SD card.
remember that every modification of the sqlite file like insert is causing that whole file is rewritten.
If you are using SD card as backend storage depends on how many hosts is monitored your proxy you may have enough big number of IOs on only writing data to sqlite DB file.
This why I've asked you peak on this host number of IO/s
Back .. in case of some cheap SD cards higher possibility is that you are not saturating max IO/s but max bytes/s on writing to the card and this is causes that you proxy is chocking.

[Massimiliano]

You are right, the root cause seems to be the SD I/O slowness.

Now I'm trying to put the sqlite storage into memory, but the special in-memory storage placeholder :memory: usually recognized by the sqlite library seems not allowed by the proxy configuration :-(

P.S. I'm passing ":memory:" through the DBName configuration variable

Thanks,

[kloczek]

I'm not 100% but IIRC zabbix may not support this method.