Problems with proxy setup

[selioitalo]

Hello everyone!

My Zabbix Proxy has recently decided it should send data hourly, or not send data at all. It is a small environment with ~7k items, mainly icmp and snmp checks.

Server and Proxy are 2.2.1, proxy runs active with sqlite. I tried changing to MySQL, going to 2.2.9 but the result is pretty much the same. If i drop the database and start a fresh one data gets sent in the first wave and then gets queued forever.

The setup worked for a month or two until this started to happen. Server and Proxy machines are pretty solid with small load and a lot of free memory, bigger pollers with values from 10 to 30.

There seems to be no communication issues, telnet goes back and forth fine, also i have run an NC proxy->server every 3s for a while with no errors.

Proxy is configured to 120s, tried bigger and smaller values to no change.

Here goes some logs from proxy:

  5828:20150708:170236.718 End of proxy_get_history_data():1000 lastid:10000
  5828:20150708:170236.718 In connect_to_server() [ZBX_SERVER]:10051 [timeout:600]
  5828:20150708:170236.748 In put_data_to_server() datalen:171648
  5828:20150708:170236.748 In send_data_to_server() [{
  5828:20150708:170236.883 In zbx_recv_response()
  5828:20150708:170321.638 zbx_recv_response() '{
  5828:20150708:170321.638 End of zbx_recv_response():SUCCEED
  5828:20150708:170321.638 End of put_data_to_server():SUCCEED
  5828:20150708:170321.638 query [txnlev:1] [begin;]
  5828:20150708:170321.638 In proxy_set_lastid() [proxy_history.history_lastid:10000]
  5828:20150708:170321.638 query [txnlev:1] [select 1 from ids where table_name='proxy_history' and field_name='history_lastid']
  5828:20150708:170321.638 query [txnlev:1] [update ids set nextid=10000 where table_name='proxy_history' and field_name='history_lastid']
  5828:20150708:170321.638 End of proxy_set_lastid()
  5828:20150708:170321.638 query [txnlev:1] [commit;]
  5828:20150708:170321.639 End of history_sender()
  5828:20150708:170321.639 In history_sender()
  5828:20150708:170321.639 In proxy_get_history_data()
  5828:20150708:170321.639 In proxy_get_lastid() field:'proxy_history.history_lastid'
  5828:20150708:170321.639 query [txnlev:0] [select nextid from ids where table_name='proxy_history' and field_name='history_lastid']
  5828:20150708:170321.639 End of proxy_get_lastid():10000
  5828:20150708:170321.639 query [txnlev:0] [select id,itemid,clock,ns,timestamp,source,severity,value,logeventid,state from proxy_history where id>10000 order by id limit 1000]
  5828:20150708:170321.650 End of proxy_get_history_data():1000 lastid:11000
  5828:20150708:170321.650 In connect_to_server() [ZBX_SERVER]:10051 [timeout:600]
  5828:20150708:170321.679 In put_data_to_server() datalen:175400
  5828:20150708:170321.679 In send_data_to_server() [{
  5828:20150708:170321.821 In zbx_recv_response()
  5828:20150708:170417.579 zbx_recv_response() '{
  5828:20150708:170417.579 End of zbx_recv_response():SUCCEED
  5828:20150708:170417.579 End of put_data_to_server():SUCCEED
  5828:20150708:170417.579 query [txnlev:1] [begin;]
  5828:20150708:170417.579 In proxy_set_lastid() [proxy_history.history_lastid:11000]
  5828:20150708:170417.579 query [txnlev:1] [select 1 from ids where table_name='proxy_history' and field_name='history_lastid']
  5828:20150708:170417.579 query [txnlev:1] [update ids set nextid=11000 where table_name='proxy_history' and field_name='history_lastid']
  5828:20150708:170417.579 End of proxy_set_lastid()
  5828:20150708:170417.579 query [txnlev:1] [commit;]
  5828:20150708:170417.579 End of history_sender()
  5828:20150708:170417.579 In history_sender()
  5828:20150708:170417.580 In proxy_get_history_data()
  5828:20150708:170417.580 In proxy_get_lastid() field:'proxy_history.history_lastid'
  5828:20150708:170417.580 query [txnlev:0] [select nextid from ids where table_name='proxy_history' and field_name='history_lastid']
  5828:20150708:170417.580 End of proxy_get_lastid():11000
  5828:20150708:170417.580 query [txnlev:0] [select id,itemid,clock,ns,timestamp,source,severity,value,logeventid,state from proxy_history where id>11000 order by id limit 1000]

From the server:

  1080:20150708:165059.035 sending configuration data to proxy "Proxy", datalen 1195179
  1099:20150708:165602.598 sending configuration data to proxy "Proxy", datalen 1195179
  1078:20150708:170106.819 sending configuration data to proxy "Proxy", datalen 1195179
  1083:20150708:170609.636 sending configuration data to proxy "Proxy", datalen 1195179

Any input is welcome. Thanks in advance!

[selioitalo]

More information,

In the rare moments where it decides to send data, it goes like this:

 20656:20150709:154110.349 In connect_to_server() [####]:10051 [timeout:600]
 20656:20150709:154110.368 In put_data_to_server() datalen:99953
 20656:20150709:154110.368 In send_data_to_server() [{
 20656:20150709:154110.422 In zbx_recv_response()
 20656:20150709:154111.090 zbx_recv_response() '{
 20656:20150709:154111.090 End of zbx_recv_response():SUCCEED
 20656:20150709:154111.090 End of put_data_to_server():SUCCEED
 20656:20150709:154111.091 query [txnlev:1] [begin;]
 20656:20150709:154111.091 In proxy_set_lastid() [proxy_history.history_lastid:61216]
 20656:20150709:154111.091 query [txnlev:1] [select 1 from ids where table_name='proxy_history' and field_name='history_lastid']
 20656:20150709:154111.091 query [txnlev:1] [update ids set nextid=61216 where table_name='proxy_history' and field_name='history_lastid']
 20656:20150709:154111.091 End of proxy_set_lastid()
 20656:20150709:154111.091 query [txnlev:1] [commit;]
 20656:20150709:154111.091 End of history_sender()
 20656:20150709:154111.091 In history_sender()
 20656:20150709:154111.091 In proxy_get_history_data_simple() table:'proxy_dhistory'
 20656:20150709:154111.092 In proxy_get_lastid() field:'proxy_dhistory.dhistory_lastid'
 20656:20150709:154111.092 query [txnlev:0] [select nextid from ids where table_name='proxy_dhistory' and field_name='dhistory_lastid']
 20656:20150709:154111.092 End of proxy_get_lastid():0
 20656:20150709:154111.092 query [txnlev:0] [select id,clock,druleid,dcheckid,type,ip,dns,port,key_,value,status from proxy_dhistory where id>0 order by id limit 1000]
 20656:20150709:154111.092 End of proxy_get_history_data_simple():0 lastid:0
 20656:20150709:154111.092 End of history_sender()
 20656:20150709:154111.092 In history_sender()
 20656:20150709:154111.092 In proxy_get_history_data_simple() table:'proxy_autoreg_host'
 20656:20150709:154111.092 In proxy_get_lastid() field:'proxy_autoreg_host.autoreg_host_lastid'
 20656:20150709:154111.092 query [txnlev:0] [select nextid from ids where table_name='proxy_autoreg_host' and field_name='autoreg_host_lastid']
 20656:20150709:154111.092 End of proxy_get_lastid():0
 20656:20150709:154111.092 query [txnlev:0] [select id,clock,host,listen_ip,listen_dns,listen_port,host_metadata from proxy_autoreg_host where id>0 order by id limit 1000]
 20656:20150709:154111.093 End of proxy_get_history_data_simple():0 lastid:0
 20656:20150709:154111.093 End of history_sender()
 20656:20150709:154111.093 data sender [sent 2679 values in 2.964939 sec, idle 30 sec]

I noticed it uses another function, proxy_get_history_data_simple()