server cannot get the data of log file monitored via proxy
My zabbix monitoring system is: zabbix agent <-> zabbix proxy <-> zabbix server.
Version of agent, proxy and server is the same: 3.0.8
My scenario is to monitor "WARNING" in a log file on the agent.
On server web, create an item:
When "WARNING" appears in the log, I can see it was caught and sent to proxy correctly from agent log:
"118423:20190528:154127.546 setup_old2new: is_same_file(/tmp/realError, /tmp/realError) = 1
118423:20190528:154127.546 process_logrt() old file list:
118423:20190528:154127.546 nr:0 filename:'/tmp/realError' mtime:1559028441 size:11723 processed_size:11723 seq:1 incomplete:0 dev:2051 ino_hi:0 ino_lo:1074296837 md5size:512 md5buf:05bb8ef15532d547d039e95e94b63713
118423:20190528:154127.546 process_logrt() new file list: (mtime:0 lastlogsize:11723 start_idx:0)
118423:20190528:154127.546 nr:0 filename:'/tmp/realError' mtime:1559029234 size:11812 processed_size:11723 seq:0 incomplete:0 dev:2051 ino_hi:0 ino_lo:1074296837 md5size:512 md5buf:05bb8ef15532d547d039e95e94b63713
118423:20190528:154127.546 In process_log() filename:'/tmp/realError' lastlogsize:11723 mtime:0
118423:20190528:154127.546 In process_value() key:'10.0.0.87:log[/tmp/realError,WARNING,,,skip,]' value:'2019-05-28 14:15:01 [WARNING] for debug, 4 proc lost: cctvnews, cctv15, cctv14, cctv10,'
118423:20190528:154127.546 In send_buffer() host:'10.0.0.225' port:10052 entries:0/100
118423:20190528:154127.546 End of send_buffer():SUCCEED
118423:20190528:154127.546 buffer: new element 0
118423:20190528:154127.546 End of process_value():SUCCEED
118423:20190528:154127.546 End of process_log() filename:'/tmp/realError' lastlogsize:11812 mtime:0 ret:SUCCEED
118423:20190528:154127.546 End of process_logrt():SUCCEED
118423:20190528:154127.546 In need_meta_update() key:log[/tmp/realError,WARNING,,,skip,]
118423:20190528:154127.546 End of need_meta_update():FAIL
118423:20190528:154127.546 End of process_active_checks()
118423:20190528:154127.546 In get_min_nextcheck()
118423:20190528:154127.546 End of get_min_nextcheck():1559029347
118423:20190528:154127.546 __zbx_zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
118421:20190528:154127.674 __zbx_zbx_setproctitle() title:'listener #2 [processing request]'
118421:20190528:154127.675 Requested [net.if.in[em3]]
118421:20190528:154127.675 Sending back [0]
118421:20190528:154127.675 __zbx_zbx_setproctitle() title:'listener #2 [waiting for connection]'
118419:20190528:154128.001 __zbx_zbx_setproctitle() title:'collector [processing data]'
118419:20190528:154128.002 In update_cpustats()
118419:20190528:154128.003 End of update_cpustats()
118419:20190528:154128.003 __zbx_zbx_setproctitle() title:'collector [idle 1 sec]'
118423:20190528:154128.546 In send_buffer() host:'10.0.0.225' port:10052 entries:1/100
118423:20190528:154128.547 JSON before sending [{"request":"agent data","data":[{"host":"10.0.0.87","key":"log[/tmp/realError,WARNING,,,skip,]","value":"2019-05-28 14:15:01 [WARNING] for debug, 4 proc lost: cctvnews, cctv15, cctv14, cctv10,","lastlogsize":11812,"clock":1559029287,"n s":546645149}],"clock":1559029288,"ns":547286281}]
118423:20190528:154128.548 JSON back [{"response":"success","info":"processed: 1; failed: 0; total: 1; seconds spent: 0.000391"}]
118423:20190528:154128.548 In check_response() response:'{"response":"success","info":"processed: 1; failed: 0; total: 1; seconds spent: 0.000391"}'
118423:20190528:154128.548 info from server: 'processed: 1; failed: 0; total: 1; seconds spent: 0.000391'
118423:20190528:154128.548 End of check_response():SUCCEED
118423:20190528:154128.548 OK
118423:20190528:154128.548 End of send_buffer():SUCCEED
"
And from the proxy log, I can see the proxy got the data correctly and no fail:
"
15236:20190528:154223.177 __zbx_zbx_setproctitle() title:'trapper #3 [processing data]'
15236:20190528:154223.177 trapper got '{"request":"agent data","data":[{"host":"10.0.0.87","key":"log[/tmp/realError,WARNING,,,skip,]","value":"2019-05-28 14:15:01 [WARNING] for debug, 4 proc lost: cctvnews, cctv15, cctv14, cctv10,","lastlogsize":11812,"clock":1559029287,"n s":546645149}],"clock":1559029288,"ns":547286281}'
15236:20190528:154223.177 In recv_agenthistory()
15236:20190528:154223.178 In process_hist_data()
15236:20190528:154223.178 In process_mass_data()
15236:20190528:154223.178 In calc_timestamp()
15236:20190528:154223.178 calc_timestamp() 00:00:00 00/00/0000
15236:20190528:154223.178 End of calc_timestamp() timestamp:0
15236:20190528:154223.178 End of process_mass_data()
15236:20190528:154223.178 End of process_hist_data():SUCCEED
15236:20190528:154223.178 In zbx_send_response()
15236:20190528:154223.178 zbx_send_response() '{"response":"success","info":"processed: 1; failed: 0; total: 1; seconds spent: 0.000391"}'
15236:20190528:154223.178 End of zbx_send_response():SUCCEED
15236:20190528:154223.178 End of recv_agenthistory()
15236:20190528:154223.178 __zbx_zbx_setproctitle() title:'trapper #3 [processed data in 0.000859 sec, waiting for connection]'
"
But I see nothing when I check the latest data and history on the server web:
This problem confuse me a lot for a long time. Any tips will be appreciated.
Version of agent, proxy and server is the same: 3.0.8
My scenario is to monitor "WARNING" in a log file on the agent.
On server web, create an item:
When "WARNING" appears in the log, I can see it was caught and sent to proxy correctly from agent log:
"118423:20190528:154127.546 setup_old2new: is_same_file(/tmp/realError, /tmp/realError) = 1
118423:20190528:154127.546 process_logrt() old file list:
118423:20190528:154127.546 nr:0 filename:'/tmp/realError' mtime:1559028441 size:11723 processed_size:11723 seq:1 incomplete:0 dev:2051 ino_hi:0 ino_lo:1074296837 md5size:512 md5buf:05bb8ef15532d547d039e95e94b63713
118423:20190528:154127.546 process_logrt() new file list: (mtime:0 lastlogsize:11723 start_idx:0)
118423:20190528:154127.546 nr:0 filename:'/tmp/realError' mtime:1559029234 size:11812 processed_size:11723 seq:0 incomplete:0 dev:2051 ino_hi:0 ino_lo:1074296837 md5size:512 md5buf:05bb8ef15532d547d039e95e94b63713
118423:20190528:154127.546 In process_log() filename:'/tmp/realError' lastlogsize:11723 mtime:0
118423:20190528:154127.546 In process_value() key:'10.0.0.87:log[/tmp/realError,WARNING,,,skip,]' value:'2019-05-28 14:15:01 [WARNING] for debug, 4 proc lost: cctvnews, cctv15, cctv14, cctv10,'
118423:20190528:154127.546 In send_buffer() host:'10.0.0.225' port:10052 entries:0/100
118423:20190528:154127.546 End of send_buffer():SUCCEED
118423:20190528:154127.546 buffer: new element 0
118423:20190528:154127.546 End of process_value():SUCCEED
118423:20190528:154127.546 End of process_log() filename:'/tmp/realError' lastlogsize:11812 mtime:0 ret:SUCCEED
118423:20190528:154127.546 End of process_logrt():SUCCEED
118423:20190528:154127.546 In need_meta_update() key:log[/tmp/realError,WARNING,,,skip,]
118423:20190528:154127.546 End of need_meta_update():FAIL
118423:20190528:154127.546 End of process_active_checks()
118423:20190528:154127.546 In get_min_nextcheck()
118423:20190528:154127.546 End of get_min_nextcheck():1559029347
118423:20190528:154127.546 __zbx_zbx_setproctitle() title:'active checks #1 [idle 1 sec]'
118421:20190528:154127.674 __zbx_zbx_setproctitle() title:'listener #2 [processing request]'
118421:20190528:154127.675 Requested [net.if.in[em3]]
118421:20190528:154127.675 Sending back [0]
118421:20190528:154127.675 __zbx_zbx_setproctitle() title:'listener #2 [waiting for connection]'
118419:20190528:154128.001 __zbx_zbx_setproctitle() title:'collector [processing data]'
118419:20190528:154128.002 In update_cpustats()
118419:20190528:154128.003 End of update_cpustats()
118419:20190528:154128.003 __zbx_zbx_setproctitle() title:'collector [idle 1 sec]'
118423:20190528:154128.546 In send_buffer() host:'10.0.0.225' port:10052 entries:1/100
118423:20190528:154128.547 JSON before sending [{"request":"agent data","data":[{"host":"10.0.0.87","key":"log[/tmp/realError,WARNING,,,skip,]","value":"2019-05-28 14:15:01 [WARNING] for debug, 4 proc lost: cctvnews, cctv15, cctv14, cctv10,","lastlogsize":11812,"clock":1559029287,"n s":546645149}],"clock":1559029288,"ns":547286281}]
118423:20190528:154128.548 JSON back [{"response":"success","info":"processed: 1; failed: 0; total: 1; seconds spent: 0.000391"}]
118423:20190528:154128.548 In check_response() response:'{"response":"success","info":"processed: 1; failed: 0; total: 1; seconds spent: 0.000391"}'
118423:20190528:154128.548 info from server: 'processed: 1; failed: 0; total: 1; seconds spent: 0.000391'
118423:20190528:154128.548 End of check_response():SUCCEED
118423:20190528:154128.548 OK
118423:20190528:154128.548 End of send_buffer():SUCCEED
"
And from the proxy log, I can see the proxy got the data correctly and no fail:
"
15236:20190528:154223.177 __zbx_zbx_setproctitle() title:'trapper #3 [processing data]'
15236:20190528:154223.177 trapper got '{"request":"agent data","data":[{"host":"10.0.0.87","key":"log[/tmp/realError,WARNING,,,skip,]","value":"2019-05-28 14:15:01 [WARNING] for debug, 4 proc lost: cctvnews, cctv15, cctv14, cctv10,","lastlogsize":11812,"clock":1559029287,"n s":546645149}],"clock":1559029288,"ns":547286281}'
15236:20190528:154223.177 In recv_agenthistory()
15236:20190528:154223.178 In process_hist_data()
15236:20190528:154223.178 In process_mass_data()
15236:20190528:154223.178 In calc_timestamp()
15236:20190528:154223.178 calc_timestamp() 00:00:00 00/00/0000
15236:20190528:154223.178 End of calc_timestamp() timestamp:0
15236:20190528:154223.178 End of process_mass_data()
15236:20190528:154223.178 End of process_hist_data():SUCCEED
15236:20190528:154223.178 In zbx_send_response()
15236:20190528:154223.178 zbx_send_response() '{"response":"success","info":"processed: 1; failed: 0; total: 1; seconds spent: 0.000391"}'
15236:20190528:154223.178 End of zbx_send_response():SUCCEED
15236:20190528:154223.178 End of recv_agenthistory()
15236:20190528:154223.178 __zbx_zbx_setproctitle() title:'trapper #3 [processed data in 0.000859 sec, waiting for connection]'
"
But I see nothing when I check the latest data and history on the server web:
This problem confuse me a lot for a long time. Any tips will be appreciated.