Ad Widget

Collapse

Zabbix Proxy "Last Seen" isn´t Updated since update to 3.4.7

Collapse
X
Collapse
 
  • Page of 2
  • Time
  • Show
Clear All
new posts
Previous 1 2 template Next
  • zt1
    Junior Member
    • Apr 2018
    • 6
    #16
    Server debug level = 5

    Few interesting lines:
    Code:
     value:0 state:1 error:'Invalid second parameter.' lastchange:0
(...)
7561:20180415:142144.450   value:0 state:0 error:'' lastchange:1517075036
  7561:20180415:142144.451   value:0 state:0 error:'' lastchange:1517513410
[U][B]7561:20180415:142144.451   value:0 state:1 error:'Invalid second parameter.' lastchange:0[/B][/U]
  7561:20180415:142144.451   value:1 state:0 error:'' lastchange:1492532232
[U][B]7561:20180415:142144.452   value:0 state:1 error:'Invalid second parameter.' lastchange:0[/B][/U]
  7561:20180415:142144.452   value:0 state:0 error:'' lastchange:1517513474
  7561:20180415:142144.452   value:0 state:0 error:'' lastchange:1517513535
    Another logs:
    Code:
    tail -f /var/log/zabbix/zabbix_server.log | grep proxy | grep error
  7561:20180415:143050.607 query [txnlev:0] [select hostid,proxy_hostid,host,ipmi_authtype,ipmi_privilege,ipmi_username,ipmi_password,maintenance_status,maintenance_type,maintenance_from,errors_from,available,disable_until,snmp_errors_from,snmp_available,snmp_disable_until,ipmi_errors_from,ipmi_available,ipmi_disable_until,jmx_errors_from,jmx_available,jmx_disable_until,status,name,lastaccess,error,snmp_error,ipmi_error,jmx_error,tls_connect,tls_accept,tls_issuer,tls_subject,tls_psk_identity,tls_psk from hosts where status in (0,1,5,6) and flags<>2]
  7561:20180415:143051.368   value:0 state:1 error:'Cannot evaluate expression: "Cannot evaluate function "Zabbix proxy UG2:zabbix[process,ipmi poller,avg,busy].avg(10m)": item is not supported.".' lastchange:0
  7561:20180415:143051.368   value:0 state:1 error:'Cannot evaluate expression: "Cannot evaluate function "Zabbix proxy UG2:zabbix[process,java poller,avg,busy].avg(10m)": item is not supported.".' lastchange:0
  7561:20180415:143051.369   value:0 state:1 error:'Cannot evaluate expression: "Cannot evaluate function "Zabbix proxy UG2:zabbix[process,snmp trapper,avg,busy].avg(10m)": item is not supported.".' lastchange:0
  7561:20180415:143051.371   value:0 state:1 error:'Cannot evaluate expression: "Cannot evaluate function "Zabbix proxy UG2:zabbix[wcache,text,pfree].min(10m)": item is not supported.".' lastchange:0
  7561:20180415:143151.431 query [txnlev:0] [select hostid,proxy_hostid,host,ipmi_authtype,ipmi_privilege,ipmi_username,ipmi_password,maintenance_status,maintenance_type,maintenance_from,errors_from,available,disable_until,snmp_errors_from,snmp_available,snmp_disable_until,ipmi_errors_from,ipmi_available,ipmi_disable_until,jmx_errors_from,jmx_available,jmx_disable_until,status,name,lastaccess,error,snmp_error,ipmi_error,jmx_error,tls_connect,tls_accept,tls_issuer,tls_subject,tls_psk_identity,tls_psk from hosts where status in (0,1,5,6) and flags<>2]
  7561:20180415:143152.186   value:0 state:1 error:'Cannot evaluate expression: "Cannot evaluate function "Zabbix proxy UG2:zabbix[process,ipmi poller,avg,busy].avg(10m)": item is not supported.".' lastchange:0
  7561:20180415:143152.186   value:0 state:1 error:'Cannot evaluate expression: "Cannot evaluate function "Zabbix proxy UG2:zabbix[process,java poller,avg,busy].avg(10m)": item is not supported.".' lastchange:0
  7561:20180415:143152.187   value:0 state:1 error:'Cannot evaluate expression: "Cannot evaluate function "Zabbix proxy UG2:zabbix[process,snmp trapper,avg,busy].avg(10m)": item is not supported.".' lastchange:0
  7561:20180415:143152.189   value:0 state:1 error:'Cannot evaluate expression: "Cannot evaluate function "Zabbix proxy UG2:zabbix[wcache,text,pfree].min(10m)": item is not supported.".' lastchange:0
  7561:20180415:143252.244 query [txnlev:0] [select hostid,proxy_hostid,host,ipmi_authtype,ipmi_privilege,ipmi_username,ipmi_password,maintenance_status,maintenance_type,maintenance_from,errors_from,available,disable_until,snmp_errors_from,snmp_available,snmp_disable_until,ipmi_errors_from,ipmi_available,ipmi_disable_until,jmx_errors_from,jmx_available,jmx_disable_until,status,name,lastaccess,error,snmp_error,ipmi_error,jmx_error,tls_connect,tls_accept,tls_issuer,tls_subject,tls_psk_identity,tls_psk from hosts where status in (0,1,5,6) and flags<>2]
  7561:20180415:143253.025   value:0 state:1 error:'Cannot evaluate expression: "Cannot evaluate function "Zabbix proxy UG2:zabbix[process,ipmi poller,avg,busy].avg(10m)": item is not supported.".' lastchange:0
  7561:20180415:143253.025   value:0 state:1 error:'Cannot evaluate expression: "Cannot evaluate function "Zabbix proxy UG2:zabbix[process,java poller,avg,busy].avg(10m)": item is not supported.".' lastchange:0
  7561:20180415:143253.026   value:0 state:1 error:'Cannot evaluate expression: "Cannot evaluate function "Zabbix proxy UG2:zabbix[process,snmp trapper,avg,busy].avg(10m)": item is not supported.".' lastchange:0
  7561:20180415:143253.028   value:0 state:1 error:'Cannot evaluate expression: "Cannot evaluate function "Zabbix proxy UG2:zabbix[wcache,text,pfree].min(10m)": item is not supported.".' lastchange:0
    I'im not sure (I'm not a developer) but bold and underline rows in first quoted logs may be the key: invalid second parameter? What parameter?

    Code:
    tail -f /var/log/zabbix/zabbix_server.log | grep Invalid
  7561:20180415:144200.069   state:1 error:'Invalid second parameter.'
  7561:20180415:144200.072   state:1 error:'Invalid second parameter.'
  7561:20180415:144200.088   state:1 error:'Invalid second parameter.'
  7561:20180415:144200.509   state:1 error:'Invalid second parameter.'
  7561:20180415:144200.526   value:0 state:1 error:'Invalid second parameter.' lastchange:0
  7561:20180415:144200.527   value:0 state:1 error:'Invalid second parameter.' lastchange:0
  7561:20180415:144200.532   value:0 state:1 error:'Invalid second parameter.' lastchange:0
  7561:20180415:144300.944   state:1 error:'Invalid second parameter.'
  7561:20180415:144300.946   state:1 error:'Invalid second parameter.'
  7561:20180415:144300.962   state:1 error:'Invalid second parameter.'
  7561:20180415:144301.366   state:1 error:'Invalid second parameter.'
  7561:20180415:144301.382   value:0 state:1 error:'Invalid second parameter.' lastchange:0
  7561:20180415:144301.383   value:0 state:1 error:'Invalid second parameter.' lastchange:0
  7561:20180415:144301.388   value:0 state:1 error:'Invalid second parameter.' lastchange:0
    Nope... this is not this:
    Code:
    7561:20180415:144200.526 triggerid:13682 description:'Zabbix data sender processes more than 75% busy' type:0 status:0 priority:3
  7561:20180415:144200.526   expression:'({TRIGGER.VALUE}=0 and {13285}>75) or ({TRIGGER.VALUE}=1 and {13285}>65)' recovery_expression:''
  7561:20180415:144200.526   value:0 state:1 error:'Invalid second parameter.' lastchange:0
  7561:20180415:144200.526   correlation_tag:'' recovery_mode:'0' correlation_mode:'0'
  7561:20180415:144200.526   topoindex:1 functional:0 locked:0
    Any sugesstions?

    How about this?
    Code:
    (...)
item [Zabbix proxy UG2:zabbix[wcache,text,pfree]] error: Invalid second parameter.
(...)
    For now we know only this: db is not updating in 'lastaccess' column of proxies.
    Last edited by zt1; 15-04-2018, 15:25.

      Comment

      • vso
        Zabbix developer
        • Aug 2016
        • 190
        #17
        Can you please grep for zbx_recv_proxy_data

          Comment

          • zt1
            Junior Member
            • Apr 2018
            • 6
            #18
            As you wish:
            Debug level 5.
            Code:
            6655:20180415:191603.811 In zbx_recv_proxy_data()
  6655:20180415:191603.817 End of zbx_recv_proxy_data():SUCCEED
  6655:20180415:191603.873 In zbx_recv_proxy_data()
  6655:20180415:191603.884 End of zbx_recv_proxy_data():SUCCEED
  6657:20180415:191604.824 In zbx_recv_proxy_data()
  6657:20180415:191604.829 End of zbx_recv_proxy_data():SUCCEED
  6655:20180415:191604.942 In zbx_recv_proxy_data()
  6655:20180415:191604.947 End of zbx_recv_proxy_data():SUCCEED
  6658:20180415:191605.835 In zbx_recv_proxy_data()
  6658:20180415:191605.842 End of zbx_recv_proxy_data():SUCCEED
  6656:20180415:191605.999 In zbx_recv_proxy_data()
  6656:20180415:191606.014 End of zbx_recv_proxy_data():SUCCEED
  6656:20180415:191606.849 In zbx_recv_proxy_data()
            0 errors. All succeed.
            Any other suggestions or ideas?

              Comment

              • vso
                Zabbix developer
                • Aug 2016
                • 190
                #19
                Originally posted by zt1
                As you wish:
                Debug level 5.
                Code:
                6655:20180415:191603.811 In zbx_recv_proxy_data()
6655:20180415:191603.817 End of zbx_recv_proxy_data():SUCCEED
6655:20180415:191603.873 In zbx_recv_proxy_data()
6655:20180415:191603.884 End of zbx_recv_proxy_data():SUCCEED
6657:20180415:191604.824 In zbx_recv_proxy_data()
6657:20180415:191604.829 End of zbx_recv_proxy_data():SUCCEED
6655:20180415:191604.942 In zbx_recv_proxy_data()
6655:20180415:191604.947 End of zbx_recv_proxy_data():SUCCEED
6658:20180415:191605.835 In zbx_recv_proxy_data()
6658:20180415:191605.842 End of zbx_recv_proxy_data():SUCCEED
6656:20180415:191605.999 In zbx_recv_proxy_data()
6656:20180415:191606.014 End of zbx_recv_proxy_data():SUCCEED
6656:20180415:191606.849 In zbx_recv_proxy_data()
                0 errors. All succeed.
                Any other suggestions or ideas?
                This means that data from proxy is indeed received successfully, was there anything interesting between
                In zbx_recv_proxy_data() and End of zbx_recv_proxy_data():SUCCEED ?

                Are you open to patching your zabbix server if I provide patch with additional debug ?
                Last edited by vso; 15-04-2018, 19:57.

                  Comment

                  • pc-nico
                    Junior Member
                    • May 2014
                    • 29
                    #20
                    i found this in the Increased Logfile:

                    Code:
                    1880:20180416:105520.001 __zbx_zbx_setproctitle() title:'trapper #8 [processed data in 0.040734 sec, waiting for connection]'
1873:20180416:105520.119 In zbx_tls_accept()
1873:20180416:105520.121 zbx_psk_server_cb() requested PSK identity "co-monitoring-XXXX"
1873:20180416:105520.121 End of zbx_tls_accept():SUCCEED (established TLSv1.2 PSK-AES128-CBC-SHA)
1873:20180416:105520.121 __zbx_zbx_setproctitle() title:'trapper #1 [processing data]'
1873:20180416:105520.123 trapper got '{"request":"proxy data","host":"co-monitoring-XXXX","version":"3.4.8","clock":1523868920,"ns":16 8147009}'
1873:20180416:105520.123 In zbx_recv_proxy_data()
1873:20180416:105520.123 query [txnlev:1] [begin;]
1873:20180416:105520.123 query [txnlev:1] [update hosts set lastaccess=1523868919 where hostid=47721381625856;
update hosts set lastaccess=1523868919 where hostid=[COLOR=#FF0000]47738561495040[/COLOR];
update hosts set lastaccess=1523868919 where hostid=[COLOR=#FF0000]47747151429632[/COLOR];
update hosts set lastaccess=1523868920 where hostid=[COLOR=#FF0000]49370649067520[/COLOR];
update hosts set lastaccess=1523868919 where hostid=50594714746880;
update hosts set lastaccess=1523868919 where hostid=52746493362176;
update hosts set lastaccess=1523868919 where hostid=53536767344640;
update hosts set lastaccess=1523868919 where hostid=54481660149760;
update hosts set lastaccess=1523868919 where hostid=54713588383744;
update hosts set lastaccess=1523868919 where hostid=55078660603904;
update hosts set lastaccess=1523868919 where hostid=55366423412736;
update hosts set lastaccess=1523868919 where hostid=58046483005440;
update hosts set lastaccess=1523868919 where hostid=58450209931264;
update hosts set lastaccess=1523868919 where hostid=58926951301120;
update hosts set lastaccess=1523868919 where hostid=60211146522624;
update hosts set lastaccess=1523868919 where hostid=63294933041152;
update hosts set lastaccess=1523868919 where hostid=63333587746816;
update hosts set lastaccess=1523868919 where hostid=63990717743104;
update hosts set lastaccess=1523868919 where hostid=64252710748160;
]
1873:20180416:105520.125 query [txnlev:1] [commit;]
1873:20180416:105520.125 In process_proxy_data()
1873:20180416:105520.125 End of process_proxy_data():SUCCEED
1873:20180416:105520.125 query [txnlev:0] [select t.taskid,t.type,t.clock,t.ttl,c.command_type,c.exe cute_on,c.port,c.authtype,c.username,c.password,c. publickey,c.privatekey,c.command,c.alertid,c.paren t_taskid,c.hostid from task t left join task_remote_command c on t.taskid=c.taskid where t.status=1 and t.proxy_hostid=11495 and t.clock+t.ttl>1523868920 and c.command_type is not null order by t.taskid]
1873:20180416:105520.126 End of zbx_recv_proxy_data():SUCCEED



                    all these Host IDs don´t exist in my Database... the highest Host ID is 14700
                    Last edited by pc-nico; 16-04-2018, 11:13.

                      Comment

                      • vso
                        Zabbix developer
                        • Aug 2016
                        • 190
                        #21
                        Could you please increase log level to trace, reload cache or restart zabbix server and provide what is between In DCdump_proxies() and End of DCdump_proxies()

                        Also please do:
                        select hostid from hosts where status in (5,6);

                          Comment

                          • pc-nico
                            Junior Member
                            • May 2014
                            • 29
                            #22
                            i increase the loglevel and reload the cache:

                            Code:
                            root@vzabbix:~# zabbix_server  -R log_level_increase=trapper
zabbix_server [2212]: command sent successfully
root@zabbix:~# zabbix_server  -R log_level_increase=trapper
zabbix_server [2218]: command sent successfully
root@vzabbix:~# zabbix_server  -R log_level_increase=trapper
zabbix_server [2221]: command sent successfully
root@vzabbix:~# zabbix_server -R config_cache_reload
zabbix_server [2235]: command sent successfully
                            but nothing like DCdump is in the logfile

                            tail -f /var/log/zabbix/zabbix_server.log | grep DCdump_proxies (while config_cache_reload)
                            cat /var/log/zabbix/zabbix_server.log | grep DCdump_proxies


                            select hostid from hosts where status in (5,6);

                            all host are status 5:

                            11111
                            11115
                            11117
                            11495
                            11780
                            12281
                            12465
                            12685
                            12739
                            12824
                            12891
                            12972
                            13515
                            13592
                            13609
                            13720
                            14019
                            14737
                            14746
                            14899
                            14960

                              Comment

                              • kloczek
                                Senior Member
                                • Jun 2006
                                • 1771
                                #23
                                Generally you can access to the server internal metrics only from hos which is NOT monitored over proxy.
                                In your logs seems you are receiving from the proxy errors about issues about server internal metrics. zabbix[wcache,text,pfree]] it is server (not proxy) metric.
                                So question is did your server monitoring is on monitored or dummy host not monitored over proxy?

                                http://uk.linkedin.com/pub/tomasz-k%...zko/6/940/430/
                                https://kloczek.wordpress.com/
                                zapish - Zabbix API SHell binding https://github.com/kloczek/zapish
                                My zabbix templates https://github.com/kloczek/zabbix-templates

                                  Comment

                                  • pc-nico
                                    Junior Member
                                    • May 2014
                                    • 29
                                    #24
                                    kloczek
                                    do you ask me or zt1?

                                      Comment

                                      • vso
                                        Zabbix developer
                                        • Aug 2016
                                        • 190
                                        #25
                                        Originally posted by pc-nico
                                        i increase the loglevel and reload the cache:

                                        Code:
                                        root@vzabbix:~# zabbix_server -R log_level_increase=trapper
zabbix_server [2212]: command sent successfully
root@zabbix:~# zabbix_server -R log_level_increase=trapper
zabbix_server [2218]: command sent successfully
root@vzabbix:~# zabbix_server -R log_level_increase=trapper
zabbix_server [2221]: command sent successfully
root@vzabbix:~# zabbix_server -R config_cache_reload
zabbix_server [2235]: command sent successfully
                                        but nothing like DCdump is in the logfile

                                        tail -f /var/log/zabbix/zabbix_server.log | grep DCdump_proxies (while config_cache_reload)
                                        cat /var/log/zabbix/zabbix_server.log | grep DCdump_proxies


                                        select hostid from hosts where status in (5,6);

                                        all host are status 5:

                                        11111
                                        11115
                                        11117
                                        11495
                                        11780
                                        12281
                                        12465
                                        12685
                                        12739
                                        12824
                                        12891
                                        12972
                                        13515
                                        13592
                                        13609
                                        13720
                                        14019
                                        14737
                                        14746
                                        14899
                                        14960
                                        Please do
                                        zabbix_server -R log_level_increase='configuration syncer'
                                        ./sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf -Rconfig_cache_reload
                                        Then you should see DCdump_proxies

                                          Comment

                                          • pc-nico
                                            Junior Member
                                            • May 2014
                                            • 29
                                            #26
                                            Code:
                                              8022:20180416:145843.108 In DCdump_proxies()
  8022:20180416:145843.108 hostid:11111 timediff:-1 location:0
  8022:20180416:145843.108 hostid:11115 timediff:-82 location:0
  8022:20180416:145843.108 hostid:11117 timediff:-1 location:0
  8022:20180416:145843.108 hostid:11495 timediff:-1 location:0
  8022:20180416:145843.108 hostid:11780 timediff:-77 location:0
  8022:20180416:145843.108 hostid:12281 timediff:1 location:0
  8022:20180416:145843.108 hostid:12465 timediff:-14 location:0
  8022:20180416:145843.108 hostid:12685 timediff:-3 location:0
  8022:20180416:145843.108 hostid:12739 timediff:106 location:0
  8022:20180416:145843.108 hostid:12824 timediff:0 location:0
  8022:20180416:145843.108 hostid:12891 timediff:0 location:0
  8022:20180416:145843.108 hostid:12972 timediff:0 location:0
  8022:20180416:145843.108 hostid:13515 timediff:-15 location:0
  8022:20180416:145843.109 hostid:13592 timediff:0 location:0
  8022:20180416:145843.109 hostid:13609 timediff:1 location:0
  8022:20180416:145843.109 hostid:13720 timediff:-285 location:0
  8022:20180416:145843.109 hostid:14019 timediff:-47 location:0
  8022:20180416:145843.109 hostid:14737 timediff:-1 location:0
  8022:20180416:145843.109 hostid:14746 timediff:-1 location:0
  8022:20180416:145843.109 hostid:14899 timediff:-2 location:0
  8022:20180416:145843.109 hostid:14960 timediff:-1 location:0
  8022:20180416:145843.109 End of DCdump_proxies()

                                              Comment

                                              • vso
                                                Zabbix developer
                                                • Aug 2016
                                                • 190
                                                #27
                                                Do you use 32 bit linux or any special compilation flags ?
                                                If it possible then you could try this patch:

                                                Code:
                                                Index: src/libs/zbxdbhigh/proxy.c
===================================================================
--- src/libs/zbxdbhigh/proxy.c    (revision 79690)
+++ src/libs/zbxdbhigh/proxy.c    (working copy)
@@ -396,7 +396,7 @@
         zbx_snprintf_alloc(&sql, &sql_alloc, &sql_offset, "update hosts"
                 " set lastaccess=%d"
                 " where hostid=" ZBX_FS_UI64 ";\n",
-                pair.second, pair.first);
+                (int)pair.second, pair.first);

         DBexecute_overflowed_sql(&sql, &sql_alloc, &sql_offset);
     }
                                                Last edited by vso; 16-04-2018, 16:06.

                                                  Comment

                                                  • pc-nico
                                                    Junior Member
                                                    • May 2014
                                                    • 29
                                                    #28
                                                    i use standard debian and Zabbix from Repo:

                                                    Code:
                                                    root@vm26-zabbix:~# uname -a
Linux vm26-zabbix 3.16.0-5-686-pae #1 SMP Debian 3.16.51-3+deb8u1 (2018-01-08) i686 GNU/Linux
root@vm26-zabbix:~# cat /etc/debian_version
8.10

                                                      Comment

                                                      • vso
                                                        Zabbix developer
                                                        • Aug 2016
                                                        • 190
                                                        #29
                                                        Originally posted by pc-nico
                                                        i use standard debian and Zabbix from Repo:

                                                        Code:
                                                        root@vm26-zabbix:~# uname -a
Linux vm26-zabbix 3.16.0-5-686-pae #1 SMP Debian 3.16.51-3+deb8u1 (2018-01-08) i686 GNU/Linux
root@vm26-zabbix:~# cat /etc/debian_version
8.10
                                                        Yes, it's 32bit, created bug report https://support.zabbix.com/browse/ZBX-13755

                                                          Comment

                                                          Previous 1 2 template Next
                                                          Working...