Ad Widget

Collapse

Proxy stop to work after 1.6 -> 1.8 upgrade

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • tootai
    Junior Member
    • Jan 2010
    • 8

    #1

    Proxy stop to work after 1.6 -> 1.8 upgrade

    Hi,

    I have a zabbix server with 2 local zabbix agent, 1 zabbix agent on internet and 1 proxy on internet with 3 local agents. This set up was working well in 1.6, all debian lenny with zabbix from backports.

    9 days ago I upgraded to 1.8.1 zabbix server, agent and proxy. Since this upgrade, no one of my 3 servers connecting to proxy have updated datas on server. All others -remember, only agent local or on Internet- are OK. Interesting is that discovery show th3 servers up!

    Connection between zabbix-server (VM in kvm) and zabbix-proxy (VM in kvm) is done through a VPN (OpenVPN) which is mounted on both real machines.

    I followed the flow of datas using tshark and datas arrive as they should on Zabbix server network interface. FYI, the working agent on Internet is using the same way to send datas.

    If someone had an idea on how to debug further the problem -or how to solve it if known ;-)- I would appreciate.

    Finally, I have a second network with approximately the same setup and face the same problem.

    Thanks for help

    --
    Daniel
  • tchjts1
    Senior Member
    • May 2008
    • 1605

    #2
    What are your agent, server and proxy logs saying? Should be some indication in there of what is happening.

    Comment

    • tootai
      Junior Member
      • Jan 2010
      • 8

      #3
      On proxy side, all is smooth. I see datas from each server which are coming as they should.

      On server side, nothing, only datas from agent :-( Remember that in 1.6 it was working like a charm.

      As told, tshark shows on eth0 packets which are coming from proxy (zabbix-trapper) like this

      2521.178355 10.99.3.2 -> 10.0.70.12 TCP 59057 > zabbix-trapper [PSH, ACK] Seq=6 Ack=1 Win=5888 Len=64 TSV=996381706 TSER=200383713
      2521.178388 10.0.70.12 -> 10.99.3.2 TCP zabbix-trapper > 59057 [ACK] Seq=1 Ack=70 Win=5824 Len=0 TSV=200383716 TSER=996381706
      2521.180431 10.0.70.12 -> 10.99.3.2 TCP zabbix-trapper > 59057 [PSH, ACK] Seq=1 Ack=70 Win=5824 Len=5 TSV=200383717 TSER=996381706
      2521.180525 10.0.70.12 -> 10.99.3.2 TCP zabbix-trapper > 59057 [FIN, PSH, ACK] Seq=6 Ack=70 Win=5824 Len=32 TSV=200383717 TSER=996381706
      2521.193109 10.99.3.2 -> 10.0.70.12 TCP 59057 > zabbix-trapper [ACK] Seq=70 Ack=6 Win=5888 Len=0 TSV=996381710 TSER=200383717
      2521.193212 10.99.3.2 -> 10.0.70.12 TCP 59057 > zabbix-trapper [FIN, ACK] Seq=70 Ack=39 Win=5888 Len=0 TSV=996381710 TSER=200383717
      2521.193231 10.0.70.12 -> 10.99.3.2 TCP zabbix-trapper > 59057 [ACK] Seq=39 Ack=71 Win=5824 Len=0 TSV=200383720 TSER=996381710
      ^C597 packets captured
      dh@zone-s:/etc/zabbix$

      3 questions:

      1. Hostname of proxy is PROXY blabla with a space between the 2 words. Could this a problem in 1.8?
      2. Nowhere in zabbix I find a definition of PROXY blabla in configuration->host but it's proposed in my hosts in "monitored by" Is this ok?
      3. in administration -> queue I see value 192 in column more then 10 minutes for PROXY blabla: what does it mean?

      Thanks for your help

      --
      Daniel

      Comment

      • tchjts1
        Senior Member
        • May 2008
        • 1605

        #4
        Originally posted by tootai
        3 questions:

        1. Hostname of proxy is PROXY blabla with a space between the 2 words. Could this a problem in 1.8?
        2. Nowhere in zabbix I find a definition of PROXY blabla in configuration->host but it's proposed in my hosts in "monitored by" Is this ok?
        3. in administration -> queue I see value 192 in column more then 10 minutes for PROXY blabla: what does it mean?

        Thanks for your help

        --
        Daniel
        (I didn't mean your Tshark logs... I meant your Zabbix logs)

        1. Don't know. Easy way to find out, remove the sapce from the name on the proxy and change the name in the frontend to match.

        2. Administration->DM->Proxies

        3. Those are items that for some reason are not returning data back to Zabbix. You can change the dropdown box from "View by proxy" to "View details" and you will see exactly what those items are and on what host.

        A good example would be if you have an item to check freespace of the E: drive on a server, but there is no E: drive. That would be an unsupported item and would show in the queue.

        Comment

        • tootai
          Junior Member
          • Jan 2010
          • 8

          #5
          1. I replace the space with underscore, no changes
          2. Noted where to find Proxies definition
          3. I checked detail and saw
          Prochaine vérification Retardé par Hôte Description
          15 Feb 2010 12:30:35 1s 3j 1h wwwmail Incoming traffic on interface eth0
          15 Feb 2010 12:30:35 1s 3j 1h wwwmail Outgoing traffic on interface lo
          15 Feb 2010 12:30:35 1s 3j 1h wwwmail Processor load
          15 Feb 2010 12:30:35 1s 3j 1h wwwmail Used disk space on /opt
          15 Feb 2010 12:30:35 1s 3j 1h pabx Used disk space on /var

          [...]

          Why the hell is next verification to 15 of februar? Second column means 1 week, 3 days 1 hour which could be correct if I calculate with daylight +1hour

          I only find 2 of my 3 servers in this queue but effectively those are servers behind the proxy!

          Is there a way to unblock the date of next verification?

          Thanks for your support

          --
          Daniel
          Last edited by tootai; 25-02-2010, 12:49.

          Comment

          • tchjts1
            Senior Member
            • May 2008
            • 1605

            #6
            Originally posted by tootai
            Is there a way to unblock the date of next verification?

            Thanks for your support

            --
            Daniel
            Have you checked your Zabbix logs yet? (Not tshark).
            zabbix_server.log
            zabbix_agentd.log
            zabbix_proxy.log

            As far as unblocking the date of next verification, the only way I am aware of is to fix the issue with the item. Look at the items that are reported in the Queue and compare to the items attached to your host. May be that they are coming up as "unsupported". If that is the case, either fix the synatx so they become supported or disable the item.

            I have some items that are delayed by 40 years
            Next check................. Delayed by
            31 Dec 16:00:30......... 40y 9w 2d

            This is because I created some items and attached to the template for a group of servers in order to monitor a specific process that will be installed next week. But since it does not exist yet, Zabbix is choking on it. Why they selected a delayed date of 40 years, I don't know. But when the process is installed on the hosts, that will clear from the queue.

            In regards to adding an underline to your proxy server name on the proxy itself, did you also make that same change in the frontend?

            Comment

            • tootai
              Junior Member
              • Jan 2010
              • 8

              #7
              I went back to PROXY blabla eg with a space.

              Proxy logs in debug mode (remember that in 1.6 all whent smooth, so my items are OK. Anyway, at least network should be supported on eth0 ;-))

              796:20100225:153524.730 Get value from agent result: '1686089728'
              796:20100225:153524.730 End of get_value():SUCCEED
              796:20100225:153524.730 In calculate_item_nextcheck (22794,30,"",1267108524)
              796:20100225:153524.730 End calculate_item_nextcheck (result:1267108554)
              796:20100225:153524.730 In DCflush_nextchecks()
              796:20100225:153524.730 End of get_values()
              796:20100225:153524.730 In DCconfig_get_normal_poller_nextcheck() poller_type:0 poller_num:4
              796:20100225:153524.730 End of DCconfig_get_normal_poller_nextcheck():1267108529
              796:20100225:153524.730 Poller #4 spent 0.060972 seconds while updating 9 values. Sleeping for 5 seconds
              799:20100225:153525.131 Trapper got [{
              "request":"proxy heartbeat",
              "host":"PROXY Kumquat"}] len 56
              799:20100225:153525.131 In process_proxy_heartbeat()
              799:20100225:153525.131 Query [txnlev:0] [select hostid from hosts where host='PROXY Kumquat' and status in (5) and hostid between 000000000000000 and 099999999999999]
              799:20100225:153525.131 Query without transaction detected [update hosts set lastaccess=1267108525 where hostid=10049]
              799:20100225:153525.131 Query [txnlev:0] [update hosts set lastaccess=1267108525 where hostid=10049]
              792:20100225:153525.287 In get_values()
              792:20100225:153525.287 In DCinit_nextchecks()
              792:20100225:153525.287 In DCconfig_get_poller_items() poller_type:0 poller_num:0
              792:20100225:153525.287 End of DCconfig_get_poller_items():13
              792:20100225:153525.287 In substitute_simple_macros (data:'net.if.in[eth0,bytes]')
              792:20100225:153525.287 In get_value() key:'net.if.in[eth0,bytes]'
              792:20100225:153525.287 In get_value_agent() host:'ZABBIX Server' addr:'127.0.0.1' key:'net.if.in[eth0,bytes]'
              792:20100225:153525.287 Sending [net.if.in[eth0,bytes]
              ]


              If this is the same transaction, why it involved ZABBIX Server? Hope you can give me more infos ...

              Thanks for your help

              --
              Daniel

              Comment

              • tootai
                Junior Member
                • Jan 2010
                • 8

                #8
                Anyone with an idea?

                --
                Daniel

                Comment

                • t-bear
                  Junior Member
                  • Mar 2010
                  • 1

                  #9
                  Hi

                  Had the same problem.
                  I did the following and now it works:
                  1: Stop the zabbix_proxy
                  2: Flush all data in the tables on the zabbix_proxy database
                  3: restart zabbix_proxy

                  All my data from the proxy now comes through to the "master".

                  hope this helps

                  Comment

                  • tootai
                    Junior Member
                    • Jan 2010
                    • 8

                    #10
                    t-bear, you're a genious

                    I stopped zabbix_proxy, removed database zabbix_proxy, create it once again, start zabbix_proxy and datas comes through to the master.

                    I did it on both my networks and on both it solved the problem.

                    Thanks a lot

                    --
                    Daniel

                    Comment

                    Working...