Ad Widget

Collapse

Move to distributed model

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • s_peter_g
    Junior Member
    • Dec 2012
    • 5

    #1

    Move to distributed model

    Hi
    I intend to migrate from single zabbix server to distributed model.
    Currently deployment:

    Hardware:
    RAM: 16 GB
    HDD: 1 HDD (SATA - 7200 RPM)
    CPU: Quad Core - Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
    Software:
    Zabbix: 1.8.3
    Hosts: 400(next month apprx: 800)
    Items: 100K(next month approx: 150K)
    triggers: 0 (next month approx: 5000)
    NVPS: ~400
    Queue Performance: more than 200 items with 10 minute delay
    DB: Mysql - InnoDB
    Size DB: 40GB

    I decided to add another hardware with the same spec in order to create distributed model
    But I have some concern about migrate from standard zabbix server to distributed model because this server is production.

    My concerns:

    1- which solution is sophisticated for my needs,Node or Proxy.of course I do not have good understanding about difference of them
    2- If I choose Node distributed model, Is there any problem in converting standard database to node database for 40GB size?for example lost data or corruption database.
    3- what type data does child node send to master node?
    4- Does Master Node have copy of all child's data (including configuration and history data)?
    5- Can admin config node via master web UI?for example add new host or create trigger or any configuration from master node.
    6- Is it possible that move hosts from currently standard zabbix server to child node? if so, How can I do it?
    7- Is there any limitation on converting standard zabbix server to a master node?
    8- Can admin access via graph and other data about one host that are monitored by child node?

    Thanks
  • heaje
    Senior Member
    Zabbix Certified Specialist
    • Sep 2009
    • 325

    #2
    Before reading my answers, please be aware that I do not currently run distributed monitoring with Nodes in our production environment. We use Proxies (we currently have 7 of them). However, I did do some testing with Nodes while evaluating their uses and such.

    which solution is sophisticated for my needs,Node or Proxy.of course I do not have good understanding about difference of them
    This depends entirely on what you expect to get out of a distributed model. If you're simply trying to spread the checks around, then proxies are excellent, lightweight, and extremely easy to set up.

    However, if your intent is to be able to have full blown Zabbix server functionality on another server, then a node is the best way to go. Check out these two links for more info:
    1. Distributed Monitoring Documentation
    2. Distributed Monitoring with Proxies

    If I choose Node distributed model, Is there any problem in converting standard database to node database for 40GB size?for example lost data or corruption database.
    In theory, there should not be any problem with this at all. If you want to test this out, take a backup of your production Zabbix database and then restore it to a test server. Then run the appropriate command line commands to convert the test DB to a node and see how it goes. More info on converting the DB can be found here:

    http://www.zabbix.com/documentation/...nitoring/nodes

    what type data does child node send to master node?
    As I recall, you can configure whether or not the child node sends history information to the master. That's the only data I'm aware of that goes to the master though.

    Does Master Node have copy of all child's data (including configuration and history data)?
    The documentation says this on this page (http://www.zabbix.com/documentation/...toring/nodes):

    "a node can be configured locally or through its master node, which has a copy of configuration data of all child nodes."

    Based on documentation (and experience), yes, the Master server will have the history data for child nodes (if the child node is configured to send that data to the master) and the configuration data.

    Can admin config node via master web UI?for example add new host or create trigger or any configuration from master node.
    See the answer to #4

    Is it possible that move hosts from currently standard zabbix server to child node? if so, How can I do it?
    I do not know the answer to this with a node configuration. With a proxy, it is incredibly easy to do it. Here's the steps to follow:
    1. On the Zabbix agent to be monitored, edit the zabbix_agentd.conf file and change the values for "Server=" to the server that will be doing checks on it. Also, change the "ServerActive=" value to the appropriate server (if necessary).
    2. Restart the agent after making the changes.
    3. In the Zabbix GUI, edit the host by going to "Configuration"->"Hosts" and change the "Monitored by Proxy" setting to the proxy you want to monitor the host.
    4. Save the host config and you're done!

    Is there any limitation on converting standard zabbix server to a master node?
    None that I'm aware of. Just know that running the changes for this will cause downtime for Zabbix while it runs.


    Can admin access via graph and other data about one host that are monitored by child node?
    I'm not 100% sure what you mean by this question, but I'm assuming you meant to ask if you can look at graphs and such for a host that is on a child node while in the GUI on the master node. If that is what you meant, then yes, you can do that.

    Comment

    • zalex_ua
      Senior Member
      Zabbix Certified Trainer
      Zabbix Certified SpecialistZabbix Certified Professional
      • Oct 2009
      • 1286

      #3
      Strong recommendation - do not start to use nodes now.
      FYI - https://support.zabbix.com/browse/ZBXNEXT-1343

      Comment

      • s_peter_g
        Junior Member
        • Dec 2012
        • 5

        #4
        Thanks a lot for your attention and good reply
        According to your reply and related links, I will not deploy Node model for distributed for my production server.and I have to implement proxy model.
        But unfortunately I have some concerns about proxy:
        1- in proxy mode, Will my database be offloaded ? or only zabbix engine will be offloaded?because one of my big concern is about growing database in long time.
        2- According to proxy document, apparently all data send from proxy to master server, Does proxy server store history and other gathered data in its database ?

        Comment

        • eggburt
          Junior Member
          • Jan 2010
          • 6

          #5
          Originally posted by zalex_ua
          Strong recommendation - do not start to use nodes now.
          FYI - https://support.zabbix.com/browse/ZBXNEXT-1343
          +1 on this, there's also many problems and pains with using nodes that simply don't exist with proxies.

          I'd recommend running completely separate servers over nodes at this point if you need any functionality that nodes provide..

          Comment

          • s_peter_g
            Junior Member
            • Dec 2012
            • 5

            #6
            Thanks again
            I do not think deploying separate server instead node or proxy,because operators must check 2 panel for alerts and notification and also many pain on configuration.

            thanks

            Comment

            • eggburt
              Junior Member
              • Jan 2010
              • 6

              #7
              Originally posted by s_peter_g
              Thanks again
              I do not think deploying separate server instead node or proxy,because operators must check 2 panel for alerts and notification and also many pain on configuration.

              thanks
              I'd be going with proxies in that situation then. Nodes aren't too reliable unless you're watching/tweaking/maintaining them constantly. There's also various bits that don't exist for nodes that do for proxies such as (The following are the ones off the top of my head I've had to hack up a solution for at some point)

              - Alerting that a node hasn't sent any data recently
              - Acknowledges don't sync from parent to child
              - New nodes sometimes mess up their event IDs and don't sync properly (Causing no alerts if you alert from the parent) - Not been able to find the actual cause yet but the solution requires digging around in the IDs table*
              - Child and parent can get out of sync. Edit a host on the parent and nothing triggers when there's an issue because the child doesn't even have the host in its database. I've not found a solution to this aside from delete and recreate the host..

              Don't get me wrong it is workable, and I've got a pretty stable setup but the amount of headaches to get there just aren't worth it in my mind if starting fresh


              -----
              * See: http://www.zabbix.com/documentation/...l/appendix/faq

              Disregard the following, remains only for historical reasons

              * - Just a thought in case this post ever comes up in a Google search (I had trouble finding the solution)

              Look at the IDs table, chances are your child node (in this example node 2) has eventids starting 2000[..] it needs to be 2002[..]!
              <nodeid>00<nodeid>[..]
              Last edited by eggburt; 03-12-2012, 16:31. Reason: Better info provided

              Comment

              • eggburt
                Junior Member
                • Jan 2010
                • 6

                #8
                I do apologise I pretty much started ranting instead of just answering!

                Originally posted by s_peter_g
                My concerns:

                1- which solution is sophisticated for my needs,Node or Proxy.of course I do not have good understanding about difference of them
                2- If I choose Node distributed model, Is there any problem in converting standard database to node database for 40GB size?for example lost data or corruption database.
                3- what type data does child node send to master node?
                4- Does Master Node have copy of all child's data (including configuration and history data)?
                5- Can admin config node via master web UI?for example add new host or create trigger or any configuration from master node.
                6- Is it possible that move hosts from currently standard zabbix server to child node? if so, How can I do it?
                7- Is there any limitation on converting standard zabbix server to a master node?
                8- Can admin access via graph and other data about one host that are monitored by child node?

                Thanks
                1- For the reasons already mentioned in the thread, I'd go with proxies. Worst case with proxies you can switch to nodes, it's not as easy to switch from nodes to proxies. (Your database will never be the same anyway)

                2- It will probably take a fair while and for the love of <insert deity here> take a backup before doing so! So easy to mess this step up and corrupt your database! As far as I know you can't continue a conversion due to the queries it runs.

                3- If working correctly and configured to do so a child node will send everything (except maybe sync info - what it has already synced, etc). With the right amount of patience you could probably re-create a child node from the parent's database (not for the faint of heart though)

                4- Yes, if you have configured it that way (i.e. Not used NodeNo* options in child config)

                5- Yep, though it's easy for people not in the know to miss, the node selector is in the top right corner, you have to Select nodes you wish to see before you can select them. Also ensure your user permissions allow access to the groups on the other node(s)

                6- No, it's not possible to move anything between nodes. Other than the central interface they're all separate servers. You can export and import (make sure to install php-xml!) or maybe at a stretch hack something up with the API but no, not as it is.

                7- Plenty mentioned in thread so far, but I would emphasise if you do go down this route anyway take a backup and make sure it works. Run the converter in a 'screen', don't allow the server to be powered off, don't cancel the script if it feels like it's taking too long. Just don't stop the script for any reason.
                Oh and make sure Zabbix isn't running before converting! You will need monitoring server downtime for as long as it takes.

                8- Yes, but you can do this with proxies anyway.


                Feel free to chuck in any other questions! and others feel free to correct me where I may have gone wrong!

                Comment

                • zalex_ua
                  Senior Member
                  Zabbix Certified Trainer
                  Zabbix Certified SpecialistZabbix Certified Professional
                  • Oct 2009
                  • 1286

                  #9
                  Originally posted by eggburt
                  - Acknowledges don't sync from parent to child
                  implemented already, see 11.3.1.2 Acknowledge synchronisation in distributed mode
                  http://www.zabbix.com/documentation/...r_improvements

                  Originally posted by eggburt
                  - New nodes sometimes mess up their event IDs and don't sync properly (Causing no alerts if you alert from the parent) - Not been able to find the actual cause yet but the solution requires digging around in the IDs table*
                  Yes, I know this case very well, actual cause is still unknown. The IDs table is not related here.

                  Originally posted by eggburt
                  Look at the IDs table, chances are your child node (in this example node 2) has eventids starting 2000[..] it needs to be 2002[..]!
                  <nodeid>00<nodeid>[..]
                  Not very correct in cases if nodeid longer than 1 digit (people like to use nodes 10,20 ... ), it better to point to documentation, see #8
                  http://www.zabbix.com/documentation/...l/appendix/faq

                  Comment

                  • eggburt
                    Junior Member
                    • Jan 2010
                    • 6

                    #10
                    Originally posted by zalex_ua
                    implemented already, see 11.3.1.2 Acknowledge synchronisation in distributed mode

                    http://www.zabbix.com/documentation/...r_improvements
                    Unfortunately that's only Child->Parent. Parent->Child doesn't get synced. Annoying when you have staff acknowledge on the master node and the child node doesn't show it as being looked into.. Ended up hooking into the acknowledge button and sending the ack using the API..

                    Also if you have actions that rely on acknowledge (with escalations) on the child this doesn't work if ack'd on the parent

                    Originally posted by zalex_ua

                    Yes, I know this case very well, actual cause is still unknown. The IDs table is not related here.



                    Not very correct in cases if nodeid longer than 1 digit (people like to use nodes 10,20 ... ), it better to point to documentation, see #8
                    http://www.zabbix.com/documentation/...l/appendix/faq
                    Good point, updated my post to point at documentation thank you

                    In my case the parent was requesting events with IDs greater than a certain value (2002[..]) however the child was using IDs 2000[..] which means all eventids were lower than 2002[..]
                    The solution (in my case, whether it works elsewhere I don't know) to this was to update the next eventid in the ids table to be nextid*<nodeid>00000000000

                    Disclaimer of course: don't just run with this information it will probably not be the same in all cases!
                    Last edited by eggburt; 03-12-2012, 16:40.

                    Comment

                    • zalex_ua
                      Senior Member
                      Zabbix Certified Trainer
                      Zabbix Certified SpecialistZabbix Certified Professional
                      • Oct 2009
                      • 1286

                      #11
                      Originally posted by eggburt
                      Unfortunately that's only Child->Parent. Parent->Child doesn't get synced.
                      ohh, you are right !
                      I missed direction of synchronization .

                      Comment

                      • heaje
                        Senior Member
                        Zabbix Certified Specialist
                        • Sep 2009
                        • 325

                        #12
                        Originally posted by s_peter_g
                        1- in proxy mode, Will my database be offloaded ? or only zabbix engine will be offloaded?because one of my big concern is about growing database in long time.
                        Really only the checks get distributed. The proxy itself does have to have a database (can be SQLite, MySQL, etc.), however, that database is simply used to store history (if you have your proxy configured to store history after it is sent to the Zabbix server) and configuration data. In the end, all history data does end up on the Zabbix server, so it's database will grow just like it would if it were the server doing the checks.

                        Originally posted by s_peter_g
                        2- According to proxy document, apparently all data send from proxy to master server, Does proxy server store history and other gathered data in its database ?
                        If you enable the appropriate options in zabbix_proxy.conf, then yes, the proxy will keep history in its database up to a specified amount of time.

                        Comment

                        Working...