Ad Widget

Collapse

Zabbix server accepts data from proxys when its database is down

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • billcouper
    Junior Member
    • Jul 2024
    • 6

    #1

    Zabbix server accepts data from proxys when its database is down

    I know this is going to sound stupid, but bear with me for a minute here.

    We have 2 active proxys in a proxy group collecting data, in hybrid mode, configured to store data for 24 hours if the zabbix server is offline.

    We had an unexpected outage in the middle of the night on the MySQL backend database. Zabbix server could not connect to the database for a 3hr period until I logged in remotely and fixed it.
    Zabbix Server cannot possibly write the data to MySQL, but still appear to accept data from proxys!?! That data goes where? I don't think it goes anywhere. It just vanishes.

    Zabbix Server was sending emails advising the database was down, so it's not like it thought everything was OK.
    WHY would Zabbix Server continue to accept data from proxys in this scenario?
    Zabbix Server should stop accept data, causing proxys to hold it offline until Zabbix Server is ready and ABLE to receive the data again.

    Is there a way to configure Zabbix Server to NOT accept data if it can't write it anywhere?
    If not... I am considering a script on Zabbix Server that checks connectivity and read/write access to the database. If the checks fail the script could either A) Stop the Zabbix Server service, or B) Prevent proxy connections via iptables rules...
    I suspect that option B could wreak havoc on the proxy load balancing algorithm, but haven't done any testing yet.
  • billcouper
    Junior Member
    • Jul 2024
    • 6

    #2
    No comments? Is it expected behaviour, or have I missed some crucial config setting somewhere?

    Comment

    • cyber
      Senior Member
      Zabbix Certified SpecialistZabbix Certified Professional
      • Dec 2006
      • 4807

      #3
      I have no definite answer, but I think data is not going directly to DB. It is accepted, cached, and then history syncers do their work and write stuff to DB. so it is kind of able to receive... But I doubt for 3 hours...

      Comment

      • billcouper
        Junior Member
        • Jul 2024
        • 6

        #4
        Thanks for replying cyber. Agreed, it was an overly long outage and hopefully a rare occurrence - this is the first time I've ever had an issue with the zabbix database.
        I will investigate the cache parameter documentation more carefully and see if there is something I can tune to increase the period it might survive.
        I will probably also look into a database health checking script and stop the zabbix-server service if the database goes offline.

        Comment

        Working...