Ad Widget

Collapse

zabbix agent sends wrong time localtime[utc]

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    zabbix agent sends wrong time localtime[utc]

    Hi all

    I'm observing somethign strange ( I think it started after upgrading zabiix infrastructure to v. 4 )

    It seems only windows servers are affected - im using system.localtime[utc] to obtain utc localtime on windows server / agent is in active setup sending data to proxy , proxy sends the data to server

    Trigger: {t_p_os windows active agent:system.localtime[utc].fuzzytime(30)}=0

    Frequently I get events about time offset more than 30s event the time is the same on windows server, zabbi proxy, zabbix server. I was able to exclude proxy or server issue and pinpoint it to the agent.


    Data from proxy log:

    16270:20190102:122316.188 trapper got '{"request":"agent data","session":"70837e6c10552738d6c179eb076fc5a7" ,"data":[{"host":"DC2","key":"system.localtime[utc]","value":"1546431793","id":139150,"clock":1546431754,"ns":984088964}],"clock":1546431758,"ns":63913014}'

    Data from server log:


    478:20190102:122317.988 In evaluate_function() function:'DC2:system.localtime[utc].fuzzytime(30)'


    Latest data: - seems that the time of latest check 2019-01-02 12:22:34 is the time of clock 1546431754 - but the value is 1546431793 - kinda value is sent with clock timestamp 39 seconds in the past

    2019-01-02 12:22:34

    1546431793

    Any hints ? Ideas where could be the issue?

    P.S. When the agent is restarted eveything is in sync - problem is closed - seems as the delay betwren clock and value time is growing with time.









    #2
    Hello gofree,

    Thank you for submitting this issue. This can be related to https://support.zabbix.com/browse/ZBX-15301.
    It may get fixed in 4.0.4 or future release. Could you please list exact versions of components in your scenario. What is an exact version of agent, proxy and server? Are you using a 64-bit agent? What version of Windows is used?

    P.S. I did came across to a similar issue and noticed that switching items in passive mode do not create a delay. This is not even a workaround. You can just try for a sec just to relate with this bug even more.

    Keep up the good work,

    Comment


      #3
      Hi

      I just updated the zabbix infra ( server,proxy ) to v. 4.0.3 but this was happening also with 4.0.1 and 4.0.2, the agents are in version 3.4.6 and 4.0.0 ( 64 bit ) , Windows version is 2012

      thanks for hint, I'll wait for 4.0.4 - guess there is no other option

      from the quick review it looks like it will be resolved on the server, proxy side - right ? https://support.zabbix.com/browse/ZBX-15301
      Last edited by gofree; 03-01-2019, 11:03.

      Comment


        #4
        Another option is temporary switch item type from "Zabbix agent (active)" to "Zabbix agent"

        Like we can spot on the top of the ticket, the component involved right now is only Agent.

        Comment


          #5
          Well I was hoping I could avoid update of the agents somehow it didnt happen with server and proxy version 3.4.x - and thats what changed mostly. I just updated couple of agents to v.4 with hope this will be caused by agent. Seems to me that it did change somewhere in the middle ...

          passive agent is not an option in my environment - secuity guys will go mad.

          Comment


            #6
            Understood.

            In case you do not have any automation to replace the agent here is one solution. Grab the latest stable source package at https://sourceforge.net/projects/zab...atest/download .
            Extract content and go to "bin" directory.

            Place this script side by side with "win64" and "win32" directories:
            Code:
            @echo off
            
            rem set destination where the agent is installed
            set d=c:\zabbix
            
            setlocal EnableDelayedExpansion
            
            if exist "%d%\zabbix_agentd.exe" (
            echo stopping zabbix service
            sc stop "Zabbix Agent"
            if !errorlevel!==0 (
            
            :win64
            if not "%ProgramFiles(x86)%"=="" (
            echo this is 64-bit machine
            xcopy /y "%~dp0win64\zabbix_agentd.exe" "%d%"
            xcopy /y "%~dp0win64\zabbix_sender.exe" "%d%"
            xcopy /y "%~dp0win64\zabbix_get.exe" "%d%"
            )
            
            :win32
            if "%ProgramFiles(x86)%"=="" (
            echo this is 32-bit machine
            xcopy /y "%~dp0win32\zabbix_agentd.exe" "%d%"
            xcopy /y "%~dp0win32\zabbix_sender.exe" "%d%"
            xcopy /y "%~dp0win32\zabbix_get.exe" "%d%"
            )
            
            sc start "Zabbix Agent"
            
            ) else echo Cannot stop Zabbix agent
            ) else echo zabbix_agentd.exe do not exist at %d%
            
            endlocal
            It will stop Zabbix agent, replace the binary and start the agent.

            Comment


              #7
              Hello gofree,

              Some test packages have been released in the attachment of bug request:
              https://support.zabbix.com/browse/ZBX-15301

              Here are direct links
              64-bit: https://support.zabbix.com/secure/at...-win-amd64.zip
              32-bit: https://support.zabbix.com/secure/at...1-win-i386.zip

              It will be a huge favour if you can install these guys on your machines and leave us some feedback after 3 days. There should be no queue at all and no delayed active items.

              Next Zabbix agent version will be released on January 28. You got some time. Not much. The whole world is in your hands

              Comment


                #8
                just to clarify - does it mean that system.localtime[utc] will be supported as active check again eg. supported - jus because in the thread ZBX-15301 is twas said no

                ill talk to our ops guys and we try to install it on couple servers
                Last edited by gofree; 10-01-2019, 17:09. Reason: adding zbx bug number

                Comment


                  #9
                  installed on 2 servers - enabled the triggers - problems closed - ill leave it running during night - post info tomorrow and after weekend

                  Comment


                    #10
                    Thank you. Will wait for some feedback.

                    Not sure what you meant about system.localtime[utc] as an active check. I'm testing this 4.0.4rc1 package on 64-bit Windows 10 and the item was able to receive a metric while using item type: Zabbix agent (active).

                    Comment


                      #11
                      Hi Im relating to my question from the ZBX-15301 thread - here and see docu saying that this key should be used only with passive check ( starting with version 4 ) Click image for larger version

Name:	Capture.PNG
Views:	1
Size:	21.6 KB
ID:	371806 - see attachment

                      If this functionality will be restored in full I'll be happy and I guess the docu will need to be chnaged again.


                      P.S. those two agent insatlled yesterday seem fine so far on the first look

                      Comment


                        #12
                        It seems to cause other problems too. It seems any agent active with a nodata() check will generate false alerts because the timestamp of the event is lagged.

                        Many servers are blocked for passive checks.

                        Comment


                          #13
                          Not suree but maybe event delay is caused by it as weel ( agent version is 3.4.6 ) - here

                          Comment


                            #14
                            Originally posted by LenR View Post
                            It seems to cause other problems too. It seems any agent active with a nodata() check will generate false alerts because the timestamp of the event is lagged.

                            Many servers are blocked for passive checks.
                            LenR! Are you using zabbix agent version 4.0.4rc1?
                            64-bit: https://support.zabbix.com/secure/at...-win-amd64.zip
                            32-bit: https://support.zabbix.com/secure/at...1-win-i386.zip
                            4.0.4rc1 version is only one right now which can examine the nodata() function successfully.

                            Comment

                            Announcement

                            Collapse
                            No announcement yet.
                            Working...
                            X