zabbix agent sends wrong time localtime[utc]

[gofree]

Hi all

I'm observing somethign strange ( I think it started after upgrading zabiix infrastructure to v. 4 )

It seems only windows servers are affected - im using system.localtime[utc] to obtain utc localtime on windows server / agent is in active setup sending data to proxy , proxy sends the data to server

Trigger: {t_p_os windows active agent:system.localtime[utc].fuzzytime(30)}=0

Frequently I get events about time offset more than 30s event the time is the same on windows server, zabbi proxy, zabbix server. I was able to exclude proxy or server issue and pinpoint it to the agent.


Data from proxy log:

16270:20190102:122316.188 trapper got '{"request":"agent data","session":"70837e6c10552738d6c179eb076fc5a7" ,"data":[{"host":"DC2","key":"system.localtime[utc]","value":"1546431793","id":139150,"clock":1546431754,"ns":984088964}],"clock":1546431758,"ns":63913014}'

Data from server log:


478:20190102:122317.988 In evaluate_function() function:'DC2:system.localtime[utc].fuzzytime(30)'


Latest data: - seems that the time of latest check 2019-01-02 12:22:34 is the time of clock 1546431754 - but the value is 1546431793 - kinda value is sent with clock timestamp 39 seconds in the past

2019-01-02 12:22:34

1546431793

Any hints ? Ideas where could be the issue?

P.S. When the agent is restarted eveything is in sync - problem is closed - seems as the delay betwren clock and value time is growing with time.

[aigars.kadikis]

Hello gofree,

Thank you for submitting this issue. This can be related to https://support.zabbix.com/browse/ZBX-15301.
It may get fixed in 4.0.4 or future release. Could you please list exact versions of components in your scenario. What is an exact version of agent, proxy and server? Are you using a 64-bit agent? What version of Windows is used?

P.S. I did came across to a similar issue and noticed that switching items in passive mode do not create a delay. This is not even a workaround. You can just try for a sec just to relate with this bug even more.

Keep up the good work,

[gofree]

Hi

I just updated the zabbix infra ( server,proxy ) to v. 4.0.3 but this was happening also with 4.0.1 and 4.0.2, the agents are in version 3.4.6 and 4.0.0 ( 64 bit ) , Windows version is 2012

thanks for hint, I'll wait for 4.0.4 - guess there is no other option

from the quick review it looks like it will be resolved on the server, proxy side - right ? https://support.zabbix.com/browse/ZBX-15301

[aigars.kadikis]

Another option is temporary switch item type from "Zabbix agent (active)" to "Zabbix agent"

Like we can spot on the top of the ticket, the component involved right now is only Agent.

[gofree]

Well I was hoping I could avoid update of the agents somehow it didnt happen with server and proxy version 3.4.x - and thats what changed mostly. I just updated couple of agents to v.4 with hope this will be caused by agent. Seems to me that it did change somewhere in the middle ...

passive agent is not an option in my environment - secuity guys will go mad.

[aigars.kadikis]

Understood.

In case you do not have any automation to replace the agent here is one solution. Grab the latest stable source package at https://sourceforge.net/projects/zab...atest/download .
Extract content and go to "bin" directory.

Place this script side by side with "win64" and "win32" directories:

@echo off

rem set destination where the agent is installed
set d=c:\zabbix

setlocal EnableDelayedExpansion

if exist "%d%\zabbix_agentd.exe" (
echo stopping zabbix service
sc stop "Zabbix Agent"
if !errorlevel!==0 (

:win64
if not "%ProgramFiles(x86)%"=="" (
echo this is 64-bit machine
xcopy /y "%~dp0win64\zabbix_agentd.exe" "%d%"
xcopy /y "%~dp0win64\zabbix_sender.exe" "%d%"
xcopy /y "%~dp0win64\zabbix_get.exe" "%d%"
)

:win32
if "%ProgramFiles(x86)%"=="" (
echo this is 32-bit machine
xcopy /y "%~dp0win32\zabbix_agentd.exe" "%d%"
xcopy /y "%~dp0win32\zabbix_sender.exe" "%d%"
xcopy /y "%~dp0win32\zabbix_get.exe" "%d%"
)

sc start "Zabbix Agent"

) else echo Cannot stop Zabbix agent
) else echo zabbix_agentd.exe do not exist at %d%

endlocal

It will stop Zabbix agent, replace the binary and start the agent.

[aigars.kadikis]

Hello gofree,

Some test packages have been released in the attachment of bug request:
https://support.zabbix.com/browse/ZBX-15301

Here are direct links
64-bit: https://support.zabbix.com/secure/at...-win-amd64.zip
32-bit: https://support.zabbix.com/secure/at...1-win-i386.zip

It will be a huge favour if you can install these guys on your machines and leave us some feedback after 3 days. There should be no queue at all and no delayed active items.

Next Zabbix agent version will be released on January 28. You got some time. Not much. The whole world is in your hands

[gofree]

just to clarify - does it mean that system.localtime[utc] will be supported as active check again eg. supported - jus because in the thread ZBX-15301 is twas said no

ill talk to our ops guys and we try to install it on couple servers

[gofree]

installed on 2 servers - enabled the triggers - problems closed - ill leave it running during night - post info tomorrow and after weekend

[aigars.kadikis]

Thank you. Will wait for some feedback.

Not sure what you meant about system.localtime[utc] as an active check. I'm testing this 4.0.4rc1 package on 64-bit Windows 10 and the item was able to receive a metric while using item type: Zabbix agent (active).

[gofree]

Hi Im relating to my question from the ZBX-15301 thread - here and see docu saying that this key should be used only with passive check ( starting with version 4 ) - see attachment

If this functionality will be restored in full I'll be happy and I guess the docu will need to be chnaged again.


P.S. those two agent insatlled yesterday seem fine so far on the first look

[LenR]

It seems to cause other problems too. It seems any agent active with a nodata() check will generate false alerts because the timestamp of the event is lagged.

Many servers are blocked for passive checks.

[gofree]

Not suree but maybe event delay is caused by it as weel ( agent version is 3.4.6 ) - here

[aigars.kadikis]

It seems to cause other problems too. It seems any agent active with a nodata() check will generate false alerts because the timestamp of the event is lagged.

Many servers are blocked for passive checks.

LenR! Are you using zabbix agent version 4.0.4rc1?
64-bit: https://support.zabbix.com/secure/at...-win-amd64.zip
32-bit: https://support.zabbix.com/secure/at...1-win-i386.zip
4.0.4rc1 version is only one right now which can examine the nodata() function successfully.