Eventlog Item Stopped Retrieving Log

[Buxton69]

I've got three items set up in a template that retrieve data from the three different event logs, these were working correctly and retrieving the logs but this morning for some reason the Application and System logs have stopped retrieving data at 05:21 and 05:23 this morning but the Security log is still being received, the odd thing is that the same thing has happened on 4 different servers all at different times.

There are no errors, I tried recreating the items but they no longer work only the Security log works.

Anyone any ideas on what I should look at or has anyone had similar or what might I have done wrong?


Thanks

[Buxton69]

I've still got this problem, it all started working again over the weekend but stopped again this morning with still no errors.

Does anyone know of anything I need to check or require any more information from me?


Thanks

[Aly]

I would set Debug level to 4, and searched for those items in it.

[Buxton69]

I had it on level 4 before but usually by the time I get in and see it stopped the events are no longer in the event viewer but I just found on one of the servers that this type of message stops at the same time I stopped receiving any application or system log information, this happens on all servers:

Event Type: Warning
Event Source: ZABBIX Agent
Event Category: None
Event ID: 1
Date: 8/18/2009
Time: 9:02:47 AM
User: N/A
Computer:
Description:
[1064]: Info from server: Processed 100 Failed 0 Total 100 Seconds spent 0.077165

I'll set the error level back to 4 and see if I can capture it.

[Buxton69]

I restarted the agent on one server after changing the logging level on just that server and all the servers have started working again except for the system log on the server I restarted the agent but the application log started !!!!!!, this did not happen before when I restarted the agents, this seems to be increasingly random.

[Buxton69]

And it has stopped again, this is the first error in the system log after it stopped getting the application log:

[5888]: Send value error: [recv] ZBX_TCP_READ() failed [A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Question is why does this happen for only the application log and system log but not the security log?

[Buxton69]

I updated the agent so it was on the latest version, the same as the server 1.6.5 and it all started working again for 3 minutes and then:

[1640]: Send value error: [recv] ZBX_TCP_READ() failed [A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
]

Still get the security log though.

[Buxton69]

I had this working over the weekend and then yesterday for no reason it all just stopped, does anyone have any ideas at all what can be checked to see why this is happening?

[Buxton69]

Extra added odd behaviour, when you refresh the screen, the date under the last check column for the item where I have eventlog[Application] can change either backwards or forwards in time on each refresh. So, at first the date was 14:09, then 13:58 and then 14:38, this is not the first time I have noticed this, it only tends to happen when the items for the eventlog have stopped working.

Any ideas or am I wasting my time, getting really fed up with Zabbix being so flakey now and getting no responce.

[bimind]

Same Problem

I have two independent Zabbix Server. One with 1.6.2 and the eventlog work fine and the other with 1.6.5 and have the same problem with the eventlog.

The event log stops receiving data or when receiving the data is old data. When I restart the agent and zabbix server everything returns to normal.

When I step it also raises the CPU utilization.

Thanks,

Oriol

[Buxton69]

Unfortunately I can't restart the Zabbix server when I want as it's being used in a production environment.

At the moment I have it working properly on one stand alone server, at least for a couple of days now but on the servers that are clusted on one the security log is working and on the other two nodes the application log is working but the other logs are notis does seem to be very random.

[bimind]

zabbix_server.log

When it fails the EventLog on the Zabbix logs file "zabbix_server.log" is recorded the following error repeatedly.

"2368:20090902:125348 [Z3005] Query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update items set nextcheck=1251888430,prevvalue=lastvalue,lastvalue ='Se ha creado la impresora RICOH Aficio MP 161 PCL 5e (desde TRANEXI) en la sesión 3."

MYSQL then begins to consume CPU in my Zabbix server and slow down considerably.

Thanks,
Oriol

[Buxton69]

I think that might not be the same problem I've got but here is the contents, or some of the thousands of messages from the eventlog after the level is changed to 4, these are some messages when it looks as if it is working:

[16168]: In zbx_get_eventlog_message() [source:System] [which:687419]
[16168]: In zbx_get_eventlog_message() [source:System] [which:687420]
[16168]: In process_value('SERVER','eventlog[System]','[16168]: In send_buffer('zabbix.tmf-group.com','10051')
')
[16168]: In send_buffer('zabbix.tmf-group.com','10051')
[16168]: Values in the buffer 76 Max 100
[16168]: Will not send now. Now 1252413507 lastsent 1252413503 < 5


[16168]: JSON before sending [{
"request":"agent data",
"data":[
{
"host":"SERVER",
"key":"eventlog[System]",
"value":"[16168]: In send_buffer('zabbix.tmf-group.com','10051')\r\n",
"lastlogsize":686548,
"timestamp":1252413445,
"source":"ZABBIX Agent",
"severity":1,
"clock":1252413507},
{
............
"clock":1252413507}],
"clock":1252413507}]

[16168]: JSON back [{
"response":"success",
"info":"Processed 100 Failed 0 Total 100 Seconds spent 0.202748"}]
...........
[16168]: Info from server: Processed 100 Failed 0 Total 100 Seconds spent 0.202748
[16168]: OK
[16168]: Buffer: new element 0
[16168]: In zbx_open_eventlog() [source:System]
16168]: Values in the buffer 2 Max 100
[16168]: Will not send now. Now 1252413509 lastsent 1252413507 < 5
[16168]: Buffer: new element 2

----------------------------------------------------------------------------
This is the error message when there are no messages being sent, nothing had happened on the server or network at this point, it has just stopped working:


[16168]: Send value error: [recv] ZBX_TCP_READ() failed [A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
]


Now, this really need to be looked at by someone at Zabbix, it shouldn't just stop working for no reason, I am now using 1.6.6 and the same version for the agent.

Most of the time the system log shows mothing but 'waiting for 1 second' when I switch the agent on and the eventlog[application] is not working and the error level is at 4, it only gives the above error message when it decides to work for a few minutes and then decides not to. I have no problems with any of the other counters through the agent, just the eventlog part.

Please can someone at Zabbix take some time to look at this issue as no one else seems to know what it is doing either, or can someone tell me what other information they need in order to fix this.

[Buxton69]

Don't really know why but this has now been working for a week, the only things I did was to removed disabled items from the template and it has been working since then.

Question to the Zabbix team: Is this a bug?


I've now added a trigger to the template:

{Template_Testr:eventlog[System].logseverity( 4 ) }=4 means HIGH message from System event log.

The eventlog messages have stopped working again, if I remove the trigger the eventlog items all start working again.

Question to the Zabbix team or anyone: Is this a bug and will it be fixed in the next release, is there any configuration I should be checking?


These messages have started to feel like diary entries on how I struggle with Zabbix as it seems like I just update this thread and people read it but no-one contributes.


Thanks in avance for your help.

[Buxton69]

Anyone any ideas about this as it would be really good to get it working.

Thanks