Log file monitoring not working in some cases

[wagfeliz]

Hi,

I am trying to do a log monitoring in an proxy monitored agent, for some reason, some kig files are getting data with no issues, some are not.
I could not find anything wrong about it in the debug log file from the agent, and its clearly working for other files.

		test2		log[/etc/ansible/playbooks/bkpruns/logs/s1.db1818.small.bkp.log]	1m	90d		Zabbix agent (active)	Ansible Log Files
		test3		log[/tmp/zt.log]	1m	90d		Zabbix agent (active)	Ansible Log Files

[root@PROXY logs]# ls -ltr /etc/ansible/playbooks/bkpruns/logs/s1.db1818.small.bkp.log /tmp/zt.log
-rw-r--r-- 1 root root 6634 Jan 24 22:44 /etc/ansible/playbooks/bkpruns/logs/s1.db1818.small.bkp.log
-rw-r--r-- 1 root root 105 Jan 27 10:57 /tmp/zt.log

Proxy	Ansible Log Files (2 Items)
			teste2
			teste3	2020-01-27 12:03:19	2016-08-13 13:03:05 third log entry

The /tmp/zt.log is an test file I add some data, its working fine, but the other file gets no data, its running for hours now.

Obs; I can see in the debug of the agent that it hass some old itens that I already deleted, its possible it have some old active checks activated somhow ? If so, how can I force the proxy/agent to get the most update data / clean it up ?

Regards,
Wagner.

[tim.mooney]

I am trying to do a log monitoring in an proxy monitored agent, for some reason, some kig files are getting data with no issues, some are not.

[root@PROXY logs]# ls -ltr /etc/ansible/playbooks/bkpruns/logs/s1.db1818.small.bkp.log /tmp/zt.log
-rw-r--r-- 1 root root 6634 Jan 24 22:44 /etc/ansible/playbooks/bkpruns/logs/s1.db1818.small.bkp.log
-rw-r--r-- 1 root root 105 Jan 27 10:57 /tmp/zt.log [TABLE]

Your zabbix_agentd probably isn't running as root (it should not be run as root). The permissions on the s1.db1818.smal.bkp.log are readable by everyone, but the permissions on some of the directories may not be.

What happens when you run your "ls -ltr" as the user that zabbix_agentd runs as?

sudo -u zabbix ls -ltr /etc/ansible/playbooks/bkpruns/logs/s1.db1818.small.bkp.log /tmp/zt.log

Do you get the same output, or do you get "permission denied" for the first one?

[wagfeliz]

Your zabbix_agentd probably isn't running as root (it should not be run as root). The permissions on the s1.db1818.smal.bkp.log are readable by everyone, but the permissions on some of the directories may not be.

What happens when you run your "ls -ltr" as the user that zabbix_agentd runs as?

sudo -u zabbix ls -ltr /etc/ansible/playbooks/bkpruns/logs/s1.db1818.small.bkp.log /tmp/zt.log

Do you get the same output, or do you get "permission denied" for the first one?

It was the first thing I tried :

[root@PROXY bkpruns]# sudo -u zabbix ls -ltr /etc/ansible/playbooks/bkpruns/logs/bkp_sql_1818.2020-02-03_22-00-01.log /tmp/zt.log
-rw-r--r-- 1 root root 105 Jan 27 10:57 /tmp/zt.log
-rw-r--r-- 1 root root 201554 Feb 4 06:07 /etc/ansible/playbooks/bkpruns/logs/bkp_sql_1818.2020-02-03_22-00-01.log

[Markku]

Hi, so, the item is configured as:

log[/etc/ansible/playbooks/bkpruns/logs/s1.db1818.small.bkp.log]

but the file name is

/etc/ansible/playbooks/bkpruns/logs/bkp_sql_1818.2020-02-03_22-00-01.log

Is that what you are saying?

Markku

[wagfeliz]

Hi, so, the item is configured as:



but the file name is



Is that what you are saying?

Markku

The file from the test dont exist anymore, I listed an new file to show its not an permission issue.
I think it may be something about the filename size or the name itself, like, containing dots in the middle.

[Markku]

In your original message:

[root@PROXY logs]# ls -ltr /etc/ansible/playbooks/bkpruns/logs/s1.db1818.small.bkp.log /tmp/zt.log
-rw-r--r-- 1 root root 6634 Jan 24 22:44 /etc/ansible/playbooks/bkpruns/logs/s1.db1818.small.bkp.log
-rw-r--r-- 1 root root 105 Jan 27 10:57 /tmp/zt.log

Proxy	Ansible Log Files (2 Items)
			teste2
			teste3	2020-01-27 12:03:19	2016-08-13 13:03:05 third log entry

The /tmp/zt.log is an test file I add some data, its working fine, but the other file gets no data, its running for hours now.

In your demo, the timestamp of the /etc/ansible/playbooks/bkpruns/logs/s1.db1818.small.bkp.log file is three days before your message, unlike /tmp/zt.log which was recently updated. The table in the output above is apparently the Latest data window, which does not show data older than 24 hours. Reference: https://www.zabbix.com/documentation...ng/latest_data

So, can you please show the "ls -l" output with the actual files, the actual item configuration, and the actual view from the Latest data again, if you are still facing problems with one file only?

Markku

[wagfeliz]

It dosent seems to be related to last data also. I just recreate another item with the new file format :

log[/etc/ansible/playbooks/bkpruns/logs/bkp_sqlhom.lastresult.log]

[root@PROXY logs]# ps -ef | grep zabbix_proxy.conf
zabbix 14552 1 0 Feb04 ? 00:00:00 /usr/sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf
root 15539 14297 0 10:27 pts/2 00:00:00 grep --color=auto zabbix_proxy.conf
[root@PROXY logs]# su - zabbix
Last login: Mon Jan 27 11:20:47 -03 2020 on pts/0
[zabbix@PROXY ~]$ cat /etc/ansible/playbooks/bkpruns/logs/bkp_sqlhom.lastresult.log
192.168.25.30 : ok=21 changed=7 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
192.168.25.30 : ok=21 changed=7 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

The configuration of the Item is very vanilla :
Type of Information = Log
Update Interval = 1m
Enabled

This is assigned to the proxy zabbix host, it worked for the /tmp/zt.log file so I dont think the problem is comunication or configuration.

Still no data is generated even after I add new lines to it.

Ideas ?

[Markku]

Sorry I have to ask this to be sure as you didn't show the full configuration: In your latest test you created a new log item, did you select the item type "Zabbix agent (active)"?

This is how it looks like if Zabbix agent cannot access the file:

There is also error in zabbix_agentd.log:

31342:20200206:155427.313 active check "log[/tmp/logtest]" is not supported: Cannot open file "/tmp/logtest": [13] Permission denied

People are always talking about SElinux problems, I don't know about those, maybe worth trying disabling it for testing if in use.

Re-read the manual in https://www.zabbix.com/documentation...ypes/log_items, if that gives you some idea. Otherwise I don't have new ideas. Some difference there apparently is if you say that one of the two identical items doesn't work.

Markku

[wagfeliz]

I just found out something diferent, I did notice that after a few hours we have the following log about this :

9643:20200206:132416.826 item "PROXY:log[/etc/ansible/playbooks/bkpruns/logs/bkp_sqlhom.lastresult.log]" became supported
9642:20200206:132616.905 item "PROXY:log[/etc/ansible/playbooks/bkpruns/logs/bkp_sqlhom.lastresult.log]" became not supported: Cannot obtain information for file "/etc/ansible/playbooks/bkpruns/logs/bkp_sqlhom.lastresult.log": [2] No such file or directory

Maybe there is something wrong in the key format ? :::
log[/etc/ansible/playbooks/bkpruns/logs/bkp_sqlhom.lastresult.log]

Should I add some parameters or maybe quotes ?

[Markku]

I don't see any problem with the item or file name. Have you talked with the admins of the software that is writing the log files?

Markku

[wagfeliz]

I don't see any problem with the item or file name. Have you talked with the admins of the software that is writing the log files?

Markku

I am not sure I understood what you asked... The file was generated by the ansible, by an script that I made my self.
I am getting worried that this may be an bug, I looked to this error "became not supported: Cannot obtain information for file" and "No such file or directory" for log operations in zabbix and I found thousands of complains, and no solution, I will try to open another bug about this.

I also did notice that for the log[/tmp/logtest], it was working before, now its not working anymore ...

[Markku]

What I meant is that since the documentation mentions lots of things to consider when monitoring files, maybe the log file is created or modified in such a way that the file monitoring does not work as intended.

If you file a bug, please have the ticket mentioned here as well so that the interested readers can see what happens with that route.

Markku

[wagfeliz]

What I meant is that since the documentation mentions lots of things to consider when monitoring files, maybe the log file is created or modified in such a way that the file monitoring does not work as intended.

If you file a bug, please have the ticket mentioned here as well so that the interested readers can see what happens with that route.

Markku

Sure, but I am not sure we need to file a new bug, I found this bug that seems to be the case :

https://support.zabbix.com/browse/ZBX-13254

As I understood its not fixed yet, its resolved and have hotfixes for even the older version 3.0, but its not applied to the 4.0 ( its in plan state ).

[tim.mooney]

https://support.zabbix.com/browse/ZBX-13254

As I understood its not fixed yet, its resolved and have hotfixes for even the older version 3.0, but its not applied to the 4.0 ( its in plan state ).

The way I read that bug report, it's specifically related to the behavior when "skip" is one of the arguments, which isn't the case here.

Also, it was integrated into all of the active branches of development, including 4.0.0alpha2, so if you're on any 4.x version, your version already includes the fix.

I guess I don't understand why you think this is a bug. If you read the "Comments" for the log[] item, the 2nd line says

If file is missing or permissions do not allow access, item turns unsupported.

Since that's exactly what's happening here, isn't it likely that the problem is that Zabbix can't read (for some reason) the file you've specified?

Have you considered adding a second item, vfs.file.exists[] for the same file? That way, when the log[] item becomes unsupported, you can look at the results from the vfs.file.exists[] item to see if the file still exists. That won't tell you if the issue is a permission problem, but it will help narrow down what the problem is. Since Zabbix is specifically saying "No such file or directory", that pretty clearly indicates that the file isn't there.