Log file monitor repeatedly sends old messages
Hi everyone!
I am struggling with our logfile monitor.
At some, seemingly random, times Zabbix reports old log messages.
For instance, we had an outage on Friday September 14th:
Timestamp Local time Value
2018-09-14 20:20:53 2018-09-14 20:01:27 14/09/2018 20:01:27 [0984] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
2018-09-14 20:20:53 2018-09-14 19:54:48 14/09/2018 19:54:48 [0984] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
2018-09-14 20:20:53 2018-09-14 19:51:21 14/09/2018 19:51:21 [12a0] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ).
2018-09-14 20:20:53 2018-09-14 19:50:51 14/09/2018 19:50:51 [0d48] CACPM049E Error getting Safe PasswordManager_workspace from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
2018-09-14 20:20:53 2018-09-14 19:47:35 14/09/2018 19:47:35 [0984] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
2018-09-14 19:48:49 2018-09-14 19:47:35 14/09/2018 19:47:35 [0984] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
All log entries were reported correctly by Zabbix. We configured actions to automatically create problem tickets and send pager and e-mail alerts.
Everything worked as expected.
But since then we sometimes get these old messages again:
Like this morning at 06:37
Timestamp Local time Value
2018-09-20 06:37:26 2018-09-14 20:01:27 14/09/2018 20:01:27 [0984] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
2018-09-20 06:37:26 2018-09-14 19:54:48 14/09/2018 19:54:48 [0984] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
2018-09-20 06:37:26 2018-09-14 19:51:21 14/09/2018 19:51:21 [12a0] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ).
2018-09-20 06:37:26 2018-09-14 19:50:51 14/09/2018 19:50:51 [0d48] CACPM049E Error getting Safe PasswordManager_workspace from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
2018-09-20 06:37:26 2018-09-14 19:47:35 14/09/2018 19:47:35 [0984] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
And again, ofcourse, new problem tickets were created etc.
This happens on an average 3 times per day at random intervals.
I know that the Log file pointer is reset to 0 if the file gets smaller, so I added a file size monitor to keep track of this.
But the logfile is behaving normally, its size didn't change at all around 06:37:
Timestamp Filesize
2018-09-20 06:40:49 46176001
2018-09-20 06:39:49 46176001
2018-09-20 06:38:49 46176001
2018-09-20 06:37:49 46176001
2018-09-20 06:36:49 46176001
2018-09-20 06:35:49 46176001
2018-09-20 06:34:49 46176001
2018-09-20 06:33:49 46176001
2018-09-20 06:32:49 46176001
2018-09-20 06:31:49 46176001
2018-09-20 06:30:49 46176001
Further logfile characteristics:
Last entry: 2018-09-20 08:50:09
Created: 2018-07-02 10:27:23
Modified: 2018-09-20 08:50:09
Accessed: 2018-07-02 10:27:23
Item configuration:
Type: Zabbix agent (active)
Key: log["D:\Program Files (x86)\CyberArk\Password Manager\Logs\pm.log", "(.*?\] \S+[WE] .*)",,,,\1 ]
Type of information: Log
Log time format: dd/MM/yyyy hh:mm:ss
Trigger expression: {Template CA CPM Specific:log["D:\Program Files (x86)\CyberArk\Password Manager\Logs\pm.log", "(.*?\] \S+[WE] .*)",,,,\1 ].str(CACPM049E)}=1
Can anyone please help me out with this issue?
I am struggling with our logfile monitor.
At some, seemingly random, times Zabbix reports old log messages.
For instance, we had an outage on Friday September 14th:
Timestamp Local time Value
2018-09-14 20:20:53 2018-09-14 20:01:27 14/09/2018 20:01:27 [0984] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
2018-09-14 20:20:53 2018-09-14 19:54:48 14/09/2018 19:54:48 [0984] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
2018-09-14 20:20:53 2018-09-14 19:51:21 14/09/2018 19:51:21 [12a0] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ).
2018-09-14 20:20:53 2018-09-14 19:50:51 14/09/2018 19:50:51 [0d48] CACPM049E Error getting Safe PasswordManager_workspace from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
2018-09-14 20:20:53 2018-09-14 19:47:35 14/09/2018 19:47:35 [0984] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
2018-09-14 19:48:49 2018-09-14 19:47:35 14/09/2018 19:47:35 [0984] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
All log entries were reported correctly by Zabbix. We configured actions to automatically create problem tickets and send pager and e-mail alerts.
Everything worked as expected.
But since then we sometimes get these old messages again:
Like this morning at 06:37
Timestamp Local time Value
2018-09-20 06:37:26 2018-09-14 20:01:27 14/09/2018 20:01:27 [0984] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
2018-09-20 06:37:26 2018-09-14 19:54:48 14/09/2018 19:54:48 [0984] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
2018-09-20 06:37:26 2018-09-14 19:51:21 14/09/2018 19:51:21 [12a0] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ).
2018-09-20 06:37:26 2018-09-14 19:50:51 14/09/2018 19:50:51 [0d48] CACPM049E Error getting Safe PasswordManager_workspace from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
2018-09-20 06:37:26 2018-09-14 19:47:35 14/09/2018 19:47:35 [0984] CACPM049E Error getting Safe PasswordManager from Vault CPM Vault (error: ITACM062S Communication error (Diagnostic information: 562) ).
And again, ofcourse, new problem tickets were created etc.
This happens on an average 3 times per day at random intervals.
I know that the Log file pointer is reset to 0 if the file gets smaller, so I added a file size monitor to keep track of this.
But the logfile is behaving normally, its size didn't change at all around 06:37:
Timestamp Filesize
2018-09-20 06:40:49 46176001
2018-09-20 06:39:49 46176001
2018-09-20 06:38:49 46176001
2018-09-20 06:37:49 46176001
2018-09-20 06:36:49 46176001
2018-09-20 06:35:49 46176001
2018-09-20 06:34:49 46176001
2018-09-20 06:33:49 46176001
2018-09-20 06:32:49 46176001
2018-09-20 06:31:49 46176001
2018-09-20 06:30:49 46176001
Further logfile characteristics:
Last entry: 2018-09-20 08:50:09
Created: 2018-07-02 10:27:23
Modified: 2018-09-20 08:50:09
Accessed: 2018-07-02 10:27:23
Item configuration:
Type: Zabbix agent (active)
Key: log["D:\Program Files (x86)\CyberArk\Password Manager\Logs\pm.log", "(.*?\] \S+[WE] .*)",,,,\1 ]
Type of information: Log
Log time format: dd/MM/yyyy hh:mm:ss
Trigger expression: {Template CA CPM Specific:log["D:\Program Files (x86)\CyberArk\Password Manager\Logs\pm.log", "(.*?\] \S+[WE] .*)",,,,\1 ].str(CACPM049E)}=1
Can anyone please help me out with this issue?
Comment