Ad Widget

Collapse

Log Monitoring: Overwritten log NOT in append mode

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • emviz
    Junior Member
    • Jun 2023
    • 2

    #1

    Log Monitoring: Overwritten log NOT in append mode

    Hi,
    I am new to Zabbix and I am trying to fit my system architecture into Zabbix items and triggers. I am running Zabbix v6.4.
    I have a bash script running on cronjob each X minutes which is filling up a "log template" in the following way:

    SysDate|Component Item|Instance(test,prod)|MessageDetails

    so for example in my logfile to monitor (same hostmachine, same item component but multiple instances to check the messages) I will have the following entries:

    20/06/2023-12:06:18|Component1|T1| OrgA has Link1 in Error
    20/06/2023-12:16:19|Component1|T1| OrgB has Link2 in Error
    20/06/2023-13:54:10|Component1|T3| OrgA has Link4 in Error
    20/06/2023-13:54:10|Component1|T3| OrgC has Link3 in Error

    At the moment I was able to create an item for each "instance" associated to the single host i am trying to monitor.
    So for these 4 lines i want to have 2 items.
    Each item is using the following config:
    Type: Zabbix agent (active)
    Key: Log with global regex (A="Component" and B=[P,T]1 ) in skip mode
    Type of info: Log
    Update: 1s
    Storageperiod 90d
    Log time format: dd/MM/yyyyphh:mm:ss

    so far so good i see that for each Item Component (two in this example) i can collect in "Last data" multiple lines
    Component1 - T1 has:
    OrgA has Link1 in Error
    OrgB has Link2 in Error
    Component1 - T3 has:
    OrgA has Link4 in Error
    OrgC has Link3 in Error
    Trigger is using find(ITEM , like, "Error")=1
    and the recovery expression is find(ITEM , like, "Online")=1
    Problem setting in MULTIPLE

    If i want to see the "Problems" each line is correctly associated to each component and instance.
    If I edit the log file by removing a line, the problem is also getting removed.
    So my log at each refresh print a new state of the situation and Zabbix is reading it (with small logic if item already exist to not overwrite timestamp)
    If I add a new line a new problem is correctly shown to the correct component and instance.
    So this log is not "flowing" in append mode. Is a snapshot in time each X minutes.

    Everything works fine but I have two big problems which i am not able to resolve.

    1)
    In case i have one last entry for a specific component-instance.
    E.g.
    20/06/2023-12:06:18|Component1|T1| OrgA has Link1 in Error
    20/06/2023-13:54:10|Component1|T3| OrgA has Link4 in Error
    20/06/2023-13:54:10|Component1|T3| OrgC has Link3 in Error

    I need to write in the log:
    20/06/2023-12:06:18|Component1|T1| OrgA has all links Online
    20/06/2023-13:54:10|Component1|T3| OrgA has Link4 in Error
    20/06/2023-13:54:10|Component1|T3| OrgC has Link3 in Error

    So I can clear and recover the component1-instance1
    I have no other way to differentiate what is "OrgA ... LinkX" cause there is no fixed naming convention so this part is very dynamic and this is the solution i found works better.

    2)
    The real big problem is the ALERT associated with the trigger.
    By editing\refreshing the logfile the alerts are sent each time the trigger is fired.
    So if i clear T1 i still get alarms on T3 even if they were already sent X minutes ago.

    Any hint on how i can make this working better than that?
    Appraciate in advance your help.

    Cheers​
Working...