Ad Widget

Collapse

No timestamp parsing with external scripts?

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • jbcheck
    Junior Member
    • May 2022
    • 15

    #1

    No timestamp parsing with external scripts?

    Hi there,

    another question from a rather old man in Germany:

    I'm reading the system log of my fritz!box with a bash sript in externalscripts. This works fine so far, Zabbix receives the data via stdout.
    To get the local timestamp I specified the appropriate log time format. Unfortunately the timestamp will not be parsed.
    What I have tried so far:
    01.04.23 22:33:44 ... --> ddMMyy hh:mm:ss
    2023-04-01 22:33:44 ... --> yyyy-MM-dd hh:mm:ss
    20230401:223344 ... --> yyyyMMdd:hhmmss

    Typ of information is configured as 'Log'.

    A layer 8 problem?

    Thanks in advance!
    jbcheck
  • Hamardaban
    Senior Member
    Zabbix Certified SpecialistZabbix Certified Professional
    • May 2019
    • 2713

    #2
    01.04.23 22:33:44 ... --> dd.MM.yyphh:mm:ss
    2023-04-01 22:33:44 ... --> yyyy-MM-ddphh:mm:ss
    20230401:223344 ... --> yyyyMMdd:hhmmss​

    There is an inaccuracy in the first example.
    And it might be worth replacing spaces with "p​"

    Comment

    • jbcheck
      Junior Member
      • May 2022
      • 15

      #3
      Thanks for the quick response!

      I tried it right away and it doesn't changed the output. 'p' as a placeholder for '-' and ':' doesn't change anything either.
      Debug mode says 'timestamp -> 0'. Unfortunately I've no idea if this has something to do with my 'mistery'.

      Ciao for now
      jbcheck

      Comment

      • Hamardaban
        Senior Member
        Zabbix Certified SpecialistZabbix Certified Professional
        • May 2019
        • 2713

        #4
        What type is your item?

        Show a few lines that you are sending to the server exactly as they are.
        Last edited by Hamardaban; 16-03-2023, 19:28.

        Comment

        • jbcheck
          Junior Member
          • May 2022
          • 15

          #5
          Type: External check
          Key: fritzbox-syslog.sh
          Type of information: Log

          Raw data look like this:
          2023-03-16 19:37:20 {"mq_log":[["16.03.23 19:37:18 Anmeldung des Benutzers zabbix an der FRITZ!Box-Benutzeroberfläche von IP-Adresse 192.168.1.222.","504","1","0"],["16.03.23 19:36:44 Anmeldung an der FRITZ!Box-Benutzeroberfläche von IP-Adresse 192.168.1.84.","500","1","0"],["16.03.23 19:36:18 Anmeldung des Benutzers zabbix an der FRITZ!Box-Benutzeroberfläche von IP-Adresse 192.168.1.222. [3 Meldungen seit 16.03.23 19:34:18]","504","1","0"],["16.03.23 19:34:05 Die Systemzeit wurde erfolgreich aktualisiert von Zeitserver xxx.xxx.32.33.","2104","1","0"],["16.03.23 19:34:05 Internetverbindung wurde erfolgreich hergestellt. IP-Adresse: xxx.xxx.192.225, DNS-Server: xxx.xxx.148.70 und xxx.xxx.150.115, Gateway: xxx.xxx.244.211, Breitband-PoP: xxx","24","2","0"],["16.03.23 19:34:01 DSL ist verfügbar (DSL-Synchronisierung besteht mit 63671\/12736 kbit\/s).","11","2","0"],["16.03.23 19:33:18 Anmeldung des Benutzers zabbix an der FRITZ!Box-Benutzeroberfläche von IP-Adresse 192.168.1.222.","504","1","0"],["16.03.23 19:32:52 DSL-Synchronisierung beginnt (Training).","12","2","0"]]}

          Processed (some sed commands) data look like this:
          2023-03-16 19:39:18 2023-03-16 19:39:17 Anmeldung des Benutzers zabbix an der FRITZ!Box-Benutzeroberfläche von IP-Adresse 192.168.1.222. [3 Meldungen seit 16.03.23 19:37:18],504,1,0
          2023-03-16 19:36:44 Anmeldung an der FRITZ!Box-Benutzeroberfläche von IP-Adresse 192.168.1.84.,500,1,0
          2023-03-16 19:36:18 Anmeldung des Benutzers zabbix an der FRITZ!Box-Benutzeroberfläche von IP-Adresse 192.168.1.222. [3 Meldungen seit 16.03.23 19:34:18],504,1,0
          2023-03-16 19:34:05 Die Systemzeit wurde erfolgreich aktualisiert von Zeitserver xxx.xxx.32.33.,2104,1,0
          2023-03-16 19:34:06 Internetverbindung wurde erfolgreich hergestellt. IP-Adresse: xxx.xxx.192.225, DNS-Server: xxx.xxx.148.70 und xxx.xxx.150.115, Gateway: xxx.xxx.244.211, Breitband-PoP: xxx,24,2,0
          2023-03-16 19:34:02 DSL ist verfügbar (DSL-Synchronisierung besteht mit 63671\/12736 kbit\/s).,11,2,0
          2023-03-16 19:33:19 Anmeldung des Benutzers zabbix an der FRITZ!Box-Benutzeroberfläche von IP-Adresse 192.168.1.222.,504,1,0
          2023-03-16 19:32:53 DSL-Synchronisierung beginnt (Training).,12,2,0

          Is a special data format needed perhaps?

          Comment

          • Hamardaban
            Senior Member
            Zabbix Certified SpecialistZabbix Certified Professional
            • May 2019
            • 2713

            #6
            I think that in order to selection the timestamp and its normal processing, it is necessary that the item receives values in the form of a single line at the beginning of which the time is written.
            Before any processing in the item.

            PS
            When I wrote
            Show a few lines that you are sending to the server exactly as they are.
            I asked about what data the item receives.
            I'm sorry, but I didn't understand that from your answer.

            Comment

            • jbcheck
              Junior Member
              • May 2022
              • 15

              #7
              Please excuse the misunderstanding I'm rather old an German. ;-)

              Some more light in the darkness, I hope:

              The only way to get the system log from my router appliance is to query it via TR-064.
              My script queries the whole system log (no other way to get the information) and passes it via stdout.
              What can be seen above (right column) are once the raw data and once the prepared data, as passed exactly.
              The left column shows the zabbix server timestamp. The 'local time' is currently missing / not processed (My understanding which might ge wrong?).
              What data: appliance start and stop messagen, logon, LAN / WAN connection status, DHCP messages, bla

              Is it bad to pass more than one line to the server per script call?
              I'm confused.

              Thanks a lot!
              jb​

              Comment

              • cyber
                Senior Member
                Zabbix Certified SpecialistZabbix Certified Professional
                • Dec 2006
                • 4807

                #8
                That timestamp thing works, if your raw data is just one line of text like
                Code:
                2023-03-16 19:39:17 Anmeldung des Benutzers zabbix an der FRITZ!Box-Benutzeroberfläche von IP-Adresse 192.168.1.222. [3 Meldungen seit 16.03.23 19:37:18],504,1,0
                If you send in json, then that does not match that... Even if you do some preprocessing to that json, you do not end up with single line starting with a timestamp, but as you showed ... you have item value as
                Code:
                2023-03-16 19:39:17 Anmeldung des Benutzers zabbix an der FRITZ!Box-Benutzeroberfläche von IP-Adresse 192.168.1.222. [3 Meldungen seit 16.03.23 19:37:18],504,1,0
                2023-03-16 19:36:44 Anmeldung an der FRITZ!Box-Benutzeroberfläche von IP-Adresse 192.168.1.84.,500,1,0
                2023-03-16 19:36:18 Anmeldung des Benutzers zabbix an der FRITZ!Box-Benutzeroberfläche von IP-Adresse 192.168.1.222. [3 Meldungen seit 16.03.23 19:34:18],504,1,0
                2023-03-16 19:34:05 Die Systemzeit wurde erfolgreich aktualisiert von Zeitserver xxx.xxx.32.33.,2104,1,0
                2023-03-16 19:34:06 Internetverbindung wurde erfolgreich hergestellt. IP-Adresse: xxx.xxx.192.225, DNS-Server: xxx.xxx.148.70 und xxx.xxx.150.115, Gateway: xxx.xxx.244.211, Breitband-PoP: xxx,24,2,0
                2023-03-16 19:34:02 DSL ist verfügbar (DSL-Synchronisierung besteht mit 63671\/12736 kbit\/s).,11,2,0
                2023-03-16 19:33:19 Anmeldung des Benutzers zabbix an der FRITZ!Box-Benutzeroberfläche von IP-Adresse 192.168.1.222.,504,1,0
                2023-03-16 19:32:53 DSL-Synchronisierung beginnt (Training).,12,2,0


                preprocessing does not split it up into multiple values, which would look like single lines...

                Comment

                • Hamardaban
                  Senior Member
                  Zabbix Certified SpecialistZabbix Certified Professional
                  • May 2019
                  • 2713

                  #9
                  That's what I've been trying to figure out and explain!

                  Comment

                  • jbcheck
                    Junior Member
                    • May 2022
                    • 15

                    #10
                    Ah, I learn and understand. Thank you!

                    So I tried to make things simple and wrote a very small (external) script:

                    Code:
                    echo "2023/03/17 15:53:18 dummy entry"
                    And it seems that I'm still wrong in my assumption of how it works:

                    Timestamp: / Local time (empty): / Value:
                    2023-03-17 19:07:18 2023/03/17 15:53:18 dummy entry
                    Log time format: yyyypMMpddphhpmmpss

                    Should this work or am I completely wrong?
                    Last edited by jbcheck; 17-03-2023, 20:15.

                    Comment

                    • jbcheck
                      Junior Member
                      • May 2022
                      • 15

                      #11
                      I gave it another try with jsonpath:

                      Click image for larger version

Name:	grafik.png
Views:	400
Size:	19.4 KB
ID:	461301

                      Click image for larger version

Name:	grafik.png
Views:	410
Size:	7.8 KB
ID:	461302


                      Click image for larger version

Name:	grafik.png
Views:	409
Size:	6.5 KB
ID:	461303

                      In this case, should the date appear in 'local time'?

                      Ciao for now
                      jb

                      Comment

                      • jbcheck
                        Junior Member
                        • May 2022
                        • 15

                        #12
                        Could it be that parsing the event date from log is not supported for external checks?
                        Even if I just let the script output the date (with and without time), it doesn't seem to be evaluated. I even tried the same format as in the example.

                        Is there another way to evaluate the log? It doesn't have to be as perfect as with the Zabbix Agent.

                        Thank your very much and a great time!
                        jb​

                        Comment


                        • Hamardaban
                          Hamardaban commented
                          Editing a comment
                          Yes - it looks like it is. "localtime" is guaranteed to work for the log and eventlog keys.
                          I also tried several options and was surprised.
                      • cyber
                        Senior Member
                        Zabbix Certified SpecialistZabbix Certified Professional
                        • Dec 2006
                        • 4807

                        #13
                        How the value arrives should not matter. I tested with a trapper item.. works as intended...
                        Click image for larger version  Name:	image.png Views:	0 Size:	16.3 KB ID:	461352

                        and then sent value to it..
                        Code:
                        echo ""<hostname>" logitem "2023/03/17 15:53:18 dummy entry"" | zabbix_sender -c /path/to/agent2.conf -i -
                        Click image for larger version  Name:	image.png Views:	0 Size:	4.7 KB ID:	461353​​
                        Last edited by cyber; 20-03-2023, 11:27.

                        Comment

                        • Hamardaban
                          Senior Member
                          Zabbix Certified SpecialistZabbix Certified Professional
                          • May 2019
                          • 2713

                          #14
                          I'm sorry - I tried my tests in the morning - and everything works!
                          Mysticism.......
                          Checked "Userparameter", "external checks" and "zabbix trapper" - "Local time" is present.​

                          zabbix_server + zabbix_proxy 6.2.7

                          Comment

                          • jbcheck
                            Junior Member
                            • May 2022
                            • 15

                            #15
                            Hm, this is myterious indeed.

                            Click image for larger version  Name:	grafik.png Views:	0 Size:	25.6 KB ID:	461835

                            Click image for larger version  Name:	grafik.png Views:	0 Size:	7.8 KB ID:	461836

                            The output part of my script:

                            curl -s "http://192.168.1.1/query.lua?mq_log=logger:status/log&sid=${sid}" | sed -e "s/\]\,\[/\n/g" | sed -e "s/\]\]\}//g" | sed "s/[^\[\[]*\[\[//" | sed "s/"//g" | sed -e "s/^\([0-9]\{2\}\).\([0-9]\{2\}\).\([0-9]\{2\}\)\(.*\)$/20\3\-\2\-\1\4/g" | sed '1!G;h;$!d' 2>&1​

                            What this line does:
                            - Fetch the log
                            - Insert line breaks
                            - Adjust the date format
                            - Reverse the line order​

                            And if I try something really simple in my script:

                            echo "2023-03-22 21:40:44 bla bla blub"

                            I get this:

                            Click image for larger version  Name:	grafik.png Views:	0 Size:	2.7 KB ID:	461837

                            That drives my crazy. Currently.

                            So if I wirte the output into a file on my zabbix server (Raspberry Pi 3, Zabbix 6.4.0) and process this file with log / logrt I also get a local time stamp.
                            I'm working on a solution with this ...

                            Thanks and ciao for now
                            jb

                            Comment

                            Working...