Monitoring Log File with ANSI Color Codes

[slash1978]

Hi Folks,

I need to monitor some log files generated by Java application using log4j2 library. The log file has ANSI color codes in front of the timestamp.

Some examples:
^[[m^[[36m2024-04-15 08:45:54,918 => Debug log
^[[m^[[32m2024-04-15 08:45:54,325 => Info log
^[[m^[[1;31m2024-04-15 08:46:04,408 => Error log

As you can see from the examples above, different log type will produce different ANSI color code prefix and the length is not consistent.
In this case, how do I specify the log timestamp format?

[cyber]

you can try to preprocess all of it and remove all that crap... (Who even does such awful things, like adding colors to loglines...)
Add regex preprocessing and match "^\^\[\[m\^\[\[(.*)m(.*)" and output "\2" after that you should be able to use normal timestamp definition...

[tim.mooney]

cyber's answer should work, but I would pursue the first part of his answer: this shouldn't be in the logs in the first place. log4j2 is quite configurable, so you could make an adjustment to its config so that the application doesn't log the color codes: https://stackoverflow.com/questions/...ging-in-log4j2

[slash1978]

Just a quick question, will regex preprocessing cause any performance issue if the log files size are large (1-2 GB)?
Unfortunately I do not have control over how the log files are generated.

[cyber]

you can try to preprocess all of it and remove all that crap... (Who even does such awful things, like adding colors to loglines...)
Add regex preprocessing and match "^\^\[\[m\^\[\[(.*)m(.*)" and output "\2" after that you should be able to use normal timestamp definition...

[tim.mooney]

cyber's answer should work, but I would pursue the first part of his answer: this shouldn't be in the logs in the first place. log4j2 is quite configurable, so you could make an adjustment to its config so that the application doesn't log the color codes: https://stackoverflow.com/questions/...ging-in-log4j2

[slash1978]

Just a quick question, will regex preprocessing cause any performance issue if the log files size are large (1-2 GB)?
Unfortunately I do not have control over how the log files are generated.

[cyber]

Preprocessing does not take place in agent, so there should be no issue on agent. On server side... maybe.. depends how much you actually gather those logs...

[slash1978]

Hi Cyber, thanks again for your response. It was very helpful.
Cheers!