Max perfomance log monitoring

[arcenciel4]

Hi! Zabbix monitors 2 logs on Windows. Logs are recorded 24 hours a day. And contain a lot of information. The only problem is the search of regular expressions and record of string with them. But zabbix lost a part of strings. I can't understand why.
Zabbix agent:

# This is a config file for the Zabbix agent daemon (Windows)
# To get more information about Zabbix, go to http://www.zabbix.com

############ GENERAL PARAMETERS #################

### Option: LogFile
#	Name of log file.
#	If not set, Windows Event Log is used.
#
# Mandatory: no
# Default:
# LogFile=

LogFile=J:\Zabbix-Agent\log\zabbix_agentd.log

### Option: LogFileSize
#	Maximum size of log file in MB.
#	0 - disable automatic log rotation.
#
# Mandatory: no
# Range: 0-1024
# Default:
 LogFileSize=50

### Option: DebugLevel
#	Specifies debug level 
#	0 - no debug
#	1 - critical information
#	2 - error information
#	3 - warnings
#	4 - for debugging (produces lots of information)
#
# Mandatory: no
# Range: 0-4
# Default:
# DebugLevel=3
DebugLevel=3

### Option: SourceIP
#	Source IP address for outgoing connections.
#
# Mandatory: no
# Default:
# SourceIP=

### Option: EnableRemoteCommands
#	Whether remote commands from Zabbix server are allowed.
#	0 - not allowed
#	1 - allowed
#
# Mandatory: no
# Default:
 EnableRemoteCommands=1

### Option: LogRemoteCommandsc 
#	Enable logging of executed shell commands as warnings.
#	0 - disabled
#	1 - enabled
#
# Mandatory: no
# Default:
LogRemoteCommands=1

##### Passive checks related

### Option: Server
#	List of comma delimited IP addresses (or hostnames) of Zabbix servers.
#	Incoming connections will be accepted only from the hosts listed here.
#	If IPv6 support is enabled then '127.0.0.1', '::127.0.0.1', '::ffff:127.0.0.1' are treated equally.
#
# Mandatory: no
# Default:
# Server=

Server=195.***.***.7

### Option: ListenPort
#	Agent will listen on this port for connections from the server.
#
# Mandatory: no
# Range: 1024-32767
# Default:
ListenPort=10050

### Option: ListenIP
#       List of comma delimited IP addresses that the agent should listen on.
#       First IP address is sent to Zabbix server if connecting to it to retrieve list of active checks.
#
# Mandatory: no
# Default:
# ListenIP=0.0.0.0


### Option: StartAgents
#	Number of pre-forked instances of zabbix_agentd that process passive checks.
#	If set to 0, disables passive checks and the agent will not listen on any TCP port.
#
# Mandatory: no
# Range: 0-100
# Default:
# StartAgents=3
StartAgents=5

##### Active checks related

### Option: ServerActive
#	List of comma delimited IP:port (or hostname:port) pairs of Zabbix servers for active checks.
#	If port is not specified, default port is used.
#	IPv6 addresses must be enclosed in square brackets if port for that host is specified.
#	If port is not specified, square brackets for IPv6 addresses are optional.
#	If this parameter is not specified, active checks are disabled.
#	Example: ServerActive=127.0.0.1:20051,zabbix.domain,[::1]:30051,::1,[12fc::1]
#
# Mandatory: no
# Default:
# ServerActive=
ServerActive=195.***.***.7:10051


### Option: Hostname
#	Unique, case sensitive hostname.
#	Required for active checks and must match hostname as configured on the server.
#	Value is acquired from HostnameItem if undefined.
#
# Mandatory: no
# Default:
# Hostname=

Hostname=MT4_MTXAPCL1

### Option: HostnameItem
#	Item used for generating Hostname if it is undefined.
#	Ignored if Hostname is defined.
#
# Mandatory: no
# Default:
# HostnameItem=system.hostname

### Option: HostMetadata
#	Optional parameter that defines host metadata.
#	Host metadata is used at host auto-registration process.
#	An agent will issue an error and not start if the value is over limit of 255 characters.
#	If not defined, value will be acquired from HostMetadataItem.
#
# Mandatory: no
# Range: 0-255 characters
# Default:
# HostMetadata=

### Option: HostMetadataItem
#	Optional parameter that defines an item used for getting host metadata.
#	Host metadata is used at host auto-registration process.
#	During an auto-registration request an agent will log a warning message if
#	the value returned by specified item is over limit of 255 characters.
#	This option is only used when HostMetadata is not defined.
#
# Mandatory: no
# Default:
# HostMetadataItem=

### Option: RefreshActiveChecks
#	How often list of active checks is refreshed, in seconds.
#
# Mandatory: no
# Range: 60-3600
# Default:
 RefreshActiveChecks=60

### Option: BufferSend
#	Do not keep data longer than N seconds in buffer.
#
# Mandatory: no
# Range: 1-3600
# Default:
 BufferSend=5

### Option: BufferSize
#	Maximum number of values in a memory buffer. The agent will send
#	all collected data to Zabbix server or Proxy if the buffer is full.
#
# Mandatory: no
# Range: 2-65535
# Default:
 BufferSize=2000

### Option: MaxLinesPerSecond
#	Maximum number of new lines the agent will send per second to Zabbix Server
#	or Proxy processing 'log', 'logrt' and 'eventlog' active checks.
#	The provided value will be overridden by the parameter 'maxlines',
#	provided in 'log', 'logrt' or 'eventlog' item keys.
#
# Mandatory: no
# Range: 1-1000
# Default:
 MaxLinesPerSecond=1000

############ ADVANCED PARAMETERS #################

### Option: Alias
#	Sets an alias for an item key. It can be used to substitute long and complex item key with a smaller and simpler one.
#	Multiple Alias parameters may be present. Multiple parameters with the same Alias key are not allowed.
#	Different Alias keys may reference the same item key.
#	For example, to retrieve paging file usage in percents from the server:
#	Alias=pg_usage:perf_counter[\Paging File(_Total)\% Usage]
#	Now shorthand key pg_usage may be used to retrieve data.
#	Aliases can be used in HostMetadataItem but not in HostnameItem or PerfCounter parameters.
#
# Mandatory: no
# Range:
# Default:

### Option: Timeout
#	Spend no more than Timeout seconds on processing
#
# Mandatory: no
# Range: 1-30
# Default:
 Timeout=30

### Option: PerfCounter
#	Syntax: <parameter_name>,"<perf_counter_path>",<period>
#	Defines new parameter <parameter_name> which is an average value for system performance counter <perf_counter_path> for the specified time period <period> (in seconds).
#	For example, if you wish to receive average number of processor interrupts per second for last minute, you can define new parameter "interrupts" as following:
#	PerfCounter = interrupts,"\Processor(0)\Interrupts/sec",60
#	Please note double quotes around performance counter path.
#	Samples for calculating average value will be taken every second.
#	You may run "typeperf -qx" to get list of all performance counters available in Windows.
#
# Mandatory: no
# Range:
# Default:

### Option: Include
#	You may include individual files in the configuration file.
#
# Mandatory: no
# Default:
# Include=

 #Include=d:\Alfa monitoring\zabbix\zabbix_agentd.userparams.conf

####### USER-DEFINED MONITORED PARAMETERS #######

### Option: UnsafeUserParameters
#	Allow all characters to be passed in arguments to user-defined parameters.
#	0 - do not allow
#	1 - allow
#
# Mandatory: no
# Range: 0-1
# Default:
# UnsafeUserParameters=0

### Option: UserParameter
#	User-defined parameter to monitor. There can be several user-defined parameters.
#	Format: UserParameter=<key>,<shell command>
#
# Mandatory: no
# Default:
# UserParameter=

In zabbix server config I just changed:

StartPollers = 60
StartTrappers = 60

Settings of Items:

Triggers use nodata function.
Agent version is 2.2.1 х64.
Why zabbix lost data?

[jan.garaj]

Did you read the important notes about log monitoring? https://www.zabbix.com/documentation...ypes/log_items

It can be anything? For example:
- your log lines were longer than 256kB
- you have had more than 1k matched log lines per second (for log monitoring is only allocated "only" 50% of BufferSize=2000)
- agent lost connection and it took more than 5 seconds (BufferSend=5)
- you have had more than 250 matched lines per check (MaxLinesPerSecond=1000 and check interval=1)

[arcenciel4]

Did you read the important notes about log monitoring? https://www.zabbix.com/documentation...ypes/log_items

It can be anything? For example:
- your log lines were longer than 256kB
- you have had more than 1k matched log lines per second (for log monitoring is only allocated "only" 50% of BufferSize=2000)
- agent lost connection and it took more than 5 seconds (BufferSend=5)
- you have had more than 250 matched lines per check (MaxLinesPerSecond=1000 and check interval=1)

Yes, I did. Lines are not longer 256kB. But what with other parametrs? I must to increase BufferSize and BufferSend? In logs I saw sometimes 1500 lines per minute.

[jan.garaj]

At first discover your max log rate per second - (avg per second is 25, but peak is important) and then set up MaxLinesPerSecond, Update interval and BufferSize accordingly

To find the required string Zabbix will process 4 times more new lines than set in MaxLinesPerSecond. Thus, for example, if a log[] or logrt[] item has Update interval of 1 second, by default the agent will analyse no more than 400 log file records and will send no more than 100 matching records to Zabbix server in one check. By increasing MaxLinesPerSecond in the agent configuration file or setting maxlines parameter in the item key, the limit can be increased up to 4000 analysed log file records and 1000 matching records sent to Zabbix server in one check. If the Update interval is set to 2 seconds the limits for one check would be set 2 times higher than with Update interval of 1 second.

[arcenciel4]

At first discover your max log rate per second - (avg per second is 25, but peak is important) and then set up MaxLinesPerSecond, Update interval and BufferSize accordingly

To find the required string Zabbix will process 4 times more new lines than set in MaxLinesPerSecond. Thus, for example, if a log[] or logrt[] item has Update interval of 1 second, by default the agent will analyse no more than 400 log file records and will send no more than 100 matching records to Zabbix server in one check. By increasing MaxLinesPerSecond in the agent configuration file or setting maxlines parameter in the item key, the limit can be increased up to 4000 analysed log file records and 1000 matching records sent to Zabbix server in one check. If the Update interval is set to 2 seconds the limits for one check would be set 2 times higher than with Update interval of 1 second.

Thank a lot! I will try!

[arcenciel4]

At first discover your max log rate per second - (avg per second is 25, but peak is important) and then set up MaxLinesPerSecond, Update interval and BufferSize accordingly

To find the required string Zabbix will process 4 times more new lines than set in MaxLinesPerSecond. Thus, for example, if a log[] or logrt[] item has Update interval of 1 second, by default the agent will analyse no more than 400 log file records and will send no more than 100 matching records to Zabbix server in one check. By increasing MaxLinesPerSecond in the agent configuration file or setting maxlines parameter in the item key, the limit can be increased up to 4000 analysed log file records and 1000 matching records sent to Zabbix server in one check. If the Update interval is set to 2 seconds the limits for one check would be set 2 times higher than with Update interval of 1 second.

I have some question.
Buffer size allocate place only for matched string or for other too?
Update interval increase all limits of parameters?(BufferSize, MaxLinesPerSecond)
Maximum lines per minute 1500, per second 25.
For matched lines is less 20 per check.
I should set:
MaxLinesPerSecond = 30
UpdateInterval = 1
BufferSize = 200 (it's correct?)
BufferSend = 50 (it's enough for lost connection?)

[jan.garaj]

Buffer size allocate place only for matched string or for other too?
- only matched
Update interval increase all limits of parameters?(BufferSize, MaxLinesPerSecond)
- no, none - processed lines!=matched line
Maximum lines per minute 1500, per second 25.
- that's average,your app can flush 1500 log lines in one second and then 59 seconds will be quiet - you should to know how your app works and then setup your tools accordingly ;-)

My recommendation:
MaxLinesPerSecond = 200
UpdateInterval = 1
BufferSize = 400
BufferSend = 5

=> max analyzed lines: 800/s, max sent lines: 200/s
But it weaker setting than your original config - so I don't understand what and why zabbix missed some logs.
Anyway you can go to source code and study implementation

https://www.zabbix.org/websvn/wsvn/zabbix.com/trunk/src/zabbix_agent/logfiles.c

[arcenciel4]

buffer size allocate place only for matched string or for other too?
- only matched
update interval increase all limits of parameters?(buffersize, maxlinespersecond)
- no, none - processed lines!=matched line
maximum lines per minute 1500, per second 25.
- that's average,your app can flush 1500 log lines in one second and then 59 seconds will be quiet - you should to know how your app works and then setup your tools accordingly ;-)

my recommendation:
Maxlinespersecond = 200
updateinterval = 1
buffersize = 400
buffersend = 5

=> max analyzed lines: 800/s, max sent lines: 200/s
but it weaker setting than your original config - so i don't understand what and why zabbix missed some logs.
Anyway you can go to source code and study implementation
https://www.zabbix.org/websvn/wsvn/z...ent/logfiles.c

thank you!

[arcenciel4]

Buffer size allocate place only for matched string or for other too?
- only matched
Update interval increase all limits of parameters?(BufferSize, MaxLinesPerSecond)
- no, none - processed lines!=matched line
Maximum lines per minute 1500, per second 25.
- that's average,your app can flush 1500 log lines in one second and then 59 seconds will be quiet - you should to know how your app works and then setup your tools accordingly ;-)

My recommendation:
MaxLinesPerSecond = 200
UpdateInterval = 1
BufferSize = 400
BufferSend = 5

=> max analyzed lines: 800/s, max sent lines: 200/s
But it weaker setting than your original config - so I don't understand what and why zabbix missed some logs.
Anyway you can go to source code and study implementation
https://www.zabbix.org/websvn/wsvn/z...ent/logfiles.c

Unfortunately, it did not help