Random Timeouts on Multiple Items and Hosts

[automiketic]

I have an environment of about 100 hosts with around 14k items monitored. I am using SNMP, SSH, ICMP, and HTTP requests, with most items being HTTP or SSH. All hosts are agentless. I am not sure on the correlation, but I am finding that many of my hosts are regularly showing timeouts and collecting data for particular items randomly. I have gone in and ran a test on the individual item right after the error and its always successful. Yet I am seeing these random timeouts and no network changes were made. I did have an issue where my value cache was getting full and causing a bunch of issues so I increased it all the way up to 8Gb and that resolved the issue. Looking at my zabbix logs it seems the issue started shortly after that change. I have attached my config. Any advice is greatly appreciated.

# This is a configuration file for Zabbix server daemon
# To get more information about Zabbix, visit http://www.zabbix.com

############ GENERAL PARAMETERS #################

### Option: ListenPort
# Listen port for trapper.
#
# Mandatory: no
# Range: 1024-32767
# Default:
# ListenPort=10051

### Option: SourceIP
# Source IP address for outgoing connections.
#
# Mandatory: no
# Default:
# SourceIP=

### Option: LogType
# Specifies where log messages are written to:
# system - syslog
# file - file specified with LogFile parameter
# console - standard output
#
# Mandatory: no
# Default:
# LogType=file

### Option: LogFile
# Log file name for LogType 'file' parameter.
#
# Mandatory: yes, if LogType is set to file, otherwise no
# Default:
# LogFile=

LogFile=/var/log/zabbix/zabbix_server.log

### Option: LogFileSize
# Maximum size of log file in MB.
# 0 - disable automatic log rotation.
#
# Mandatory: no
# Range: 0-1024
# Default:
# LogFileSize=1

LogFileSize=0

### Option: DebugLevel
# Specifies debug level:
# 0 - basic information about starting and stopping of Zabbix processes
# 1 - critical information
# 2 - error information
# 3 - warnings
# 4 - for debugging (produces lots of information)
# 5 - extended debugging (produces even more information)
#
# Mandatory: no
# Range: 0-5
# Default:
# DebugLevel=3

### Option: PidFile
# Name of PID file.
#
# Mandatory: no
# Default:
# PidFile=/tmp/zabbix_server.pid

PidFile=/run/zabbix/zabbix_server.pid

### Option: SocketDir
# IPC socket directory.
# Directory to store IPC sockets used by internal Zabbix services.
#
# Mandatory: no
# Default:
# SocketDir=/tmp

SocketDir=/run/zabbix

### Option: DBHost
# Database host name.
# If set to localhost, socket is used for MySQL.
# If set to empty string, socket is used for PostgreSQL.
# If set to empty string, the Net Service Name connection method is used to connect to Oracle database; also see
# the TNS_ADMIN environment variable to specify the directory where the tnsnames.ora file is located.
#
# Mandatory: no
# Default:

#DBHost=localhost

### Option: DBName
# Database name.
# If the Net Service Name connection method is used to connect to Oracle database, specify the service name from
# the tnsnames.ora file or set to empty string; also see the TWO_TASK environment variable if DBName is set to
# empty string.
#
# Mandatory: yes
# Default:
# DBName=

DBName=zabbix

### Option: DBSchema
# Schema name. Used for PostgreSQL.
#
# Mandatory: no
# Default:
# DBSchema=

### Option: DBUser
# Database user.
#
# Mandatory: no
# Default:
# DBUser=

DBUser=zabbix

### Option: DBPassword
# Database password.
# Comment this line if no password is used.
#
# Mandatory: no
# Default:

DBPassword=Pass1w0rd

### Option: DBSocket
# Path to MySQL socket.
#
# Mandatory: no
# Default:
# DBSocket=

### Option: DBPort
# Database port when not using local socket.
# If the Net Service Name connection method is used to connect to Oracle database, the port number from the
# tnsnames.ora file will be used. The port number set here will be ignored.
#
# Mandatory: no
# Range: 1024-65535
# Default:
# DBPort=

### Option: AllowUnsupportedDBVersions
# Allow server to work with unsupported database versions.
# 0 - do not allow
# 1 - allow
#
# Mandatory: no
# Default:
# AllowUnsupportedDBVersions=0

### Option: HistoryStorageURL
# History storage HTTP[S] URL.
#
# Mandatory: no
# Default:
# HistoryStorageURL=

### Option: HistoryStorageTypes
# Comma separated list of value types to be sent to the history storage.
#
# Mandatory: no
# Default:
# HistoryStorageTypes=uint,dbl,str,log,text

### Option: HistoryStorageDateIndex
# Enable preprocessing of history values in history storage to store values in different indices based on date.
# 0 - disable
# 1 - enable
#
# Mandatory: no
# Default:
# HistoryStorageDateIndex=0

### Option: ExportDir
# Directory for real time export of events, history and trends in newline delimited JSON format.
# If set, enables real time export.
#
# Mandatory: no
# Default:
# ExportDir=

### Option: ExportFileSize
# Maximum size per export file in bytes.
# Only used for rotation if ExportDir is set.
#
# Mandatory: no
# Range: 1M-1G
# Default:
# ExportFileSize=1G

### Option: ExportType
# List of comma delimited types of real time export - allows to control export entities by their
# type (events, history, trends) individually.
# Valid only if ExportDir is set.
#
# Mandatory: no
# Default:
# ExportType=events,history,trends

############ ADVANCED PARAMETERS ################

### Option: StartPollers
# Number of pre-forked instances of pollers.
#
# Mandatory: no
# Range: 0-1000
# Default:

StartPollers=100

### Option: StartIPMIPollers
# Number of pre-forked instances of IPMI pollers.
# The IPMI manager process is automatically started when at least one IPMI poller is started.
#
# Mandatory: no
# Range: 0-1000
# Default:
# StartIPMIPollers=0

### Option: StartPreprocessors
# Number of pre-forked instances of preprocessing workers.
# The preprocessing manager process is automatically started when preprocessor worker is started.
#
# Mandatory: no
# Range: 1-1000
# Default:
StartPreprocessors=15

### Option: StartPollersUnreachable
# Number of pre-forked instances of pollers for unreachable hosts (including IPMI and Java).
# At least one poller for unreachable hosts must be running if regular, IPMI or Java pollers
# are started.
#
# Mandatory: no
# Range: 0-1000
# Default:
StartPollersUnreachable=50

### Option: StartHistoryPollers
# Number of pre-forked instances of history pollers.
# Only required for calculated checks.
# A database connection is required for each history poller instance.
#
# Mandatory: no
# Range: 0-1000
# Default:
# StartHistoryPollers=5

### Option: StartTrappers
# Number of pre-forked instances of trappers.
# Trappers accept incoming connections from Zabbix sender, active agents and active proxies.
# At least one trapper process must be running to display server availability and view queue
# in the frontend.
#
# Mandatory: no
# Range: 0-1000
# Default:
StartTrappers=10

### Option: StartPingers
# Number of pre-forked instances of ICMP pingers.
#
# Mandatory: no
# Range: 0-1000
# Default:

StartPingers=50

### Option: StartDiscoverers
# Number of pre-forked instances of discoverers.
#
# Mandatory: no
# Range: 0-250
# Default:
StartDiscoverers=15

### Option: StartHTTPPollers
# Number of pre-forked instances of HTTP pollers.
#
# Mandatory: no
# Range: 0-1000
# Default:
StartHTTPPollers=5

### Option: StartTimers
# Number of pre-forked instances of timers.
# Timers process maintenance periods.
# Only the first timer process handles host maintenance updates. Problem suppression updates are shared
# between all timers.
#
# Mandatory: no
# Range: 1-1000
# Default:
StartTimers=2

### Option: StartEscalators
# Number of pre-forked instances of escalators.
#
# Mandatory: no
# Range: 1-100
# Default:
StartEscalators=2

### Option: StartAlerters
# Number of pre-forked instances of alerters.
# Alerters send the notifications created by action operations.
#
# Mandatory: no
# Range: 1-100
# Default:
StartAlerters=5

### Option: JavaGateway
# IP address (or hostname) of Zabbix Java gateway.
# Only required if Java pollers are started.
#
# Mandatory: no
# Default:
# JavaGateway=

### Option: JavaGatewayPort
# Port that Zabbix Java gateway listens on.
#
# Mandatory: no
# Range: 1024-32767
# Default:
# JavaGatewayPort=10052

### Option: StartJavaPollers
# Number of pre-forked instances of Java pollers.
#
# Mandatory: no
# Range: 0-1000
# Default:
# StartJavaPollers=0

### Option: StartVMwareCollectors
# Number of pre-forked vmware collector instances.
#
# Mandatory: no
# Range: 0-250
# Default:
# StartVMwareCollectors=0

### Option: VMwareFrequency
# How often Zabbix will connect to VMware service to obtain a new data.
#
# Mandatory: no
# Range: 10-86400
# Default:
# VMwareFrequency=60

### Option: VMwarePerfFrequency
# How often Zabbix will connect to VMware service to obtain performance data.
#
# Mandatory: no
# Range: 10-86400
# Default:
# VMwarePerfFrequency=60

### Option: VMwareCacheSize
# Size of VMware cache, in bytes.
# Shared memory size for storing VMware data.
# Only used if VMware collectors are started.
#
# Mandatory: no
# Range: 256K-2G
# Default:
# VMwareCacheSize=8M

### Option: VMwareTimeout
# Specifies how many seconds vmware collector waits for response from VMware service.
#
# Mandatory: no
# Range: 1-300
# Default:
# VMwareTimeout=10

### Option: SNMPTrapperFile
# Temporary file used for passing data from SNMP trap daemon to the server.
# Must be the same as in zabbix_trap_receiver.pl or SNMPTT configuration file.
#
# Mandatory: no
# Default:
SNMPTrapperFile=/tmp/zabbix_traps.tmp

#SNMPTrapperFile=/var/log/snmptrap/snmptrap.log

### Option: StartSNMPTrapper
# If 1, SNMP trapper process is started.
#
# Mandatory: no
# Range: 0-1
# Default:
StartSNMPTrapper=1

### Option: ListenIP
# List of comma delimited IP addresses that the trapper should listen on.
# Trapper will listen on all network interfaces if this parameter is missing.
#
# Mandatory: no
# Default:
# ListenIP=0.0.0.0

### Option: HousekeepingFrequency
# How often Zabbix will perform housekeeping procedure (in hours).
# Housekeeping is removing outdated information from the database.
# To prevent Housekeeper from being overloaded, no more than 4 times HousekeepingFrequency
# hours of outdated information are deleted in one housekeeping cycle, for each item.
# To lower load on server startup housekeeping is postponed for 30 minutes after server start.
# With HousekeepingFrequency=0 the housekeeper can be only executed using the runtime control option.
# In this case the period of outdated information deleted in one housekeeping cycle is 4 times the
# period since the last housekeeping cycle, but not less than 4 hours and not greater than 4 days.
#
# Mandatory: no
# Range: 0-24
# Default:
# HousekeepingFrequency=1

### Option: MaxHousekeeperDelete
# The table "housekeeper" contains "tasks" for housekeeping procedure in the format:
# [housekeeperid], [tablename], [field], [value].
# No more than 'MaxHousekeeperDelete' rows (corresponding to [tablename], [field], [value])
# will be deleted per one task in one housekeeping cycle.
# If set to 0 then no limit is used at all. In this case you must know what you are doing!
#
# Mandatory: no
# Range: 0-1000000
# Default:
# MaxHousekeeperDelete=5000

### Option: CacheSize
# Size of configuration cache, in bytes.
# Shared memory size for storing host, item and trigger data.
#
# Mandatory: no
# Range: 128K-64G
# Default:
CacheSize=8G

### Option: CacheUpdateFrequency
# How often Zabbix will perform update of configuration cache, in seconds.
#
# Mandatory: no
# Range: 1-3600
# Default:
# CacheUpdateFrequency=60

### Option: StartDBSyncers
# Number of pre-forked instances of DB Syncers.
#
# Mandatory: no
# Range: 1-100
# Default:
# StartDBSyncers=4

### Option: HistoryCacheSize
# Size of history cache, in bytes.
# Shared memory size for storing history data.
#
# Mandatory: no
# Range: 128K-2G
# Default:
HistoryCacheSize=64M

### Option: HistoryIndexCacheSize
# Size of history index cache, in bytes.
# Shared memory size for indexing history cache.
#
# Mandatory: no
# Range: 128K-2G
# Default:
HistoryIndexCacheSize=32M

### Option: TrendCacheSize
# Size of trend write cache, in bytes.
# Shared memory size for storing trends data.
#
# Mandatory: no
# Range: 128K-2G
# Default:
TrendCacheSize=32M

### Option: TrendFunctionCacheSize
# Size of trend function cache, in bytes.
# Shared memory size for caching calculated trend function data.
#
# Mandatory: no
# Range: 128K-2G
# Default:
# TrendFunctionCacheSize=4M

### Option: ValueCacheSize
# Size of history value cache, in bytes.
# Shared memory size for caching item history data requests.
# Setting to 0 disables value cache.
#
# Mandatory: no
# Range: 0,128K-64G
# Default:
ValueCacheSize=8G

### Option: Timeout
# Specifies how long we wait for agent, SNMP device or external check (in seconds).
#
# Mandatory: no
# Range: 1-30
# Default:
# Timeout=3

Timeout=15

### Option: TrapperTimeout
# Specifies how many seconds trapper may spend processing new data.
#
# Mandatory: no
# Range: 1-300
# Default:
# TrapperTimeout=300

### Option: UnreachablePeriod
# After how many seconds of unreachability treat a host as unavailable.
#
# Mandatory: no
# Range: 1-3600
# Default:
# UnreachablePeriod=45

### Option: UnavailableDelay
# How often host is checked for availability during the unavailability period, in seconds.
#
# Mandatory: no
# Range: 1-3600
# Default:
# UnavailableDelay=60

### Option: UnreachableDelay
# How often host is checked for availability during the unreachability period, in seconds.
#
# Mandatory: no
# Range: 1-3600
# Default:
# UnreachableDelay=15

### Option: AlertScriptsPath
# Full path to location of custom alert scripts.
# Default depends on compilation options.
# To see the default path run command "zabbix_server --help".
#
# Mandatory: no
# Default:
# AlertScriptsPath=/usr/lib/zabbix/alertscripts

### Option: ExternalScripts
# Full path to location of external scripts.
# Default depends on compilation options.
# To see the default path run command "zabbix_server --help".
#
# Mandatory: no
# Default:
# ExternalScripts=/usr/lib/zabbix/externalscripts

### Option: FpingLocation
# Location of fping.
# Make sure that fping binary has root ownership and SUID flag set.
#
# Mandatory: no
# Default:
# FpingLocation=/usr/sbin/fping

### Option: Fping6Location
# Location of fping6.
# Make sure that fping6 binary has root ownership and SUID flag set.
# Make empty if your fping utility is capable to process IPv6 addresses.
#
# Mandatory: no
# Default:
# Fping6Location=/usr/sbin/fping6

### Option: SSHKeyLocation
# Location of public and private keys for SSH checks and actions.
#
# Mandatory: no
# Default:
# SSHKeyLocation=

### Option: LogSlowQueries
# How long a database query may take before being logged (in milliseconds).
# Only works if DebugLevel set to 3, 4 or 5.
# 0 - don't log slow queries.
#
# Mandatory: no
# Range: 1-3600000
# Default:
# LogSlowQueries=0

LogSlowQueries=3000

### Option: TmpDir
# Temporary directory.
#
# Mandatory: no
# Default:
# TmpDir=/tmp

### Option: StartProxyPollers
# Number of pre-forked instances of pollers for passive proxies.
#
# Mandatory: no
# Range: 0-250
# Default:
# StartProxyPollers=1

### Option: ProxyConfigFrequency
# How often Zabbix Server sends configuration data to a Zabbix Proxy in seconds.
# This parameter is used only for proxies in the passive mode.
#
# Mandatory: no
# Range: 1-3600*24*7
# Default:
# ProxyConfigFrequency=300

### Option: ProxyDataFrequency
# How often Zabbix Server requests history data from a Zabbix Proxy in seconds.
# This parameter is used only for proxies in the passive mode.
#
# Mandatory: no
# Range: 1-3600
# Default:
# ProxyDataFrequency=1

### Option: StartLLDProcessors
# Number of pre-forked instances of low level discovery processors.
#
# Mandatory: no
# Range: 1-100
# Default:
# StartLLDProcessors=2

### Option: AllowRoot
# Allow the server to run as 'root'. If disabled and the server is started by 'root', the server
# will try to switch to the user specified by the User configuration option instead.
# Has no effect if started under a regular user.
# 0 - do not allow
# 1 - allow
#
# Mandatory: no
# Default:
# AllowRoot=0

### Option: User
# Drop privileges to a specific, existing user on the system.
# Only has effect if run as 'root' and AllowRoot is disabled.
#
# Mandatory: no
# Default:
# User=zabbix

### Option: Include
# You may include individual files or all files in a directory in the configuration file.
# Installing Zabbix will create include directory in /usr/local/etc, unless modified during the compile time.
#
# Mandatory: no
# Default:
# Include=

# Include=/usr/local/etc/zabbix_server.general.conf
# Include=/usr/local/etc/zabbix_server.conf.d/
# Include=/usr/local/etc/zabbix_server.conf.d/*.conf

### Option: SSLCertLocation
# Location of SSL client certificates.
# This parameter is used only in web monitoring.
# Default depends on compilation options.
# To see the default path run command "zabbix_server --help".
#
# Mandatory: no
# Default:
# SSLCertLocation=${datadir}/zabbix/ssl/certs

### Option: SSLKeyLocation
# Location of private keys for SSL client certificates.
# This parameter is used only in web monitoring.
# Default depends on compilation options.
# To see the default path run command "zabbix_server --help".
#
# Mandatory: no
# Default:
# SSLKeyLocation=${datadir}/zabbix/ssl/keys

### Option: SSLCALocation
# Override the location of certificate authority (CA) files for SSL server certificate verification.
# If not set, system-wide directory will be used.
# This parameter is used in web monitoring, SMTP authentication, HTTP agent items and for communication with Vault.
#
# Mandatory: no
# Default:
# SSLCALocation=

### Option: StatsAllowedIP
# List of comma delimited IP addresses, optionally in CIDR notation, or DNS names of external Zabbix instances.
# Stats request will be accepted only from the addresses listed here. If this parameter is not set no stats requests
# will be accepted.
# If IPv6 support is enabled then '127.0.0.1', '::127.0.0.1', '::ffff:127.0.0.1' are treated equally
# and '::/0' will allow any IPv4 or IPv6 address.
# '0.0.0.0/0' can be used to allow any IPv4 address.
# Example: StatsAllowedIP=127.0.0.1,192.168.1.0/24,::1,2001:db8::/32,zabbix.example.com
#
# Mandatory: no
# Default:
# StatsAllowedIP=
StatsAllowedIP=127.0.0.1

####### LOADABLE MODULES #######

### Option: LoadModulePath
# Full path to location of server modules.
# Default depends on compilation options.
# To see the default path run command "zabbix_server --help".
#
# Mandatory: no
# Default:
# LoadModulePath=${libdir}/modules

### Option: LoadModule
# Module to load at server startup. Modules are used to extend functionality of the server.
# Formats:
# LoadModule=<module.so>
# LoadModule=<path/module.so>
# LoadModule=</abs_path/module.so>
# Either the module must be located in directory specified by LoadModulePath or the path must precede the module name.
# If the preceding path is absolute (starts with '/') then LoadModulePath is ignored.
# It is allowed to include multiple LoadModule parameters.
#
# Mandatory: no
# Default:
# LoadModule=

####### TLS-RELATED PARAMETERS #######

### Option: TLSCAFile
# Full pathname of a file containing the top-level CA(s) certificates for
# peer certificate verification.
#
# Mandatory: no
# Default:
# TLSCAFile=

### Option: TLSCRLFile
# Full pathname of a file containing revoked certificates.
#
# Mandatory: no
# Default:
# TLSCRLFile=

### Option: TLSCertFile
# Full pathname of a file containing the server certificate or certificate chain.
#
# Mandatory: no
# Default:
# TLSCertFile=

### Option: TLSKeyFile
# Full pathname of a file containing the server private key.
#
# Mandatory: no
# Default:
# TLSKeyFile=

####### For advanced users - TLS ciphersuite selection criteria #######

### Option: TLSCipherCert13
# Cipher string for OpenSSL 1.1.1 or newer in TLS 1.3.
# Override the default ciphersuite selection criteria for certificate-based encryption.
#
# Mandatory: no
# Default:
# TLSCipherCert13=

### Option: TLSCipherCert
# GnuTLS priority string or OpenSSL (TLS 1.2) cipher string.
# Override the default ciphersuite selection criteria for certificate-based encryption.
# Example for GnuTLS:
# NONE:+VERS-TLS1.2:+ECDHE-RSA:+RSA:+AES-128-GCM:+AES-128-CBC:+AEAD:+SHA256:+SHA1:+CURVE-ALL:+COMP-NULL:+SIGN-ALL:+CTYPE-X.509
# Example for OpenSSL:
# EECDH+aRSA+AES128:RSA+aRSA+AES128
#
# Mandatory: no
# Default:
# TLSCipherCert=

### Option: TLSCipherPSK13
# Cipher string for OpenSSL 1.1.1 or newer in TLS 1.3.
# Override the default ciphersuite selection criteria for PSK-based encryption.
# Example:
# TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA25 6
#
# Mandatory: no
# Default:
# TLSCipherPSK13=

### Option: TLSCipherPSK
# GnuTLS priority string or OpenSSL (TLS 1.2) cipher string.
# Override the default ciphersuite selection criteria for PSK-based encryption.
# Example for GnuTLS:
# NONE:+VERS-TLS1.2:+ECDHE-PSK:+PSK:+AES-128-GCM:+AES-128-CBC:+AEAD:+SHA256:+SHA1:+CURVE-ALL:+COMP-NULL:+SIGN-ALL
# Example for OpenSSL:
# kECDHEPSK+AES128:kPSK+AES128
#
# Mandatory: no
# Default:
# TLSCipherPSK=

### Option: TLSCipherAll13
# Cipher string for OpenSSL 1.1.1 or newer in TLS 1.3.
# Override the default ciphersuite selection criteria for certificate- and PSK-based encryption.
# Example:
# TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA25 6:TLS_AES_128_GCM_SHA256
#
# Mandatory: no
# Default:
# TLSCipherAll13=

### Option: TLSCipherAll
# GnuTLS priority string or OpenSSL (TLS 1.2) cipher string.
# Override the default ciphersuite selection criteria for certificate- and PSK-based encryption.
# Example for GnuTLS:
# NONE:+VERS-TLS1.2:+ECDHE-RSA:+RSA:+ECDHE-PSK:+PSK:+AES-128-GCM:+AES-128-CBC:+AEAD:+SHA256:+SHA1:+CURVE-ALL:+COMP-NULL:+SIGN-ALL:+CTYPE-X.509
# Example for OpenSSL:
# EECDH+aRSA+AES128:RSA+aRSA+AES128:kECDHEPSK+AES128 :kPSK+AES128
#
# Mandatory: no
# Default:
# TLSCipherAll=

### Option: DBTLSConnect
# Setting this option enforces to use TLS connection to database.
# required - connect using TLS
# verify_ca - connect using TLS and verify certificate
# verify_full - connect using TLS, verify certificate and verify that database identity specified by DBHost
# matches its certificate
# On MySQL starting from 5.7.11 and PostgreSQL following values are supported: "required", "verify_ca" and
# "verify_full".
# On MariaDB starting from version 10.2.6 "required" and "verify_full" values are supported.
# Default is not to set any option and behavior depends on database configuration
#
# Mandatory: no
# Default:
# DBTLSConnect=

### Option: DBTLSCAFile
# Full pathname of a file containing the top-level CA(s) certificates for database certificate verification.
# Supported only for MySQL and PostgreSQL
#
# Mandatory: no
# (yes, if DBTLSConnect set to one of: verify_ca, verify_full)
# Default:
# DBTLSCAFile=

### Option: DBTLSCertFile
# Full pathname of file containing Zabbix server certificate for authenticating to database.
# Supported only for MySQL and PostgreSQL
#
# Mandatory: no
# Default:
# DBTLSCertFile=

### Option: DBTLSKeyFile
# Full pathname of file containing the private key for authenticating to database.
# Supported only for MySQL and PostgreSQL
#
# Mandatory: no
# Default:
# DBTLSKeyFile=

### Option: DBTLSCipher
# The list of encryption ciphers that Zabbix server permits for TLS protocols up through TLSv1.2
# Supported only for MySQL
#
# Mandatory no
# Default:
# DBTLSCipher=

### Option: DBTLSCipher13
# The list of encryption ciphersuites that Zabbix server permits for TLSv1.3 protocol
# Supported only for MySQL, starting from version 8.0.16
#
# Mandatory no
# Default:
# DBTLSCipher13=

### Option: Vault
# Specifies vault:
# HashiCorp - HashiCorp KV Secrets Engine - Version 2
# CyberArk - CyberArk Central Credential Provider
#
# Mandatory: no
# Default:
# Vault=HashiCorp

### Option: VaultToken
# Vault authentication token that should have been generated exclusively for Zabbix server with read only permission
# to paths specified in Vault macros and read only permission to path specified in optional VaultDBPath
# configuration parameter.
# It is an error if VaultToken and VAULT_TOKEN environment variable are defined at the same time.
#
# Mandatory: no
# (yes, if Vault is explicitly set to HashiCorp)
# Default:
# VaultToken=

### Option: VaultURL
# Vault server HTTP[S] URL. System-wide CA certificates directory will be used if SSLCALocation is not specified.
#
# Mandatory: no
# Default:
# VaultURL=https://127.0.0.1:8200

### Option: VaultDBPath
# Vault path or query depending on the Vault from where credentials for database will be retrieved by keys.
# Keys used for HashiCorp are 'password' and 'username'.
# Example path:
# secret/zabbix/database
# Keys used for CyberArk are 'Content' and 'UserName'.
# Example query:
# AppID=zabbix_server&Query=Safe=passwordSafe;Object =zabbix_server_database
# This option can only be used if DBUser and DBPassword are not specified.
#
# Mandatory: no
# Default:
# VaultDBPath=

### Option: VaultTLSCertFile
# Name of the SSL certificate file used for client authentication. The certificate file must be in PEM1 format.
# If the certificate file contains also the private key, leave the SSL key file field empty. The directory
# containing this file is specified by configuration parameter SSLCertLocation.
#
# Mandatory: no
# Default:
# VaultTLSCertFile=

### Option: VaultTLSKeyFile
# Name of the SSL private key file used for client authentication. The private key file must be in PEM1 format.
# The directory containing this file is specified by configuration parameter SSLKeyLocation.
#
# Mandatory: no
# Default:
# VaultTLSKeyFile=

### Option: StartReportWriters
# Number of pre-forked report writer instances.
#
# Mandatory: no
# Range: 0-100
# Default:
StartReportWriters=3

### Option: WebServiceURL
# URL to Zabbix web service, used to perform web related tasks.
# Example: http://localhost:10053/report
#
# Mandatory: no
# Default:
WebServiceURL=127.0.0.1:10053/report

### Option: ServiceManagerSyncFrequency
# How often Zabbix will synchronize configuration of a service manager (in seconds).
#
# Mandatory: no
# Range: 1-3600
# Default:
# ServiceManagerSyncFrequency=60

### Option: ProblemHousekeepingFrequency
# How often Zabbix will delete problems for deleted triggers (in seconds).
#
# Mandatory: no
# Range: 1-3600
# Default:
# ProblemHousekeepingFrequency=60

## Option: StartODBCPollers
# Number of pre-forked ODBC poller instances.
#
# Mandatory: no
# Range: 0-1000
# Default:
# StartODBCPollers=1

####### For advanced users - TCP-related fine-tuning parameters #######

## Option: ListenBacklog
# The maximum number of pending connections in the queue. This parameter is passed to
# listen() function as argument 'backlog' (see "man listen").
#
# Mandatory: no
# Range: 0 - INT_MAX (depends on system, too large values may be silently truncated to implementation-specified maximum)
# Default: SOMAXCONN (hard-coded constant, depends on system)
# ListenBacklog=


####### High availability cluster parameters #######

## Option: HANodeName
# The high availability cluster node name.
# When empty, server is working in standalone mode; a node with empty name is registered with address for the frontend to connect to.
#
# Mandatory: no
# Default:
# HANodeName=

## Option: NodeAddress
# IP or hostname with optional port to specify how frontend should connect to the server.
# Format: <address>[:port]
#
# This option can be overridden by address specified in frontend configuration.
#
# Mandatory: no
# Default:
# NodeAddress=localhost:10051

[tim.mooney]

How frequently are the items being gathered? Did you do any network tuning, for example tuning for a system with a high # of outgoing connections? Is it possible you're running out of outgoing sockets while waiting for sockets to recycle?

Are you seeing any unexpected messages in the Zabbix server's zabbix_server.log, or in the kernel log ring buffer for the system (run 'dmesg -T')?

[automiketic]

Id say 70% of the items are gathered every 5 mins 15% every minute and 15% every hour or every 24 hours.

No network tuning has been done and it is worth mentioning this is running on VM with a NIC in bridged mode.

I am not sure how to check outgoing sockets and how saturated they are, can you point me in the right direction to check that?

The only messages I am seeing in the zabbix log are:
1966:20221129:094402.956 item "xxxxxxxxxxxx" became not supported: Cannot perform request: OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
1971:20221129:094411.962 error reason for "xxxxxxxx" changed: Cannot perform request: Operation timed out after 10000 milliseconds with 0 bytes received

These types of errors roll fairly consistently.

[Markku]

I have written down these related to Zabbix tuning (source is unknown but these make sense):

sysctl -w net.ipv4.tcp_max_syn_backlog=4096
sysctl -w net.ipv4.tcp_keepalive_time=600
sysctl -w net.ipv4.tcp_fin_timeout=30

Defaults can be something like this (on one of my servers):

$ cat /proc/sys/net/ipv4/tcp_max_syn_backlog
256
$ cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
$ cat /proc/sys/net/ipv4/tcp_fin_timeout
60

You may also want to scale the installation by installing a Zabbix proxy and offloading some of the hosts there.

Markku

[Markku]

ss -s

command can use used to show socket statistics.

Markku

[markfree]

You can also try "netstat -s" if your system supports it. It is very detailed, though.

[tim.mooney]

Id say 70% of the items are gathered every 5 mins 15% every minute and 15% every hour or every 24 hours.

Ok, that's a lower # of outgoing connections than I was expecting, but it's probably still worth checking your socket usage.

As far as socket utilization, do some web searching for something like "linux network socket utilization" or similar. An example I found that has a little info on the "ss" utility: https://www.networkworld.com/article...nnections.html

Most stuff is probably just going to look in /proc/net/sockstat, which is probably what I would do too, but you need to read the Linux kernel proc filesystem documentation to understand what each number means in that file.

Mainly I'm wondering how close you're getting to 65536 in all states that make them (currently) unusable.

The only messages I am seeing in the zabbix log are:
1966:20221129:094402.956 item "xxxxxxxxxxxx" became not supported: Cannot perform request: OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
1971:20221129:094411.962 error reason for "xxxxxxxx" changed: Cannot perform request: Operation timed out after 10000 milliseconds with 0 bytes received

These types of errors roll fairly consistently.

errno 104 is ECONNRESET, connection reset by peer. That the errors are happening frequently probably mean they are related to your overall problem.

As far as the tuning settings Markku posted, the first two are fine, but if the issue is outgoing socket churn, the first two changes are not going to help your issue. The third setting, however, is potentially very beneficial for socket reuse.

[automiketic]

I actually had already made the timeout change following your prior advice about tuning for higher outgoing connections. I've also just gone ahead and increased the timeout on the HTTP items to see if that helps as well though they are all primarily discovered item prototypes so will need to wait till the next discovery check for that to trickle down I assume. Attached image shows some more of the errors. I know that one says connection reset by peer by they often switch back and forth either timing out or saying connection reset or even socket error. As to the sockets used I'm only seeing 1361 if I am understanding that correctly but have attached a screen.

[cyber]

With SNMP, please verify that your credentials are correct... You can also try to turn off bulk requests and see, if it helps something...

[automiketic]

SNMP actually seems to be the only thing doesn't have any issues. I've also now run into an issue where a few of my ssh calls that typically run once a day a specified time dont work until i restart the zabbix-server service manually. I've increased logging to hopefully figure out why this happening, but any insight is greatly appreciated.

[tim.mooney]

As to the sockets used I'm only seeing 1361 if I am understanding that correctly

Socket usage might have been higher before the change to the 'tcp_fin_timeout', but you're nowhere near running out of sockets now, so that theory doesn't look like it's the problem.

You've provided good info from the zabbix_server.log, but did you ever look through the kernel logs in the kernel ring buffer? Although most of those messages should get written to a system log somewhere, I find that it's sometimes easier to spot problems by looking at the ring buffer via

dmesg -T

since everything in there is effectively a kernel message. Keep in mind that there's going to be lots of purely informational and "boot" style messages in there, so most of the messages aren't any cause for concern. Repeated stuff after the boot messages finish, especially stuff that looks like a warning or error, may be worth further investigation. If you can correlate times for kernel messages with times for errors in the zabbix_server.log, that would be a strong indicator that they're related.

[automiketic]

I did review the kernel logs on the zabbix server and client machine and didnt really see any errors what-so-ever besides the typical boot messages

[tim.mooney]

Thanks for confirming you checked the kernel log too.

With nothing being logged there, it becomes more difficult to debug. I'm not sure what I would look for next. Maybe try catch something with a network packet capture, to see if that gives any clues, but with the volume of packets you may have to capture to get an example of one of the failures, it might be like looking for a needle in a haystack.