Get value from agent failed: cannot connect to [[220.226.186.190]:10051]: [4] Interru

[Heilig]

Interesting.
Could you share the error message here (Configuration --> Hosts --> put mouse over the red "Z" for a "problem" host)?

[lgalford]

Did you find a solution

Good morning

thanks for your interest in the case. the following message is the one that I get when I put the mouse over the Z in red.

Receive empty resonse from Zabbix Agent [10.xx.xx.xx]. Assuming that agent dropped connection because of access permisions.

the firewall is down on the server and I added the exception for the port 10051 for incomming, outgoing connections and for the tcp and udp port.

any idea..

[Heilig]

Add Zabbix server IP in the "Server" parameter in zabbix_agentd.conf and restart the agent processes (https://www.zabbix.com/documentation.../zabbix_agentd).

[lgalford]

Still with the problem

hi,

The Zabbix server parameter is configured, see my configuration file.

# This is a config file for the Zabbix agent daemon (Windows)
# To get more information about Zabbix, go to http://www.zabbix.com

############ GENERAL PARAMETERS #################

### Option: LogFile
# Name of log file.
# If not set, Windows Event Log is used.
#
# Mandatory: no
# Default:
# LogFile=

LogFile=c:\zabbix_agentd.log

### Option: LogFileSize
# Maximum size of log file in MB.
# 0 - disable automatic log rotation.
#
# Mandatory: no
# Range: 0-1024
# Default:
LogFileSize=1024

### Option: DebugLevel
# Specifies debug level
# 0 - no debug
# 1 - critical information
# 2 - error information
# 3 - warnings
# 4 - for debugging (produces lots of information)
#
# Mandatory: no
# Range: 0-4
# Default:
DebugLevel=3

### Option: SourceIP
# Source IP address for outgoing connections.
#
# Mandatory: no
# Default:
SourceIP=10.58.61.65

### Option: EnableRemoteCommands
# Whether remote commands from Zabbix server are allowed.
# 0 - not allowed
# 1 - allowed
#
# Mandatory: no
# Default:
# EnableRemoteCommands=0

### Option: LogRemoteCommands
# Enable logging of executed shell commands as warnings.
# 0 - disabled
# 1 - enabled
#
# Mandatory: no
# Default:
# LogRemoteCommands=0

##### Passive checks related

### Option: Server
# List of comma delimited IP addresses (or hostnames) of Zabbix servers.
# Incoming connections will be accepted only from the hosts listed here.
# If IPv6 support is enabled then '127.0.0.1', '::127.0.0.1', '::ffff:127.0.0.1' are treated equally.
#
# Mandatory: no
# Default:
# Server=

Server=10.0.0.55

### Option: ListenPort
# Agent will listen on this port for connections from the server.
#
# Mandatory: no
# Range: 1024-32767
# Default:
# ListenPort=10050

### Option: ListenIP
# List of comma delimited IP addresses that the agent should listen on.
# First IP address is sent to Zabbix server if connecting to it to retrieve list of active checks.
#
# Mandatory: no
# Default:
# ListenIP=0.0.0.0

### Option: StartAgents
# Number of pre-forked instances of zabbix_agentd that process passive checks.
# If set to 0, disables passive checks and the agent will not listen on any TCP port.
#
# Mandatory: no
# Range: 0-100
# Default:
# StartAgents=3

##### Active checks related

### Option: ServerActive
# List of comma delimited IPort (or hostnameort) pairs of Zabbix servers for active checks.
# If port is not specified, default port is used.
# IPv6 addresses must be enclosed in square brackets if port for that host is specified.
# If port is not specified, square brackets for IPv6 addresses are optional.
# If this parameter is not specified, active checks are disabled.
# Example: ServerActive=127.0.0.1:20051,zabbix.domain,[::1]:30051,::1,[12fc::1]
#
# Mandatory: no
# Default:
# ServerActive=

ServerActive=127.0.0.1

### Option: Hostname
# Unique, case sensitive hostname.
# Required for active checks and must match hostname as configured on the server.
# Value is acquired from HostnameItem if undefined.
#
# Mandatory: no
# Default:
# Hostname=

Hostname=Windows host

### Option: HostnameItem
# Item used for generating Hostname if it is undefined. Ignored if Hostname is defined.
# Does not support UserParameters or aliases.
#
# Mandatory: no
# Default:
# HostnameItem=system.hostname

### Option: HostMetadata
# Optional parameter that defines host metadata.
# Host metadata is used at host auto-registration process.
# An agent will issue an error and not start if the value is over limit of 255 characters.
# If not defined, value will be acquired from HostMetadataItem.
#
# Mandatory: no
# Range: 0-255 characters
# Default:
# HostMetadata=

### Option: HostMetadataItem
# Optional parameter that defines an item used for getting host metadata.
# Host metadata is used at host auto-registration process.
# During an auto-registration request an agent will log a warning message if
# the value returned by specified item is over limit of 255 characters.
# This option is only used when HostMetadata is not defined.
#
# Mandatory: no
# Default:
# HostMetadataItem=

### Option: RefreshActiveChecks
# How often list of active checks is refreshed, in seconds.
#
# Mandatory: no
# Range: 60-3600
# Default:
# RefreshActiveChecks=120

### Option: BufferSend
# Do not keep data longer than N seconds in buffer.
#
# Mandatory: no
# Range: 1-3600
# Default:
# BufferSend=5

### Option: BufferSize
# Maximum number of values in a memory buffer. The agent will send
# all collected data to Zabbix server or Proxy if the buffer is full.
#
# Mandatory: no
# Range: 2-65535
# Default:
# BufferSize=100

### Option: MaxLinesPerSecond
# Maximum number of new lines the agent will send per second to Zabbix Server
# or Proxy processing 'log', 'logrt' and 'eventlog' active checks.
# The provided value will be overridden by the parameter 'maxlines',
# provided in 'log', 'logrt' or 'eventlog' item keys.
#
# Mandatory: no
# Range: 1-1000
# Default:
# MaxLinesPerSecond=100

############ ADVANCED PARAMETERS #################

### Option: Alias
# Sets an alias for an item key. It can be used to substitute long and complex item key with a smaller and simpler one.
# Multiple Alias parameters may be present. Multiple parameters with the same Alias key are not allowed.
# Different Alias keys may reference the same item key.
# For example, to retrieve paging file usage in percents from the server:
# Alias=pg_usageerf_counter[\Paging File(_Total)\% Usage]
# Now shorthand key pg_usage may be used to retrieve data.
# Aliases can be used in HostMetadataItem but not in HostnameItem or PerfCounter parameters.
#
# Mandatory: no
# Range:
# Default:

### Option: Timeout
# Spend no more than Timeout seconds on processing
#
# Mandatory: no
# Range: 1-30
# Default:
# Timeout=3

### Option: PerfCounter
# Syntax: <parameter_name>,"<perf_counter_path>",<period>
# Defines new parameter <parameter_name> which is an average value for system performance counter <perf_counter_path> for the specified time period <period> (in seconds).
# For example, if you wish to receive average number of processor interrupts per second for last minute, you can define new parameter "interrupts" as following:
# PerfCounter = interrupts,"\Processor(0)\Interrupts/sec",60
# Please note double quotes around performance counter path.
# Samples for calculating average value will be taken every second.
# You may run "typeperf -qx" to get list of all performance counters available in Windows.
#
# Mandatory: no
# Range:
# Default:

### Option: Include
# You may include individual files in the configuration file.
#
# Mandatory: no
# Default:
# Include=

# Include=c:\zabbix\zabbix_agentd.userparams.conf

####### USER-DEFINED MONITORED PARAMETERS #######

### Option: UnsafeUserParameters
# Allow all characters to be passed in arguments to user-defined parameters.
# 0 - do not allow
# 1 - allow
#
# Mandatory: no
# Range: 0-1
# Default:
# UnsafeUserParameters=0

### Option: UserParameter
# User-defined parameter to monitor. There can be several user-defined parameters.
# Format: UserParameter=<key>,<shell command>
#
# Mandatory: no
# Default:
# UserParameter=
###############################################

Is there a way to change the default trapper port of the zabbix server...
lets say change it fron 10051 to 10049 just for testing...

I try to do that change it on the zabbix_server.conf and no luck is there another file where I should try to change it...

Waiting for you replay...

[Heilig]

It's a some misconfiguration.

You have configured next parameters:
Server=10.0.0.55
Is 10.0.0.55 Zabbix server IP?

ServerActive=127.0.0.1
Please note, active agent checks will not work with this settings.

Could you share Zabbix server configuration file it could help to find the cause of your issue?
=============================================
You must set "ListenPort=0049" in zabbix_server.conf and restart Zabbix server processes to apply changes.

[lgalford]

config file

yes the 10.0.0.55 is the zabbix server ip address.

in the Server Active what ip addres should go?

the zabbix server config file....

[root@srv-58000001055 ~]# more /etc/zabbix/zabbix_server.conf
# This is a configuration file for Zabbix Server process
# To get more information about Zabbix,
# visit http://www.zabbix.com

############ GENERAL PARAMETERS #################

### Option: NodeID
# Unique NodeID in distributed setup.
# 0 - standalone server
#
# Mandatory: no
# Range: 0-999
# Default:
# NodeID=0

### Option: ListenPort
# Listen port for trapper.
#
# Mandatory: no
# Range: 1024-32767
# Default:
ListenPort=10051

### Option: SourceIP
# Source IP address for outgoing connections.
#
# Mandatory: no
# Default:
# SourceIP=

### Option: LogFile
# Name of log file.
# If not set, syslog is used.
#
# Mandatory: no
# Default:
# LogFile=

LogFile=/var/log/zabbix/zabbix_server.log

### Option: LogFileSize
# Maximum size of log file in MB.
# 0 - disable automatic log rotation.
#
# Mandatory: no
# Range: 0-1024
# Default:
LogFileSize=1024

### Option: DebugLevel
# Specifies debug level
# 0 - no debug
# 1 - critical information
# 2 - error information
# 3 - warnings
# 4 - for debugging (produces lots of information)
#
# Mandatory: no
# Range: 0-4
# Default
DebugLevel=3

### Option: PidFile
# Name of PID file.
#
# Mandatory: no
# Default:
PidFile=/var/run/zabbix/zabbix_server.pid

### Option: DBHost
# Database host name.
# If set to localhost, socket is used for MySQL.
# If set to empty string, socket is used for PostgreSQL.
#
# Mandatory: no
# Default:
# DBHost=localhost

### Option: DBName
# Database name.
# For SQLite3 path to database file must be provided. DBUser and DBPassword are ignored.
#
# Mandatory: yes
# Default:
# DBName=

DBName=zabbix

### Option: DBSchema
# Schema name. Used for IBM DB2.
#
# Mandatory: no
# Default:
# DBSchema=

### Option: DBUser
# Database user. Ignored for SQLite.
#
# Mandatory: no
# Default:
# DBUser=

DBUser=zabbix

### Option: DBPassword
# Database password. Ignored for SQLite.
# Comment this line if no password is used.
#
# Mandatory: no
# Default:
DBPassword=XXXXXXXXX

### Option: DBSocket
# Path to MySQL socket.
#
# Mandatory: no
# Default:
DBSocket=/var/lib/mysql/mysql.sock

### Option: DBPort
# Database port when not using local socket. Ignored for SQLite.
#
# Mandatory: no
# Range: 1024-65535
# Default (for MySQL):
# DBPort=3306

############ ADVANCED PARAMETERS ################

### Option: StartPollers
# Number of pre-forked instances of pollers.
#
# Mandatory: no
# Range: 0-1000
# Default:
StartPollers=1

### Option: StartIPMIPollers
# Number of pre-forked instances of IPMI pollers.
#
# Mandatory: no
# Range: 0-1000
# Default:
# StartIPMIPollers=0

### Option: StartPollersUnreachable
# Number of pre-forked instances of pollers for unreachable hosts (including IPMI).
#
# Mandatory: no
# Range: 0-1000
# Default:
StartPollersUnreachable=500

### Option: StartTrappers
# Number of pre-forked instances of trappers.
# Trappers accept incoming connections from Zabbix sender, active agents, active proxies and child nodes.
# At least one trapper process must be running to display server availability in the frontend.
#
# Mandatory: no
# Range: 0-1000
# Default:
StartTrappers=100

### Option: StartPingers
# Number of pre-forked instances of ICMP pingers.
#
# Mandatory: no
# Range: 0-1000
# Default:
StartPingers=100

### Option: StartDiscoverers
# Number of pre-forked instances of discoverers.
#
# Mandatory: no
# Range: 0-250
# Default:
StartDiscoverers=1

### Option: StartHTTPPollers
# Number of pre-forked instances of HTTP pollers.
#
# Mandatory: no
# Range: 0-1000
# Default:
# StartHTTPPollers=1

### Option: JavaGateway
# IP address (or hostname) of Zabbix Java gateway.
# Only required if Java pollers are started.
#
# Mandatory: no
# Default:
# JavaGateway=

### Option: JavaGatewayPort
# Port that Zabbix Java gateway listens on.
#
# Mandatory: no
# Range: 1024-32767
# Default:
# JavaGatewayPort=10052

### Option: StartJavaPollers
# Number of pre-forked instances of Java pollers.
#
# Mandatory: no
# Range: 0-1000
# Default:
# StartJavaPollers=0

### Option: SNMPTrapperFile
# Temporary file used for passing data from SNMP trap daemon to the server.
# Must be the same as in zabbix_trap_receiver.pl or SNMPTT configuration file.
#
# Mandatory: no
# Default:
# SNMPTrapperFile=/tmp/zabbix_traps.tmp

### Option: StartSNMPTrapper
# If 1, SNMP trapper process is started.
#
# Mandatory: no
# Range: 0-1
# Default:
# StartSNMPTrapper=0

### Option: ListenIP
# List of comma delimited IP addresses that the trapper should listen on.
# Trapper will listen on all network interfaces if this parameter is missing.
#
# Mandatory: no
# Default:
# ListenIP=0.0.0.0

# ListenIP=127.0.0.1

### Option: HousekeepingFrequency
# How often Zabbix will perform housekeeping procedure (in hours).
# Housekeeping is removing unnecessary information from history, alert, and alarms tables.
#
# Mandatory: no
# Range: 1-24
# Default:
# HousekeepingFrequency=1

### Option: MaxHousekeeperDelete
# The table "housekeeper" contains "tasks" for housekeeping procedure in the format:
# [housekeeperid], [tablename], [field], [value].
# No more than 'MaxHousekeeperDelete' rows (corresponding to [tablename], [field], [value])
# will be deleted per one task in one housekeeping cycle.
# SQLite3 does not use this parameter, deletes all corresponding rows without a limit.
# If set to 0 then no limit is used at all. In this case you must know what you are doing!
#
# Mandatory: no
# Range: 0-1000000
# Default:
# MaxHousekeeperDelete=500

### Option: DisableHousekeeping
# If set to 1, disables housekeeping.
#
# Mandatory: no
# Range: 0-1
# Default:
# DisableHousekeeping=0

### Option: SenderFrequency
# How often Zabbix will try to send unsent alerts (in seconds).
#
# Mandatory: no
# Range: 5-3600
# Default:
# SenderFrequency=30

### Option: CacheSize
# Size of configuration cache, in bytes.
# Shared memory size for storing host, item and trigger data.
#
# Mandatory: no
# Range: 128K-2G
# Default:
# CacheSize=8M

### Option: CacheUpdateFrequency
# How often Zabbix will perform update of configuration cache, in seconds.
#
# Mandatory: no
# Range: 1-3600
# Default:
# CacheUpdateFrequency=60

### Option: StartDBSyncers
# Number of pre-forked instances of DB Syncers
#
# Mandatory: no
# Range: 1-100
# Default:
# StartDBSyncers=4

### Option: HistoryCacheSize
# Size of history cache, in bytes.
# Shared memory size for storing history data.
#
# Mandatory: no
# Range: 128K-2G
# Default:
# HistoryCacheSize=8M

### Option: TrendCacheSize
# Size of trend cache, in bytes.
# Shared memory size for storing trends data.
#
# Mandatory: no
# Range: 128K-2G
# Default:
# TrendCacheSize=4M

### Option: HistoryTextCacheSize
# Size of text history cache, in bytes.
# Shared memory size for storing character, text or log history data.
#
# Mandatory: no
# Range: 128K-2G
# Default:
# HistoryTextCacheSize=64M

### Option: NodeNoEvents
# If set to '1' local events won't be sent to master node.
# This won't impact ability of this node to propagate events from its child nodes.
#
# Mandatory: no
# Range: 0-1
# Default:
# NodeNoEvents=0

### Option: NodeNoHistory
# If set to '1' local history won't be sent to master node.
# This won't impact ability of this node to propagate history from its child nodes.
#
# Mandatory: no
# Range: 0-1
# Default:
# NodeNoHistory=0

### Option: Timeout
# Specifies how long we wait for agent, SNMP device or external check (in seconds).
#
# Mandatory: no
# Range: 1-30
# Default:
Timeout=2

### Option: TrapperTimeout
# Specifies how many seconds trapper may spend processing new data.
#
# Mandatory: no
# Range: 1-300
# Default:
# TrapperTimeout=300

### Option: UnreachablePeriod
# After how many seconds of unreachability treat a host as unavailable.
#
# Mandatory: no
# Range: 1-3600
# Default:
UnreachablePeriod=1

### Option: UnavailableDelay
# How often host is checked for availability during the unavailability period, in seconds.
#
# Mandatory: no
# Range: 1-3600
# Default:
# UnavailableDelay=30

### Option: UnreachableDelay
# How often host is checked for availability during the unreachability period, in seconds.
#
# Mandatory: no
# Range: 1-3600
# Default:
UnreachableDelay=1

### Option: AlertScriptsPath
# Full path to location of custom alert scripts.
# Default depends on compilation options.
#
# Mandatory: no
# Default:
AlertScriptsPath=/var/lib/zabbixsrv/alertscripts

### Option: ExternalScripts
# Full path to location of external scripts.
# Default depends on compilation options.
#
# Mandatory: no
# Default:
ExternalScripts=/etc/zabbix/externalscripts

### Option: FpingLocation
# Location of fping.
# Make sure that fping binary has root ownership and SUID flag set.
#
# Mandatory: no
# Default:
# FpingLocation=/usr/sbin/fping

### Option: Fping6Location
# Location of fping6.
# Make sure that fping6 binary has root ownership and SUID flag set.
# Make empty if your fping utility is capable to process IPv6 addresses.
#
# Mandatory: no
# Default:
# Fping6Location=/usr/sbin/fping6

### Option: SSHKeyLocation
# Location of public and private keys for SSH checks and actions
#
# Mandatory: no
# Default:
# SSHKeyLocation=

### Option: LogSlowQueries
# How long a database query may take before being logged (in milliseconds).
# Only works if DebugLevel set to 3 or 4.
# 0 - don't log slow queries.
#
# Mandatory: no
# Range: 1-3600000
# Default:
# LogSlowQueries=0

### Option: TmpDir
# Temporary directory.
#
# Mandatory: no
# Default:
TmpDir=/var/lib/zabbixsrv/tmp

### Option: Include
# You may include individual files or all files in a directory in the configuration file.
# Installing Zabbix will create include directory in /etc, unless modified during the compile time.
#
# Mandatory: no
# Default:
# Include=

# Include=/etc/zabbix_server.general.conf
# Include=/etc/zabbix_server.conf.d/

### Option: StartProxyPollers
# Number of pre-forked instances of pollers for passive proxies.
#
# Mandatory: no
# Range: 0-250
# Default:
# StartProxyPollers=1

### Option: ProxyConfigFrequency
# How often Zabbix Server sends configuration data to a Zabbix Proxy in seconds.
# This parameter is used only for proxies in the passive mode.
#
# Mandatory: no
# Range: 1-3600*24*7
# Default:
# ProxyConfigFrequency=3600

### Option: ProxyDataFrequency
# How often Zabbix Server requests history data from a Zabbix Proxy in seconds.
# This parameter is used only for proxies in the passive mode.
#
# Mandatory: no
# Range: 1-3600
# Default:
# ProxyDataFrequency=1

[lgalford]

Zabbix server 10051 can't be changed

hi,

I change the zabbix port to 0049 as you wrote and reboot the server and it did not work then i guess maybe a typo so I change it to 10049 and I see this message

See the attachment

regards
Luis

[lgalford]

The error is

the error that I see is:

Zabbix server is not running information may not be current

[yukra]

the error that I see is:

Zabbix server is not running information may not be current

as root: service zabbix-server start
and see /var/log/zabbix/zabbix-server.log

[lgalford]

zabbix server

The zabbix server log does not show any error.. but the messages still

Rgds
Luis Alford

[jlrd]

I'm seeing some similar, possibly related behavior. Almost entirely all of my checks are Zabbix Agent (Active) checks as opposed to the passive style. So my agents push their data from the checks to my proxies, as opposed to my proxies opening connections to my agents asking for updated data from the checks. Consequently all my monitoring activities are operating "normally".

However I often find that a handful of my clients, typically Windows agents running SQL Server, usually in a clustered and/or AlwaysOn configuration, often get that "Red Z" on the Availability column on the Zabbix frontend. The popup usually reports something like "cannot connect" or "connection refused". The agents are up happy and regularly sending their active data to proxies and onto the database. Some agents show these errors while others, configured identically as best I can tell, both Zabbix wise and OS/app wise, do not. There does not appear to be any rhyme or reason other than most often they appear on SQL server based hosts, but NOT ALL SQL based hosts. Other agents with the "Red Z" are almost exclusively firewall or network connectivity related errors.

As my monitoring is functioning, these "Red Z's" mostly offend my OCD nature, i've not invested significant time in digging in further. However I have found one odd commonality amongst my "Red Z" SQL Server agents. If you run "netstat -a" you will see that the Zabbix agent is binding to 10050 "twice". I'm not even sure how that is possible. I have confirmed multiple times that 1 and only 1 agent is installed/present on these systems. Also every tool I've used to inspect the port binding says that it is the same agentd exe that is creating both binds. Everywhere I have a "Red Z" and SQL I see the double bind. However not every SQL Server agentd I run shows this double bind. Many show a single bind and a green Z.

Odd, and frustrating to say the least but maybe what I have said/found can trigger something that will move this issue/conversation forward.

My Zabbix agentd conf is very generic and very, very standardized across my hosts. My "Red Z's" and my "Green Z's" have the same config and the same proxies. Additionally I see the issue in multiple data centers so multiple proxies as well, so not a single "bum" proxy.

My agentd config:

EnableRemoteCommands=0
LogRemoteCommands=0
Server=zabbixproxy,127.0.0.1
StartAgents=3
ServerActive=zabbixproxy
HostnameItem=system.hostname
Timeout=30

My proxy config:

Server=xxx.xxx.xxx.xxx
Hostname=zabbixproxy
LogFile=/var/log/zabbix/zabbix_proxy.log
LogFileSize=100
DBName=zabbix
DBUser=xxxxxxxxxx
ProxyLocalBuffer=24
ProxyOfflineBuffer=24
ConfigFrequency=300
StartPollers=20
StartIPMIPollers=1
StartPollersUnreachable=20
StartTrappers=20
StartPingers=15
JavaGateway=localhost
StartJavaPollers=5
Timeout=30
ExternalScripts=/etc/zabbix/externalscripts
Include=/usr/local/etc/zabbix_proxy.conf.d/

screen cap of the odd double bind:

[lgalford]

still not working

I have tested the server and it does not seem to be a firewall issue. I don't know what may be causing this problem i need to fix this.. please if anyone have an idea I would really apreciate

[lgalford]

Please Help

Any idea how can we fix this...

[Heilig]

i believe, this schema should help:

https://zabbix.org/wiki/Troubleshooting#Zabbix_says_.22Agent_unreachable.22.2C_but_the_host_is_up._How_do_I_debug_that.3F

[lgalford]

I finally resolved

Hi,

I figured out as I suppected from day one.. it was a networking issue. the link betewen sites was blocking the trafic. when i get more detail i will let you know.
thanks for all the people that share some thoughts and ideas thanks

rgds
Luis Alford