server and proxy - reading first byte from connection failed

[batchen_regev]

Hey,
im using zabbix server 4.0.4
proxy 4.0.8

i have defined TLS connection using a certificate:
proxy conf:
TLSConnect=cert
TLSAccept=cert
TLSCAFile=/etc/zabbix/keys/zabbix-ca.crt
TLSCertFile=/etc/zabbix/keys/zabbix-server.crt
TLSKeyFile=/etc/zabbix/keys/zabbix-server.key


agent_conf:
TLSConnect=cert
TLSAccept=cert
TLSCAFile=/etc/zabbix/keys/zabbix-ca.crt
TLSCertFile=/etc/zabbix/keys/zabbix-server.crt
TLSKeyFile=/etc/zabbix/keys/zabbix-server.key

at gui - >
Administration - > proxies -> proxy1 - > Encryption -> cert & no encrypt (cause its passive)
hosts - > proxy1 -> Encryption cert & cert

i get these errors in the server :
4707:20190721:125820.041 failed to accept an incoming connection: from proxy1: reading first byte from connection failed: [104] Connection reset by peer
cannot connect to proxy "proxy1": TCP successful, cannot establish TLS to [[proxy1]:10051]: SSL_connect() I/O error: [104] Connection reset by peer

and at proxy logs :
31308:20190721:060419.537 active check configuration update from [zabbixserver:10051] started to fail (TCP successful, cannot establish TLS to [[zabbixserver]:10051]: SSL_connect() I/O error: [104] Connection reset by peer)

telnet both ways works :
10050
10051

one way from proxy to server:
443

i dont get why 443 is not open to listen immidietly when i set TLS configuration at proxy setting.
i dont know if its related but i have another proxy same setting, 443 one way and works


thanks

[batchen_regev]

Ill add that the difference between proxy1 which have errors and proxy2 which works are that
proxy1 is not connected to intenet and cannot resolve anything and working proxy2 is connect.

i added the needed values in zabbix-server and proxy1 /etc/hosts but still nothing works regarding TLS.



edit:
server debug mode :
27925:20190721:153601.845 cannot connect to proxy "proxy1": TCP successful, cannot establish TLS to [[proxy1]:10051]: SSL_connect() I/O error: [104] Connection reset by peer
27925:20190721:153601.845 End of connect_to_proxy():NETWORK_ERROR
27925:20190721:153601.845 End of get_data_from_proxy():NETWORK_ERROR
27925:20190721:153601.845 End of proxy_get_data():NETWORK_ERROR
27925:20190721:153601.845 In DCrequeue_proxy() update_nextcheck:6
27925:20190721:153601.845 End of DCrequeue_proxy()
27925:20190721:153601.845 End of process_proxy()
27925:20190721:153601.845 In DCconfig_get_proxypoller_nextcheck()
27925:20190721:153601.845 End of DCconfig_get_proxypoller_nextcheck():1563712561
27925:20190721:153601.845 In process_proxy()
27925:20190721:153601.845 In DCconfig_get_proxypoller_hosts()

[ingus.vilnis]

https://www.zabbix.com/documentation/4.0/manual/encryption/troubleshooting/certificate_problems#crl_expired_or_expires_during_server_operation

Check out if the documentation gives you any hints.

And don't bother checking port 443 here as it is not used for communication between Zabbix elements, only ports 10050 and 10051 are used.

[batchen_regev]

https://www.zabbix.com/documentation...rver_operation
Check out if the documentation gives you any hints.

And don't bother checking port 443 here as it is not used for communication between Zabbix elements, only ports 10050 and 10051 are used.

Hey, thanks for the replay.
i checked 443 because the TLS works with it, doesn't it ?
and i have checked this page but i dont get this full error message with the explanation of what is the issue, i only get :
27925:20190721:153601.845 cannot connect to proxy "proxy1": TCP successful, cannot establish TLS to [[proxy1]:10051]: SSL_connect() I/O error: [104] Connection reset by peer

any more ideas?

[ingus.vilnis]

443 is used for HTTPS requests.

Zabbix even encrypted uses the same ports as unencrypted.

Is the proxy supposed to be active or passive? Can you check that you have set the right types in both zabbix_proxy.conf and in gui?

[andris]

You can also check with your network administrators what is between server and proxy in your network. There are cases when firewalls (or intrusion detection systems) do not like TLS traffic on unusual ports (e.g.10051). Also you can check does it work with PSK instead of certificates.

[batchen_regev]

443 is used for HTTPS requests.

Zabbix even encrypted uses the same ports as unencrypted.

Is the proxy supposed to be active or passive? Can you check that you have set the right types in both zabbix_proxy.conf and in gui?

Hey,
the proxy suppused to be passive and is set to passive under -> administration-> proxis

here is proxy conf :
ProxyMode=1
Server=zabbixserverip
Hostname=proxy-zabbix
LogFile=/var/log/zabbix/zabbix_proxy.log
LogFileSize=0
EnableRemoteCommands=1
PidFile=/var/run/zabbix/zabbix_proxy.pid
SocketDir=/var/run/zabbix
DBName=zabbix_proxy
DBUser=zabbix
DBPassword=zabbix
SNMPTrapperFile=/var/log/snmptrap/snmptrap.log
Timeout=4
ExternalScripts=/usr/lib/zabbix/externalscripts
LogSlowQueries=3000
TLSConnect=cert
TLSAccept=cert
TLSCAFile=/etc/zabbix/keys/zabbix-ca.crt
TLSCertFile=/etc/zabbix/keys/zabbix-server.crt
TLSKeyFile=/etc/zabbix/keys/zabbix-server.key





proxy agent:
PidFile=/var/run/zabbix/zabbix_agentd.pid
LogFile=/var/log/zabbix/zabbix_agentd.log
LogFileSize=0
Server=zabbixserverip
ServerActive=zabbixserverip
Hostname=proxy-zabbix
HostMetadataItem=system.uname
TLSConnect=cert
TLSAccept=cert
TLSCAFile=/etc/zabbix/keys/zabbix-ca.crt
TLSCertFile=/etc/zabbix/keys/zabbix-server.crt
TLSKeyFile=/etc/zabbix/keys/zabbix-server.key

please help.

[batchen_regev]

You can also check with your network administrators what is between server and proxy in your network. There are cases when firewalls (or intrusion detection systems) do not like TLS traffic on unusual ports (e.g.10051). Also you can check does it work with PSK instead of certificates.

We checked with network and we have the same open ports on both working proxy and problem proxy.
they both give the same network traffic logs.

[Markku]

the proxy suppused to be passive and is set to passive under -> administration-> proxis
here is proxy conf :
ProxyMode=1
Server=proxyip

Server= should have the IP address of the server that is allowed to connect to this passive proxy.

Markku

[batchen_regev]

Server= should have the IP address of the server that is allowed to connect to this passive proxy.

Markku

Sorry, my mistake i meant zabbixserverip.
i fixed that..

[Markku]

In the first message your errors show that the proxy seems to be in active mode, in the second message it is in passive mode, so you have changed the setup between those, let's assume that you have restarted the proxy to get the configuration changes running.

When you get the "connection reset by peer" message on the server, do you see anything in the logs in proxy?

In the first message you say that telnet to port 10051 works (but was it before or after your proxy configuration changes). That leads to say that I have never set up TLS with certificates in Zabbix, but have you validated the TLS configuration somehow, like is the other proxy working with the exactly same TLS configuration? If in doubt, try without TLS first. As mentioned earlier, TCP ports will be the same regardless of using TLS or not (= Zabbix server will connect to the passive proxy to TCP port 10051).

"Connection reset by peer" /usually/ means that the destination (= proxy) has denied the connection, as firewalls /usually/ just drop the offending connection attempts (and again, you said that telnet to port 10051 worked, even though next-gen firewalls can mislead telnet tests).

Markku

[andris]

You can try it with PSK - it is easier to get a working PSK-encryption than with certificates.

[batchen_regev]

In the first message your errors show that the proxy seems to be in active mode, in the second message it is in passive mode, so you have changed the setup between those, let's assume that you have restarted the proxy to get the configuration changes running.

When you get the "connection reset by peer" message on the server, do you see anything in the logs in proxy?

In the first message you say that telnet to port 10051 works (but was it before or after your proxy configuration changes). That leads to say that I have never set up TLS with certificates in Zabbix, but have you validated the TLS configuration somehow, like is the other proxy working with the exactly same TLS configuration? If in doubt, try without TLS first. As mentioned earlier, TCP ports will be the same regardless of using TLS or not (= Zabbix server will connect to the passive proxy to TCP port 10051).

"Connection reset by peer" /usually/ means that the destination (= proxy) has denied the connection, as firewalls /usually/ just drop the offending connection attempts (and again, you said that telnet to port 10051 worked, even though next-gen firewalls can mislead telnet tests).

Markku

Hey,
my proxy is in passive mode both in guy and in conf file.
port 10050 and 10051 was open before everything,

what else can i check?
in firewall working proxy and problem proxy shows the same packet transfer.
the only different between them is working proxy can nslookup / ping internet and the other cannot.
problem proxy is connected to internet but is blocked for ping and resolves..

thanks

[batchen_regev]

You can try it with PSK - it is easier to get a working PSK-encryption than with certificates.

they want cert, and also there is no reason why both machine same version that 1 works and 1 doesnt..

[Markku]

what else can i check?
in firewall working proxy and problem proxy shows the same packet transfer.
the only different between them is working proxy can nslookup / ping internet and the other cannot.
problem proxy is connected to internet but is blocked for ping and resolves..

thanks

What is shown in the proxy logs?

According to the firewall logs, did the not-working proxy try to access something during the Zabbix server connection attempts?

Markku