server and proxy - reading first byte from connection failed

[batchen_regev]

Added debug : from proxy log:
15551:20190728:104435.730 OpenSSL library (version OpenSSL 1.0.1e-fips 11 Feb 2013) initialized
15557:20190728:104435.730 proxy #15 started [unreachable poller #1]
15551:20190728:104435.731 zbx_tls_init_child() loaded CA certificate(s) from file "/etc/zabbix/keys/zabbix-ca.crt"
15551:20190728:104435.731 zbx_tls_init_child() loaded certificate(s) from file "/etc/zabbix/keys/zabbix-server.crt"
15551:20190728:104435.731 zbx_tls_init_child() loaded private key from file "/etc/zabbix/keys/zabbix-server.key"
15551:20190728:104435.731 zbx_tls_init_child() certificate ciphersuites: ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA AES128-GCM-SHA256 AES128-SHA256 AES128-SHA
15551:20190728:104435.731 zbx_tls_init_child() PSK ciphersuites: PSK-AES128-CBC-SHA
15551:20190728:104435.731 zbx_tls_init_child() certificate and PSK ciphersuites: ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA AES128-GCM-SHA256 AES128-SHA256 AES128-SHA PSK-AES128-CBC-SHA
15551:20190728:104435.731 End of zbx_tls_init_child()
15551:20190728:104435.731 __zbx_zbx_setproctitle() title:'task manager [connecting to the database]'
15551:20190728:104435.731 In DBconnect() flag:0
15562:20190728:104435.732 proxy #20 started [trapper #5]
15562:20190728:104435.732 In zbx_tls_init_child()
15554:20190728:104435.732 proxy #12 started [poller #3]
15562:20190728:104435.734 OpenSSL library (version OpenSSL 1.0.1e-fips 11 Feb 2013) initialized
15562:20190728:104435.734 zbx_tls_init_child() loaded CA certificate(s) from file "/etc/zabbix/keys/zabbix-ca.crt"
15562:20190728:104435.735 zbx_tls_init_child() loaded certificate(s) from file "/etc/zabbix/keys/zabbix-server.crt"
15562:20190728:104435.735 zbx_tls_init_child() loaded private key from file "/etc/zabbix/keys/zabbix-server.key"
15562:20190728:104435.735 zbx_tls_init_child() certificate ciphersuites: ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA AES128-GCM-SHA256 AES128-SHA256 AES128-SHA
15562:20190728:104435.735 zbx_tls_init_child() PSK ciphersuites: PSK-AES128-CBC-SHA
15562:20190728:104435.735 zbx_tls_init_child() certificate and PSK ciphersuites: ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA AES128-GCM-SHA256 AES128-SHA256 AES128-SHA PSK-AES128-CBC-SHA
15562:20190728:104435.735 End of zbx_tls_init_child()
15562:20190728:104435.735 __zbx_zbx_setproctitle() title:'trapper #5 [connecting to the database]'
15562:20190728:104435.735 In DBconnect() flag:0
15563:20190728:104435.735 proxy #21 started [icmp pinger #1]
15563:20190728:104435.735 __zbx_zbx_setproctitle() title:'icmp pinger #1 [getting values]'
15563:20190728:104435.735 In get_pinger_hosts()
15563:20190728:104435.735 In DCconfig_get_poller_items() poller_type:3
15563:20190728:104435.735 End of DCconfig_get_poller_items():0
15563:20190728:104435.736 End of get_pinger_hosts():0
15563:20190728:104435.736 In process_pinger_hosts()
15563:20190728:104435.736 End of process_pinger_hosts()
15563:20190728:104435.736 In DCconfig_get_poller_nextcheck() poller_type:3
15563:20190728:104435.736 End of DCconfig_get_poller_nextcheck():-1
15563:20190728:104435.736 __zbx_zbx_setproctitle() title:'icmp pinger #1 [got 0 values in 0.000277 sec, idle 5 sec]'
15551:20190728:104435.737 End of DBconnect():0
15551:20190728:104435.737 __zbx_zbx_setproctitle() title:'task manager [started, idle 5 sec]'
15558:20190728:104435.739 proxy #16 started [trapper #1]
15558:20190728:104435.739 In zbx_tls_init_child()
15558:20190728:104435.741 OpenSSL library (version OpenSSL 1.0.1e-fips 11 Feb 2013) initialized
15562:20190728:104435.741 End of DBconnect():0
15562:20190728:104435.741 __zbx_zbx_setproctitle() title:'trapper #5 [processed data in 0.000000 sec, waiting for connection]'
15559:20190728:104435.742 proxy #17 started [trapper #2]
15559:20190728:104435.742 In zbx_tls_init_child()
15558:20190728:104435.742 zbx_tls_init_child() loaded CA certificate(s) from file "/etc/zabbix/keys/zabbix-ca.crt"
15558:20190728:104435.742 zbx_tls_init_child() loaded certificate(s) from file "/etc/zabbix/keys/zabbix-server.crt"
15558:20190728:104435.742 zbx_tls_init_child() loaded private key from file "/etc/zabbix/keys/zabbix-server.key"
15558:20190728:104435.743 zbx_tls_init_child() certificate ciphersuites: ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA AES128-GCM-SHA256 AES128-SHA256 AES128-SHA
15558:20190728:104435.743 zbx_tls_init_child() PSK ciphersuites: PSK-AES128-CBC-SHA
15561:20190728:104435.743 proxy #19 started [trapper #4]
15561:20190728:104435.743 In zbx_tls_init_child()
15560:20190728:104435.743 proxy #18 started [trapper #3]
15560:20190728:104435.743 In zbx_tls_init_child()

[Markku]

they want cert, and also there is no reason why both machine same version that 1 works and 1 doesnt..

Have you crosschecked all the configuration details between the working and not-working proxy already? I mean all, like the permissions in the TLS-related files and everything? (Ok your response above appeared while I was writing so apparently proxy can read the files at least)

It is a good idea to try without TLS or with TLS-PSK just to narrow down the possibilities, and then return the cert configurations.

Markku

[Markku]

In the proxy debug log you provided there is no sign about an incoming connection. Are you sure your server is really trying to connect this proxy?

Also, the shown debug was not even one second in length, so you need to work more on that.

Markku

[batchen_regev]

they saw the same packages go throw to the zabbix server from working and not working proxy, and both have the same firewall rules.
here is some more info :

Working proxy tcptump on port 10051:
06:42:53.969649 IP 10.0.31.179.34926 > proxyus: Flags [R], seq 2829924188, win 0, length 0
06:42:54.835692 IP zabbix server > proxyus: Flags [S], seq 940561844, win 29200, options [mss 1388,sackOK,TS val 86381550 ecr 0,nop,wscale 7], length 0
06:42:54.835726 IP proxyus > zabbix server: Flags [S.], seq 507662809, ack 940561845, win 28960, options [mss 1460,sackOK,TS val 1213232989 ecr 86381550,nop,wscale 7], length 0
06:42:54.972073 IP zabbix server > proxyus: Flags [.], ack 1, win 229, options [nop,nop,TS val 86381687 ecr 1213232989], length 0
06:42:54.972503 IP zabbix server > proxyus: Flags [P.], seq 1:128, ack 1, win 229, options [nop,nop,TS val 86381687 ecr 1213232989], length 127
06:42:54.972520 IP proxyus > zabbix server: Flags [.], ack 128, win 227, options [nop,nop,TS val 1213233125 ecr 86381687], length 0
06:42:54.973898 IP proxyus > zabbix server: Flags [.], seq 1:2753, ack 128, win 227, options [nop,nop,TS val 1213233127 ecr 86381687], length 2752
06:42:54.973945 IP proxyus > zabbix server: Flags [P.], seq 2753:3178, ack 128, win 227, options [nop,nop,TS val 1213233127 ecr 86381687], length 425
06:42:55.111187 IP zabbix server > proxyus: Flags [.], ack 1377, win 251, options [nop,nop,TS val 86381826 ecr 1213233127], length 0
06:42:55.111216 IP zabbix server > proxyus: Flags [.], ack 1377, win 273, options [nop,nop,TS val 86381826 ecr 1213233127,nop,nop,sack 1 {2753:3178}], length 0
06:42:55.111298 IP zabbix server > proxyus: Flags [.], ack 3178, win 295, options [nop,nop,TS val 86381826 ecr 1213233127], length 0
06:42:55.115079 IP zabbix server > proxyus: Flags [.], seq 128:1504, ack 3178, win 295, options [nop,nop,TS val 86381829 ecr 1213233127], length 1376
06:42:55.115153 IP zabbix server > proxyus: Flags [.], seq 1504:2880, ack 3178, win 295, options [nop,nop,TS val 86381829 ecr 1213233127], length 1376
06:42:55.115173 IP proxyus > zabbix server: Flags [.], ack 2880, win 272, options [nop,nop,TS val 1213233268 ecr 86381829], length 0
06:42:55.115179 IP zabbix server > proxyus: Flags [P.], seq 2880:3244, ack 3178, win 295, options [nop,nop,TS val 86381829 ecr 1213233127], length 364
06:42:55.115820 IP proxyus > zabbix server: Flags [P.], seq 3178:3229, ack 3244, win 293, options [nop,nop,TS val 1213233269 ecr 86381829], length 51
06:42:55.252610 IP zabbix server > proxyus: Flags [P.], seq 3244:3318, ack 3229, win 295, options [nop,nop,TS val 86381967 ecr 1213233269], length 74
06:42:55.254439 IP proxyus > zabbix server: Flags [P.], seq 3229:3934, ack 3318, win 293, options [nop,nop,TS val 1213233407 ecr 86381967], length 705
06:42:55.391579 IP zabbix server > proxyus: Flags [P.], seq 3318:3390, ack 3934, win 317, options [nop,nop,TS val 86382106 ecr 1213233407], length 72
06:42:55.391611 IP zabbix server > proxyus: Flags [FP.], seq 3390:3421, ack 3934, win 317, options [nop,nop,TS val 86382106 ecr 1213233407], length 31
06:42:55.393510 IP proxyus > zabbix server: Flags [P.], seq 3934:3965, ack 3422, win 293, options [nop,nop,TS val 1213233546 ecr 86382106], length 31
06:42:55.393576 IP proxyus > zabbix server: Flags [F.], seq 3965, ack 3422, win 293, options [nop,nop,TS val 1213233546 ecr 86382106], length 0
06:42:55.393605 IP proxyus > zabbix server: Flags [R.], seq 3966, ack 3422, win 293, options [nop,nop,TS val 0 ecr 86382106], length 0
06:42:55.529947 IP zabbix server > proxyus: Flags [R], seq 940565266, win 0, length 0

problem proxy :
13:42:21.766371 IP zabbix server.34948 > problem_proxy: Flags [R.], seq 1, ack 1, win 229, length 0
13:42:22.324203 IP zabbix server.35104 > problem_proxy: Flags [S], seq 834353790, win 29200, options [mss 1460,sackOK,TS val 86355661 ecr 0,nop,wscale 7], length 0
13:42:22.324249 IP problem_proxy > zabbix server.35104: Flags [S.], seq 2789995399, ack 834353791, win 28960, options [mss 1460,sackOK,TS val 536278193 ecr 86355661,nop,wscale 7], length 0
13:42:22.324499 IP zabbix server.35104 > problem_proxy: Flags [.], ack 1, win 229, options [nop,nop,TS val 86355662 ecr 536278193], length 0
13:42:22.324715 IP zabbix server.35104 > problem_proxy: Flags [R.], seq 1, ack 1, win 229, length 0
13:42:23.326990 IP zabbix server.35534 > problem_proxy: Flags [S], seq 3182279670, win 29200, options [mss 1460,sackOK,TS val 86356664 ecr 0,nop,wscale 7], length 0
13:42:23.327026 IP problem_proxy > zabbix server.35534: Flags [S.], seq 3656895336, ack 3182279671, win 28960, options [mss 1460,sackOK,TS val 536279195 ecr 86356664,nop,wscale 7], length 0
13:42:23.327322 IP zabbix server.35534 > problem_proxy: Flags [.], ack 1, win 229, options [nop,nop,TS val 86356665 ecr 536279195], length 0
13:42:23.327500 IP zabbix server.35534 > problem_proxy: Flags [R.], seq 1, ack 1, win 229, length 0
13:42:24.964621 IP zabbix server.35754 > problem_proxy: Flags [S], seq 1570537035, win 29200, options [mss 1460,sackOK,TS val 86358302 ecr 0,nop,wscale 7], length 0
13:42:24.964663 IP problem_proxy > zabbix server.35754: Flags [S.], seq 1021176106, ack 1570537036, win 28960, options [mss 1460,sackOK,TS val 536280833 ecr 86358302,nop,wscale 7], length 0
13:42:24.964954 IP zabbix server.35754 > problem_proxy: Flags [.], ack 1, win 229, options [nop,nop,TS val 86358302 ecr 536280833], length 0
13:42:24.965162 IP zabbix server.35754 > problem_proxy: Flags [R.], seq 1, ack 1, win 229, length 0
13:42:25.446320 IP zabbix server.35912 > problem_proxy: Flags [S], seq 632257268, win 29200, options [mss 1460,sackOK,TS val 86358783 ecr 0,nop,wscale 7], length 0
13:42:25.446360 IP problem_proxy > zabbix server.35912: Flags [S.], seq 365587049, ack 632257269, win 28960, options [mss 1460,sackOK,TS val 536281315 ecr 86358783,nop,wscale 7], length 0
13:42:25.446660 IP zabbix server.35912 > problem_proxy: Flags [.], ack 1, win 229, options [nop,nop,TS val 86358784 ecr 536281315], length 0
13:42:25.446821 IP zabbix server.35912 > problem_proxy: Flags [R.], seq 1, ack 1, win 229, length 0
13:42:26.448137 IP zabbix server.36204 > problem_proxy: Flags [S], seq 1879487986, win 29200, options [mss 1460,sackOK,TS val 86359785 ecr 0,nop,wscale 7], length 0
13:42:26.448173 IP problem_proxy > zabbix server.36204: Flags [S.], seq 685994527, ack 1879487987, win 28960, options [mss 1460,sackOK,TS val 536282317 ecr 86359785,nop,wscale 7], length 0
13:42:26.448410 IP zabbix server.36204 > problem_proxy: Flags [.], ack 1, win 229, options [nop,nop,TS val 86359786 ecr 536282317], length 0
13:42:26.448533 IP zabbix server.36204 > problem_proxy: Flags [R.], seq 1, ack 1, win 229, length 0
13:42:27.007570 IP zabbix server.36220 > problem_proxy: Flags [S], seq 325175554, win 29200, options [mss 1460,sackOK,TS val 86360345 ecr 0,nop,wscale 7], length 0
13:42:27.007607 IP problem_proxy > zabbix server.36220: Flags [S.], seq 219143086, ack 325175555, win 28960, options [mss 1460,sackOK,TS val 536282876 ecr 86360345,nop,wscale 7], length 0
13:42:27.007850 IP zabbix server.36220 > problem_proxy: Flags [.], ack 1, win 229, options [nop,nop,TS val 86360345 ecr 536282876], length 0
13:42:27.008014 IP zabbix server.36220 > problem_proxy: Flags [R.], seq 1, ack 1, win 229, length 0
================================================== ================================================== ================================================== =
#Main diff i see in this tcpdump logs are :
zabbix server in problem proxy changes zabbix server.<36220> the last port ? or is it connection number ?
And also lots of R resets and no P push commands ( I dont really understand networking that much)

problem proxy logs :
#problem_proxy : firewall-cmd --state
not running

#problem_proxy :iptables -nL
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

#problem_proxy :netstat -ntplu
tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN 15926/zabbix_agentd
tcp 0 0 0.0.0.0:10051 0.0.0.0:* LISTEN 15944/zabbix_proxy
tcp6 0 0 :::10050 :::* LISTEN 15926/zabbix_agentd
tcp6 0 0 :::10051 :::* LISTEN 15944/zabbix_proxy

SELINUX=disabled

Proxyagent logs :
16762:20190728:140922.088 End of zbx_tls_connect():FAIL error:'SSL_connect() I/O error: [104] Connection reset by peer'
16762:20190728:140922.088 active check configuration update from [10.0.31.179:10051] started to fail (TCP successful, cannot establish TLS to [[10.0.31.179]:10051]: SSL_connect() I/O error: [104] Connection reset by peer)
16762:20190728:140922.088 End of refresh_active_checks():FAIL

Zabbixserver logs:
16599:20190728:140232.814 cannot connect to proxy_proxy: TCP successful, cannot establish TLS to [[problem_proxy:10051]: SSL_connect() I/O error: [104] Connection reset by peer
with debug :
20529:20190728:141545.046 zbx_send_response_ext() '{"response":"success","info":"processed: 1; failed: 0; total: 1; seconds spent: 0.000621"}'
20420:20190728:141545.062 In zbx_tls_connect(): issuer:"" subject:""
20420:20190728:141545.062 End of zbx_tls_connect():FAIL error:'SSL_connect() I/O error: [104] Connection reset by peer'
20420:20190728:141545.062 cannot connect to proxy "problem_proxy": TCP successful, cannot establish TLS to [[problem_proxy]:10051]: SSL_connect() I/O error: [104] Connection reset by peer
20420:20190728:141545.062 End of connect_to_proxy():NETWORK_ERROR
20420:20190728:141545.062 End of get_data_from_proxy():NETWORK_ERROR
20420:20190728:141545.062 End of proxy_get_data():NETWORK_ERROR
20420:20190728:141545.064 In zbx_tls_connect(): issuer:"" subject:""
20420:20190728:141545.065 End of zbx_tls_connect():FAIL error:'SSL_connect() I/O error: [104] Connection reset by peer'
20420:20190728:141545.065 cannot connect to proxy "problem_proxy": TCP successful, cannot establish TLS to [[problem_proxy]:10051]: SSL_connect() I/O error: [104] Connection reset by peer
20420:20190728:141545.065 End of connect_to_proxy():NETWORK_ERROR
20420:20190728:141545.065 End of get_data_from_proxy():NETWORK_ERROR
20420:20190728:141545.065 End of proxy_get_data():NETWORK_ERROR
20413:20190728:141545.075 End of zbx_process_trigger():FAIL flags:0
20413:20190728:141545.075 End of zbx_process_trigger():FAIL flags:0
20413:20190728:141545.099 End of zbx_process_trigger():FAIL flags:0
20413:20190728:141545.099 End of zbx_process_trigger():FAIL flags:0

Proxy logs :
16644:20190728:140507.340 zbx_tls_init_child() loaded CA certificate(s) from file "/etc/zabbix/keys/zabbix-ca.crt"
16641:20190728:140507.340 zbx_tls_init_child() loaded CA certificate(s) from file "/etc/zabbix/keys/zabbix-ca.crt"
16644:20190728:140507.340 zbx_tls_init_child() loaded certificate(s) from file "/etc/zabbix/keys/zabbix-server.crt"
16644:20190728:140507.340 zbx_tls_init_child() loaded private key from file "/etc/zabbix/keys/zabbix-server.key"
16642:20190728:140507.340 zbx_tls_init_child() loaded CA certificate(s) from file "/etc/zabbix/keys/zabbix-ca.crt"
16641:20190728:140507.340 zbx_tls_init_child() loaded certificate(s) from file "/etc/zabbix/keys/zabbix-server.crt"
16644:20190728:140507.340 zbx_tls_init_child() certificate ciphersuites: ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA AES128-GCM-SHA256 AES128-SHA256 AES128-SHA
16644:20190728:140507.340 zbx_tls_init_child() PSK ciphersuites: PSK-AES128-CBC-SHA
16641:20190728:140507.340 zbx_tls_init_child() loaded private key from file "/etc/zabbix/keys/zabbix-server.key"
16644:20190728:140507.340 zbx_tls_init_child() certificate and PSK ciphersuites: ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA AES128-GCM-SHA256 AES128-SHA256 AES128-SHA PSK-AES128-CBC-SHA
16644:20190728:140507.340 End of zbx_tls_init_child()
16629:20190728:140507.342 __zbx_zbx_setproctitle() title:'discoverer #1 [processed 0 rules in 0.000000 sec, performing discovery]'
16629:20190728:140507.342 query [txnlev:0] [select distinct r.druleid,r.iprange,r.name,c.dcheckid,r.proxy_host id,r.delay from drules r left join dchecks c on c.druleid=r.druleid and c.uniq=1 where r.status=0 and r.nextcheck<=1564311907 and mod(r.druleid,1)=0]
16641:20190728:140507.342 __zbx_zbx_setproctitle() title:'poller #1 [got 0 values in 0.000000 sec, getting values]'
16641:20190728:140507.342 In get_values()
16641:20190728:140507.342 In DCconfig_get_poller_items() poller_type:0
16641:20190728:140507.342 End of DCconfig_get_poller_items():0
16641:20190728:140507.342 In DCconfig_get_poller_nextcheck() poller_type:0
16641:20190728:140507.342 End of DCconfig_get_poller_nextcheck():-1
16641:20190728:140507.342 End of get_values():0
16641:20190728:140507.342 __zbx_zbx_setproctitle() title:'poller #1 [got 0 values in 0.000295 sec, idle 5 sec]'
16644:20190728:140507.342 __zbx_zbx_setproctitle() title:'poller #4 [got 0 values in 0.000000 sec, getting values]'
16645:20190728:140507.342 zbx_tls_init_child() loaded CA certificate(s) from file "/etc/zabbix/keys/zabbix-ca.crt"
16651:20190728:140507.342 End of DBconnect():0
16651:20190728:140507.343 __zbx_zbx_setproctitle() title:'trapper #5 [processed data in 0.000000 sec, waiting for connection]'
16629:20190728:140507.343 query [txnlev:0] [select count(*),min(nextcheck) from drules where status=0 and mod(druleid,1)=0]
16645:20190728:140507.343 zbx_tls_init_child() loaded certificate(s) from file "/etc/zabbix/keys/zabbix-server.crt"
16642:20190728:140507.343 __zbx_zbx_setproctitle() title:'poller #2 [got 0 values in 0.000000 sec, getting values]'
16644:20190728:140507.343 In get_values()
16645:20190728:140507.343 zbx_tls_init_child() loaded private key from file "/etc/zabbix/keys/zabbix-server.key"
16642:20190728:140507.343 In get_values()
16642:20190728:140507.343 In DCconfig_get_poller_items() poller_type:0
16642:20190728:140507.343 End of DCconfig_get_poller_items():0
16642:20190728:140507.343 In DCconfig_get_poller_nextcheck() poller_type:0
16642:20190728:140507.343 End of DCconfig_get_poller_nextcheck():-1
16642:20190728:140507.343 End of get_values():0
16642:20190728:140507.343 __zbx_zbx_setproctitle() title:'poller #2 [got 0 values in 0.001181 sec, idle 5 sec]'
16629:20190728:140507.343 get_minnextcheck(): no items to update
16645:20190728:140507.343 zbx_tls_init_child() certificate ciphersuites: ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA AES128-GCM-SHA256 AES128-SHA256 AES128-SHA
16644:20190728:140507.343 In DCconfig_get_poller_items() poller_type:0
16646:20190728:140507.343 In zbx_tls_init_child()
16629:20190728:140507.343 __zbx_zbx_setproctitle() title:'discoverer #1 [processed 0 rules in 0.001100 sec, idle 60 sec]'
16645:20190728:140507.343 zbx_tls_init_child() PSK ciphersuites: PSK-AES128-CBC-SHA
16646:20190728:140507.343 OpenSSL library (version OpenSSL 1.0.1e-fips 11 Feb 2013) initialized
16644:20190728:140507.343 End of DCconfig_get_poller_items():0
16644:20190728:140507.343 In DCconfig_get_poller_nextcheck() poller_type:0
16644:20190728:140507.344 End of DCconfig_get_poller_nextcheck():-1
16644:20190728:140507.344 End of get_values():0
16645:20190728:140507.344 zbx_tls_init_child() certificate and PSK ciphersuites: ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA AES128-GCM-SHA256 AES128-SHA256 AES128-SHA PSK-AES128-CBC-SHA
16645:20190728:140507.344 End of zbx_tls_init_child()
16645:20190728:140507.344 __zbx_zbx_setproctitle() title:'poller #5 [connecting to the database]'
16643:20190728:140507.344 In zbx_tls_init_child()
16644:20190728:140507.344 __zbx_zbx_setproctitle() title:'poller #4 [got 0 values in 0.001629 sec, idle 5 sec]'
16645:20190728:140507.344 In DBconnect() flag:0
16643:20190728:140507.344 OpenSSL library (version OpenSSL 1.0.1e-fips 11 Feb 2013) initialized
16646:20190728:140507.344 zbx_tls_init_child() loaded CA certificate(s) from file "/etc/zabbix/keys/zabbix-ca.crt"
16646:20190728:140507.345 In get_values()
16646:20190728:140507.345 In DCconfig_get_poller_items() poller_type:1
16646:20190728:140507.345 End of DCconfig_get_poller_items():0
16646:20190728:140507.345 In DCconfig_get_poller_nextcheck() poller_type:1
16646:20190728:140507.345 End of DCconfig_get_poller_nextcheck():-1
16646:20190728:140507.345 End of get_values():0
16646:20190728:140507.346 __zbx_zbx_setproctitle() title:'unreachable poller #1 [got 0 values in 0.000278 sec, idle 5 sec]'
16643:20190728:140507.346 End of DBconnect():0
16643:20190728:140507.346 __zbx_zbx_setproctitle() title:'poller #3 [got 0 values in 0.000000 sec, getting values]'
16643:20190728:140507.346 In get_values()
16643:20190728:140507.346 In DCconfig_get_poller_items() poller_type:0
16643:20190728:140507.346 End of DCconfig_get_poller_items():0
16643:20190728:140507.346 In DCconfig_get_poller_nextcheck() poller_type:0
16643:20190728:140507.346 End of DCconfig_get_poller_nextcheck():-1
16643:20190728:140507.346 End of get_values():0
16643:20190728:140507.346 __zbx_zbx_setproctitle() title:'poller #3 [got 0 values in 0.000271 sec, idle 5 sec]'
16631:20190728:140508.321 __zbx_zbx_setproctitle() title:'history syncer #1 [processed 0 values in 0.001091 sec, syncing history]'
16631:20190728:140508.321 In zbx_sync_history_cache() history_num:0
16631:20190728:140508.321 End of zbx_sync_history_cache()
16634:20190728:140508.321 __zbx_zbx_setproctitle() title:'history syncer #3 [processed 0 values in 0.000551 sec, syncing history]'
16634:20190728:140508.321 In zbx_sync_history_cache() history_num:0
16634:20190728:140508.321 End of zbx_sync_history_cache()
16631:20190728:140508.321 __zbx_zbx_setproctitle() title:'history syncer #1 [processed 0 values in 0.000218 sec, idle 1 sec]'
16634:20190728:140508.321 __zbx_zbx_setproctitle() title:'history syncer #3 [processed 0 values in 0.000316 sec, idle 1 sec]'
16638:20190728:140508.321 __zbx_zbx_setproctitle() title:'self-monitoring [processing data]'
16638:20190728:140508.321 In collect_selfmon_stats()
16632:20190728:140508.321 __zbx_zbx_setproctitle() title:'history syncer #2 [processed 0 values in 0.001438 sec, syncing history]'
16638:20190728:140508.322 End of collect_selfmon_stats()
16638:20190728:140508.322 __zbx_zbx_setproctitle() title:'self-monitoring [processed data in 0.000232 sec, idle 1 sec]'
16632:20190728:140508.322 In zbx_sync_history_cache() history_num:0
16632:20190728:140508.322 End of zbx_sync_history_cache()
16632:20190728:140508.322 __zbx_zbx_setproctitle() title:'history syncer #2 [processed 0 values in 0.000333 sec, idle 1 sec]'
16636:20190728:140508.322 __zbx_zbx_setproctitle() title:'history syncer #4 [processed 0 values in 0.001016 sec, syncing history]'
16636:20190728:140508.322 In zbx_sync_history_cache() history_num:0
16636:20190728:140508.322 End of zbx_sync_history_cache()
16636:20190728:140508.322 __zbx_zbx_setproctitle() title:'history syncer #4 [processed 0 values in 0.000120 sec, idle 1 sec]'
16649:20190728:140508.961 failed to accept an incoming connection: from zabbix_server: reading first byte from connection failed: [104] Connection reset by peer
16649:20190728:140508.961 __zbx_zbx_setproctitle() title:'trapper #3 [processed data in 0.000000 sec, waiting for connection]'
16647:20190728:140509.118 failed to accept an incoming connection: from zabbix_server: reading first byte from connection failed: [104] Connection reset by peer
16647:20190728:140509.118 __zbx_zbx_setproctitle() title:'trapper #1 [processed data in 0.000000 sec, waiting for connection]'
16631:20190728:140509.321 __zbx_zbx_setproctitle() title:'history syncer #1 [processed 0 values in 0.000218 sec, syncing history]'
16631:20190728:140509.322 In zbx_sync_history_cache() history_num:0
16631:20190728:140509.322 End of zbx_sync_history_cache()

problem proxy~]# egrep -v '^$|^#' /etc/zabbix/zabbix_proxy.conf
ProxyMode=1
Server=zabbixServerIp
Hostname=cyberarktest-zabbix
LogFile=/var/log/zabbix/zabbix_proxy.log
LogFileSize=0
DebugLevel=4
EnableRemoteCommands=1
PidFile=/var/run/zabbix/zabbix_proxy.pid
SocketDir=/var/run/zabbix
DBName=zabbix_proxy
DBUser=zabbix
DBPassword=zabbix
SNMPTrapperFile=/var/log/snmptrap/snmptrap.log
Timeout=4
ExternalScripts=/usr/lib/zabbix/externalscripts
LogSlowQueries=3000
TLSConnect=cert
TLSAccept=cert
TLSCAFile=/etc/zabbix/keys/zabbix-ca.crt
TLSCertFile=/etc/zabbix/keys/zabbix-server.crt
TLSKeyFile=/etc/zabbix/keys/zabbix-server.key

working proxy : ~]# egrep -v '^$|^#' /etc/zabbix/zabbix_proxy.conf
ProxyMode=1
Server=zabbixServerIp
Hostname=usntwpvzbxp01
LogFile=/var/log/zabbix/zabbix_proxy.log
LogFileSize=0
DebugLevel=3
EnableRemoteCommands=1
PidFile=/var/run/zabbix/zabbix_proxy.pid
SocketDir=/var/run/zabbix
DBName=zabbix_proxy
DBUser=zabbix
DBPassword=zabbix
SNMPTrapperFile=/var/log/snmptrap/snmptrap.log
Timeout=4
ExternalScripts=/usr/lib/zabbix/externalscripts
LogSlowQueries=3000
TLSConnect=cert
TLSAccept=cert
TLSCAFile=/etc/zabbix/keys/zabbix-ca.crt
TLSCertFile=/etc/zabbix/keys/zabbix-server.crt
TLSKeyFile=/etc/zabbix/keys/zabbix-server.key

[batchen_regev]

In the proxy debug log you provided there is no sign about an incoming connection. Are you sure your server is really trying to connect this proxy?

Also, the shown debug was not even one second in length, so you need to work more on that.

Markku

please look at my latest comment. thanks!

[batchen_regev]

even without TLS settings i get this errors it seems.

18232:20190728:145630.069 failed to accept an incoming connection: from 10.0.111.10: TLS handshake set result code to 1: file s3_srvr.c line 1435: error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher: TLS write fatal alert "handshake failure"
18229:20190728:145648.914 temporarily disabling Zabbix agent checks on host "problem proxy": host unavailable

[Markku]

Here is the key observation:

(one cycle only)
13:42:27.007570 IP zabbix server.36220 > problem_proxy: Flags [S], seq 325175554, win 29200, options [mss 1460,sackOK,TS val 86360345 ecr 0,nop,wscale 7], length 0
13:42:27.007607 IP problem_proxy > zabbix server.36220: Flags [S.], seq 219143086, ack 325175555, win 28960, options [mss 1460,sackOK,TS val 536282876 ecr 86360345,nop,wscale 7], length 0
13:42:27.007850 IP zabbix server.36220 > problem_proxy: Flags [.], ack 1, win 229, options [nop,nop,TS val 86360345 ecr 536282876], length 0
13:42:27.008014 IP zabbix server.36220 > problem_proxy: Flags [R.], seq 1, ack 1, win 229, length 0

This means that the TCP three-way handshake has been completed in rows 1-3. No data has yet been transferred, but the server decides to Reset the connection right away in row 4, without any practical delay.

So, the question basically is: What can be a reason for the Zabbix server to initiate the connection to the proxy but then immediately reset the connection? (Without sending or receiving a single byte of application-level data)

Thanks for testing with no TLS as well.

Markku

[Markku]

even without TLS settings i get this errors it seems.

18232:20190728:145630.069 failed to accept an incoming connection: from 10.0.111.10: TLS handshake set result code to 1: file s3_srvr.c line 1435: error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher: TLS write fatal alert "handshake failure"
18229:20190728:145648.914 temporarily disabling Zabbix agent checks on host "problem proxy": host unavailable

"Incoming connection"? Where is this log from?

I mean, this looks like the *agent* connection from proxy to server, not the *proxy* connection.

Markku

[batchen_regev]

"Incoming connection"? Where is this log from?

I mean, this looks like the *agent* connection from proxy to server, not the *proxy* connection.

Markku

Hey,
i checked again now. proxy works with no TLS
agent isnt because i havent changed the agentconf yet.

so now we see proxy does works with no TLS, what can we do ?

[batchen_regev]

Here is the key observation:

(one cycle only)
13:42:27.007570 IP zabbix server.36220 > problem_proxy: Flags [S], seq 325175554, win 29200, options [mss 1460,sackOK,TS val 86360345 ecr 0,nop,wscale 7], length 0
13:42:27.007607 IP problem_proxy > zabbix server.36220: Flags [S.], seq 219143086, ack 325175555, win 28960, options [mss 1460,sackOK,TS val 536282876 ecr 86360345,nop,wscale 7], length 0
13:42:27.007850 IP zabbix server.36220 > problem_proxy: Flags [.], ack 1, win 229, options [nop,nop,TS val 86360345 ecr 536282876], length 0
13:42:27.008014 IP zabbix server.36220 > problem_proxy: Flags [R.], seq 1, ack 1, win 229, length 0

This means that the TCP three-way handshake has been completed in rows 1-3. No data has yet been transferred, but the server decides to Reset the connection right away in row 4, without any practical delay.

So, the question basically is: What can be a reason for the Zabbix server to initiate the connection to the proxy but then immediately reset the connection? (Without sending or receiving a single byte of application-level data)

Thanks for testing with no TLS as well.

Markku

i wish i knew
im really lost in this

[batchen_regev]

Certificate in problem proxy are the same in working proxy and same in zabbix server:
drwxr-xr-x 2 zabbix zabbix 75 Jul 28 14:45 keys

-r-x------ 1 zabbix zabbix 2114 Jul 21 13:54 zabbix-ca.crt
-r-x------ 1 zabbix zabbix 1655 Jul 21 13:54 zabbix-proxy.crt
-r-x------ 1 zabbix zabbix 1675 Jul 21 13:55 zabbix-proxy.key

same permissions..

[batchen_regev]

https://www.zabbix.com/documentation...rver_operation
Check out if the documentation gives you any hints.

And don't bother checking port 443 here as it is not used for communication between Zabbix elements, only ports 10050 and 10051 are used.

can you please help ? please observe the logs i have sent here. we got to a point where proxy works with no TLS but have no idea why it doenst work with TLS.

[Markku]

From your Zabbix server logs:

20420:20190728:141545.064 In zbx_tls_connect(): issuer:"" subject:""
20420:20190728:141545.065 End of zbx_tls_connect():FAIL error:'SSL_connect() I/O error: [104] Connection reset by peer'
20420:20190728:141545.065 cannot connect to proxy "problem_proxy": TCP successful, cannot establish TLS to [[problem_proxy]:10051]: SSL_connect() I/O error: [104] Connection reset by peer

Can you check your Issuer and Subject configurations in the Encryption tab for the not-working proxy in Zabbix GUI?

(Again, I've not configured TLS certicates with Zabbix, just working with the data you've provided)

Markku

[batchen_regev]

From your Zabbix server logs:

20420:20190728:141545.064 In zbx_tls_connect(): issuer:"" subject:""
20420:20190728:141545.065 End of zbx_tls_connect():FAIL error:'SSL_connect() I/O error: [104] Connection reset by peer'
20420:20190728:141545.065 cannot connect to proxy "problem_proxy": TCP successful, cannot establish TLS to [[problem_proxy]:10051]: SSL_connect() I/O error: [104] Connection reset by peer

Can you check your Issuer and Subject configurations in the Encryption tab for the not-working proxy in Zabbix GUI?

(Again, I've not configured TLS certicates with Zabbix, just working with the data you've provided)

Markku

Hey, it is empty like the working proxy.
i understand that if i dont set the "TLSServerCertIssuer" and "TLSServerCertSubject" then i dont need to specify,
i also tried putting this issuer and subject yesterday using the manual https://www.zabbix.com/documentation...g_certificates

but it didnt help.

[batchen_regev]

are there keys :
"/etc/zabbix/keys/zabbix-proxy.crt"
"/etc/zabbix/keys/zabbix-ca.crt"
"/etc/zabbix/keys/zabbix-proxy.key"

supposed to be copy- paste from zabbix server keys ? because thats what i did.

server keys :
-r-x------ 1 zabbix zabbix 2114 Mar 11 14:14 zabbix-ca.crt - copied
-r-x------ 1 zabbix zabbix 3326 Mar 11 14:14 zabbix-ca.key
-r-x------ 1 zabbix zabbix 17 Mar 11 14:14 zabbix-ca.srl
-r-x------ 1 zabbix zabbix 1655 Mar 11 14:14 zabbix-server.crt - copied
-r-x------ 1 zabbix zabbix 1062 Mar 11 14:14 zabbix-server.csr
-r-x------ 1 zabbix zabbix 1675 Mar 11 14:14 zabbix-server.key -copied