Zabbix-agent fails to accept connections on GNUTLS hosts

[nprdev]

Ever since some updates, a few zabbix agents (still version 5.0.44) are failing to connect with ssl-related errors.
Trying to make Zabbix-agent and server communicate between two hosts, but the zabbix-agent host is using gnuTLS, which is much more particular than openssl;
I even tried setting the SAN properly, to no avail.

Anyone know more details in particular about 'what' could be wrong with the cert?

In the web interface, I see:

Zabbix agent on trap647.coyote.com.au is unreachable for 5 minutes

I've properly flipped around (inverted) the TLS Subject string, and changed EmailAddress to EMAIL as wanted by GNUtls. It used to work; It's configured like this on the client in zabbix_agentd.conf (any identifiable information has been changed in this example, it's a ficticious value but should get the point across):

TLSServerCertIssuer = [email protected],CN=zabbixca.acme.com,OU=Sysadmin,O=ACME,L=Melbourne,ST=Victoria,C=AU
TLSServerCertSubject = [email protected],CN=zabbix.coyote.com.au,OU=HQ,O=Wiley Enterprises,L=Strzelecki Desert,ST=NSW,C=AU

On the client:

/usr/sbin/zabbix_agentd --version | grep GnuTLS

Compiled with GnuTLS 3.7.1
Running with GnuTLS 3.7.1

tail -n 1 /var/log/zabbix-agent/zabbix_agentd.log

2479919:20241112:102954.051 failed to accept an incoming connection: certificate subject does not match for 88.88.88.88

On the server:

cat /etc/hosts | grep zabbix.coyote.com.au

88.88.88.88 zabbix.coyote.com.au

openssl x509 -text -in server.crt -noout

Certificate:
Data:
Version: 3 (0x2)
Serial Number:
3f:de:5e:21:a4:72:c4:0e:0e:17:6a:7d:89:24:8f:07:e1 :56:b1:ac
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = AU, ST = Victoria, L = Melbourne, O = ACME, OU = Sysadmin, CN = zabbixca.acme.com, emailAddress = [email protected]
Validity
Not Before: Nov 12 09:03:38 2024 GMT
Not After : Nov 12 09:03:38 2027 GMT
Subject: C = AU, ST = NSW, L = Strzelecki Desert, O = Wiley Enterprises, OU = HQ, CN = zabbix.coyote.com.au, emailAddress = [email protected]
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:ac:99:8f:1c:30:96:6f:56:b0:dc:a7:05:54:16:
17:50:c5:7d:5b:59:80:6d:6b:93:63:8c:36:38:f2:
ee:30:8b:89:9a:85:42:ce:58:62:4c:08:88:f5:15:
06:33:ef:48:68:08:8c:af:79:57:ec:1b:33:60:a7:
59:83:4c:ee:85:7f:48:c6:a3:0d:f0:1d:a0:2c:6e:
af:65:74:bf:00:24:89:c4:48:3d:62:39:14:72:32:
3b:5a:a0:df:ba:d9:8e:42:a5:a9:73:f4:81:f1:b2:
eb:c7:a6:86:2f:db:6d:e2:9c:00:75:78:c9:f3:a4:
e6:3d:99:9a:81:75:7a:ba:78:02:7d:86:c3:a5:93:
e4:0a:be:e5:2b:fb:e1:87:36:03:f1:36:c2:9f:67:
f6:f2:27:09:24:36:88:ff:4d:eb:03:d0:6c:90:a6:
e1:56:38:a4:2b:ba:9c:8d:4d:41:43:08:35:db:72:
eb:d7:b5:12:1f:d7:f6:b3:b3:05:40:a0:25:28:63:
00:0c:5c:3c:ba:37:45:13:03:b1:df:02:51:a4:d1:
da:d9:23:0c:b7:53:73:27:b7:cc:83:bb:50:4a:40:
68:9c:d4:55:38:af:d9:71:38:7f:b0:0e:ea:04:c1:
fc:1f:70:b3:c4:0a:b7:37:e5:ba:a6:5a:be:e1:b1:
48:f1
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
F4:7A:1A:3D:12:76:5D:DE:79:6A:59:6D:0E:14:CD:CD:45:EF:7C:21
X509v3 Authority Key Identifier:
keyid:9E:F2:28:9B:32:F5:18:34:25:94:E6:3E:0E:DC:00:BB:83:CA:AF:5F

X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Key Encipherment
X509v3 Extended Key Usage: critical
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Alternative Name:
DNS:zabbix.coyote.com.au, IP Address:88.88.88.88, IP Address:2008:8007:BEE9:BEE9::12A
Signature Algorithm: sha256WithRSAEncryption
4a:ab:cc:87:b3:55:f4:2c:0d:ae:18:12:7b:d0:ee:d3:07 :58:
c7:a5:5d:c7:cd:db:cd:1c:fe:60:2c:75:aa:22:81:8b:d5 :5f:
69:3b:a0:00:a3:53:77:d7:7b:77:46:8c:16:4e:6a:4b:67 :8c:
c6:4d:3c:6d:3e:48:04:36:cf:c2:89:95:34:17:50:21:60 :8b:
33:a7:ec:75:b0:62:12:dd:b9:6d:21:27:80:d2:49:dc:09 :80:
50:40:11:51:ea:84:03:03:7a:b9:59:23:6a:f9:07:28:7f :41:
6c:70:1f:ff:66:b2:31:6a:a3:e5:28:b1:eb:f2:e4:a2:f1 :6c:
fd:62:b2:51:93:c6:d0:58:72:ba:0a:42:51:af:b1:2b:ff :15:
f7:87:3e:c9:47:21:74:36:e8:0f:a3:e6:e5:34:30:68:ee :bd:
26:72:d8:15:02:de:fd:18:43:8a:02:0d:b2:f3:4d:30:8f :68:
02:e5:e1:2e:53:0a:78:4f:d1:33:9b:3b:8a:60:51:59:10 :ae:
70:22:50:e3:2e:cf:97:ab:22:45:2d:5b:57:b1:bc:99:b6 :a1:
06:b8:b1:69:48:d6:4c:5c:84:10:f8:02:0e:c5:7d:0f:dd :b7:
f4:86:89:a4:c3:e8:ae:fe:75:b4:5a:32:95:2a:90:a9:db :86:
de:87:9b:2d:d9:06:32:11:1a:92:0c:e2:cc:63:c1:c3:eb :ce:
1b:1e:18:37:86:79:28:93:c2:aa:15:91:e0:b3:58:b0:43 :48:
ab:4b:6e:15:98:6c:4f:03:c0:09:14:9d:dc:5a:b0:73:1f :a4:
54:1d:92:9d:21:a1:ed:56:78:01:19:7b:1d:a3:24:38:1b :3c:
d0:96:79:4c:54:bf:bc:aa:2e:ed:bf:6b:99:99:9f:2e:1c :60:
5d:2f:8c:a3:e0:a7:04:80:e8:d4:8e:5a:2f:28:fa:c2:14 :8b:
c4:82:76:77:66:61:c6:07:77:60:38:88:4b:0d:9e:99:18 :08:
6c:2f:b3:cf:ff:e9:8e:0e:f3:fa:00:1d:58:e8:df:f9:6e :c9:
b1:50:2d:1f:fd:36:25:e6:3b:2b:fe:e3:b7:18:4a:44:e3 :64:
f1:35:92:ad:33:d0:5d:41:12:c9:31:f7:24:0b:fd:fe:1e :97:
71:d3:e1:e3:61:bb:72:03:a4:b6:5b:18:38:a3:12:3b:79 :33:
56:e4:17:be:46:55:21:29:4f:d8:3e:4a:93:af:4f:e6:75 :a7:
cc:98:19:de:85:b7:5b:88:97:40:c4:7c:72:96:ed:02:70 :3d:
8d:65:e2:fa:f6:ee:2a:2f:a7:08:2d:3a:ad:99:ea:4f:14 :ff:
8a:90:7b:3e:80:80:09:ab

[nprdev]

I found and fixed the problem, you would not be able to spot it from the example (as much as I gave). That's due to the problem being an invisible typo in the actual subject data (which has been masked).

In some cases, the error message is rather unhelpful; 'subject does not match' is very generic. Other times though you do see this as well (when you restart the agent);

Active check configuration update from [zabbix.coyote.com.au:10051] started to fail (TCP succesful, cannot establish TLS to [[zabbix.coyote.com.au]:10051]: issuer: peer: "[email protected],CN=zabbixca.acme.c om,O U=Sysadmin,O=ACME,L=Melbourne,ST=Victoria,C=AU", required: "[email protected],CN=zabbixca.acme.c om,O U=Sysadmin,O=ACME,L=Melbourne,ST=Victoria,C=AU", subject: peer: "[email protected],CN=zabbix.coyote.com.au,OU=Headquarters,O=Wiley Enterprises,L=Strzelecki Desert,ST=New South Wales,C=AU", required: "[email protected],CN=zabbix.coyote.c om.a u,OU=Headqua)

The message gets cut off at exactly 600 characters. This hid the error (causing quite a lot of wasted time trying to find it).

Some notes:

1. subject means it can be anything, and everything is checked.
2. EmailAddress should be EMAIL when using GNUtls
3. Don't put spaces in there, they matter too.
4. The order of these values is important and checked (even though that isn't in the RFC at all).
5. The order should be in the exact reverse of what openSSL puts out.

All of this is a PITA to get right.