Zabbix packets' connection state marked as invalid

[Johan Ehnberg]

Zabbix 3.2 in a Ubuntu Xenial environment.

I started noticing the following messages generated by UFW on LXD "hypervisors". They show up only on hosts at the same site as our Zabbix instance:

[10283266.317242] [UFW BLOCK] IN=br1 OUT= MAC=0c:c4:7a:c2:f3:d6:00:16:3e:94:a2:4c:08:00 SRC=10.1.1.16 DST=10.1.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=10051 DPT=57438 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
[10283270.317519] [UFW BLOCK] IN=br1 OUT= MAC=0c:c4:7a:c2:f3:d6:00:16:3e:94:a2:4c:08:00 SRC=10.1.1.16 DST=10.1.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=10051 DPT=57438 WINDOW=28960 RES=0x00 ACK SYN URGP=0 
[10283294.319086] [UFW BLOCK] IN=br1 OUT= MAC=0c:c4:7a:c2:f3:d6:00:16:3e:94:a2:4c:08:00 SRC=10.1.1.16 DST=10.1.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=10051 DPT=57438 WINDOW=28960 RES=0x00 ACK SYN URGP=0

I tracked down the rules that UFW generates that log the messages:

-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
-A ufw-before-input -m conntrack --ctstate INVALID -j DROP
-A ufw-logging-deny -m conntrack --ctstate INVALID -m limit --limit 3/min --limit-burst 10 -j RETURN
-A ufw-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "

Looking at it with tcpdump reveals the following:

12:52:39.626614 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.1.16.zabbix-trapper > 10.1.0.2.57800: Flags [S.], cksum 0x1542 (incorrect -> 0x7ace), seq 2839701961, ack 3496694385, win 28960, options [mss 1460,sackOK,TS val 2570447683 ecr 2570447433,nop,wscale 7], length 0
12:52:39.626748 IP (tos 0x0, ttl 63, id 47162, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.0.2.57800 > 10.1.1.16.zabbix-trapper: Flags [S], cksum 0x107b (correct), seq 3496694384, win 29200, options [mss 1460,sackOK,TS val 2570447683 ecr 0,nop,wscale 7], length 0
12:52:39.626775 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.1.16.zabbix-trapper > 10.1.0.2.57800: Flags [S.], cksum 0x1542 (incorrect -> 0x7ace), seq 2839701961, ack 3496694385, win 28960, options [mss 1460,sackOK,TS val 2570447683 ecr 2570447433,nop,wscale 7], length 0
12:52:41.626598 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.1.16.zabbix-trapper > 10.1.0.2.57800: Flags [S.], cksum 0x1542 (incorrect -> 0x78da), seq 2839701961, ack 3496694385, win 28960, options [mss 1460,sackOK,TS val 2570448183 ecr 2570447433,nop,wscale 7], length 0
12:52:45.626614 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.1.16.zabbix-trapper > 10.1.0.2.57800: Flags [S.], cksum 0x1542 (incorrect -> 0x74f2), seq 2839701961, ack 3496694385, win 28960, options [mss 1460,sackOK,TS val 2570449183 ecr 2570447433,nop,wscale 7], length 0
12:52:53.626612 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.1.16.zabbix-trapper > 10.1.0.2.57800: Flags [S.], cksum 0x1542 (incorrect -> 0x6d22), seq 2839701961, ack 3496694385, win 28960, options [mss 1460,sackOK,TS val 2570451183 ecr 2570447433,nop,wscale 7], length 0
12:53:09.626575 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.1.16.zabbix-trapper > 10.1.0.2.57800: Flags [S.], cksum 0x1542 (incorrect -> 0x5d82), seq 2839701961, ack 3496694385, win 28960, options [mss 1460,sackOK,TS val 2570455183 ecr 2570447433,nop,wscale 7], length 0

UFW has limits that allow some packages through, and Zabbix works properly. The messages are a nuisance however, and since they are caused by UFW default generic rules, they cannot be changed by adding custom rules which run only afterwards. Also, changing the defaults of UFW would impact more than Zabbix.

Any thoughts on why this is happening?

Any ideas on how to work around it, short of disabling logging?