Ad Widget

Collapse

random-cased AAAA DNS queries

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • ddrucker
    Member
    • Feb 2019
    • 35
    #1

    random-cased AAAA DNS queries

    Zabbix 7.0.2 is sending hundred and hundreds of DNS AAAA record requests for my monitored hosts ... all with their HoSTnAMES RANdoMlY CAseD.

    Why?


    Code:
    → Q: PROXALone.mclEAn.harVArd.EDu.MClEAN.HaRvaRd.EDu IN AAAA
← S: NXDOMAIN
← A: MCLeaN.harvaRD.EdU IN SOA ecorepridns1.partners.org domainadmin.partners.org 2032176641 3600 3600 2592000 900

→ Q: MicVEeaM.MCLEaN.HaRVaRd.EDU.mclean.HarvARd.edU IN AAAA
← S: NXDOMAIN
← A: mCLEaN.harvarD.EDu IN SOA ecorepridns1.partners.org domainadmin.partners.org 2032176693 3600 3600 2592000 900

→ Q: x5BaCKup.MClEAN.haRvarD.EDu.McleAN.Harvard.eDU IN AAAA
← S: NXDOMAIN
← A: MClEAN.hARVarD.eDU IN SOA ecorepridns1.partners.org domainadmin.partners.org 2032176683 3600 3600 2592000 900

→ Q: iRIs.MCLEan.HARVARD.EDu.mCleAn.HarvARd.eDU IN AAAA
← S: NXDOMAIN
← A: McLean.haRvard.edu IN SOA ecorepridns1.partners.org domainadmin.partners.org 2032176703 3600 3600 2592000 900

→ Q: MicVNa.MClEaN.HArVARD.Edu.MClEan.haRVARD.EDu IN AAAA
← S: NXDOMAIN
← A: MClEAN.harvArD.Edu IN SOA ecorepridns1.partners.org domainadmin.partners.org 2032176693 3600 3600 2592000 900

→ Q: pROXMox01.mclEaN.hArVArD.edu.McLEAn.harVard.Edu IN AAAA
← S: NXDOMAIN
← A: MCLEAn.HarVaRd.edu IN SOA ecorepridns1.partners.org domainadmin.partners.org 2032176641 3600 3600 2592000 900

→ Q: DeLL-sCG.MclEan.hArvARd.eDu.McLean.hArvaRD.EDu IN AAAA
← S: NXDOMAIN
← A: MClEan.HArVaRd.eDU IN SOA ecorepridns1.partners.org domainadmin.partners.org 2032176663 3600 3600 2592000 900
    Tags: dns
    • Markku
      Senior Member
      Zabbix Certified SpecialistZabbix Certified ProfessionalZabbix Certified Expert
      • Sep 2018
      • 1781
      #2
      I see your post was eventually accepted even though it was surely looking like spam to the forum software

      This looks like being a feature of libevent that Zabbix 7.0 uses for async DNS resolution:

      Source of async_poller.c - Zabbix - ZABBIX GIT
      https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/src/libs/zbxpoller/async_poller.c#440


      https://libevent.org/doc/dns_8h.html...4574d85ae18cc8​

      http://www.wangafu.net/~nickm/libeve.../Ref9_dns.html​

      Markku

        Comment

        • ddrucker
          Member
          • Feb 2019
          • 35
          #3
          Is there any way to prevent IPv6 lookups so we're not spamming the upstream with dozens of requests per second? We don't use IPv6, so it's a cache miss every time.

            Comment

            • Markku
              Senior Member
              Zabbix Certified SpecialistZabbix Certified ProfessionalZabbix Certified Expert
              • Sep 2018
              • 1781
              #4
              For some reason systemd-resolved seems to not cache the NXDOMAIN responses. If I get it correctly, a normal recursive resolver (your upstream resolver for example) will cache that response, with TTL=900 in your example.

              Markku

                Comment

                • lumarel
                  Junior Member
                  • Mar 2022
                  • 2
                  #5
                  Hi there,

                  having the same "issue" with thousands of requests per minute.
                  Looks like also nscd does not cache these requests.

                  Also interesting is, the duplicated domain in the requests.

                  Cheers, lumarel

                    Comment

                    • HyperVize
                      Junior Member
                      • Sep 2024
                      • 1
                      #6
                      I also run into the same issue.

                      Code:
                      21:04:38.013379 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 49444+ AAAA? medIA01.hidden.intErNaL. (44)
21:04:38.013547 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 44040+ AAAA? Pve01.hidden.INTeRnaL.hidden.InteRNal. (61)
21:04:38.020215 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 28180+ AAAA? pvE01.hidden.inTERnAL.hidden.iNTERNal. (61)
21:04:38.020365 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 49504+ AAAA? PVE01.hidden.INtERNaL.hidden.inTernAL. (61)
21:04:38.020467 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 32173+ A? GuaRd01.hidden.iNTerNaL. (44)
21:04:38.020528 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 46333+ AAAA? GUARD01.hidden.INternal. (44)
21:04:38.020586 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 10855+ A? MEDia01.hidden.intErNal. (44)
21:04:38.020646 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 43336+ AAAA? Media01.hidden.INTeRNaL. (44)
21:04:38.020702 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 63535+ AAAA? PVE01.hidden.inTeRnAL.hidden.IntERNal. (61)
21:04:38.020827 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 55640+ AAAA? PvE01.hidden.InTErNal.hidden.InteRnAL. (61)
21:04:38.020923 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 8447+ AAAA? pve01.hidden.INtErNAL.hidden.IntERNal. (61)
21:04:38.021050 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 39844+ A? ruNDECk01.hidden.INTeRnAl. (46)
21:04:38.021115 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 45666+ AAAA? RUNDeCK01.hidden.INtERnAL. (46)
21:04:41.010003 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 42476+ A? pVe01.hidden.iNTERnal. (42)
21:04:41.010536 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 43469+ AAAA? mEDIa01.hidden.INtERnaL.hidden.INTERNAl. (63)
21:04:41.017604 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 9971+ AAAA? meDIA01.hidden.iNTeRnAl.hidden.INTErNAl. (63)
21:04:41.017716 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 23335+ AAAA? RuNdEcK01.hidden.inTeRNAL.hidden.iNtErnAL. (65)
21:04:41.017739 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 42347+ AAAA? gUard01.hidden.INterNal.hidden.IntErNaL. (63)
21:04:41.017760 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 20157+ A? guARd01.hidden.INTERNAl. (44)
21:04:41.017782 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 8073+ AAAA? GUaRd01.hidden.INTErNAL. (44)
21:04:41.019188 IP zabbix01.hidden.internal.38396 > 10.8.0.1.domain: 21367+ AAAA? guArD01.hidden.INtErNAL.hidden.IntERnal. (63)
21:04:41.161374 IP zabbix01.hidden.internal.40032 > 10.8.0.1.domain: 60769+ A? pve02.hidden.internal. (42)
                      I often lose connection to my servers due to this behaviour.

                      I would also like to add that this DNS lookups are happening extreme frequent resulting in thousands (!) requests per minute.

                        Comment

                        • siempie
                          Junior Member
                          • Sep 2024
                          • 1
                          #7
                          I see the same on my environment. Im running 7.0.3 on Debian12.
                          It is not as much as HyperVize experiences but definitely strange.

                          What is also strange is the extra search domain.

                          Code:
                          10:54:18.432588 IP 10.0.10.93.43416 > 10.0.10.254.53: 17661+ A? PVE0.example.com. (36)
10:54:18.432628 IP 10.0.10.93.43416 > 10.0.10.254.53: 33382+ AAAA? PVE0.example.com. (36)
10:54:18.443763 IP 10.0.10.93.43416 > 10.0.10.254.53: 40148+ AAAA? PvE0.example.com.example.InTernAl. (53)
10:54:19.433195 IP 10.0.10.93.43416 > 10.0.10.254.53: 48476+ A? pvE2.example.com. (36)
10:54:19.433220 IP 10.0.10.93.43416 > 10.0.10.254.53: 35532+ AAAA? PVe2.example.com. (36)
10:54:19.433237 IP 10.0.10.93.43416 > 10.0.10.254.53: 43224+ A? pVE2.example.com. (36)
10:54:19.433249 IP 10.0.10.93.43416 > 10.0.10.254.53: 35715+ AAAA? pVE2.example.com. (36)
10:54:19.433263 IP 10.0.10.93.43416 > 10.0.10.254.53: 33733+ A? PvE0.example.com. (36)
10:54:19.433274 IP 10.0.10.93.43416 > 10.0.10.254.53: 44824+ AAAA? PvE0.example.com. (36)
10:54:19.444375 IP 10.0.10.93.43416 > 10.0.10.254.53: 52496+ AAAA? pvE2.example.com.example.INterNaL. (53)
10:54:19.446308 IP 10.0.10.93.43416 > 10.0.10.254.53: 59604+ AAAA? pVE2.example.com.example.iNTERnal. (53)
10:54:19.446988 IP 10.0.10.93.43416 > 10.0.10.254.53: 28308+ AAAA? pVE0.example.com.example.interNaL. (53)
10:54:20.432632 IP 10.0.10.93.43416 > 10.0.10.254.53: 35283+ A? PvE2.example.com. (36)
10:54:20.432662 IP 10.0.10.93.43416 > 10.0.10.254.53: 29833+ AAAA? PVe2.example.com. (36)
10:54:20.432688 IP 10.0.10.93.43416 > 10.0.10.254.53: 22837+ A? PVe1.example.com. (36)
10:54:20.432700 IP 10.0.10.93.43416 > 10.0.10.254.53: 38772+ AAAA? pve1.example.com. (36)
10:54:20.432720 IP 10.0.10.93.43416 > 10.0.10.254.53: 25340+ A? Pve2.example.com. (36)
10:54:20.432731 IP 10.0.10.93.43416 > 10.0.10.254.53: 20411+ AAAA? PVe2.example.com. (36)
10:54:20.444327 IP 10.0.10.93.43416 > 10.0.10.254.53: 3916+ AAAA? pVE2.example.com.example.intErnaL. (53)
10:54:20.444901 IP 10.0.10.93.43416 > 10.0.10.254.53: 19320+ AAAA? pVE1.example.com.example.InTERnal. (53)
10:54:20.448082 IP 10.0.10.93.43416 > 10.0.10.254.53: 55794+ AAAA? pve2.example.com.example.iNTeRNAl. (53)
10:54:21.432585 IP 10.0.10.93.43416 > 10.0.10.254.53: 20868+ A? pve2.example.com. (36)
10:54:21.432634 IP 10.0.10.93.43416 > 10.0.10.254.53: 9204+ AAAA? pvE2.example.com. (36)
10:54:21.432655 IP 10.0.10.93.43416 > 10.0.10.254.53: 62672+ A? PvE2.example.com. (36)
10:54:21.432668 IP 10.0.10.93.43416 > 10.0.10.254.53: 7858+ AAAA? PVe2.example.com. (36)
10:54:21.432688 IP 10.0.10.93.43416 > 10.0.10.254.53: 32485+ A? PvE1.example.com. (36)
10:54:21.432699 IP 10.0.10.93.43416 > 10.0.10.254.53: 2148+ AAAA? PvE1.example.com. (36)
10:54:21.444256 IP 10.0.10.93.43416 > 10.0.10.254.53: 54479+ AAAA? PVE2.example.com.example.iNTernaL. (53)
10:54:21.444763 IP 10.0.10.93.43416 > 10.0.10.254.53: 14606+ AAAA? pVe2.example.com.example.IntERNAl. (53)
10:54:21.447818 IP 10.0.10.93.43416 > 10.0.10.254.53: 23818+ AAAA? PvE1.example.com.example.InTernAL. (53)​

                            Comment

                            • lumarel
                              Junior Member
                              • Mar 2022
                              • 2
                              #8
                              The solution/workaround right now seems to be adding a resolver inbetween for the system, added a PowerDNS recursor here now, which also caches the NXDOMAIN requests well enough.

                                Comment

                                Previous template Next
                                Working...