That does seem to be a good solution, thank you for the recommendation!

Running an AS and obtaining a /48 through a sponsoring LIR seems to cost about $150/year in my country, but hardly anyone (except hosting providers and large companies) does this, since self-hosting, especially more complicated stuff, isn’t really that popular here; people are mostly uninformed that it even exists.

But route64 is apparently completely free (donations are welcome); they provide a /56 PA-like IPv6 block carved out of their PI, and they also handle BGP stuff, and I don’t even have to pay for any of this - overall, a great choice, considering my constraints. Anycast to multiple tunnels costs money (maybe that’s what you meant with 2 EUR/month?) but that’s pretty much lunch money so it would be OK with me.

At this point I pretty much think this may be the exact case - and it may work in reverse: “since ns2 has AAAA listed then it’s an IPv6-capable deployment and we are allowed to fall back to IPv4, but since ns1 lacks the AAAA record, then we won’t even consider falling back to ns1 if ns2 is down”.

Or maybe another scenario:

• let’s check ns2’s AAAA record
• oh it’s down! Let’s check ns2’s A record
• it’s down too! Let’s switch to ns1…
• … but at that time the DNS resolution timeout expires, since we took too long resolving the records; and on the second try, no one remembers that ns2 was down, repeating the cycle

Basically my internal DNS zonefile is configured like this (it should be easier to read than my explanation):

coredns_config:
  zoneFiles:
    - filename: int.pootis.network.zone
      domain: int.pootis.network
      contents: |
        $ORIGIN int.pootis.network.
        $TTL 300

        @       SOA     ns.int.pootis.network. admin.pootis.network. (
          2026082001
          1200
          300
          1209600
          300
        )

        @       NS      ns.int.pootis.network.
        ns      AAAA    {{ k8s_dns_ipv6 }}
        ns      A       {{ k8s_dns_ipv4 }}

        vault     CNAME   internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}.
        cinny     CNAME   internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}.
        grafana   CNAME   internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}.
        stalwart  CNAME   internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}.
        lemmy     CNAME   internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}.
        mail      CNAME   stalwart.stalwart.svc.{{ k8s_cluster_domain }}.
        webmail   CNAME   internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}.

        ; ACME challenges
        ; int.pootis.network + *.int.pootis.network
        _acme-challenge      CNAME 1ca12d2c-8034-4a09-a940-dc59bd4f38d8.acme-dns.pootis.network.
        ; mail.int.pootis.network
        _acme-challenge.mail CNAME d175b66d-909f-46ba-8572-636ebc235d4c.acme-dns.pootis.network.

I actually already have a local DNS resolver… well, sort of: most of my cluster’s internal services, such as Vaultwarden and other stuff, are available only over an in-cluster IKEv2 VPN.

What I mean is, I have to connect to the VPN first, which pushes a Configuration Payload to my IKEv2 client, and that payload contains a DNS server IP, and that DNS server (exposed by a Kubernetes Service) serves a custom “internal” zone (.int.pootis.network), and proxies everything else either to k8s coredns, or to upstream DNS forwarders.

Okay, that explanation may have been somewhat complicated (it sounded simpler in my head) but the end result is that I can resolve and reach stuff like lemmy.int.pootis.network (my Lemmy frontend) only while connected to the in-cluster VPN. It is completely hidden from external users (no CT log record, no external DNS record, and only proxyable by internal Traefik instance).

The issue only occurs with my externally-facing DNS, which is needed to, for example, to reach my Lemmy backend instance (lemmy.pootis.network) and the pictrs deployment. But yeah, internal DNS works perfectly for me and it easily survives a node failure.