Symptoms#

I ran apt update on my VPS and got a puzzling result: only one repository failed; every other repository worked fine.

Err:1 http://mirrors.tencentyun.com/ubuntu jammy InRelease
  Could not resolve 'mirrors.tencentyun.com'

Switching to a public mirror such as archive.ubuntu.com or mirrors.aliyun.com worked. Pinging other hosts worked too. In other words, the network connection was fine—the only thing that failed was DNS resolution for this private mirror’s domain.

Root Cause#

The machine was running mihomo with TUN mode enabled. In this setup, mihomo intercepted the machine’s DNS requests on port 53. The DNS servers listed in /etc/resolv.conf were not actually handling those queries; instead, they were sent to the upstream DNS servers configured in mihomo. Mine were all public resolvers, including DoH services and 1.1.1.1.

That was the problem: mirrors.tencentyun.com is a mirror domain intended for Tencent Cloud’s private network. It needs to be resolved through the DNS servers inside the VPC:

183.60.83.19 / 183.60.82.98

Asking 1.1.1.1 to resolve a domain that exists only on a private network was never going to work. The public resolver did not know about it, so the lookup failed.

First Wrong Turn: fake-ip-filter Did Not Help#

My first thought was: “If fake IPs are involved, shouldn’t excluding this domain fix it?” So I added it to fake-ip-filter:

dns:
  fake-ip-filter:
    - 'mirrors.tencentyun.com'

It did not help.

The reason is straightforward: fake-ip-filter only controls whether a domain receives a fake IP address. It determines what kind of address is returned, not which upstream DNS server performs the lookup. Whether the query goes to 1.1.1.1 or another resolver has nothing to do with this setting.

The Fix: Explicitly Select the Upstream with nameserver-policy#

What needed to change was who resolved this domain. I added a nameserver-policy entry to send its queries to the VPC DNS servers:

dns:
  nameserver-policy:
    '+.mirrors.tencentyun.com': ['183.60.83.19', '183.60.82.98']
    # Add other private-network domains here as needed.

After hot-reloading the configuration, I checked the result:

Terminal window
$ dig +short mirrors.tencentyun.com
169.254.0.3

169.254.0.3 was the fake IP that mihomo returned to the application. The actual DNS lookup had already been handled by the VPC DNS servers, and the TUN layer mapped the connection to the real address when it was established. The fake IP did not prevent the service from working.

I ran apt update again, and it worked immediately.

Lessons Learned#

  • DNS servers supplied by DHCP can effectively be bypassed when TUN DNS interception is enabled. In my setup, mihomo took over all DNS requests on port 53. Simply putting the VPC DNS servers in /etc/resolv.conf did not help.
  • Private-network domains need an explicit DNS resolution policy. Whether it is Tencent Cloud’s mirrors.tencentyun.com, a corporate intranet domain, or a private service, a domain known only to private DNS servers needs a nameserver-policy entry pointing to those servers. Otherwise, the query goes to a public resolver and fails.
  • Keep the roles of fake-ip-filter and nameserver-policy separate:
    • fake-ip-filter decides whether to return a real IP or a fake IP.
    • nameserver-policy decides which upstream DNS server resolves the domain.
    • When DNS resolution itself is failing, it is usually the latter that needs changing.

One misplaced setting cost me half a day of troubleshooting. Writing this down so my future self does not fall into the same trap again.