DNS resolution failures are one of those problems that are easy to diagnose but surprisingly disruptive while you wait for someone else to fix them. The server is up, your application is running, but outbound requests start failing because the DNS server your hosting provider assigned can’t resolve certain domains.
That’s exactly what happened here. A production server began returning SERVFAIL errors when trying to resolve certain domains through the hosting provider’s default nameserver. A quick dig confirmed the problem was isolated to that resolver — querying Google’s public DNS at 8.8.8.8 returned the correct records immediately.
$ dig MX ky.gov @10.0.80.11
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17255
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
$ dig MX ky.gov @8.8.8.8
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22479
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; ANSWER SECTION:
ky.gov. 21600 IN MX 10 mxa-00377201.gslb.pphosted.com.
ky.gov. 21600 IN MX 10 mxb-00377201.gslb.pphosted.com.
A support ticket was filed, but waiting on the hosting provider wasn’t an option. Fortunately, Ubuntu makes it straightforward to override the resolver without touching /etc/resolv.conf directly.
How DNS Resolution Works on Ubuntu
Modern Ubuntu servers use systemd-resolved to manage DNS resolution. Rather than editing /etc/resolv.conf directly — which systemd-resolved owns and will overwrite — the right approach is to drop a configuration file into /etc/systemd/resolved.conf.d/. Files in that directory are merged into the resolver’s configuration at startup, letting you override defaults cleanly without editing the main config.
Step 1: Create the Configuration File
Create a new configuration file in the drop-in directory. The filename can be anything ending in .conf:
sudo vim /etc/systemd/resolved.conf.d/custom-dns.conf
Add the following contents:
[Resolve]
DNS=8.8.8.8 1.1.1.1
FallbackDNS=9.9.9.9
This tells systemd-resolved to use Google’s DNS (8.8.8.8) and Cloudflare’s DNS (1.1.1.1) as the primary resolvers, with Quad9 (9.9.9.9) as a fallback. All three are reliable public resolvers that handle billions of queries per day.
Step 2: Reload the Daemon and Restart the Resolver
After saving the file, restart systemd-resolved:
sudo systemctl restart systemd-resolved
Step 3: Verify the Change
Confirm the new resolvers are active:
resolvectl status
Look for the DNS Servers line in the output. It should now reflect 8.8.8.8 and 1.1.1.1 rather than the old nameserver.
You can also run the same dig query that originally failed to confirm it now resolves correctly:
dig MX ky.gov
A Note on Reverting
If the hosting provider fixes their DNS server and you want to go back to the default, simply delete the file you created and restart the resolver:
sudo rm /etc/systemd/resolved.conf.d/custom-dns.conf
sudo systemctl restart systemd-resolved
The drop-in approach makes this kind of change easy to apply and just as easy to roll back, which is exactly what you want when working around a temporary infrastructure issue.