Fix for kubernetes on DigitalOcean Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 11.111.11.1 11.111.11.2 11.111.11.1" (example IP addresses used)

After ignoring the error for a bit, I finally had to absolutely solve it due to being unable to deploy a prometheus chart inside my kubernetes cluster.

This isn't really a step-by-step but more an accumulation of resources used/found to finally help solve my problem

TL;DR

• Modify /etc/netplan/50-cloud-init.yaml removing some entries from IFACE.nameservers.address
  • netplan apply to apply changes
  • resolvctl to see changes to DNS Servers stanza for the IFACE modified
  • Review /run/systemd/resolve/resolv.conf to see less nameservers
• Modify /etc/systemd/resolved.conf.d/DigitalOcean.conf either mving/commenting it out
  • systemctl restart systemd-resolved.service to apply changes
  • resolvctl to see changes to DNS Servers stanza for Global modified
  • Review /run/systemd/resolve/resolv.conf to see less nameservers
• (Optionally) Using resolvectl command
  • resolvectl dns IFACE DNS_ADDRESS to update the DNS servers that appear in resolvectl and /run/systemd/resolve/stub-resolv.conf
  • resolvectl dns IFACE to view the nameservers for the interface
  • This is similar to editing the /etc/netplan files, but most likely will not persist.
  • To persist these changes, edit the netplan file

I won't recommend how you modify these files, whether you want to try googles 8.8.8.8 8.8.4.4, or attempting to use only the global entry and completely removing the entries from the cloud-init.yaml file etc. Thats more on the individual to work with and test. The only goal really is lowering your nameservers in /run/systemd/resolve/resolv.conf to 3 or below, how its done is up to the reader.

The bottom titled "My thoughts" is a formatted version of above with slightly more explanation (since Im not really knowledgeable on the subject of DNS). Everything else below is kind of all over copy+pasting of links and some notes on them with only a bit of coherency as I worked backwards after finally getting the solution.

---

https://wiki.archlinux.org/title/systemd-networkd
Under section 2 Configuration files
Configuration files are located in /usr/lib/systemd/network/, the volatile runtime network directory /run/systemd/network/ and the local administration network directory /etc/systemd/network/. Files in /etc/systemd/network/ have the highest priority.

https://askubuntu.com/questions/1128536/how-to-make-persistent-changes-to-etc-resolv-conf-ubuntu-18-10
https://linuxize.com/post/how-to-set-dns-nameservers-on-ubuntu-18-04/
https://ubuntu.com/server/docs/network-configuration netplan apply

I found the file I needed under /run/systemd/network/10-netplan-eth0.network
I think this only gets read on system boot/restart
Using resolvectl dns IFACE DNS_ADDRESS will add the DNS entrys to both /run/systemd/resolve/resolv.conf and youll see it under resolvectl

I think I saw this command somwhere else but I was determined to find a config but if youre fed up, issuing the first command
will set the nameservers for the eth0 interface
https://gist.github.com/brasey/fa2277a6d7242cdf4e4b7c720d42b567?permalink_comment_id=4417602#gistcomment-4417602

https://discussion.fedoraproject.org/t/systemd-resolved-duplicate-entries/31676/3 /etc/systemd/resolved.conf.d/DigitalOcean.conf

https://unix.stackexchange.com/questions/328131/how-to-troubleshoot-dns-with-systemd-resolved

Notable files/dirs:

/etc/systemd/resolved.conf
/etc/systemd/resolved.conf.d/
/lib/systemd/resolv.conf

/run/systemd/resolve/resolv.conf
/run/systemd/resolve/stub-resolv.conf
/var/run/systemd/resolve/resolv.conf
/var/run/systemd/resolve/stub-resolv.conf

/run/systemd/network/*
/run/systemd/network/10-netplan-IFACE.network

/etc/netplan/50-cloud-init.yaml

Notable commands:

systemctl restart systemd-resolved.service
systemctl status systemd-resolved.service
resolvectl dns IFACE DNS_SERVER
resolvectl flush-caches
systemd-analyze cat-config systemd/resolved.conf
systemctl restart network

DO NOT run these
ip addr flush eth0
ip addr flush eth1

My thoughts/observation

The bottom will have a quick list of items to do to fix it however you choose, below is some additional info gathered.
Whatever is in the /etc/netplan/50-cloud-init.yaml gets reflected in /run/systemd/network/10-netplan-IFACE.network files which then populates /run/systemd/resolve/resolv.conf on system boot/restart as far as im aware.
You can manually apply these changes using netplan apply

For each IFACE.network file that has a DNS entry (or under nameservers.addresses in /etc/netplan/*), that will add a nameserver entry into /run/systemd/resolve/resolv.conf and you can see these entries listed under each individual IFACE in resolvectl

The entries you see in resolvectl can be modified using resolvectl dns IFACE DNS_ADDRESS as well (to persist modify the cloud-init.yaml file) which will also alter /run/systemd/resolve/resolv.conf nameserver entries (the main culprit of the error)

Under /etc/systemd/resolved.conf.d/DigitalOcean.conf theres also entries that populate the top Global entry in the resolvectl command and add additional entries to /run/systemd/resolve/resolv.conf (again the culprit).

After modifying/mving /etc/systemd/resolved.conf.d/DigitalOcean.conf, you can issue systemctl restart systemd-resolved.service to see the changes reflected in resolvectl.

If using a private network on DigitalOcean this will cause an additional 2 DNS entries.
So 2 from eth0, 2 from eth1, and 2 from Global for a total of 6, when the max available is 3.
The top TL;DR of this file gives some basic/simple instructions on what you can do to accomplish that.