Let’s imagine you have a website; we will use mine as the example, so https://timnash.co.uk very cool; lots of people visit it every day; yay, go me.
Unfortunately, so do lots of bots, and you are fed up with them hitting your login page. You already configured Tailscale and Tailscale SSH (don’t worry if you don’t know what Tailscale is, we will have a quick catch-up in a moment), so wouldn’t it be nice if you could simply only access the admin areas if you are connecting over your tailnet?
Yeah, I thought so too, and it turns out it’s not actually that complicated but does require a few hoops and a little bit of DNS, and by DNS, I mean split DNS. So get your beverage of choice and journey with me into the world of Tailscale.
What is Tailscale?
Tailscale is a VPN service that is built on Wireguards noise protocol. This is very cool; it means you can quickly build something called a mesh network, where devices can talk to each other through Tailscale on what they call “tailnets”, a virtual network. So this means your laptop and your server appear to each other to be on the same network even when they are behind really weirdly configured firewalls.
Why is Tailscale so great? Well, the stuff underneath, like Wireguard, is open-source software you can run yourself; Tailscale has just abstracted a lot of the pain away. They even offer to bring the pain back via headscale, which is an open-source version of Tailscale control server.
I am a big fan of Tailscale and use it to securely talk to servers I have control over. They have a bunch of neat features, like providing authentication for SSH and loads of other features. Some of which we will be using to get this all to work.
Introducing Caddy
This site runs on Caddy, which is an HTTP server like Nginx or Apache, it’s a little more modern with a simpler config, and I really like it for small projects like this site.
So our goal is to lock down certain areas of the site to only be accessible via Tailscale. Tailscale uses a fixed IP range, so we can do something like:
@authed {
not ip_range 100.64.0.0/10
not path /wp-admin/admin-ajax.php
path /wp-admin/
path /wp-login.php
path /xmlrpc.php
}
respond @authed "<h1>Access Denied</h1>" 401Within our Caddy config file, and job’s done!
Ah, no, we have a few problems. First off, the 100.64.0.0/10 is a HUGE subnet, and a bad actor who realises what we are doing could easily spoof it, but we have a problem if we go and ping the site.
ping timnash.co.uk
PING timnash.co.uk (139.162.211.81) 56(84) bytes of data.
64 bytes from 139-162-211-81.ip.linodeusercontent.com (139.162.211.81): icmp_seq=1 ttl=57 time=2.43 ms
64 bytes from 139-162-211-81.ip.linodeusercontent.com (139.162.211.81): icmp_seq=2 ttl=57 time=2.11 ms
64 bytes from 139-162-211-81.ip.linodeusercontent.com (139.162.211.81): icmp_seq=3 ttl=57 time=1.98 msWe can see it’s responding with 139.162.211.81, and if we visit the site and look in the server logs, our IP is our normal ISP IP, not our internal tailnet IP.
Well, that’s awkward.
You see when we make a request to timnash.co.uk, we ask our DNS server, “Hey, what’s this site’s IP?” and it responds with a public IP which we then reach out to get the site contents. Our browser and computer don’t know that this is a server inside of our tailnet; it’s gone off to the big wide internet.
So we have two problems; using an IP range while better than nothing is not ideal, we can do better, and we have to make sure we access the site through Tailscale. We will deal with that one first.
MagicDNS and the fun of Split DNS
So, to remind ourselves, we have our Tailscale network, which is called a tailnet, and this acts as a virtual network on our devices; it presents like a network device just like the wifi or wired network connection does. One of the features within Tailscale is something called MagicDNS. This is a small, inbuilt DNS service that allows us to use the tailnet names (normally the host name of the clients) so, for example, I can do ssh timnashcouk instead of using the tailnet IP.
This is really handy, but it also allows us to set multiple other DNS servers for resolving domains outside of our tailnet. This includes custom DNS servers, and one of the very useful features of this is something called split DNS
Split DNS is where we specify that any requests for domain resolving to a specific domain let’s say timnash.co.uk is sent to a specific DNS server. This might not make much sense at the moment but hopefully it will.
If we run the command dig timnash.co.uk our computer will go and ask the DNS server for the record, and it will return the public address; we know this from our ping. With Split DNS we can say to MagicDNS that we should say we know the address timnash.co.uk. This means the request is not made to the computer’s normal DNS resolver; instead, MagicDNS will handle it, only magicDNS has no idea how to handle it, so it will forward the request to the DNS server we specify which will (hopefully) return our tailnet IP.
Ok, so we need a DNS server.
DNS is one of those things that sends shivers down many people’s spines, get it wrong, and it’s a world of pain and despair. However in this case we are running a very small DNS resolver; indeed we only want it to return one domain.
My intention was to set up dnsmasq which is a super lightweight DNS server (it’s what powers pihole under the hood, if you have ever used that) on the timnash.co.uk server itself, bind it to the tailnet IP and job done. Before doing this live I thought I would test it on a small VM and so installed the software only to be told it couldn’t bind to port 53.
Hmmm, wait, the machine already has a DNS resolver installed. Now I felt sheepish; what is listening on 53?
netstat -anpe | grep ":53" | grep "LISTEN"
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 101 45705 3278/systemd-resolv Ah, SystemD, a sysadmin’s best friend, it would appear it has some sort of DNS resolver, and indeed it does. Systemd-resolved is a DNS caching service, it’s not a full DNS server, rather it caches results from other DNS servers. It’s actually really cool and supports a lot of very cool things including ironically split DNS not that we are going to use it’s split DNS unless we wanted some sort of weird chaining DNS hell. More importantly we can probably get it to give us an IP address, so could feasibly be used in place of MagicDNS on non-Tailscale VPNs; Maybe.
Given systemd-resolved is there, can it do what I want?
Keep in mind we are abusing what this thing is meant for which is to cache entries, not serve them to remote parties, but it has a couple of features that, when combined, allow us to do exactly what we want.
Our first problem is you can’t add records to it like a DNS system, it gets those records from elsewhere. However, one option is for it to get those records from its /etc/hosts file.
You can do so by editing /etc/systemd/resolved.conf and enabling ReadEtcHosts=yes
Now all we have to do is edit our /etc/hosts on that server and add in our entry for timnash.co.uk with the tailnet IP of the server with timnash.co.uk.
Yay, now when we dig timnash.co.uk on the server, we get the tailnet IP.
Next, by default systemd-resolved only responds to local requests on the machine, not the wider machines on the network. We want magicDNS to talk to it so we need to bind it to our tailnet interface and we do so by adding:
...
DNSStubListenerExtra={tailnet IP for the server}
... into the /etc/systemd/resolved.conf.
Restart, and we have a working DNS for timnash.co.uk to resolve against.
Yay!
Our next step is to get MagicDNS to use it. This is actually the very easy part, log into Tailscale account, navigate to DNS, make sure you have MagicDNS enabled (I’m 99% sure it is by default) and then add a nameserver. Select “custom nameserver” and put in the IP address for our server. Then click the “Restrict to domain” option and specify the domain, in our case timnash.co.uk, and click Save.
Once done, whenever you are using Tailscale and your machine makes a DNS request for timnash.co.uk it will ask our server for the IP, not the usual DNS server, and the server retrieves it from the host’s file.
Brilliant, except there is a problem, DNS is more than just a single A record, and I have several subdomains, such as api.timnash.co.uk, and these were failing to resolve.
The reason why is the MagicDNS is matching anything in the *.timnash.co.uk and saying “send it to our server to resolve”. Our server goes “I have record for timnash.co.uk, but not for api.timnash.co.uk I will ask the Magic DNS server for the additional records. Hey, MagicDNS do you have the IP address for api.timnash.co.uk” and. ah yes. We are now in a loop.
Worse, even if we specify a DNS server to call and fallback they are never called because of priorities.
There is a simple solution, and that is to disable MagicDNS on the server where our DNS is
tailscale up --ssh --accept-dns=falseUsing the accept-dns flag disables the DNS, and it loops to the public DNS and returns all the additional records. It does mean the server can’t make use of the MagicDNS, but in this case, this is ok and not a problem.
So to summarise where we are:
We have modified the DNS cache resolver that comes with systemd to return our tailnet IP for timnash.co.uk. This resolver is being used by Tailscale’s MagicDNS and split DNS feature to mean that when we are using Tailscale and we visit timnash.co.uk, we visit using the IP of the server on the tailnet, not the public IP.
In theory with our code block in Caddy, we could stop now; job done. But those IPs could be spoofed, wouldn’t it be nice if Caddy only let you in if you were on the tailnet?
Caddy and bind directive
I actually spent far too long trying to think of solutions because I made an assumption. Early in my research, I came across Caddy bind directive which is like the listen directive in Nginx; you say listen (bind to) this interface (by using the IP for that interface).
However, I assumed that you couldn’t have more than one site block with the same name. I was wrong. Only I didn’t test this for several hours.
When I did test, I discovered that I could have multiple blocks with the same name in Caddy, and so was able to do:
www.timnash.co.uk {
bind 139.162.211.81
redir / https://timnash.co.uk 301
}
www.timnash.co.uk {
bind 100.98.xxx.xx
redir / https://timnash.co.uk 301
}Where one is bound to the public IP and the other to the tailnet. I did the same with the main block. The only difference between them is that the public one has the additional:
@authed {
not path /wp-admin/admin-ajax.php
path /wp-admin
path /wp-login.php
path /xmlrpc.php
}
respond @authed "<h1>Access Denied</h1>" 401While the Tailscale does not.
That is it, it’s working. If you are on my Tailscale network or, more specifically, if you can talk to my server through its Tailscale interface then you can access my WordPress login area. If you are not on the network, you can not.
Tailscale is it worth doing?
Totally, I think the overheads, especially as systemd-resolved is built in and enabled on most Linux systems out of the box means there are no performance issues and having two separate configs for the tailnet version of the site and public while they have to be kept in sync (I manage mine in Ansible so this is not an issue) gives advantages not just from security.
One thing I have done is created a second PHP pool, meaning folks on the tailnet are using different PHP workers with different limits. Should there ever be an issue with the site, I should always be able to gain access even if others are struggling, I can also run debugging tools like xdebug on the live site; heresy I know, but do so safely as it will only be enabled on my pool. I can even run a completely different version of PHP!
Tailscale is an amazing service, I could get halfway there myself with WireGuard and endless tutorials but using Tailscale is so easy, and it has totally changed my way of working on servers. I really recommend you check it out and I believe it’s free now for up to 100 devices.
So that’s that; we did DNS and survived, maybe.