I used Tailscale the other day to solve a problem where I wanted to scrape a government website from inside GitHub Actions but I was being IP blocked by Cloudflare. My scraper in GitHub Actions now connects to my Tailscale network and uses my Apple TV as an exit node - works great! https://til.simonwillison.net/tailscale/tailscale-github-actions
I’ve done similar to bypass geographic restrictions on streaming services when traveling abroad. The streaming services block IP addresses from certain regions and they also typically block popular VPNs, but with Tailscale I set the exit node for my Raspberry Pi at home and I was good to go. The only issue is that if you want to watch content on a TV, you need something like an Apple TV that can connect the TV to Tailscale.
I’ve raved about it before but… we adopted Tailscale in March 2020 because of an unfortunate work-from-home situation that popped up :). There’s something truly magical about having our field team put a drone in the air and just being able to type the host name into VLC and watch live video off of it. Starlink backhaul, fancy 2.4GHz long-range/high-speed radios to the drone, Tailscale running right on the onboard imaging system.
I would like to note that you can run all this yourself.
Headscale runs just fine in k8s, letting you keep control of your network, if you’re scared of everything being tied to some sso, or if you just like an infinite free tier. It has a quirk with cloudflare tunnels, so you need somewhere to run it, like Google gcp free tier or whatever. (I work for Google)
It works with the clients in app stores, so your TV, phone, and computers don’t need anything extra. I have not gotten it to work with gl.inet routers, which is a bummer.
The only thing I haven’t gotten to work/isn’t supported is the traffic/client logging, some cases of compliance would require this. There is some obscure env variable you can tune, but no open source implementation of a receiver. Sadly haven’t found time to further dive into fixing this.
Yep. I actually set up my own thing back in the day with Tinc + ocserv (+ proper dnsmasq), so for me really Tailscale does not seem worthwhile
However, if I started from scratch nowadays I would likely make different choices, but the interesting thing about Tailscale is that they put many existing pieces together in a very usable package.
(And I believe they have some innovations on top, too.)
Tailscale wrote the new Internet, which I think is an insightful article. Another interesting project in this space is Yggdrasil. Some other projects like Nebula might also be worth exploring.
Thanks for the writeup, I’ll bookmark it for sure.
I recently started looking into wireguard to access a few of my devices behind NATs and also ended up researching Tailscale in tandem. I will gladly accept that Tailscale is amazing and easy for lots of folks but it turns out I’m a control freak when it comes to system administration and I don’t fully understand everything that Tailscale does to the host. (Particularly MagicDNS.) To put it another way, it has lots of small parts. I was also a bit turned off by the fact that it uses a userspace implementation of Wireguard even though I have a Wireguard-capable kernel of all of my boxes. The final straw was that the documentation for Headscale (the open source control plane I planned to use) seemed rather lacking and if you have any questions, you basically have no option but to log into their Discord.
So my simple poor-man’s alternative is to just deploy Wireguard to the nodes as “spokes” along with a “hub” on a VPS and route all traffic through that. Yeah, it’s going to be inefficient compared to node-to-node traffic, but I’m not asking much out of this. I’m using an IPv6 ULAs for the Wireguard IPs and each of these gets an AAAA record on my DNS server. At some point, I may look into whether I can deploy my own DERP server to somehow make it more of a mesh and less of a wheel.
Tailscaled runs its own nameserver and edits resolv.conf to use it on my linux laptop. They had a blog post where they talked about tackling all the DNS configurations they handle.
Also, if you run systemd-resolved, it simply registers its “magic” nameserver with resolved API, and then everything “just works” because resolved has first-class support for multi-homed resolution.
IIRC, that blog post also concluded with “just use resolved”.
FWIW I switched from plain WireGuard to a Headscale setup about a year ago, and it’s been working perfectly fine.
You’re right that the documentation can be a bit obtuse, and once or twice I’ve had breaking changes when upgrading the Headscale server, but that’s to be expected, and I now skim the release notes before upgrading (which I should have done all along, of course).
My poor-man’s alternative is to run autossh tunnels that connect to my home server via DDNS. Then I can open a reverse tunnel from home to the remote device.
Pros:
it works
my home server already runs an SSH server anyway, so no additional service needed
no magic involved
no external company involved
Cons:
requires working DDNS (if my home provider wouldn’t offer a public IP, I’d be screwed)
somewhat hacky?
tunneling over a TCP tunnel can be a bad experience. One day I should replace autossh by Wireguard.
How bad is the NAT traversal situation with Headscale?
I’m a happy Tailscale user (also free tier, although I almost feel guilty for the sheer amount of convenience I get for free) and I’m regularly amazed by its ability to establish direct communication in seemingly hopeless scenarios (e.g., NATs on both sides, one of which is a double NAT and the other is café Wi-Fi, sometimes in different countries). I’m assuming that at least some of the hole punching logic must be on the server side?
I wish I knew but I haven’t needed to try Headscale yet. That’s what concerns me the most, one day needing Headscale and finding it lacking. But comments from other users seem to speak highly of it.
I’ve been using Headscale for a few years now. Not many, if any, NAT issues. However when going overseas to places/areas with dodgy networking infrastructure, sometimes I would have a NAT hiccup or connection issue.
IIRC HS uses the same DERP servers as Tailscale since that list is public(?)
Correct, yeah. I haven’t experimented with doing that myself because I think the default TS servers have served me well. Haven’t had issues besides like I said, in foreign countries. Which isn’t ideal, but I’m not even sure if those issues I had are directly related to NAT – could be just related to general shaky connectivity.
Something people often tend to miss with tailscale is that it’s not an OpenVPN replacement, but rather an IPSec replacement (also see: wireguard). Back in ~2017 ish (before tailscale was founded) I was using it to connect multiple DCs together (previously if you didn’t want to do ipsec on the gateway you’d just run a literal fiber line between the locations). I’m still doing this, but tailscale makes it way easier, and also much more transparent (my homegrown set of scripts to ensure dnsmasq knew what to ask for what weren’t all that clean).
I should honestly write about this sometime because the kind of ergonomics you can achieve in the typical startup network situation (multiple kubernetes clusters and some dedicated servers thrown into the mix, so a classic “hybrid cloud” setup) are great.
I mean you are technically correct. But also, It’s totally an OpenVPN replacement in that it solves a whole lot of the same problems and it does so in a way that, compared to OpenVPN, is almost anger inducingly easy. If you’ve ever had to manage or configure OpenVPN you probably know what I mean.
Oh I don’t disagree! I mostly mean that it’s a superset; i.e. it can do way more than that. It also does even more in that respect with the authentication suite via funnel, so it’s truly the “do everything” VPN.
Not directly related, but I’ve found that tailscale pairs incredibly well with https://mutagen.io/ to sync a folder between computers.
They have their own solution (https://tailscale.com/kb/1369/taildrive), but it is based on webdav, which is almost unusable on macOS. In contrast mutagen works as expected.
The bad news is that it’s no longer possible thanks to a cursed thing called CGNAT
No longer possible for whom? My takeaway from this post was less “tailscale is good” and more “wow, I’m sure glad my ISP didn’t screw me over like this person’s did!”
Yeah that’s just my experience but with the IPv4 pool drying up it’s kind of inevitable? Also there’s still the question of exposing the home server to the internet. (Which may be desired if it’s hosting your blog, of course.)
My favorite cool application of Tailscale recently: I needed to test some software on Linux within WSL on Windows, from my Mac in a cafe. A quick install of Tailscale on the inner Linux VM made ssh immediately work, through unfathomable layers of firewalls/configuration of Linux, Windows, my home router, my ISP…
To my knowledge, it’s quite similar, although I’m not a Tailscale user. You can associate multiple devices with your account, and all of these devices become part of your mesh network. They will have local IP addresses that can be accessed from each device, along with optional nicknames (DNS domains). Additionally, you can invite other people to connect their machines to your mesh network.
From implementation pov, it’s pure WireGuard for user traffic. For linux we use the official kernel module, for Windows we use official WireGuardNT and for other systems (macOS, iOS, Android) we generally use NepTUN rust userspace WireGuard implementation (formerly CloudFlare’s boringtun). When connecting peers we try to connect them directly (with various NAT traversal techniques) and if that fails we transparently run the traffic over the relay servers.
The logic of setting it all up, nat punching, and other control-plane like responsibilities is handled by cross platform rust library libtelio used by all the clients. In case of Linux, the client is also open source: https://github.com/NordSecurity/nordvpn-linux.
Exposing a port from your laptop to your phone: When developing a web application, once in a while you’ll have to test on an actual device.
Note that if your development server is on the same local network as yours, you probably don’t even need Tailscale for this. Just point your phone at whatever the IP address of your laptop or desktop is on your local network and you’ll see the site on your phone. (If you’re at work or at a coffee shop, sometimes this won’t work, and you’d want to consider Tailscale again.)
In fact that this works so easily on LAN is part of the initial inspiration for Tailscale: Remembering the LAN
I thought about mentioning this but I wasn’t sure. Some dev servers listen on 127.0.0.1 and you need to give a flag like —host to make it listen to 0.0.0.0 so it sounded a bit distracting.
It could still be a good footnote, let me add it! Thanks!
I chose to use tailscale for my remote-first game development company and it was a choice I never came to regret.
The only downside was the limited amount of “named individuals” in the ACLs, and that the ACL’s themselves could be easier to write/debug. As it currently stands there’s at least some built-in testing you can write (but those count as named individuals ;) ) – I can see that file getting massively annoying in large enough deployments.
Regardless, for <100 people and <1,000 machines/vms/accessible-containers it really was quite excellent.
I don’t regret it even for a second. It made the network truly “flat” and eliminated issues we would have had with NAT completely.
The App Connector stuff will be useful for any UK-based people, find an exit node outside the UK IP Ranges and pin Lobsters to that. 😉
I didn’t realise that Tailscale + Mullvad was effectively double-blind to traffic, makes sense now I think about it some more. I love it because it’s just one VPN solution on every device, and I can appear from elsewhere when I want or need to without having to muck around with yet another app.
You should ask for a Hat if you don’t want to write that each time 🙃
And yes, App Connectors are super useful. It would be super cool if you could nominate a Mullvad exit node for them, but I can totally get why that isn’t a simple integration heh.
I used Tailscale on a pinch to be able to track some humidity sensors in my home while I was traveling abroad. I tried a couple of solutions first (WireGuard and the VPN built into my Meraki stack) and it was honestly a hassle. Between having to figure out what each option meant, having to download certificates and transfer them to my phone, etc. Definitely not something I’d recommend to someone who doesn’t work with computers for a living.
Then I realized that HomeAssistant has a Tailscale add-on. I installed it, installed the Tailscale app on my phone and after two minutes I could see HomeAssistant from my phone. Magical.
I used Tailscale the other day to solve a problem where I wanted to scrape a government website from inside GitHub Actions but I was being IP blocked by Cloudflare. My scraper in GitHub Actions now connects to my Tailscale network and uses my Apple TV as an exit node - works great! https://til.simonwillison.net/tailscale/tailscale-github-actions
I’ve done similar to bypass geographic restrictions on streaming services when traveling abroad. The streaming services block IP addresses from certain regions and they also typically block popular VPNs, but with Tailscale I set the exit node for my Raspberry Pi at home and I was good to go. The only issue is that if you want to watch content on a TV, you need something like an Apple TV that can connect the TV to Tailscale.
I read your article after I created my setup and found it similar to mine. Another cool use case!
I’ve raved about it before but… we adopted Tailscale in March 2020 because of an unfortunate work-from-home situation that popped up :). There’s something truly magical about having our field team put a drone in the air and just being able to type the host name into VLC and watch live video off of it. Starlink backhaul, fancy 2.4GHz long-range/high-speed radios to the drone, Tailscale running right on the onboard imaging system.
I would like to note that you can run all this yourself.
Headscale runs just fine in k8s, letting you keep control of your network, if you’re scared of everything being tied to some sso, or if you just like an infinite free tier. It has a quirk with cloudflare tunnels, so you need somewhere to run it, like Google gcp free tier or whatever. (I work for Google)
It works with the clients in app stores, so your TV, phone, and computers don’t need anything extra. I have not gotten it to work with gl.inet routers, which is a bummer.
See also: https://github.com/anderspitman/awesome-tunneling
The only thing I haven’t gotten to work/isn’t supported is the traffic/client logging, some cases of compliance would require this. There is some obscure env variable you can tune, but no open source implementation of a receiver. Sadly haven’t found time to further dive into fixing this.
Yep. I actually set up my own thing back in the day with Tinc + ocserv (+ proper dnsmasq), so for me really Tailscale does not seem worthwhile
However, if I started from scratch nowadays I would likely make different choices, but the interesting thing about Tailscale is that they put many existing pieces together in a very usable package.
(And I believe they have some innovations on top, too.)
Tailscale wrote the new Internet, which I think is an insightful article. Another interesting project in this space is Yggdrasil. Some other projects like Nebula might also be worth exploring.
Thanks for the writeup, I’ll bookmark it for sure.
I recently started looking into wireguard to access a few of my devices behind NATs and also ended up researching Tailscale in tandem. I will gladly accept that Tailscale is amazing and easy for lots of folks but it turns out I’m a control freak when it comes to system administration and I don’t fully understand everything that Tailscale does to the host. (Particularly MagicDNS.) To put it another way, it has lots of small parts. I was also a bit turned off by the fact that it uses a userspace implementation of Wireguard even though I have a Wireguard-capable kernel of all of my boxes. The final straw was that the documentation for Headscale (the open source control plane I planned to use) seemed rather lacking and if you have any questions, you basically have no option but to log into their Discord.
So my simple poor-man’s alternative is to just deploy Wireguard to the nodes as “spokes” along with a “hub” on a VPS and route all traffic through that. Yeah, it’s going to be inefficient compared to node-to-node traffic, but I’m not asking much out of this. I’m using an IPv6 ULAs for the Wireguard IPs and each of these gets an AAAA record on my DNS server. At some point, I may look into whether I can deploy my own DERP server to somehow make it more of a mesh and less of a wheel.
Tailscaled runs its own nameserver and edits resolv.conf to use it on my linux laptop. They had a blog post where they talked about tackling all the DNS configurations they handle.
Also, if you run systemd-resolved, it simply registers its “magic” nameserver with resolved API, and then everything “just works” because resolved has first-class support for multi-homed resolution.
IIRC, that blog post also concluded with “just use resolved”.
FWIW I switched from plain WireGuard to a Headscale setup about a year ago, and it’s been working perfectly fine.
You’re right that the documentation can be a bit obtuse, and once or twice I’ve had breaking changes when upgrading the Headscale server, but that’s to be expected, and I now skim the release notes before upgrading (which I should have done all along, of course).
You might enjoy https://github.com/tonarino/innernet perhaps?
Perhaps! I have it bookmarked but have yet to dig into it.
My poor-man’s alternative is to run autossh tunnels that connect to my home server via DDNS. Then I can open a reverse tunnel from home to the remote device.
Pros:
Cons:
Tailscale is the one of few services that I use where a rug pull of the free tier will really screw me over
Luckily there’s Headscale for that day! Surely there will be a disruption and self-hosting is a chore but at least all hope will not be lost.
How bad is the NAT traversal situation with Headscale?
I’m a happy Tailscale user (also free tier, although I almost feel guilty for the sheer amount of convenience I get for free) and I’m regularly amazed by its ability to establish direct communication in seemingly hopeless scenarios (e.g., NATs on both sides, one of which is a double NAT and the other is café Wi-Fi, sometimes in different countries). I’m assuming that at least some of the hole punching logic must be on the server side?
Works just fine, Headscale has an embedded DERP server: see https://headscale.net/stable/about/features/ and https://tailscale.com/kb/1232/derp-servers
I wish I knew but I haven’t needed to try Headscale yet. That’s what concerns me the most, one day needing Headscale and finding it lacking. But comments from other users seem to speak highly of it.
I’ve been using Headscale for a few years now. Not many, if any, NAT issues. However when going overseas to places/areas with dodgy networking infrastructure, sometimes I would have a NAT hiccup or connection issue.
IIRC HS uses the same DERP servers as Tailscale since that list is public(?)
It does by default, but you can also run your own.
Correct, yeah. I haven’t experimented with doing that myself because I think the default TS servers have served me well. Haven’t had issues besides like I said, in foreign countries. Which isn’t ideal, but I’m not even sure if those issues I had are directly related to NAT – could be just related to general shaky connectivity.
Something people often tend to miss with tailscale is that it’s not an OpenVPN replacement, but rather an IPSec replacement (also see: wireguard). Back in ~2017 ish (before tailscale was founded) I was using it to connect multiple DCs together (previously if you didn’t want to do ipsec on the gateway you’d just run a literal fiber line between the locations). I’m still doing this, but tailscale makes it way easier, and also much more transparent (my homegrown set of scripts to ensure dnsmasq knew what to ask for what weren’t all that clean).
I should honestly write about this sometime because the kind of ergonomics you can achieve in the typical startup network situation (multiple kubernetes clusters and some dedicated servers thrown into the mix, so a classic “hybrid cloud” setup) are great.
I mean you are technically correct. But also, It’s totally an OpenVPN replacement in that it solves a whole lot of the same problems and it does so in a way that, compared to OpenVPN, is almost anger inducingly easy. If you’ve ever had to manage or configure OpenVPN you probably know what I mean.
Oh I don’t disagree! I mostly mean that it’s a superset; i.e. it can do way more than that. It also does even more in that respect with the authentication suite via funnel, so it’s truly the “do everything” VPN.
Not directly related, but I’ve found that tailscale pairs incredibly well with https://mutagen.io/ to sync a folder between computers.
They have their own solution (https://tailscale.com/kb/1369/taildrive), but it is based on webdav, which is almost unusable on macOS. In contrast mutagen works as expected.
I wasn’t aware of Taildrive, though given its alpha state and your stability comments, it’s a good thing I didn’t mention it.
TIL https://mutagen.io , might need to play around with it since it looks pretty cool.
Care to elaborate about the macOS issues?
I think it’s remote drive in general, which adds a bunch of latency when you’re trying to develop in them.
In contrast, mutagen works by creating a copy of the files on each ends, and then use a file-watcher to sync them
No longer possible for whom? My takeaway from this post was less “tailscale is good” and more “wow, I’m sure glad my ISP didn’t screw me over like this person’s did!”
Yeah that’s just my experience but with the IPv4 pool drying up it’s kind of inevitable? Also there’s still the question of exposing the home server to the internet. (Which may be desired if it’s hosting your blog, of course.)
My favorite cool application of Tailscale recently: I needed to test some software on Linux within WSL on Windows, from my Mac in a cafe. A quick install of Tailscale on the inner Linux VM made ssh immediately work, through unfathomable layers of firewalls/configuration of Linux, Windows, my home router, my ISP…
Yes, I use Tailscale inside WSL too. Very handy because the Windows machine is more beefy than my other native Linux desktop.
It doesn’t work so great for builds because I keep getting clock skew errors a lot.
Wait, what? I am still using DDNS to do basically the same thing (hosting a site on a SBC from my home).
Lucky for you, but your days may be numbered. My ISP offers paid static IP service though, but I haven’t used it.
NordVPN (I work there) also offers free meshnet service as part of its vpn client software.
I saw no mention of WireGuard on that page. Would you be able to describe how that service works in comparison to Tailscale?
To my knowledge, it’s quite similar, although I’m not a Tailscale user. You can associate multiple devices with your account, and all of these devices become part of your mesh network. They will have local IP addresses that can be accessed from each device, along with optional nicknames (DNS domains). Additionally, you can invite other people to connect their machines to your mesh network.
From implementation pov, it’s pure WireGuard for user traffic. For linux we use the official kernel module, for Windows we use official WireGuardNT and for other systems (macOS, iOS, Android) we generally use NepTUN rust userspace WireGuard implementation (formerly CloudFlare’s boringtun). When connecting peers we try to connect them directly (with various NAT traversal techniques) and if that fails we transparently run the traffic over the relay servers.
The logic of setting it all up, nat punching, and other control-plane like responsibilities is handled by cross platform rust library libtelio used by all the clients. In case of Linux, the client is also open source: https://github.com/NordSecurity/nordvpn-linux.
Interesting, I’ve never heard of that before! The homepage doesn’t open here in Turkey by the way, probably a big target for censorship circumvention.
Note that if your development server is on the same local network as yours, you probably don’t even need Tailscale for this. Just point your phone at whatever the IP address of your laptop or desktop is on your local network and you’ll see the site on your phone. (If you’re at work or at a coffee shop, sometimes this won’t work, and you’d want to consider Tailscale again.)
In fact that this works so easily on LAN is part of the initial inspiration for Tailscale: Remembering the LAN
I thought about mentioning this but I wasn’t sure. Some dev servers listen on 127.0.0.1 and you need to give a flag like
—hostto make it listen to 0.0.0.0 so it sounded a bit distracting.It could still be a good footnote, let me add it! Thanks!
I added the footnote!
I chose to use tailscale for my remote-first game development company and it was a choice I never came to regret.
The only downside was the limited amount of “named individuals” in the ACLs, and that the ACL’s themselves could be easier to write/debug. As it currently stands there’s at least some built-in testing you can write (but those count as named individuals ;) ) – I can see that file getting massively annoying in large enough deployments.
Regardless, for <100 people and <1,000 machines/vms/accessible-containers it really was quite excellent.
I don’t regret it even for a second. It made the network truly “flat” and eliminated issues we would have had with NAT completely.
The App Connector stuff will be useful for any UK-based people, find an exit node outside the UK IP Ranges and pin Lobsters to that. 😉
I didn’t realise that Tailscale + Mullvad was effectively double-blind to traffic, makes sense now I think about it some more. I love it because it’s just one VPN solution on every device, and I can appear from elsewhere when I want or need to without having to muck around with yet another app.
Think of App Connectors like SplitDNS for specific domains. Really useful!
Note I work for Tailscale so everything I say is subject to bias.
You should ask for a Hat if you don’t want to write that each time 🙃
And yes, App Connectors are super useful. It would be super cool if you could nominate a Mullvad exit node for them, but I can totally get why that isn’t a simple integration heh.
I used Tailscale on a pinch to be able to track some humidity sensors in my home while I was traveling abroad. I tried a couple of solutions first (WireGuard and the VPN built into my Meraki stack) and it was honestly a hassle. Between having to figure out what each option meant, having to download certificates and transfer them to my phone, etc. Definitely not something I’d recommend to someone who doesn’t work with computers for a living.
Then I realized that HomeAssistant has a Tailscale add-on. I installed it, installed the Tailscale app on my phone and after two minutes I could see HomeAssistant from my phone. Magical.