An incredibly long ramp up to complaining about centralised control by rent seekers (a very reasonable complaint!) which gets bogged down in some ostensibly unrelated shade about whether client-server computing makes sense (it does) or is itself somehow responsible for the rent seeking (it isn’t; you can seek rent on proprietary peer to peer systems as well!) to then arrive at:
There’s going to be a new world of haves and have-nots. Where in 1970 you had or didn’t have a mainframe, and in 1995 you had or didn’t have the Internet, and today you have or don’t have a TLS cert, tomorrow you’ll have or not have Tailscale. And if you don’t, you won’t be able to run apps that only work in a post-Tailscale world.
I mean if only we had public, zero-trust internet infrastructure, the whole key-drop and NAT traversal thing could be hosted there. A .. gasp! tax-funded installation of headscale, e.g.
This whole complaining about rent seekers and then going about face and bragging about how everyone will need them and then they will be able to collect rent is really rubbing me the wrong way. Making me reconsider my usage of tailscale. At least there is headscale to fall back on.
So let’s look at the state of this proposed “new Internet”. It features:
A closed, undocumented, protocol.
A single open source implementation that had to reverse engineer the protocol, and which only exists at the mercy of Tailscale, because …
Tailscale owns 10 patents pertaining to this domain, which they will most certainly weaponise against any commercial implementations, and may weaponise against open source implementations at a time of their choosing.
Anyone seriously considering using this as the basis for a “new Internet” is deluding themselves. At some point this product is going to enshittify, and when it does, it’ll be messy in direct proportion to the number of otherwise open systems that have bought into it.
Could it work out? Yeah, if Tailscale publishes an RFC describing the protocol in full, and disclaims all of their related patent claims. Until that happens, this is just another proprietary tarpit waiting to trap the unwary.
The patent issue is the first one I have seen raised that seriously undermines their general “give away useful stuff” vibe; committing their parents to a defensive pool would go a long way, there.
I went searching for patents when I got to wondering … given that Tailscale has $2M in Series A funding and $100M in Series B funding, how are they planning to repay their investors, when anyone could just stand up a self-hosted headscale instance, and potentially charge for it?
I’d always assumed they were planning to make money from the big corporate players - self-hosting Headscale as a bigco means you need to do your own IDP integration & security audits, in addition to paid staff time to keep the thing running… not to mention that whoever championed “lets host it ourselves for free” is going to be in hot water every time the control plane has issues.
I’m very much reminded of the comment on the dropbox product launch - “Surely anyone who wanted this could simply write a short shell script to implement it”.
I found this a bit long-winded, and pretty grandiose if not downright creepy at times:
Well, about 1 in 20,000 people in the world uses the New Internet (that’s Tailscale). We’re not going to stop until it’s all of them.
…but still well written and entertaining. I enjoyed it. It convinced me to at least kick Tailscale’s tires at some point.
The bigger problem, though, is the problem that all “we’re disrupting X” narratives have: they’re not interested in disruption for the sake of liberation, because liberation is not profitable. Another visionary treatise corralled at the last moment into the new walled garden. I guess I’d still prefer to rent e2ee p2p networking capability for my own hardware than to rent a machine from AWS…
Except that with tailscale there’s no walled garden. Everything about the architecture and the client is open source, and they fund work on an open source version of the server too
I don’t know that I’d say there is no walled garden. The server is proprietary and the open source re-implementation Headscale is not really comparable feature wise. Plus it’s made by reverse engineering the protocol by a Tailscale employee so its not like there is a standard or reference out there to implement. There also isn’t any existing web UI or admin console that works with Headscale that is comparable to the proprietary one by any stretch outside of the most basic features. I say this as someone who’s company ran Headscale as the primary for a long time before ultimately switching to the proprietary one out of necessity.
I’m showing my ignorance here, but why does the protocol have to be reverse-engineered? They purport to be based on Wireguard. Is there a higher-level proprietary protocol on top of that?
Wireguard is the substrate. All the key handling, route management, DNS management, ACLs, heartbeats, host and user management, proxies, etc. etc. etc. is the part that Tailscale adds on top. The protocol I mention isn’t the substrate protocol, but the protocol to talk to and cooperate with the server. Additionally, it’s all the work the server and related bits do because that’s where a large chunk of the Tailscale magic resides.
There’s a bit of a paradox in your question – what does a company sell if not some extra service on top. Especially when selling networking software service to people whose job description importantly includes, configure, run (and sometime write) FOSS software. They’ve set themselves up to only compete on innovation speed, network effects, and support, something like.
They tout, and many people celebrate (including blog posts on) NAT traversal, with failover to encrypted traffic forwarding. They also provide control through SSO and network traffic policy, well beyond wireguard, not impossible, but quite expensive and tedious to replicate with say ansible scripts configuring firewalls. In addition, Tailscale is building a bunch of services to provide stickiness, lock-in, indeed just more features & a reason to pay.
The article puts a lot of blame on the connectivity (NATs), and it’s completely justified given that the IPv4 address space has been exhausted for a long time and it’ll only be getting worse. But let’s address the elephant in the room (that the author so conveniently ignores and never mentions) — IPv6 solves connectivity issues between nodes, it’ll be a long time before we run out of 2^64 LANs with 2^64 devices each (given that the current industry standard is to allocate /64 for LANs but sometimes devices get their own /64 e.g. for VMs).
Internet could be a lot better if instead of commercial VPN companies we had public NAT64 gateways to reduce the cost of adoption (so that IPv6-only networks could communicate with an already heavily NATed IPv4 world). I’m honestly surprised this isn’t the thing yet since major cloud providers could definitely pull this off on an Internet-scale with all the resources available to them. In fact, Cloudflare does this to a certain extent with WARP but that runs a layer above instead of simply configuring the system to use DNS64 server (that Cloudflare already provides) and some infrastructure to retransmit packets (that they already have since it’s a public IP tunneling service).
The article puts a lot of blame on the connectivity (NATs), and it’s completely justified given that the IPv4 address space has been exhausted for a long time and it’ll only be getting worse. But let’s address the elephant in the room (that the author so conveniently ignores and never mentions) — IPv6 solves connectivity issues between nodes, it’ll be a long time before we run out of 2^64 LANs with 2^64 devices each (given that the current industry standard is to allocate /64 for LANs but sometimes devices get their own /64 e.g. for VMs).
The founder of Tailscale really really really hates IPv6. He has written a bunch about it, but boiling it down to the essential points, his argument is basically that since IPv6 doesn’t solve all problems, we should never use it. Ironically, Tailscale has become too big for the /10 they use, so they kinda undermined some of the interesting stuff they’re doing.
The founder of Tailscale really really really hates IPv6.
Er. I think that is over-stating the case. He has some well-founded, cogently-argued comments about how it was poorly designed and that that’s why it has not succeeded and replaced IPv4. And he’s right: it hasn’t.
He has written a bunch about it,
Yes, some of the best most insightful commentary about it that I’ve seen anywhere.
but boiling it down to the essential points, his argument is basically that since IPv6 doesn’t solve all problems, we should never use it.
That does not even resemble the argument as I understood it, no.
He is saying, AFAICS, “almost everyone uses IPv4 because IPv6 is better but not better enough so here are some useful ways we are improving IPv4.”
There was so much I agreed with in this article, but something just feels off about it. I do agree the Internet and the Web has many, many layers of unnecessary complexity. I do believe that most developers think they need the scale of FAANG when they do not and never will and probably shouldn’t even want to. And so on.
I’m just… not convinced Tailscale is the solution to all of the world’s problems.
I almost posted this here when I read it, but I decided it was too corporate at the end. Tailscale is cool and what they’re doing is cool, but this feels like it doesn’t add much to David Crawshaw’s old post linked to in the piece.
I don’t think there’s anything wrong with that, but here and on HN, the reaction was super-negative, and I think the reason is that it came across as too corporate with too little technical content. Usually Apenwarr writes about really technical stuff like how NAT tunneling works or why IPv6 failed, so it disappointed people’s expectations.
I thought the first 3/4s of this article was very insightful, but as others have pointed out the proposed solution is nothing of the sort (unless you’re a tailscale investor I guess).
I do think that the analysis of the nature of cloud centralisation (i.e. connectivity-centric) is on-point… but I don’t know if tailscale (or even something like it, but more open) is the solution.
It feels like what we really need is more tools in the style of Let’s Encrypt, which make a formerly hard/expensive problem easy and accessible to smaller actors on the web… but what exactly that looks like I’m not sure. There are lots of smart people working on indieweb tech at the moment, so perhaps something (or some things) will emerge from that milieu.
Everything that makes Tailscale interesting is proprietary. Wireguard is just an implementation detail. There is a an open source clone (Headscale), albeit Headscale’s author works for Tailscale AFAIK. Mind you, Headscale just reverse engineered the API, it’s not an open standard or anything.
The clients are open source except for the ones that aren’t, such as the macOS and the Windows GUI clients.
But that doesn’t invalidate anything about what I said, everything that makes Tailscale interesting, and a SPOF, and creates vendor lock-in is the server.
Look, I’ll make it easy. Tailscale provides a cool service. They are (and will continue to be) the only ones providing this exact service. They even have a monopoly on Headscale. Maybe you are ok with this, maybe you aren’t, but pretending this is not the case is dishonest. And them wanting to monopolize the internet is downright scary.
Look, I’ll make it easy. Tailscale provides a cool service. They are (and will continue to be) the only ones providing this exact service.
We adopted Tailscale in March 2020 as it became clear that we weren’t going to be heading to the office for a while. We’re a UAV company and the company doesn’t run on the normal SaaS “lots of servers” kind of model but rather we have a bunch of embedded hardware (edge computing, I guess they’re calling it now) that we work with. Tailscale provided Linux arm32 and aarch64 clients already when we adopted it. The result is that we could send hardware to team members, they could connect it to their house WiFi or wired Ethernet, and we could collaborate on the physical hardware. A junior engineer is working on talking to a telemetry radio and I can SSH into his unit to help them debug things.
As our kit evolved from prototypes on peoples’ kitchen tables to pre-production hardware that we were field testing we kept Tailscale installed on all of the edge devices. Our field team has a Starlink antenna on the roof of the trailer they use to transport the gear around. When an aircraft is powered up in the field I get a ping on Slack. If they have any issues I can very reliably SSH into the unit and debug things. If I’m feeling curious I can even connect ground station software on my laptop in a coffee shop to the aircraft network and watch the flight in real-time. If we’ve both got a good connection I can watch the video feed off the aircraft live.
It’s incredible. Absolutely incredible. And it Just Works. Over and over and over. It works so well that is has become a completely invisible part of our network infrastructure. No one has to do any administration or maintenance on it; auth is tied to our Office 365 accounts and other than running apt update once in a while (which we have to do anyway since we distribute our own internal software through apt) it is completely maintenance-free. We disable key expiration on our edge devices because our field teams aren’t generally the kind of people you want to be SSHing into the hardware locally and trying to run commands with sudo; as a result, we’ve had old aircraft sit in a storage unit for months only to have someone pull it out, power it on, and I immediately have the ability to log into it and pull software updates from the comfort of my home office.
I would love to see some competition in this space for sure. But as a warning for anyone who’s going to try to compete with them: make sure that you make whatever you come up with as simple and bullet-proof as Tailscale is.
Absolutely, Tailscale just works and it’s a great service. I am frankly surprised why nobody competes with them. In fact I am disappointed some open source or non-profit organization doesn’t compete with them.
We had an issue with tailscale where our nodes were having issues getting direct connections to each other; this resulted in a relay connection during early startup, and the relay was having issues that tailscale wasn’t reporting as an outag(maybe it was networking later between relay and node). We solved the issue by ensuring port forwarding was available earlier (via explicit firewall rules) which in turn helped establishing direct node connections and bypassing the faulty relay.
Just a story about the the fallback to using relays causing issues.
An incredibly long ramp up to complaining about centralised control by rent seekers (a very reasonable complaint!) which gets bogged down in some ostensibly unrelated shade about whether client-server computing makes sense (it does) or is itself somehow responsible for the rent seeking (it isn’t; you can seek rent on proprietary peer to peer systems as well!) to then arrive at:
The king is dead, long live the king!
So it’s just an ads for tailscale ? Is it really relevant or just clickbait title ?
I mean it’s on the tailscale blog… There should really be a tag for corporate blog posts.
I recall finding a lot of tailscale blog content interesting in the past, which is part of what made this post so jarring.
“Centralization is bad. We will centralize and control the network instead to fix it”
One has to pay back one’s $100M Series B somehow.
It did take a very dark turn in the end, didn’t it?
I mean if only we had public, zero-trust internet infrastructure, the whole key-drop and NAT traversal thing could be hosted there. A .. gasp! tax-funded installation of headscale, e.g.
This whole complaining about rent seekers and then going about face and bragging about how everyone will need them and then they will be able to collect rent is really rubbing me the wrong way. Making me reconsider my usage of tailscale. At least there is headscale to fall back on.
Wow. Robin Hood to Darth Vader in one blog post.
So let’s look at the state of this proposed “new Internet”. It features:
A closed, undocumented, protocol.
A single open source implementation that had to reverse engineer the protocol, and which only exists at the mercy of Tailscale, because …
Tailscale owns 10 patents pertaining to this domain, which they will most certainly weaponise against any commercial implementations, and may weaponise against open source implementations at a time of their choosing.
Anyone seriously considering using this as the basis for a “new Internet” is deluding themselves. At some point this product is going to enshittify, and when it does, it’ll be messy in direct proportion to the number of otherwise open systems that have bought into it.
Could it work out? Yeah, if Tailscale publishes an RFC describing the protocol in full, and disclaims all of their related patent claims. Until that happens, this is just another proprietary tarpit waiting to trap the unwary.
More “the new AOL” than “the new Internet” ;-P
( Patent search here: https://patents.google.com/?assignee=tailscale&oq=tailscale )
The patent issue is the first one I have seen raised that seriously undermines their general “give away useful stuff” vibe; committing their parents to a defensive pool would go a long way, there.
I went searching for patents when I got to wondering … given that Tailscale has $2M in Series A funding and $100M in Series B funding, how are they planning to repay their investors, when anyone could just stand up a self-hosted headscale instance, and potentially charge for it?
I’d always assumed they were planning to make money from the big corporate players - self-hosting Headscale as a bigco means you need to do your own IDP integration & security audits, in addition to paid staff time to keep the thing running… not to mention that whoever championed “lets host it ourselves for free” is going to be in hot water every time the control plane has issues.
I’m very much reminded of the comment on the dropbox product launch - “Surely anyone who wanted this could simply write a short shell script to implement it”.
I was thinking more of competing businesses offering open source Tailscale implementations. “Amazon AWS Tailscale” or similar.
That has to be okay in order for Tailscale to become “the new Internet” as opposed to “just” a perfectly fine SaaS product.
I guess it still counts as Embrace, Extend and Extinguish when you’ve started the open-source thing you’re embracing. Smart people.
I found this a bit long-winded, and pretty grandiose if not downright creepy at times:
…but still well written and entertaining. I enjoyed it. It convinced me to at least kick Tailscale’s tires at some point.
The bigger problem, though, is the problem that all “we’re disrupting X” narratives have: they’re not interested in disruption for the sake of liberation, because liberation is not profitable. Another visionary treatise corralled at the last moment into the new walled garden. I guess I’d still prefer to rent e2ee p2p networking capability for my own hardware than to rent a machine from AWS…
Except that with tailscale there’s no walled garden. Everything about the architecture and the client is open source, and they fund work on an open source version of the server too
I don’t know that I’d say there is no walled garden. The server is proprietary and the open source re-implementation Headscale is not really comparable feature wise. Plus it’s made by reverse engineering the protocol by a Tailscale employee so its not like there is a standard or reference out there to implement. There also isn’t any existing web UI or admin console that works with Headscale that is comparable to the proprietary one by any stretch outside of the most basic features. I say this as someone who’s company ran Headscale as the primary for a long time before ultimately switching to the proprietary one out of necessity.
I’m showing my ignorance here, but why does the protocol have to be reverse-engineered? They purport to be based on Wireguard. Is there a higher-level proprietary protocol on top of that?
Wireguard is the substrate. All the key handling, route management, DNS management, ACLs, heartbeats, host and user management, proxies, etc. etc. etc. is the part that Tailscale adds on top. The protocol I mention isn’t the substrate protocol, but the protocol to talk to and cooperate with the server. Additionally, it’s all the work the server and related bits do because that’s where a large chunk of the Tailscale magic resides.
There’s a bit of a paradox in your question – what does a company sell if not some extra service on top. Especially when selling networking software service to people whose job description importantly includes, configure, run (and sometime write) FOSS software. They’ve set themselves up to only compete on innovation speed, network effects, and support, something like.
They tout, and many people celebrate (including blog posts on) NAT traversal, with failover to encrypted traffic forwarding. They also provide control through SSO and network traffic policy, well beyond wireguard, not impossible, but quite expensive and tedious to replicate with say ansible scripts configuring firewalls. In addition, Tailscale is building a bunch of services to provide stickiness, lock-in, indeed just more features & a reason to pay.
The most concise way to approximate, Wireguard (dead simple P2P network link setup) versus Tailscale, versus Headscale (in my quick search) is headscale’s feature list (and description of tailscale & design goal).
This post might be the better, simpler overview.
The article puts a lot of blame on the connectivity (NATs), and it’s completely justified given that the IPv4 address space has been exhausted for a long time and it’ll only be getting worse. But let’s address the elephant in the room (that the author so conveniently ignores and never mentions) — IPv6 solves connectivity issues between nodes, it’ll be a long time before we run out of 2^64 LANs with 2^64 devices each (given that the current industry standard is to allocate /64 for LANs but sometimes devices get their own /64 e.g. for VMs).
Internet could be a lot better if instead of commercial VPN companies we had public NAT64 gateways to reduce the cost of adoption (so that IPv6-only networks could communicate with an already heavily NATed IPv4 world). I’m honestly surprised this isn’t the thing yet since major cloud providers could definitely pull this off on an Internet-scale with all the resources available to them. In fact, Cloudflare does this to a certain extent with WARP but that runs a layer above instead of simply configuring the system to use DNS64 server (that Cloudflare already provides) and some infrastructure to retransmit packets (that they already have since it’s a public IP tunneling service).
The founder of Tailscale really really really hates IPv6. He has written a bunch about it, but boiling it down to the essential points, his argument is basically that since IPv6 doesn’t solve all problems, we should never use it. Ironically, Tailscale has become too big for the /10 they use, so they kinda undermined some of the interesting stuff they’re doing.
Er. I think that is over-stating the case. He has some well-founded, cogently-argued comments about how it was poorly designed and that that’s why it has not succeeded and replaced IPv4. And he’s right: it hasn’t.
Yes, some of the best most insightful commentary about it that I’ve seen anywhere.
That does not even resemble the argument as I understood it, no.
He is saying, AFAICS, “almost everyone uses IPv4 because IPv6 is better but not better enough so here are some useful ways we are improving IPv4.”
There was so much I agreed with in this article, but something just feels off about it. I do agree the Internet and the Web has many, many layers of unnecessary complexity. I do believe that most developers think they need the scale of FAANG when they do not and never will and probably shouldn’t even want to. And so on.
I’m just… not convinced Tailscale is the solution to all of the world’s problems.
Time for cloudflare WARP to eat tailscale, so we can watch them fight each other, instead of us.
Cloudfare WARP and Tailscale are really not interchangeable, although there are some common use cases.
Both do allow you to enroll company devices in a zero trust kind of setup.
Pretty sure CF could end up doing the same as TS, they just don’t seem to care.
I almost posted this here when I read it, but I decided it was too corporate at the end. Tailscale is cool and what they’re doing is cool, but this feels like it doesn’t add much to David Crawshaw’s old post linked to in the piece.
It was, it says, an internal company presentation which they later decided to share. What’s so wrong with that?
I don’t think there’s anything wrong with that, but here and on HN, the reaction was super-negative, and I think the reason is that it came across as too corporate with too little technical content. Usually Apenwarr writes about really technical stuff like how NAT tunneling works or why IPv6 failed, so it disappointed people’s expectations.
ISWYM. Fair point.
I thought the first 3/4s of this article was very insightful, but as others have pointed out the proposed solution is nothing of the sort (unless you’re a tailscale investor I guess).
I do think that the analysis of the nature of cloud centralisation (i.e. connectivity-centric) is on-point… but I don’t know if tailscale (or even something like it, but more open) is the solution.
It feels like what we really need is more tools in the style of Let’s Encrypt, which make a formerly hard/expensive problem easy and accessible to smaller actors on the web… but what exactly that looks like I’m not sure. There are lots of smart people working on indieweb tech at the moment, so perhaps something (or some things) will emerge from that milieu.
Is tailscale’s tech based on open standard ? Otherwise it’ll be just another broken monopoly ^^’
Yes, it’s based on wireguard.
Everything that makes Tailscale interesting is proprietary. Wireguard is just an implementation detail. There is a an open source clone (Headscale), albeit Headscale’s author works for Tailscale AFAIK. Mind you, Headscale just reverse engineered the API, it’s not an open standard or anything.
With headscale, you have both an open source server and client. The tailscale server is closed, but the clients are open.
The clients are open source except for the ones that aren’t, such as the macOS and the Windows GUI clients.
But that doesn’t invalidate anything about what I said, everything that makes Tailscale interesting, and a SPOF, and creates vendor lock-in is the server.
Look, I’ll make it easy. Tailscale provides a cool service. They are (and will continue to be) the only ones providing this exact service. They even have a monopoly on Headscale. Maybe you are ok with this, maybe you aren’t, but pretending this is not the case is dishonest. And them wanting to monopolize the internet is downright scary.
We adopted Tailscale in March 2020 as it became clear that we weren’t going to be heading to the office for a while. We’re a UAV company and the company doesn’t run on the normal SaaS “lots of servers” kind of model but rather we have a bunch of embedded hardware (edge computing, I guess they’re calling it now) that we work with. Tailscale provided Linux arm32 and aarch64 clients already when we adopted it. The result is that we could send hardware to team members, they could connect it to their house WiFi or wired Ethernet, and we could collaborate on the physical hardware. A junior engineer is working on talking to a telemetry radio and I can SSH into his unit to help them debug things.
As our kit evolved from prototypes on peoples’ kitchen tables to pre-production hardware that we were field testing we kept Tailscale installed on all of the edge devices. Our field team has a Starlink antenna on the roof of the trailer they use to transport the gear around. When an aircraft is powered up in the field I get a ping on Slack. If they have any issues I can very reliably SSH into the unit and debug things. If I’m feeling curious I can even connect ground station software on my laptop in a coffee shop to the aircraft network and watch the flight in real-time. If we’ve both got a good connection I can watch the video feed off the aircraft live.
It’s incredible. Absolutely incredible. And it Just Works. Over and over and over. It works so well that is has become a completely invisible part of our network infrastructure. No one has to do any administration or maintenance on it; auth is tied to our Office 365 accounts and other than running
apt updateonce in a while (which we have to do anyway since we distribute our own internal software through apt) it is completely maintenance-free. We disable key expiration on our edge devices because our field teams aren’t generally the kind of people you want to be SSHing into the hardware locally and trying to run commands withsudo; as a result, we’ve had old aircraft sit in a storage unit for months only to have someone pull it out, power it on, and I immediately have the ability to log into it and pull software updates from the comfort of my home office.I would love to see some competition in this space for sure. But as a warning for anyone who’s going to try to compete with them: make sure that you make whatever you come up with as simple and bullet-proof as Tailscale is.
Absolutely, Tailscale just works and it’s a great service. I am frankly surprised why nobody competes with them. In fact I am disappointed some open source or non-profit organization doesn’t compete with them.
We use zerotier at work, it seems pretty close? It also ‘just works’. It’s L2 rather than L3 which does make it slightly different I guess.
There’s some helpful comparisons by John Goerzen at Easily Accessing All Your Stuff with a Zero-Trust Mesh VPN. He has a particular aim so the pros and cons are focussed on that.
They don’t need to since headscale already exists and the clients are already open source
that’s false. A few of the clients are open source, but most of them aren’t.
You mean specifically the GUIs are not? Sure. An open source GUI project I could see value in
Yes and the clients are all open source and they fund work on an open source server as well
Enshittification strikes again.
We had an issue with tailscale where our nodes were having issues getting direct connections to each other; this resulted in a relay connection during early startup, and the relay was having issues that tailscale wasn’t reporting as an outag(maybe it was networking later between relay and node). We solved the issue by ensuring port forwarding was available earlier (via explicit firewall rules) which in turn helped establishing direct node connections and bypassing the faulty relay.
Just a story about the the fallback to using relays causing issues.
Kind of tired of reading articles promoting Tailscale.