I might use this if I weren’t worried about linux users other than my main having access to tailscale’s net interface.
Someday I’ll get around to learning nftables. Today is not that day
Actually now that I think about it, if I were to filter all outbound packets by user, how would I host services on the tailnet via a daemon user? Can I use conntrack with nftables? What about UDP? Would I even care about isolating tailnet internal services to another user? Maybe I should focus on not needing to worry about network access by other users so I can rely on loopback as a safe interface. Maybe I should go crazy with network namespaces
I hope this style of posting isn’t too bothersome to others. I’m not used to interacting like this vs lurking here discussing/rambling on discord with friends
Absolutely your comments are 100% on point and not bothersome at all!
I LOVE tailscale, but I’m a one man shop and just using it for my personal infrastructure so I wouldn’t even know to think about problems like the one you cite.
It’s been a long time since I played with nftables, but IIRC you can mark packets originating from different UIDs and then you can make decisions based on metainformation. Have not tested the theory though.
My understanding is with this feature, the SSH connection is handled by tailscaled before the Linux kernel ever sees them, so nftables wouldn’t be helpful. It can do that because the userspace daemon terminates the WireGuard connection.
Yes! This has been absolute magic. My ssh private key is securely in my vault instead of on lots of hard drives, waiting to be slurped up by some malicious npm package or whatever.
I’ve also tried this recently, but my only gripe was the frequency at which it asks for authentication. It’s kind of a pain when you use VS Code and use features like terminal split. It would ask me to authenticate every time a new shell was open, which I understand, but it’s not quite what I need for my use cases at work at least.
That’s odd I have the same setup and I don’t think I have that problem, it just asks on connection/reconnection. FWIW you can unlock 1P with Windows Hello or macOS touchid and then you can use those on ssh key prompts.
I would guess that my VS Code is configured to not spawn a new process for every terminal and yours is. There could probably be a lot of settings that would affect this, but that’s where I would look. It being intended 1Password behavior they all new processes need to re-auth with the agent totally makes sense to me.
So that seems to work a bit better, I’m starting to realize though that there is likely some sort of timeout that requires re-authentication after a while.
Tailscale has impressively convenient UX for a VPN, but assuming you’re not running a lot of high-value unauthenticated/never-updated services on your tailnet, the risks are medium-low. Enabling SSH and other service access with the same authentication moves the risk needle to “insane.”
For practical purposes, Tailscale operates a Kerberos KDC (or X.509 client-certificate CA if you prefer) that you don’t control, run by people you don’t know, who are now going to be targeted by all the hackers and LEOs in the world. The same effectively applies for the public SSO provider you have probably connected.
As far as I can tell, the Tailscale “KDC” effectively issues the equivalent of Kerberos tickets (probably more or less the thing they call a “node key” in the blog). Up to now, a ticket was only valid for the Tailscale equivalent of IKE viz. you could establish the equivalent of an IPsec Security Association which would allow you to connect to port 22 on a protected host, but go no further.
So any compromise would have limited effect, as your host is still protected by separate service-level authentication, such as a password or a separate SSH public key.
With deployment of Tailscale SSH, the ticket will also let you log in to the protected host. So a TS SSO or ticket compromise is now equivalent to a traditional Kerberos password or TGT compromise: the attacker has access to the TS “IKE” service to establish tunnels, and through the tunnel to the special SSH service which accepts TS tickets. Should Tailscale itself fall, the entire realm falls just as a Kerberos realm would.
This feature provides SSH access to users such as root, seemingly bypassing the ssh server itself. Sure an attacker would still need access to your tailnet, and the correct ACLs, but I agree that this feature will continue to raise the focus on tailscale as an attack vector.
Personally, I moved away from tailscale after I became uncomfortable with how much access a successful attack would gain.
Does tailscale have concept of users present on the devices, so that one can only allow one of the two users (foo, and bar) of computer-a to log-in to computer-b, i.e. bar@computer-a SSH into computer-b, and not foo@computer-a ? Otherwise giving whole computer computer-a access to log-in to computer-b is not quite great.
I was wondering the same thing. As far as I can tell, this is *@host-a -> user@host-b authentication. It sounds like anyone (or any process) on host-a effectively has the same SSH privileges as the Tailscale user whose token was used to authenticate host-a.
Do you have a specific loss in mind? My impression is that generally the design is that the client part is transparent for existing apps, and their customization happens mostly on the server side, where there is less variety in software.
You always have double encryption with ssh over wireguard (or any VPN). There are no client SSH keys because it knows what host you’re coming from and uses that information to match the ACL to grant access. As for the other technical details, their docs are pretty good and the code is open source.
I’d rather see them stabilize/productionize Tailscale kubernetes deployments, such as the sidecar, proxy, and subnet router.
I tried setting up the subnet router, but it just wouldn’t work, so I gave up trying to use it as a secure way to grant access cluster services access.
What did you have trouble with with the subnet router? Slightly different use-case but it works great in ECS. The most recent release also added a Docker entry point so you don’t need to roll your own.
I might use this if I weren’t worried about linux users other than my main having access to tailscale’s net interface.
Someday I’ll get around to learning nftables. Today is not that day
Actually now that I think about it, if I were to filter all outbound packets by user, how would I host services on the tailnet via a daemon user? Can I use conntrack with nftables? What about UDP? Would I even care about isolating tailnet internal services to another user? Maybe I should focus on not needing to worry about network access by other users so I can rely on loopback as a safe interface. Maybe I should go crazy with network namespaces
I hope this style of posting isn’t too bothersome to others. I’m not used to interacting like this vs lurking here discussing/rambling on discord with friends
Not at all! It’s on topic discussion and people do benefit from reading this kind of thought, in general.
Absolutely your comments are 100% on point and not bothersome at all!
I LOVE tailscale, but I’m a one man shop and just using it for my personal infrastructure so I wouldn’t even know to think about problems like the one you cite.
I replaced self-managed wireguard with tailscale. Much easier to handle and runs on Linux (arm64 and amd64) and OSX.
It’s been a long time since I played with nftables, but IIRC you can mark packets originating from different UIDs and then you can make decisions based on metainformation. Have not tested the theory though.
My understanding is with this feature, the SSH connection is handled by tailscaled before the Linux kernel ever sees them, so nftables wouldn’t be helpful. It can do that because the userspace daemon terminates the WireGuard connection.
I tried this out today and have to say it’s pretty cool. I never really had a good model for transporting ssh keys around when changing devices.
A nice alternative for recent versions of OpenSSH is to use FIDO/U2F hardware keys. Then your ssh keys live in your pocket!
If you use 1Password, the new v8 CLI will do it all for you(it runs an SSH agent backed by 1password storage).
Yes! This has been absolute magic. My ssh private key is securely in my vault instead of on lots of hard drives, waiting to be slurped up by some malicious npm package or whatever.
I’ve also tried this recently, but my only gripe was the frequency at which it asks for authentication. It’s kind of a pain when you use VS Code and use features like terminal split. It would ask me to authenticate every time a new shell was open, which I understand, but it’s not quite what I need for my use cases at work at least.
That’s odd I have the same setup and I don’t think I have that problem, it just asks on connection/reconnection. FWIW you can unlock 1P with Windows Hello or macOS touchid and then you can use those on ssh key prompts.
I just double checked and I do NOT get auth prompts for new terminals.
Interesting, I commented here and they replied that it was expected behavior https://1password.community/discussion/128261/1password-asking-for-permission-each-time
I would guess that my VS Code is configured to not spawn a new process for every terminal and yours is. There could probably be a lot of settings that would affect this, but that’s where I would look. It being intended 1Password behavior they all new processes need to re-auth with the agent totally makes sense to me.
Perhaps check this setting? https://stackoverflow.com/a/65440523/6211058
Awesome! I’ll try it today and report back
So that seems to work a bit better, I’m starting to realize though that there is likely some sort of timeout that requires re-authentication after a while.
On really? I need to dig into that.
docs: https://developer.1password.com/docs/ssh/agent/
in 1password developer preferences turn on the agent put the IdentityAgent line in ~/.ssh/config
new item a SSH key
if you generated the key, copy the public key into your ~/.authorized_keys file (or similar) on tootiehost
ssh tootiehost
Tailscale has impressively convenient UX for a VPN, but assuming you’re not running a lot of high-value unauthenticated/never-updated services on your tailnet, the risks are medium-low. Enabling SSH and other service access with the same authentication moves the risk needle to “insane.”
For practical purposes, Tailscale operates a Kerberos KDC (or X.509 client-certificate CA if you prefer) that you don’t control, run by people you don’t know, who are now going to be targeted by all the hackers and LEOs in the world. The same effectively applies for the public SSO provider you have probably connected.
It’s 11pm. Do you know where your krbtgt is?
How? The ssh server is only accessible within the tailnet. Nothing new is exposed to the world.
As far as I can tell, the Tailscale “KDC” effectively issues the equivalent of Kerberos tickets (probably more or less the thing they call a “node key” in the blog). Up to now, a ticket was only valid for the Tailscale equivalent of IKE viz. you could establish the equivalent of an IPsec Security Association which would allow you to connect to port 22 on a protected host, but go no further.
So any compromise would have limited effect, as your host is still protected by separate service-level authentication, such as a password or a separate SSH public key.
With deployment of Tailscale SSH, the ticket will also let you log in to the protected host. So a TS SSO or ticket compromise is now equivalent to a traditional Kerberos password or TGT compromise: the attacker has access to the TS “IKE” service to establish tunnels, and through the tunnel to the special SSH service which accepts TS tickets. Should Tailscale itself fall, the entire realm falls just as a Kerberos realm would.
This feature provides SSH access to users such as root, seemingly bypassing the ssh server itself. Sure an attacker would still need access to your tailnet, and the correct ACLs, but I agree that this feature will continue to raise the focus on tailscale as an attack vector.
Personally, I moved away from tailscale after I became uncomfortable with how much access a successful attack would gain.
Does tailscale have concept of users present on the devices, so that one can only allow one of the two users (
foo, andbar) ofcomputer-ato log-in tocomputer-b, i.e.bar@computer-aSSH intocomputer-b, and notfoo@computer-a? Otherwise giving whole computercomputer-aaccess to log-in tocomputer-bis not quite great.T hanks in advance!
I was wondering the same thing. As far as I can tell, this is
*@host-a->user@host-bauthentication. It sounds like anyone (or any process) onhost-aeffectively has the same SSH privileges as the Tailscale user whose token was used to authenticatehost-a.Aren’t some of the advantages of using tailscale ssh lost if they only offer a server, not a client? I would appreciate some more technical detail.
Do you have a specific loss in mind? My impression is that generally the design is that the client part is transparent for existing apps, and their customization happens mostly on the server side, where there is less variety in software.
You don’t need to deal with SSH keys unless I am massively misunderstanding the submission.
You always have double encryption with ssh over wireguard (or any VPN). There are no client SSH keys because it knows what host you’re coming from and uses that information to match the ACL to grant access. As for the other technical details, their docs are pretty good and the code is open source.
I’d rather see them stabilize/productionize Tailscale kubernetes deployments, such as the sidecar, proxy, and subnet router.
I tried setting up the subnet router, but it just wouldn’t work, so I gave up trying to use it as a secure way to grant access cluster services access.
What did you have trouble with with the subnet router? Slightly different use-case but it works great in ECS. The most recent release also added a Docker entry point so you don’t need to roll your own.
Sure, here’s the error in the logs:
Error 2022-06-15T01:39:22.460048251Z2022/06/15 01:39:22 ipnserver: conn2: ReadMsg: read unix /tmp/tailscaled.sock->@: read: connection reset by peer
That was deployed with ‘make subnet’ after running ‘make rbac’. Are you deploying in a different manner?
TS_ROUTES show up in Tailscale for the subnet router but they aren’t routable; I cannot connect to any pods or services on those subnets.
I’m using ECS, not Kubernetes, just using the Tailscale container running in Fargate. Don’t think I can help sorry!