I’m a Penetration Tester and before that was a UNIX Sys Admin, one of the best things people have done is start to use bastion hosts. They are not perfect at all, but if you think it’s worse than what people have been doing than you have no idea how disjointed things are in large software deployments. Let’s break down this a little bit so maybe it will help other people:
Use a firewall instead of a bastion: Use both. But, really if you are actually monitoring your logs have you ever gotten more useful data from a firewall or detailed logging from SSH? Did they make a full handshake or stop after the available authentication types? These matter for a couple reasons, if they do get in because someone leaked keys on GitHub will your firewall logs tell you which keys?
Bastions aren’t easier to keep hardened, because you should be doing the same thing on all machines: I sort of agree with the fact that you should be running hardening scripts everywhere, but if you have any sort of mixed environment or if you have temporal machines (Docker hype) the scripts may not even be applicable.
Zero days work on all servers running SSH: I agree with this and I don’t think any sane person would have made this argument.
You use the same SSH keys for internal and external: I wish they was wrong, but I’ve never seen anyone do it any other way. You shouldn’t be using the same keys. But what they totally failed to get is that SSH has support for user/group matching, I can apply restrictions based on the host and prevent/log more heavily. For example, when Grsecurity was still around (I suppose you could do it with SELinux as well), we used to apply policies based on the user connecting in and additionally changed logging and access control based on that. Good luck with granular rules on all app servers.
Any non trivial tool will be able to quickly port scan the local subnets and jump along, so they don’t slow attackers: Maybe you should have talked to an attacker before writing this. I almost never port scan unless it’s my last resort. ARP tables, routes, configuration files, and connected ssh-agents are the name of the game. A proper bastion shouldn’t be giving these access parameters, seriously prevent shell access, chroot, and like vbernat said use the ProxyJump command and an ssh_config.
Rant over. There are some almost okay arguments and ideas in here, but they are not good for people to get in the habit of in the real world. One of the worst ideas in here is a common theme I see in the dev ops world, vendor lock-in and non-portable configurations. I think that AWS tools should be an augmentation to good security practice and all your security policies should be able to be dropped into any other platform and still do their due diligence.
Zero days work on all servers running SSH: I agree with this and I don’t think any sane person would have made this argument.
They run on all servers running vanilla SSH when the vulnerability isn’t the protocol itself. The difference between the two may be a lot of avoided hacks. I’m a big fan of obfuscations on top of good, security practices. :)
As much as I am totally okay with obfuscation on top of good security (I used to advocate very heavily for knockknock from Moxie Marlinspike) I do think there are incremental problems from obfuscation. I’ve had to pass off what I considered “hardened” infrastructure to new people as I was off-boarding and it was a nightmare to explain to them everything that existed and why, even with what at the time I considered “good” documentation. I think that it’s harder to maintain custom infrastructure and for the amount of technical debt to security pay-offs it may not always be worth it. I’d rather people use a bastion server that someone else can maintain. I think there are situations where I know people are capable of working together and running systems, and I say go for it, but they seem to be getting more rare.
That makes a lot of sense. Depending on company or team, it’s probably a good idea to be selective about what obfuscations one uses. I’d say at least some thing on how hosts or networks connect. Maybe well-maintained, unusual OS or apps for key services. Things there’s not lots of vulnerabilities for in the black hat markets. These by themselves block the majority of opportunists.
Someone really, really has to want a specific company taken down before those become useless. Those companies are usually toast anyway at that point if small to mid-size. The asymmetry between attacker and defender motivation/resources is just too high in attacker’s favor.
Real world perspective here, I do those threat simulations, and I often do them with insane deadlines. I have run into orgs that were actually truly prepared for a targeted attack maybe ~5 times. The thing that is missing here too is that 5 SSH servers mean that now I have 5 endpoints to spray against. Why do vuln research when 99% of all groups have weak password policies that we can use simple psychology, web scraping, and OSINT. So in the end I’d rather have 2FA and a bastion server and get that first engrained in peoples head as a standard.
Oh yeah, I’m with you on giving them straight-forward, quick stuff to establish a baseline. You could say I was just adding some high-security perspective for people who might find the methods useful or at least interesting. Some Lobsters like reading about it. So, I keep adding that stuff to threads.
The first argument seems to ignore the fact you can (and IMO should) use ProxyJump/ProxyCommand instead of agent forwarding. I don’t buy the other arguments either. Automation can help to turn every host into a bastion, but this is just increasing your surface of attack.
The author can’t have worked in an environment that’s heavily audited and subject to compensating controls.
Standards like PCI-DSS require bastion hosts for accessing nodes which process customer data and you won’t get a pass without them. (Admittedly there are many things in PCI-DSS which are just box-ticking, but you might be losing customers without that certification.)
Ok. This guide is sort of about Bastion hosts and sort of AWS specific. I’m talking about the former in my reply. I will start by saying I agree with the gist that at least small businesses might want to minimize security to the essentials. They really are sacrificing growth or cutting into low profits every time they buy something. The opportunity cost can be a bigger threat than average damage from hacks. Medium-sized companies don’t fit that here since we’re talking about 1-2 servers depending on if its AWS or in a traditional DMZ. A medium-sized company’s budget should barely notice 1-2 servers if they’re already all in the cloud.
re logging
Logging enabled on all your machines which the attackers will compromise equals they might delete the logs. Whether external service or another server, the best approach to logs is using append-only and/or write-only storage in a setup where one can do nothing else without special privileges. Additionally, having a copy of the logs on an extra server, even Bastion, can help if it’s more protected. They might be able to delete some but not others.
re easier to harded “why? they are machines, too”
and re prevent 0-day flaws
This shows a profound ignorance about Security 101. Rules independently invented in high-assurance software and by hackers like Daniel J. Bernstein are to (a) minimize code in general and (b) minimize privileged code. The more (a) you have, the more attack surface. The more of (b), the more damage can be done longer. Purpose-built hosts such as Bastions might need much less code to operate reducing attacks. Another thing I noticed was the performance hit of full, memory safety might be acceptable if an entire server is dedicated to one application. So, here’s a few options:
Use OpenBSD. You get hardening, mitigations, few people targeting it to begin with, and good docs. A book even. They’ve already done a lot of work on using it for this purpose.
If medium sized, use a separation kernel. Similar reasoning with less attack surface (4-12kloc in kernel) and fewer people targeting them. I say medium-sized since they’ll cost a pile of money, developer time, or both. Once set up, you can keep reusing them for new apps. Most support POSIX and/or Linux apps, too. Some micro- and separation-kernel vendors also have high availability options.
Take Poly2 approach to TCB reduction: delete code out of your kernels and apps you don’t need. Make it return without doing anything or log-and-alert. Gotta be frustrating for hackers. A friend of mine, “Tommy the Turtle,” used this on a fully-loaded, WinXP desktop with Firefox/NoScript, a security suite, office apps, and so on. He got it down to 650MB of HD space. I warned that he might have deleted something security-critical somewhere. He did have way less attack surface.
Attempt to use something like SAFEcode or Softbound+CETS on the kernel and/or critical apps. There will be a performance hit. The reward is knocking out the root causes attackers will hit. If it’s a dedicated server, you might not even notice it. Profile it for sure, though, in common situations.
re Bastions slow down attackers
This section is really configuration-specifics. Instead, I’ll point out that one with good, automated monitoring and profiles might spot an exfiltration attempt. One of my tricks was to have apps tag their data in headers or something with stuff attackers aren’t expecting. If they use their own code, the exfiltration will look different in a trivial-to-spot way. The rise of attacks reusing an apps own code probably defeated that method, though. Worked for a while… Finally, if an exfiltration or DDOS attack are detected, the security appliance can be used to either fail-safe or rate limit followed by notice to security administrator. Consider all this just extra value on the core value of an ultra-hardened appliance absorbing most attacks.
re increased attack surface due to extra server
The labor-intensive options might increase attack surface due to people slipping up. Just occasionally updating a configuration or install of a Linux/BSD server isn’t a lot of work. What author is describing is employees that are lazy as hell. That’s a different issue. Maybe they should use turn-key, security appliances or something if they’re not even going to update one, extra server. Even BS that appliance vendors pull sounds lower risk than security team not doing updates.
re the solution: don’t use Bastion hosts
The solution: use Bastion hosts the right way when you need them or they could be useful. That’s often the case. Maybe AWS doesn’t really. I don’t know anything about AWS in detail. I got no opinion on that except to say it might be wrong due to faulty premises I found in other arguments.
I’m a Penetration Tester and before that was a UNIX Sys Admin, one of the best things people have done is start to use bastion hosts. They are not perfect at all, but if you think it’s worse than what people have been doing than you have no idea how disjointed things are in large software deployments. Let’s break down this a little bit so maybe it will help other people:
Rant over. There are some almost okay arguments and ideas in here, but they are not good for people to get in the habit of in the real world. One of the worst ideas in here is a common theme I see in the dev ops world, vendor lock-in and non-portable configurations. I think that AWS tools should be an augmentation to good security practice and all your security policies should be able to be dropped into any other platform and still do their due diligence.
They run on all servers running vanilla SSH when the vulnerability isn’t the protocol itself. The difference between the two may be a lot of avoided hacks. I’m a big fan of obfuscations on top of good, security practices. :)
As much as I am totally okay with obfuscation on top of good security (I used to advocate very heavily for knockknock from Moxie Marlinspike) I do think there are incremental problems from obfuscation. I’ve had to pass off what I considered “hardened” infrastructure to new people as I was off-boarding and it was a nightmare to explain to them everything that existed and why, even with what at the time I considered “good” documentation. I think that it’s harder to maintain custom infrastructure and for the amount of technical debt to security pay-offs it may not always be worth it. I’d rather people use a bastion server that someone else can maintain. I think there are situations where I know people are capable of working together and running systems, and I say go for it, but they seem to be getting more rare.
That makes a lot of sense. Depending on company or team, it’s probably a good idea to be selective about what obfuscations one uses. I’d say at least some thing on how hosts or networks connect. Maybe well-maintained, unusual OS or apps for key services. Things there’s not lots of vulnerabilities for in the black hat markets. These by themselves block the majority of opportunists.
Someone really, really has to want a specific company taken down before those become useless. Those companies are usually toast anyway at that point if small to mid-size. The asymmetry between attacker and defender motivation/resources is just too high in attacker’s favor.
Real world perspective here, I do those threat simulations, and I often do them with insane deadlines. I have run into orgs that were actually truly prepared for a targeted attack maybe ~5 times. The thing that is missing here too is that 5 SSH servers mean that now I have 5 endpoints to spray against. Why do vuln research when 99% of all groups have weak password policies that we can use simple psychology, web scraping, and OSINT. So in the end I’d rather have 2FA and a bastion server and get that first engrained in peoples head as a standard.
Oh yeah, I’m with you on giving them straight-forward, quick stuff to establish a baseline. You could say I was just adding some high-security perspective for people who might find the methods useful or at least interesting. Some Lobsters like reading about it. So, I keep adding that stuff to threads.
The first argument seems to ignore the fact you can (and IMO should) use ProxyJump/ProxyCommand instead of agent forwarding. I don’t buy the other arguments either. Automation can help to turn every host into a bastion, but this is just increasing your surface of attack.
The author can’t have worked in an environment that’s heavily audited and subject to compensating controls.
Standards like PCI-DSS require bastion hosts for accessing nodes which process customer data and you won’t get a pass without them. (Admittedly there are many things in PCI-DSS which are just box-ticking, but you might be losing customers without that certification.)
Ok. This guide is sort of about Bastion hosts and sort of AWS specific. I’m talking about the former in my reply. I will start by saying I agree with the gist that at least small businesses might want to minimize security to the essentials. They really are sacrificing growth or cutting into low profits every time they buy something. The opportunity cost can be a bigger threat than average damage from hacks. Medium-sized companies don’t fit that here since we’re talking about 1-2 servers depending on if its AWS or in a traditional DMZ. A medium-sized company’s budget should barely notice 1-2 servers if they’re already all in the cloud.
re logging
Logging enabled on all your machines which the attackers will compromise equals they might delete the logs. Whether external service or another server, the best approach to logs is using append-only and/or write-only storage in a setup where one can do nothing else without special privileges. Additionally, having a copy of the logs on an extra server, even Bastion, can help if it’s more protected. They might be able to delete some but not others.
re easier to harded “why? they are machines, too” and re prevent 0-day flaws
This shows a profound ignorance about Security 101. Rules independently invented in high-assurance software and by hackers like Daniel J. Bernstein are to (a) minimize code in general and (b) minimize privileged code. The more (a) you have, the more attack surface. The more of (b), the more damage can be done longer. Purpose-built hosts such as Bastions might need much less code to operate reducing attacks. Another thing I noticed was the performance hit of full, memory safety might be acceptable if an entire server is dedicated to one application. So, here’s a few options:
Use OpenBSD. You get hardening, mitigations, few people targeting it to begin with, and good docs. A book even. They’ve already done a lot of work on using it for this purpose.
If medium sized, use a separation kernel. Similar reasoning with less attack surface (4-12kloc in kernel) and fewer people targeting them. I say medium-sized since they’ll cost a pile of money, developer time, or both. Once set up, you can keep reusing them for new apps. Most support POSIX and/or Linux apps, too. Some micro- and separation-kernel vendors also have high availability options.
Take Poly2 approach to TCB reduction: delete code out of your kernels and apps you don’t need. Make it return without doing anything or log-and-alert. Gotta be frustrating for hackers. A friend of mine, “Tommy the Turtle,” used this on a fully-loaded, WinXP desktop with Firefox/NoScript, a security suite, office apps, and so on. He got it down to 650MB of HD space. I warned that he might have deleted something security-critical somewhere. He did have way less attack surface.
Attempt to use something like SAFEcode or Softbound+CETS on the kernel and/or critical apps. There will be a performance hit. The reward is knocking out the root causes attackers will hit. If it’s a dedicated server, you might not even notice it. Profile it for sure, though, in common situations.
re Bastions slow down attackers
This section is really configuration-specifics. Instead, I’ll point out that one with good, automated monitoring and profiles might spot an exfiltration attempt. One of my tricks was to have apps tag their data in headers or something with stuff attackers aren’t expecting. If they use their own code, the exfiltration will look different in a trivial-to-spot way. The rise of attacks reusing an apps own code probably defeated that method, though. Worked for a while… Finally, if an exfiltration or DDOS attack are detected, the security appliance can be used to either fail-safe or rate limit followed by notice to security administrator. Consider all this just extra value on the core value of an ultra-hardened appliance absorbing most attacks.
re increased attack surface due to extra server
The labor-intensive options might increase attack surface due to people slipping up. Just occasionally updating a configuration or install of a Linux/BSD server isn’t a lot of work. What author is describing is employees that are lazy as hell. That’s a different issue. Maybe they should use turn-key, security appliances or something if they’re not even going to update one, extra server. Even BS that appliance vendors pull sounds lower risk than security team not doing updates.
re the solution: don’t use Bastion hosts
The solution: use Bastion hosts the right way when you need them or they could be useful. That’s often the case. Maybe AWS doesn’t really. I don’t know anything about AWS in detail. I got no opinion on that except to say it might be wrong due to faulty premises I found in other arguments.