Having servers listening to ssh on publicly accessible networks feels wrong, having worked in places where bastions / jumpboxes were the norm. This is useful if, say, vulnerabilities are found in the openssh server code. This also simplifies offboarding.
I’ve heard this argument before but it’s never really made sense to me for three reasons:
There have been vastly are more remote code execution vulnerabilities in the Linux network stack than in the OpenSSH server.
OpenSSH runs privilege separated. Compromising the process that runs on port 22 doesn’t give you anything except a stepping stone for talking via a tightly controlled interface to the parent.
Perimeter security is not really security. If I can compromise your jump box, I can easily then ssh to every other machine on the network, using automated tooling.
If OpenSSH makes a noticeable difference to your attack surface, I am very impressed by your baseline security. if you’re running a general-purpose network stack, from any OS, this is almost certainly not the case, irrespective of the other services you’re running.
Offboarding should be done by disabling the accounts (globally). If you’re relying on disabling access to a jump box to disable remote access, you’d better be absolutely sure that your perimeter security is 100% secure. Spoiler: it isn’t. Worse, the kind of malicious employee that you actually want to lock out will have trivially punched a hole in it before they left (for example, by setting up a task on another box that, on boot, sshs to a machine outside your firewall and does remote port forwarding. This can be done as any unprivileged user on any system in your network and then allows them to initiate arbitrary inbound connections to your network).
Having some sort of egress control from privileged environments is so important and yet so often ignored or actively resisted. (Folks in my experience really want access to NPM and GitHub from everywhere, even if they know intellectually it’s probably a bad idea.) Even simple audit logging of processing opening outbound connections to “new” hosts could save you from a lot of bad outcomes (i.e., the backdoor you mention just sitting around idle for months until someone decides to use it).
The main argument in favor of bastion/jumphost setups IMHO is simplifying network routing. You can keep private IPs for your internal network and use the bastion as a bridge, vs. having to put boxes that may have no other reason to receive direct IP traffic from the Internet at large into a publicly-routable network.
Regardless, I agree that the odds are that your biggest hole will be elsewhere (secrets management, unsecured database and internal API endpoints, automatic installation of untrusted libraries and/or containers, etc., etc.)
Exactly. I’ve never understood why people think their product’s http server is somehow innately more secure than ssh, a battle-tested tool used for mutually authenticated remote access. I think people just get spooked when they see all the failed login attempts from sniffers and whatnot like it’s somehow different than a crawler poking around port 80.
The places I’ve worked, we had one key for the jump box and another for the hosts behind it. The automated tooling only worked if you had both. We also had different keys for dev, staging, and prod. Our internal systems could not reach the outside world, they could only reach the gateway systems. It was more work, but I think it would beat the problem you describe above.
I agree. Note that if an organization uses certs for normal day-to-day operations, offboarding would simply be to not issue the daily cert for the given user. Older certs would have expired.
Having servers listening to ssh on publicly accessible networks feels wrong, having worked in places where bastions / jumpboxes were the norm. This is useful if, say, vulnerabilities are found in the openssh server code. This also simplifies offboarding.
I’ve heard this argument before but it’s never really made sense to me for three reasons:
If OpenSSH makes a noticeable difference to your attack surface, I am very impressed by your baseline security. if you’re running a general-purpose network stack, from any OS, this is almost certainly not the case, irrespective of the other services you’re running.
Offboarding should be done by disabling the accounts (globally). If you’re relying on disabling access to a jump box to disable remote access, you’d better be absolutely sure that your perimeter security is 100% secure. Spoiler: it isn’t. Worse, the kind of malicious employee that you actually want to lock out will have trivially punched a hole in it before they left (for example, by setting up a task on another box that, on boot, sshs to a machine outside your firewall and does remote port forwarding. This can be done as any unprivileged user on any system in your network and then allows them to initiate arbitrary inbound connections to your network).
Having some sort of egress control from privileged environments is so important and yet so often ignored or actively resisted. (Folks in my experience really want access to NPM and GitHub from everywhere, even if they know intellectually it’s probably a bad idea.) Even simple audit logging of processing opening outbound connections to “new” hosts could save you from a lot of bad outcomes (i.e., the backdoor you mention just sitting around idle for months until someone decides to use it).
The main argument in favor of bastion/jumphost setups IMHO is simplifying network routing. You can keep private IPs for your internal network and use the bastion as a bridge, vs. having to put boxes that may have no other reason to receive direct IP traffic from the Internet at large into a publicly-routable network.
Regardless, I agree that the odds are that your biggest hole will be elsewhere (secrets management, unsecured database and internal API endpoints, automatic installation of untrusted libraries and/or containers, etc., etc.)
Exactly. I’ve never understood why people think their product’s http server is somehow innately more secure than ssh, a battle-tested tool used for mutually authenticated remote access. I think people just get spooked when they see all the failed login attempts from sniffers and whatnot like it’s somehow different than a crawler poking around port 80.
The places I’ve worked, we had one key for the jump box and another for the hosts behind it. The automated tooling only worked if you had both. We also had different keys for dev, staging, and prod. Our internal systems could not reach the outside world, they could only reach the gateway systems. It was more work, but I think it would beat the problem you describe above.
I agree. Note that if an organization uses certs for normal day-to-day operations, offboarding would simply be to not issue the daily cert for the given user. Older certs would have expired.