My experience with various enterprise IPMI implementations was generally pretty awful. They vary widely in quality and are generally an incredibly rich target for exploits.
If I needed to run servers with any kind of sincere security requirements at all, I’d just turn it off, and demand that we buy hardware where it COULD be turned off.
Many HPE servers have a dedicated network ports for the iLO card but can also optionally share one of the regular network ports if needed. When in shared mode, you can indeed configure a VLAN tag for the management traffic, which can be different to the VLAN tag used by the host operating system normally.
Unfortunately, in the same way that chris explained that a any compromised host might be able to switch the device IPMI mode from dedicated to shared, using a VLAN for segregation can have a similar problem. If the compromised host adds a sub-interface with the tagged VLAN to their networking stack they now can gain network access to the entire IPMI VLAN.
In addition there are other annoyance with using a shared interface. Because the OS has control of the NIC it can reset the PHY. If the PHY is interrupted while, for example, you’re connected over Serial over LAN or a virtual KVM, you lose access. If you’re lucky, that’s temporary. If you’re really unlucky the OS can continually reset the PHY making IPMI access unusable. A malicious actor could abuse this to lock out someone from remote management.
That can’t happen when you use a dedicated interface for IPMI (other than explicit IPMI commands sent over /dev/ipmi0). Generally switching a BMC from dedicated mode to shared mode requires a BIOS/UEFI configuration change and a server reset.
(Speaking from experience with shared mode and the OS resetting the NIC. The malicious actor is merely a scenario I just dreamt up.)
Indeed, although I suspect in many cases these IPMI modules are already accessible from the compromised host over SMBus/SMIC or direct serial interfaces anyway - possibly even with more privileged access than over the network. That’s how iLOs and DRACs can have their network and user/group settings configured from the operating system.
The increased risk mostly isn’t to the compromised host’s own IPMI; as you note, that’s more or less under the control of the attacker once they compromise the host (although network access might allow password extraction attacks and so on). The big risk is to all of the other IPMIs on the IPMI VLAN, which would let an attacker compromise their hosts in turn. Even if an attacker doesn’t compromise the hosts, network access to an IPMI often allows all sorts of things you won’t like, such as discovering your IPMI management passwords and accounts (which are probably common across your fleet).
The L2 feature you are looking for is called a protected port. This should be available on any managed switch, but I’ll link to the cisco documentation:
In a previous life at a large hosting we used this feature on switch ports that were connected to servers for the purposes of using our managed backup services.
Some servers auto-switch to use a shared port if there is no link on the dedicated port when the server first powers on. In that case, to get the bmc to use the dedicated port you have to issue a cold reset to the bmc.
Disabling the bmc and using serial doesn’t necessarily work because disabling the bmc can also disable the serial port. I’ve no idea who came up with this design.
Most server manufacturers use the same BMC vendor (ASPEED), so it’s likely to be common code from the chip vendor(s). The whole thing is a Pandora’s Box that I wish I never opened.
I’d noticed this as well a few years ago and found it baffling why anyone would think this was a good idea. Glad to hear I’m not just paranoid and that other people share the concern.
How much can a damn ethernet port cost on multi-thousand dollar server?
My experience with various enterprise IPMI implementations was generally pretty awful. They vary widely in quality and are generally an incredibly rich target for exploits.
If I needed to run servers with any kind of sincere security requirements at all, I’d just turn it off, and demand that we buy hardware where it COULD be turned off.
Is there no mode that would share the physical network port but tag all IPMI traffic with a VLAN you configure?
Many HPE servers have a dedicated network ports for the iLO card but can also optionally share one of the regular network ports if needed. When in shared mode, you can indeed configure a VLAN tag for the management traffic, which can be different to the VLAN tag used by the host operating system normally.
Unfortunately, in the same way that chris explained that a any compromised host might be able to switch the device IPMI mode from dedicated to shared, using a VLAN for segregation can have a similar problem. If the compromised host adds a sub-interface with the tagged VLAN to their networking stack they now can gain network access to the entire IPMI VLAN.
In addition there are other annoyance with using a shared interface. Because the OS has control of the NIC it can reset the PHY. If the PHY is interrupted while, for example, you’re connected over Serial over LAN or a virtual KVM, you lose access. If you’re lucky, that’s temporary. If you’re really unlucky the OS can continually reset the PHY making IPMI access unusable. A malicious actor could abuse this to lock out someone from remote management.
That can’t happen when you use a dedicated interface for IPMI (other than explicit IPMI commands sent over /dev/ipmi0). Generally switching a BMC from dedicated mode to shared mode requires a BIOS/UEFI configuration change and a server reset.
(Speaking from experience with shared mode and the OS resetting the NIC. The malicious actor is merely a scenario I just dreamt up.)
Indeed, although I suspect in many cases these IPMI modules are already accessible from the compromised host over SMBus/SMIC or direct serial interfaces anyway - possibly even with more privileged access than over the network. That’s how iLOs and DRACs can have their network and user/group settings configured from the operating system.
The increased risk mostly isn’t to the compromised host’s own IPMI; as you note, that’s more or less under the control of the attacker once they compromise the host (although network access might allow password extraction attacks and so on). The big risk is to all of the other IPMIs on the IPMI VLAN, which would let an attacker compromise their hosts in turn. Even if an attacker doesn’t compromise the hosts, network access to an IPMI often allows all sorts of things you won’t like, such as discovering your IPMI management passwords and accounts (which are probably common across your fleet).
(I’m the author of the linked to article.)
The L2 feature you are looking for is called a protected port. This should be available on any managed switch, but I’ll link to the cisco documentation:
https://www.cisco.com/en/US/docs/switches/lan/catalyst3850/software/release/3.2_0_se/multibook/configuration_guide/b_consolidated_config_guide_3850_chapter_011101.html
In a previous life at a large hosting we used this feature on switch ports that were connected to servers for the purposes of using our managed backup services.
Other notes on ipmi:
Some servers auto-switch to use a shared port if there is no link on the dedicated port when the server first powers on. In that case, to get the bmc to use the dedicated port you have to issue a cold reset to the bmc.
Disabling the bmc and using serial doesn’t necessarily work because disabling the bmc can also disable the serial port. I’ve no idea who came up with this design.
Most server manufacturers use the same BMC vendor (ASPEED), so it’s likely to be common code from the chip vendor(s). The whole thing is a Pandora’s Box that I wish I never opened.
I’d noticed this as well a few years ago and found it baffling why anyone would think this was a good idea. Glad to hear I’m not just paranoid and that other people share the concern.
How much can a damn ethernet port cost on multi-thousand dollar server?