Having the ability to remotely manage and monitor servers even when their main operating system becomes unresponsive is vital to enterprise IT administrators. All server manufacturers provide this functionality in firmware through a set of chips that run independent of the rest of the server and OS. These are known as baseboard management controllers (BMCs) and if they’re not secured properly, they can open the door to highly persistent and hard-to-detect rootkits.
Over the years, security researchers have found and demonstrated vulnerabilities in the BMC implementations of different server manufacturers and attackers have taken advantage of some of them. One recent example is iLOBleed, a malicious BMC implant found in the wild by an Iranian cybersecurity company that targets Hewlett Packard Enterprise (HPE) Gen8 and Gen9 servers, but this is not the only such attack found over the years.
According to an analysis by firmware security firm Eclypsium, 7,799 HPE iLO (HPE’s Integrated Lights-Out) server BMCs are exposed to the internet and most do not appear to be running the latest version of the firmware. When other vulnerabilities were found in the BMC implementation of Supermicro servers in 2019, more than 47,000 publicly exposed Supermicro BMCs from over 90 different countries were exposed. It’s safe to say that across all server vendors, the number of BMC interfaces that can be attacked from the internet is in the tens or hundreds of thousands.
“BMC vulnerabilities are also incredibly common and often overlooked when it comes to updates,” the Eclypsium researchers said in a new blog post following the iLOBleed reports. “Vulnerabilities and misconfigurations can be introduced early in the supply chain before an organization ever takes ownership of a server. Supply chain issues can still exist even after deployment due to vulnerable updates or if adversaries are able to compromise a vendor’s update process. Ultimately, this creates a challenge for enterprises in which there are many vulnerable systems, very high impacts in the case of an attack, and adversaries actively exploiting the devices in the wild.”
The iLOBleed implant
HPE’s iLO technology has existed in HPE servers for over 15 years. It’s implemented as an ARM chip that has its own dedicated network controller, RAM and flash storage. Its firmware includes a dedicated operating system that runs independently of the server’s main operating system. Like all BMCs, HPE iLO is essentially a small computer designed to control a larger computer — the server itself.
Administrators can access iLO through a web-based administration panel that’s served through the BMC’s dedicated network port, or via tools that talk with the BMC over the standardized Intelligent Platform Management Interface (IPMI) protocol. Admins can use iLO to turn the server on and off, tweak various hardware and firmware settings, access the system console, reinstall the main operating system by attaching a CD/DVD image remotely, monitoring hardware and software sensors and even deploy BIOS/UEFI updates.
The iLOBleed implant is suspected to be the creation of an advanced persistent threat (APT) group and has been used since at least 2020. It is believed to exploit known vulnerabilities such as CVE-2018-7078 and CVE-2018-7113 to inject new malicious modules into the iLO firmware that add disk wiping functionality.
Once installed, the rootkit also blocks attempts to upgrade the firmware and reports back that the newer version was installed successfully to trick administrators. However, there are ways to tell that the firmware was not upgraded. For example, the login screen in the latest available version should look slightly different. If it doesn’t, it means that the update was prevented, even if the firmware reports the latest version.
It’s also worth noting that infecting the iLO firmware is possible if an attacker gains root (administrator) privileges to the host operating system since this allows flashing the firmware. If the server’s iLO firmware has no known vulnerabilities, it is possible to downgrade the firmware to a vulnerable version. On Gen10 it is possible to prevent downgrade attacks by enabling a firmware setting, but this is not turned on by default and not possible on older generations.
“Attackers can abuse these [BMC] capabilities in a variety of ways,” the Eclypsium researchers said. “iLOBleed has demonstrated the ability to use the BMC to wipe the disks of a server. The attacker could just as easily steal data, install additional payloads, control the server in any way, or disable it entirely. It is also important to note that compromising physical servers can put not only workloads but entire clouds at risk.”
Past BMC attacks
In 2016, researchers from Microsoft documented the activities of an APT group dubbed PLATINUM that used Intel’s Active Management Technology (AMT) Serial-over-LAN (SOL) to set up a covert communication channel to transfer files. AMT is a component of Intel Management Engine (Intel ME), a BMC-like solution that exists in most Intel desktop and server CPUs. Most firewalls and network monitoring tools are not configured to inspect AMT SOL or IPMI traffic in general, allowing attackers like PLATINUM to evade detection.
In 2018, BleepingComputer reported attacks against Linux servers with a ransomware program called JungleSec which, based on reports from victims, was deployed through insecure IPMI interfaces using default manufacturer credentials.
In 2020, a security researcher showed how he could leverage insecure BMC interfaces on an organization’s Openstack cloud to take over virtualized servers during a penetration testing engagement.
“iLOBleed provides an incredibly clear case study not only on the importance of firmware security in BMCs, but for firmware security in general,” the Eclypsium researchers said. “Many organizations today have adopted concepts such as zero trust, which defines the need to independently assess and verify the security of every asset and action. Yet, in most cases, these ideas have not made their way to the most fundamental code of a device.”
Mitigating BMC attacks
The standard security practice for IPMI interfaces, whether built-in or added via expansion cards, is to not expose them directly to the internet or even the main corporate network. BMCs should be placed in their own isolated network segment that’s intended for management purposes. Access to this segment can be restricted by using VLANs, firewalls, VPNs and other similar security technologies.
Organizations should periodically check with their server manufacturers for updates to the BMC firmware and more generally track the CVEs discovered in the firmware of all their critical assets. The lack of firmware version tracking and vulnerability scanning creates a big blindspot on corporate networks and low-level rootkits like iLOBleed can provide attackers with a highly persistent and powerful foothold inside environments.
If BMC firmware offers the option to block the deployment of older firmware versions — downgrades — like in the case of HPE Gen10/iLO5 servers, this option should be turned on. Other firmware security features such as digital signature verification should also be enabled.
The default administrative credentials for BMC interfaces and admin panels should be changed and security features such as traffic encryption and authentication should always be enabled.
Finally, many BMCs have logging capabilities that allow changes to servers to be monitored and recorded through specifications such as Redfish and other XML interfaces. These logs should periodically be audited to detect any unauthorized changes.
Copyright © 2022 IDG Communications, Inc.
