Unknown attackers managed to break into the central code repository of the PHP project and add malicious code with the intention to insert a backdoor into the runtime that powers most websites on the internet. The hackers impersonated two high-profile PHP developers, but the code commits were not very subtle and were detected within hours when other developers reviewed them.
The incident didn’t have a widespread impact like the recent SolarWinds compromise or other supply chain attacks where backdoors made it into stable releases of software products and were pushed out to regular users. However, it made the PHP Group, the organization that maintains PHP, reconsider how its code infrastructure is run.
Open-source code sits at the core of critical internet services and accounts for most of the codebase in modern applications. The number of supply chain attacks against open-source projects, which are generally run by volunteers with limited resources, has sharply risen over the past few years and experts expect this attack vector to grow in popularity among attackers since it’s hard to defend against.
What happened with the PHP repository?
On Sunday, a code commit with the name “[skip-ci] Fix typo” was pushed to the php-src repository on git.php.net. The commit was made in the name of PHP founder and core developer Rasmus Lerdorf. Two hours later, another PHP contributor commented that the code had a typo and another inquired about what the code did. This attracted the attention of a developer with security background who commented: “This line executes PHP code from within the useragent HTTP header, if the string starts with ‘zerodium’.”
Essentially the code was a backdoor that would have allowed an attacker to execute arbitrary code on any web server running this Trojanized version of PHP by simply sending requests to it with a specifically crafted string in the HTTP header. Fortunately, the malicious code was spotted quickly thanks to good code review practices and only impacted the development branch of PHP 8.1 which is not expected to be released until November and not yet used in production.
What happened next was even more intriguing: The rogue commit made in Lerdorf’s name was quickly reverted by PHP core developer Nikita Popov, but shortly after, Popov’s reversion was itself reverted from what appeared to be his own account. This was actually the hacker impersonating Popov and reinstating the malicious code.
After a third core developer stepped in and reverted the rogue commit and the server was taken offline, Popov made an announcement on the PHP mailing list that read: “Yesterday (2021-03-28) two malicious commits were pushed to the php-src repo from the names of Rasmus Lerdorf and myself. We don’t yet know how exactly this happened, but everything points towards a compromise of the git.php.net server (rather than a compromise of an individual git account).”
According to Popov, following the attack, the team decided to discontinue its own git server and move development to GitHub because maintaining its own git infrastructure that used a home-grown karma system for access control was “an unnecessary security risk.” The PHP Group was already using GitHub as a mirror for its self-managed git repository, but it has now made it the primary repository and is in the process of migrating all contributors over to the PHP organization on GitHub, which requires two-factor authentication for accounts.
Was this a grey hat hacker’s proof-of-concept attack?
“This seems to have all the hallmarks of somebody trying to send a message,” Ilkka Turunen, field CTO at DevOps automation and open-source governance company Sonatype, tells CSO via email. “Although it definitely crosses the ethical hacking boundary by a long shot, it had some pretty overt code as well as typos […] The message, it seems, is there is unnecessary attack surface present due to outdated infrastructure like the self-hosted git server, and that can lead to disastrous consequences.”
According to many commentators on Twitter and Hacker News the commit was not subtle at all. It had [skip-ci] in the name, a string that causes many CI tools to skip their building and testing workflows. While skip-ci is a well-known convention in the developer community, its use would attract attention if it’s not often used by a project.
The malicious code also included the string “zerodium” as a trigger for the backdoor and the message “REMOVETHIS: sold to zerodium, mid 2017.” Zerodium is a company that buys zero-days exploits for a variety of operating systems and popular desktop and Web applications, including for PHP itself. According to its website, the company is offering up to $250,000 for a remote code execution vulnerability in PHP.
Chaouki Bekrar, Zerodium’s founder and CEO, said on Twitter that the company had nothing to do with the incident and described the hackers as trolls.
Whether they intended or not for the rogue commit to be easily discoverable, the level of access the attackers had could have been used to introduce a more subtle vulnerability in the code — like in the Linux backdoor attempt of 2003 — maybe by impersonating a developer who is not as visible as Lerdorf or Popov in the PHP community. In fact, Popov noted in his announcement that the project’s investigation continues and will include a review of all repositories for any additional “corruption.” He didn’t answer additional questions sent via email by CSO.
Validating the source of code commits
Git supports signing code commits with GPG keys and this feature is also present on GitHub for both code committed through the web-based interface and from a local machine. The goal of signing code commits is to ensure code originates from accounts or individuals that are trusted by the organization. It is a form of identity verification that relies on public key cryptography, and GitHub will attach a green “Verified” indicator to commits that are digitally signed.
While some PHP contributors already sign their commits, the project did not enforce signature verification as a requirement for all commits. This is not necessarily unusual when it comes to large collaborative projects as there has been a long-running debate in the open-source community on whether all commits need to be signed or just tags (code freezes that usually signify a release). However, following the attack, PHP developers are considering enforcing signed commits for at least the php-src repository.
“From a verification perspective, as the stewards of the [Maven] Central Repository, Sonatype has been requiring signing code as a method of verifying identity since the beginning, exactly due to the identity verification question,” Turunen says. “It by no means is a silver bullet to these sorts of issues, as Central also uses other methods to verify identity, but it should seriously be considered as a part of a complete breakfast. It adds a layer of friction and certainly won’t be without its engineering challenges and ultimately it’s for each project to decide the level of integrity they need, but as we are talking about critical digital infrastructure we would advocate strong identification of committers.”
Signing code commits will not prevent all impersonation attacks. For example, if an individual developer’s machine is compromised and has their private GPG key, the attacker could be able to sign malicious code locally and then commit it from their machine. There are ways to mitigate such risks by using additional security mechanisms, like storing keys on external hardware tokens or hardware security modules, but that can create complexity and open-source projects are generally anxious when it comes to potential hurdles that might discourage volunteer contributions. That said, big projects tend to only give commit privileges to a small number of regular and trusted contributors, not everyone who submits code to be reviewed for inclusion.
“This is a tough balance to strike — with the ease of access and contribution as core tenants of the open-source ethos weighed against the need for security and enforcement,” Turunen says. “There is no bulletproof security, and by nature open-source ecosystems will continue to be targets of supply chain attack attempts, as we have seen across all the major ecosystems time and again. We advocate any steps that can be taken to ensure and verify the identity of the committers for at least critical parts of the infrastructure and code should be one, as we have chosen to do with Maven Central.”
Self-hosted vs third-party service
Despite being widely used and relied on, many open-source projects suffer from a shortage of human and financial resources. Maintaining the infrastructure for a large development project and ensuring that it’s secure on top of that is a big undertaking that even large commercial companies often fail at. Many open-source projects are run by volunteers who have other full-time jobs. For example, when the critical Heartbleed bug was found in OpenSSL in 2014, people were surprised to learn that the OpenSSL project only had two full-time developers in charge of its huge and complex codebase that was used by billions of devices, servers and applications. Following that incident, the Linux Foundation in collaboration with major internet companies launched the Core Infrastructure Initiative (CII) to fund projects that are critical to the internet.
By moving their code infrastructure to a third-party service like GitHub, open-source projects can outsource server administration and security to a service provider that has paid full-time employees with the expertise to do those tasks reasonably well since the success of its business and reputation depends on it. Decisions like the one taken by the PHP Group to retire some of its self-hosted services might become inevitable for similar projects as they look to better protect their users from potential supply chain compromises.
“Hosting any infrastructure is a maintenance commitment that goes beyond just updating the software or services,” Turunen says. “We regularly see known vulnerabilities being exploited less than 24 hours after disclosure that puts immense pressure on self-hosted infrastructure to be kept up to date. In this case, the larger providers can offer an economy of scale in infrastructure and software maintenance smaller maintainer teams just cannot reach as it is not in their core competency. It needs to be said though, sometimes these decisions are made with philosophy and wider open-source ethos in mind. There is no general answer, but something every major project should definitely consider as an option going forward.”
Copyright © 2021 IDG Communications, Inc.
