The IT security community has been hard at work for the past week to investigate a critical and easy-to-exploit vulnerability in a hugely popular Java component called Log4j that’s present in millions of applications and products. Since the flaw was first disclosed and attackers started exploiting it, security researchers have discovered additional security issues in Log4j and various ways to bypass some of the proposed mitigations, leaving security teams scrambling for the correct ways to protect their applications, servers and networks.
Updating the affected component to the latest version — currently 2.16.0 for Java 8 and newer and 2.12.2 for Java 7 and older (yet to be released) — is the best way to mitigate the two flaws identified so far: CVE-2021-44228, also known as Log4Shell, which leads to remote code execution, and CVE-2021-45046, which can cause a denial-of-service condition.
Unfortunately, immediate patching is not viable in all scenarios. Packaged products from third-party vendors might contain vulnerable versions of the popular logging library that users can’t modify without updating the whole product, so they are dependent on vendors to release updates.
Business critical servers and applications might not be able to restart immediately or applications might run in containers for which new container images must be built. Like with most vulnerabilities, alternative mitigations are very useful for security teams, but it’s important to understand their limitations and the false sense of security some of them can induce.
Removing the JndiLookup class
This vulnerability is caused by the way Log4j uses a Java feature called JNDI (Java Naming and Directory Interface) that was designed to allow the loading of additional Java objects during runtime execution. JNDI can be used to load such objects from remote naming services over several protocols. The original exploit used LDAP (Lightweight Directory Access Protocol), which is the most common one, but others are also supported: DNS (Domain Name System), RMI (Remote Method Invocation), NDS (Novell Directory Services), NIS (Network Information Service), and CORBA (Common Object Request Broker Architecture).
One way to fix the vulnerability is to disable the use of JNDI message lookups, which is what Log4j 2.16.0 does. However, this can also be achieved by essentially ripping out the entire JndiLookup class, which implements this functionality, from an affected Log4j package. Since Java components are essentially ZIP archives, administrators can run the following command to modify and patch a vulnerable package instance:
zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
Hotpatching using a Java agent
Hotpatching is the process of deploying a patch to a running process without having to restart it. Java supports the on-the-fly modification of byte-code that’s already running in a Java Virtual Machine (JVM) through an instrumentation API and so-called Java agents. A Java agent is essentially a JAR (Java Archive) file that can be dynamically attached to a JVM during runtime.
In response to the Log4j vulnerabilities, the Corretto team from Amazon Web Services developed a Java agent that attempts to patch the lookup() method of all loaded org.apache.logging.log4j.core.lookup.JndiLookup instances to return the string “Patched JndiLookup::lookup()” instead of connecting to a remote server.
The agent is available on GitHub and can also be deployed as an ephemeral container to an existing Kubernetes pod to patch applications that are already running in other containers. Ephemeral containers are supported in Kubernetes v1.16 and later.
Exploiting the flaw itself to temporarily prevent exploitation
It’s possible to leverage the vulnerability itself on affected servers to make certain changes to the live system and application that would prevent further exploitation. Researchers from security firm Cybereason developed such an immunization exploit and researchers from LunaSec further improved it and hosted it on a live server as a public service.
One use case for something like this are all those third-party vendor products — packaged applications, embedded devices and appliances — that don’t have patches available yet or vulnerable products that have reached end-of-life and will never receive an official update. Using the exploit against itself could be a viable short-term solution.
It’s important to understand that using this has some significant caveats. First, the fix is transient because the changes the exploit makes apply to the running Java process and will be reverted when the JVM restarts. This means the immunization needs to be reapplied if the server is restarted. Second, while the method has been tested on various configurations and systems, it is possible that it won’t work on all and could result in crashes on some. Recovering from a crash might involve a server reboot, so it’s not a good idea to run this on critical systems where downtime is not an option.
Finally, using this against servers that you don’t have ownership of and you don’t control is likely to be illegal since it’s exploiting the vulnerability, even though for non-malicious purposes.
Identifying vulnerable systems
Before any response strategy is developed and any of the aforementioned mitigation paths can be used, organizations need to first identify all the applications and systems they have that could be vulnerable to Log4j exploits. That’s not easy to do, considering that each application can bundle its own instance of Log4j and could also load it dynamically as part of some other third-party dependency.
Several lists of Log4Shell vendor advisories are maintained by the security community and CERTs, but they’re likely to be incomplete. Unfortunately, until software bills of materials (SBOMs) become widely adopted by software developers, security teams will be faced with the time-consuming and error-prone task of identifying impacted systems in their organizations in response to every new vulnerability. It’s likely that more researchers and attackers will start digging for flaws in other widely used components in the wake of this vulnerability.
The security community responded fast by developing open-source tools to automate finding vulnerable servers and instances of the Log4j package. LunaSec’s log4shell tool can check .jar and .war files in a project directory and report if any are vulnerable. Support for the Log4j vulnerabilities have been added to other open-source and commercial vulnerability scanners and tools.
Inefficient Log4j mitigations
Since the first Log4j vulnerability was announced, several proposed mitigations have been shown to be ineffective and should no longer be relied upon.
Upgrading the Java version is not enough. The initial exploit didn’t work on Java versions newer than 6u212, 7u202, 8u192 or 11.0.2 because the default configuration in these versions prevent class loading via JNDI (Java Naming and Directory Interface) from remote servers. However, security researchers have since shown that attackers can build payloads that take advantage of classes in the application’s own classpath instead of remote ones, so this doesn’t prevent all attacks.
The formatMsgNoLookups flag doesn’t prevent all exploits. Early mitigation advice, including from the Log4j developers, was to set a property called formatMsgNoLookups in Log4j versions higher than 2.10 to true or an environment variable called LOG4J_FORMAT_MSG_NO_LOOKUPS. This was proven inefficient by the second denial-of-service vulnerability, which works even with this flag enabled. “There are still code paths in Log4j where message lookups could occur: known examples are applications that use Logger.printf(“%s”, userInput), or applications that use a custom message factory, where the resulting messages do not implement StringBuilderFormattable,” the developers said.
Disabling message lookups by modifying log statement formats. One proposed mitigation was to modify all log statement formats in the application from %m, %msg or %message to %m{nolookups}, %msg{nolookups} or %message{nolookups} to disable the message lookup feature. This doesn’t work for the same reasons as the formatMsgNoLookups flag, but is also dangerous because it creates a false sense of security. It’s very easy to miss updating a logging statement in a dependency or to reintroduce a vulnerable %m statement later without realizing. If you’ve already used this mitigation, you should not rely on it.
Using web application firewalls (WAFs) to filter malicious requests. While blocking the known exploits with a web application firewall is possible, it’s very hard to catch all possible exploit strings. Since the flaw was announced, researchers have demonstrated many ways to build nested and obfuscated payloads that would bypass proposed WAF filtering rules. That said, WAF and IPS vendors are constantly updating their Log4Shell signatures, so this can be used as an immediate and temporary response to block known exploits or as an additional defense layer on top of other mitigations. It’s worth noting that WAFs are normally used for publicly exposed assets, but there are internal exploit paths and scenarios to this vulnerability that would not go through a WAF to be blocked.
As the security community continues to investigate this vulnerability and its impact it’s possible that existing mitigation strategies will be changed or retracted. It’s therefore best to constantly check the Log4j project’s security page and advisories from organizations like CISA for any updates.
Copyright © 2021 IDG Communications, Inc.
