The emergence of the DevOps culture over the past several years has fundamentally changed software development, allowing companies to push code faster and to automatically scale the infrastructure needed to support new features and innovations. The increased push toward DevSecOps, which bakes security into the development and operations pipelines, is now changing the state of application security, but gaps still remain according to data from new industry reports.
Security testing ownership
A new report by the Enterprise Strategy Group (ESG), which surveyed 378 application developers and application security professionals in North America, found that many organizations continue to push code with known vulnerabilities into production despite viewing their own application security programs as solid.
Releasing vulnerable code is never good but doing so knowingly is better than doing it without knowing, since the decision usually involves some risk assessment, a plan to fix, and maybe temporary mitigations. Half of respondents said their organizations do this regularly and a third said they do it occasionally. The most often cited reasons were meeting a critical deadline, the vulnerabilities being low risk or the issues being discovered too late in the release cycle (45%).
The findings highlight why integrating security testing as early in the development process as possible is important, but also that releasing vulnerable code is not necessarily a sign of not having a good security program because this can happen for different reasons and no single type of security testing will catch all bugs. However, the report also found that many organizations are still in the process of expanding their application security programs, with only a third saying their programs cover more than three quarters of their codebase and a third saying their programs cover less than half.
Who takes responsibility for the decision of pushing vulnerable code into production can vary from organization to organization, the survey found. In 28% of organizations the decision is taken by the development manager together with a security analyst, in 24% by the development manager alone and in 21% by a security analyst.
This could actually be a sign of application security programs maturing, because DevSecOps is about moving security testing as early as possible in the development pipeline, whereas in the past security testing fell solely in the sphere of security teams who used to perform it after the product was complete.
In organizations where the development team does the security testing as a result of integrations into their processes and also consumes the results, it’s normal for the development manager to make decisions regarding which vulnerabilities are acceptable, either in collaboration with the security team or even inside their own organization if they have a security champion — a developer with application security knowledge and training — on their team. Such decisions, however, should still be taken based on policies put in place by the CISO organization, which is ultimately responsible for managing the entire company’s information security risk and can, for example, decide which applications are more exposed to attacks or contain more sensitive information that hackers could target. Those applications might have stricter rules in place when it comes to patching.
If the risk is not evaluated correctly, shipping code with known vulnerabilities can have serious consequences. Sixty percent of respondents admitted that their production applications were exploited through vulnerabilities listed in the OWASP Top-10 over the past 12 months. The OWASP Top-10 contains the most critical security risks to web applications and include problems like SQL injection, broken authentication, sensitive data exposure, broken access controls, security misconfigurations, the use of third-party components with known vulnerabilities and more. These are issues that should not generally be allowed to exist in production code.
According to ESG’s report, companies use a variety of application security testing tools: API security vulnerability (ASV) scanning (56%), infrastructure-as-code security tools to protect against misconfigurations (40%), static application security testing (SAST) tools (40%), software composition analysis (SCA) testing tools (38%), interactive application security testing (IAST) tools (38%), dynamic application security testing (DAST) tools (36%), plugins for integrated development environments (IDEs) that assist with security issue identification and resolution (29%), scanning tools for images used in containers, repositories and microservices (29%), fuzzing tools (16%) and container runtime configuration security tools (15%).
However, among the top challenges in using these tools, respondents listed developers lacking the knowledge to mitigate the identified issues (29%), developers not using tools the company invested in effectively (24%), security testing tools adding friction and slowing down development cycles (26%) and lack of integration between application security tools from different vendors (26%).
While almost 80% of organizations report that their security analysts are directly engaged with their developers by working directly to review features and code, by working with developers to do threat modelling or by participating in daily development scrum meetings, developers themselves don’t seem to get a lot of security training. This is why in only 19% of organizations the application security testing task is formally owned by individual developers and in 26% by development managers. A third of organizations still have this task assigned to dedicated security analysts and in another 29% it’s jointly owned by the development and security teams.
In a third of organizations less than half of developers are required to take formal security training and only 15% such training is required for all developers. Less than half of organizations require developers to engage in formal security training more than once a year, 16% expecting developers to self-educate and 20% only offering training when a developer joins the team.
Furthermore, even when training is provided or required, the effectiveness of such training is not properly tracked in most organizations. Only 40% of organizations track security issue introduction and continuous improvement metrics for development teams or individual developers.
Veracode, one of the application security vendors who sponsored the ESG research, recently launched the Veracode Security Labs Community Edition, an in-browser platform where developers can get free access to dozens of application security courses and containerized apps that they can exploit and patch for practice.
Open-source components targeted in supply chain attacks
Any mature application security program should also cover any open-source components and frameworks because these make up a large percentage of modern application code bases and carry risks of inherited vulnerabilities and supply chain attacks. Almost half of respondents in ESG’s survey said that open-source components make up over 50% of their code base and 8% said they account for two thirds of their code. Despite that, only 48% of organizations have invested in controls to deal with open-source vulnerabilities.
In its 2020 State of the Software Supply Chain report, open-source governance company Sonatype noted a 430% year-over-year growth in attacks targeting open-source software projects. These attacks are no longer passive where attackers exploit vulnerabilities after they’ve been publicly disclosed, but ones where attackers try to compromise and inject malware into upstream open-source projects whose code is then pulled by developers into their own applications.
In May, the GitHub security team issued a warning about a malware campaign dubbed Octopus Scanner that was backdooring NetBeans IDE projects. Malicious or compromised components have also been regularly distributed on package repositories like npm or PyPi.
The complex web of dependencies makes dealing with this issue difficult. In 2019, researchers from Darmstadt University analyzed the npm ecosystem, which is the primary source for JavaScript components. They found that any typical package loaded an average of 79 other third-party packages from 39 different maintainers. The top five packages on npm had a reach of between 134,774 and 166,086 other packages.
“When malicious code is deliberately and secretly injected upstream into open source projects, it is highly likely that no one knows the malware is there, except for the person that planted it,” Sonatype said in its report. “This approach allows adversaries to surreptitiously set traps upstream, and then carry out attacks downstream once the vulnerability has moved through the supply chain and into the wild.”
According to the company, between February 2015 and June 2019, 216 such “next-generation” supply chain attacks were reported, but from July 2019 to May 2020 an additional 929 attacks were documented, so this has become a very popular attack vector.
In terms of traditional attacks where hackers exploit known vulnerabilities in components, companies seem unprepared to respond quickly enough. In the case of the Apache Struts2 vulnerability that ultimately led to the Equifax breach in 2017, attackers started exploiting the vulnerability within 72 hours after it became known. More recently, a vulnerability reported in SaltStack was also exploited within three days after being announced, catching many companies unprepared.
A Sonatype survey of 679 software development professionals revealed that only 17% of organizations learn about open-source vulnerabilities within a day of public disclosure. A third learn within the first week and almost half after a week’s time. Furthermore, around half of organizations required more than a week to respond to a vulnerability after learning about it and half of those took more than a month.
Both the availability and consumption of open-source components is increasing with every passing year. The JavaScript community introduced over 500,000 new component releases over the past year pushing the npm directory to 1.3 million packages. Until May developers downloaded packages 86 billion times from npm, Sonatype projecting that by the end of the year the figure will reach 1 trillion downloads. It’s concerning that the University of Darmstadt research published last year revealed that nearly 40% of all npm packages contain or depend code with known vulnerabilities and that 66% vulnerabilities in npm packages remain unpatched.
In the Java ecosystem, developers downloaded 226 billion open-source software components from the Maven Central Repository in 2019, which was a 55% increase compared to 2018. Given the statistics seen in 2020, Sonatype estimates that Java components downloads will reach 376 billion this year. The company, which maintains the Central Repository and has deep insights into the data, reports that one in ten downloads was for a component with a known vulnerability.
A further analysis of 1,700 enterprise applications revealed that on average they contained 135 third-party software components, of which 90% were open source. Eleven percent of those open-source components had at least one vulnerability, but applications had on average 38 known vulnerabilities inherited from such components. It was also not uncommon to see applications assembled from 2,000 to 4,000 open-source components, highlighting the major role the open-source ecosystem plays in modern software development.
Similar component consumption trends were observed in the .NET ecosystem and the microservice ecosystem, with DockerHub receiving 2.2 container images over the past year and being on track to seeing 96 billion image pull requests by developers this year. Publicly reported supply chain attacks have involved malicious container images hosted on DockerHub and the possibility of having images with misconfigurations or vulnerabilities is also high.
Infrastructure-as-code needs remediation-as-code
The DevOps movement has fundamentally changed software development and made possible the new microservice architecture where traditional monolith applications are broken down into individually maintained services that run in their own containers. Applications no longer contain just the code necessary for their features, but also the configuration files that dictate and automate their deployment on cloud platforms, along with the resources they need. Under DevSecOps, development teams are not only responsible for writing secure code, but also deploying secure infrastructure.
In a new report, cloud security firm Accurics, which operates a platform that can detect vulnerable configurations in infrastructure-as-code templates and cloud deployments, 41% of organizations had hardcoded keys with privileges in their configurations that were used to provision computing resources, 89% deployments had resources provisioned and running with overly permissive identity and access management (IAM) policies and nearly all of them had misconfigured routing rules.
