A hot discussion topic among CIOs is making organizations more efficient with AI and automation. It’s a mandate coming from your investors, board members, and CEO. But where do you start?
Many CIOs are tasked with reducing costs while mitigating risk for the business. Preventing disruptions to digital services stands out as a key area for improvement as it impacts the team, the customers, and the bottom line. And incidents often filter in through one central point: the Network Operations Center. However, NOCs are inundated with noise and manual tasks, and costly to staff with L1 and L2 responders only to have those responders escalate to subject matter experts (SMEs). To combat this, IT leaders are incorporating event-driven automation in their NOC.
Defining event-driven automation
Event-driven automation is kick-started at the event level. This means NOCs can trigger event-processing actions immediately as data comes in from trusted sources such as monitoring tools. Uses for event-driven automation in the NOC are numerous, ranging from adding critical context to an incident to better target the responder, to determining the next best action, to running diagnostics, and even declaring a major incident.
The beauty of event-driven automation is that it doesn’t start and stop at just reducing human intervention, it can eliminate it entirely for certain issues. Event-driven automation can be tightly integrated end-to-end, meaning organizations can rely on self-healing processes with machines as the first line of defense. This ensures teams are only involved when human skills and knowledge are necessary. And it can be fine-tuned over time to address more use cases.
Teams upskilled with event-driven automation
For many organizations, the NOC is responsible for incident detection, classification, and triaging issues to get services back online without SME involvement. With event-driven automation, incident response becomes more proactive, completing these steps and resolving problems before customers notice the impact. This is because much of the manual work NOCs do is relegated to automation. As a natural consequence, L1 NOC engineers have to spend less time watching dashboards and escalating incidents manually.
But these benefits don’t just help L1s. With automation pulling context and diagnostics, there’s more (and better) information for L2s to understand the root cause as well, further reducing SME effort. And there’s a trickle-down effect as more event-driven automation is applied. Many teams also see a positive impact from NOC modernization efforts, including simply a lower impact on their day-to-day work.
Site Reliability Engineering (SRE) and Platform Engineering teams: SRE and Platform Engineering teams can build out automation so that a human is never bothered with a task a machine can complete. Sometimes this takes the form of auto-remediation, completely solving incidents ahead of human involvement. Other times, this means creating runbooks that can help NOCs and other teams troubleshoot faster with the help of automated scripts. This helps responders come up to speed faster on issues, reducing MTTR and customer impact.
Major Incident Management (MIM): MIM teams should only receive confirmed customer-impacting incidents that cannot be auto-remediated and rely on the NOC to correctly categorize and immediately route these issues. When this happens, MIM teams need the incident populated with automated diagnostics and triage information to have the right context for immediate response. After the incident, MIM teams are often the drivers of continuous learning, ensuring that any further opportunities for automation are built back into the technical system to improve the process next time.
Engineering: Event-driven automation ensures that only the very minimum of engineering SMEs receive notifications for incidents that they need to work on with the right context already applied. And engineering teams can create auto-remediation for well-understood problems so they can preserve their time for more innovation.
Support: Auto-remediation resolves issues before customers notice the impact and without having to engage humans unnecessarily. And, with better data and the correct teams on the incident from the start, support will receive fewer cases from upset customers.
With teams across the organization able to see improvements from event-driven automation in the NOC, it sounds like an easy win, right? But you still need to do one thing: prove the value with concrete data.
Quantifying the value of event-driven automation
The key thing about proving the value of an initiative is aligning it to the company’s top goals. In this case, we can use ” reduce costs while mitigating risk.”
Risk comes in many forms, but one that organizations feel and can quantify at the bottom line is SLA penalties. As you apply more automation and your MTTR improves, how much less are you paying out in SLAs? Even one big incident that’s averted or reduced can have six-figure impacts.
Quantifying projected revenue and growth can be tricky, but it boils down to time. Time to resolve, actually. The faster an incident resolves, the less impact on revenue and the more time for innovation. For instance, an incident that previously took an hour to resolve and cost the company $100K in lost revenue might be caught quicker and auto-remediated, resulting in no loss of revenue.
The future of event-driven automation
In 2024, one thing is for sure: automation is a powerful tool when used to achieve concrete goals. Understanding new technology and assessing where it best fits into your operational processes is critical to avoid slipping into a laggard position, so start small and iterate quickly. Event-driven automation in your NOC, which drives consistent and incremental improvements across the entire organization, is a great place to start.
To learn more, visit us here.
