Reducing MTTR is not resolving incidents faster

The metric everyone wants to reduce

Throughout this series on Modern IT Operations, we have seen that many organizations still face challenges that go far beyond the availability of tools. We discussed why so many operations still spend a good part of their time putting out fires, how the growth of observability ended up being accompanied by an explosion in alert volume, why detecting an incident is relatively simple, but building the context needed to identify its root cause remains one of the greatest challenges of operations, and how observability stopped being just a set of tools and became a new way of operating.

These reflections lead us naturally to one of the metrics most closely followed by organizations: MTTR (Mean Time to Recovery or Mean Time to Restore), the metric that represents the time needed to restore a service after an incident occurs. Among the main indicators used to assess operational efficiency, it is part of the daily routine of infrastructure, SRE, NOC, and platform engineering teams.

Not by chance, MTTR is also among the metrics tracked by modern reliability engineering frameworks, such as the DORA metrics, which use service recovery time as one of the parameters to assess organizations’ operational performance.

For many years, reducing MTTR seemed a natural consequence of technological evolution.

As new tools appeared, we believed incidents would be resolved faster and faster. We automated operational procedures, developed rollback mechanisms, improved deployment pipelines, expanded monitoring coverage, and invested in platforms able to identify anomalies seconds after they appear. In parallel, teams began to document procedures, create playbooks, and standardize incident response processes.

It was reasonable to imagine that all this evolution would produce significantly faster operations.

In many respects, it really did.

Today we can detect outages in a matter of seconds, automatically page the responsible teams, run recovery routines without human intervention, and track practically everything that happens inside a technology environment. If we compare these capabilities with those available fifteen or twenty years ago, the difference is impressive.

Even so, all it takes is to talk to professionals who live the day-to-day of a NOC, an SRE team, or an operations center to notice a quite familiar feeling. Despite all this evolution, some incidents still consume much more time than they should. Not because the technical solution is particularly complex, but because finding out which solution should be applied has become an ever greater challenge.

This perception began to catch my attention while following projects in organizations from different sectors. The tools changed, the architectures changed, the levels of operational maturity changed. The pattern, however, remained practically the same. When a major incident happened, most of the initial time was not spent fixing the problem, but trying to understand it.

At first glance, this difference seems merely a matter of interpretation. In practice, it completely changes how we see MTTR itself.

Whenever we analyze this metric, our natural tendency is to imagine that it measures the speed at which a team manages to restore a service. The word “recovery” immediately takes us to technical execution. We think about restarting applications, restoring servers, rolling back deployments, increasing compute capacity, or replacing faulty components.

That image, however, represents only a part of reality.

Before any of those actions happens, there is a silent stage, barely visible and often responsible for the largest share of the time consumed during an incident. It is the period in which the team is still trying to answer the most important question of all:

What is really happening?

The longest part of an incident rarely shows up in the metrics

Imagine a large e-commerce company on a high-demand day, during a promotional campaign that concentrates thousands of simultaneous visits and drives a relevant share of daily revenue.

Shortly after sales begin, monitoring starts to record an increase in the response time of some services. Almost simultaneously, the observability platform identifies growing latency in certain APIs. The Service Desk starts receiving tickets from customers who cannot complete their purchases. The team responsible for the databases notices an increase in processing queues, while infrastructure professionals observe unusual resource consumption in part of the cloud environment. At the same time, business indicators begin to show an unexpected drop in sales conversion.

None of this information is wrong. All of it represents real symptoms. The problem is that none of it explains, on its own, the origin of the incident.

At that moment, nobody can say whether all those signals are part of the same problem, whether multiple incidents are happening at the same time, or whether some alerts are merely consequences of a failure that started at another point in the architecture.

It is exactly at this moment that the real work of the operation begins.

Analysts start comparing dashboards, reviewing logs, checking recent changes, verifying dependencies between applications, reviewing deployments made in the last hours, and exchanging hypotheses with other teams. Little by little, what looked like a disorderly set of events starts to gain coherence. Some possibilities are discarded, others start to make more sense, and new evidence appears as earlier hypotheses stop explaining the observed behavior.

Only after this understanding begins to form can the team decide what the first technical action should be.

Curiously, when the incident ends, this entire process usually disappears behind a single number.

The executive report will say that MTTR was sixty, ninety, or one hundred twenty minutes. The metric will be compared with previous periods, targets will be discussed, and new action plans will be defined. Almost never, however, does that number reveal where the time was actually consumed.

This difference helps explain why organizations with similar tools, equally experienced professionals, and comparable levels of automation show such different results when facing similar incidents. In many cases, the speed of recovery is less related to how quickly a solution is executed than to how quickly the team manages to understand the problem it needs to solve.

This may be one of the most important transformations that IT operations have gone through in recent years. For a long time, reducing MTTR meant speeding up recovery procedures. Today, more and more, it means reducing the time needed to turn dozens or hundreds of seemingly disconnected signals into a consistent explanation of what really happened.

This change did not happen because teams lost efficiency. It happened because the very nature of technology environments changed. And it is precisely this transformation that we will begin to explore next.

Complexity moved elsewhere

For a long time, the complexity of IT operations was concentrated in the infrastructure itself. Large datacenters, specialized equipment, redundant links, and highly centralized environments required deep technical knowledge to keep running.

The operation was challenging because the technology was complex. Today, that scenario has changed significantly.

Infrastructure has become more standardized, more automated, and, in many respects, simpler to manage. Cloud computing reduced the effort to provision new environments. Containers standardized application deployment. Managed platforms took over activities that previously required specialized teams. Infrastructure as code brought repeatability to processes that historically depended on manual intervention.

It would be natural to imagine that, given this evolution, operations would also become simpler.

What happened, however, was something different. Complexity did not disappear. It migrated from the infrastructure to the relationships between the components that make up modern environments.

Instead of being concentrated inside a server, a network device, or a specific database, it came to exist in how hundreds of components need to work in a coordinated way. A seemingly simple service may depend on dozens of microservices, APIs, message queues, distributed databases, managed cloud services, and integrations with external vendors.

Each of these elements may operate correctly in isolation and yet the final user experience be completely compromised. This change is subtle, but it deeply alters how an incident needs to be investigated.

When cause and symptom stop coinciding

A few years ago, when an application was unavailable, it was relatively common for the origin of the problem to be close to the symptom itself. An overloaded server, a failing network interface, an unavailable database, or a configuration error normally explained a good part of the observed behavior.

Naturally there were exceptions, but in most situations the relationship between cause and effect was direct enough to quickly guide the investigation. In today’s architectures, that proximity has practically ceased to exist.

An increase in the response time perceived by the user may originate in a configuration change made hours earlier in a service with apparently no relation to that application.

A small increase in latency in an API can cause queues in another component that, in turn, degrades a third service and produces symptoms completely different from those generated by the original cause. In a few minutes, the incident stops being a localized event and starts to propagate across the whole architecture as a chain of successive effects.

In this context, the challenge is no longer identifying where the first alert appeared. The real challenge becomes understanding how all those events are related.

This may be one of the biggest differences between operating a traditional infrastructure and operating a modern distributed environment.

A scarcity of context, not of data

The volume of information is no longer the main problem.

Today, practically every organization can collect metrics, logs, traces, infrastructure events, application indicators, and information about user experience. In many environments there is, in fact, too much information.

What remains scarce is the ability to turn all of it into a consistent explanation of what is really happening. This distinction completely changes how we interpret an incident.

When different tools present dozens of simultaneous alerts, our first reaction is usually to find out which one “found” the problem first. In practice, however, no tool found the complete problem. Each one recorded only what it was able to observe from its own perspective.

Monitoring identifies availability changes. Observability evidences changes in behavior. APM tools show performance degradation. Logs record specific events. ITSM platforms report the impact perceived by users.

All of this information is correct. None of it, on its own, represents the incident.

That is precisely why more mature organizations stopped investing only in expanding data collection and started concentrating efforts on building context. The goal stopped being simply to see more components of the architecture. It became understanding how these components relate during an operational event.

This difference may seem merely conceptual, but it produces extremely concrete effects in the daily life of teams.

When context takes time to be built, different specialists start parallel investigations, each focused on the part of the environment they are responsible for. Infrastructure analyzes infrastructure. Database analyzes databases. Networks investigate networks. Development reviews applications. Cloud checks compute resources. Everyone works intensely, but not always on the same problem.

It is not a matter of lack of competence or lack of dedication. What is often missing is a shared view able to connect all these pieces before time keeps running.

It is at this moment that we begin to realize that MTTR depends not only on the quality of the people or the efficiency of the tools. It depends, above all, on the speed with which an organization can turn fragmented information into a collective understanding of the incident.

This may be the hardest activity in modern operations, not because data is missing, but because we have never produced so much data, spread across so many different places, to explain a single problem.

This is the new complexity operations live with every day. A complexity that is not hidden inside technology, but distributed among the countless relationships that technology itself has come to establish. It is precisely in this environment that the invisible time of MTTR starts to be consumed, long before the first corrective action is executed.

When each team sees only a part of the problem

There is a characteristic present in practically all modern operations that rarely receives the attention it deserves. The larger the organization, the higher the level of specialization of the teams responsible for sustaining it tends to be.

This evolution was necessary. After all, it would be unfeasible to expect a single professional to master, with the same depth, cloud infrastructure, networks, databases, Kubernetes, security, observability, application development, continuous integration, and user experience. Specialization made teams more efficient, raised technical quality, and allowed organizations to keep up with the growing technological sophistication of recent years.

It also produced an important side effect.

As each team came to know one part of the architecture deeply, it became increasingly difficult for anyone to understand the architecture as a whole.

This situation becomes particularly evident during a critical incident.

Infrastructure professionals start by checking CPU, memory, disk, virtual machine, and cluster consumption. Database specialists analyze connections, slow queries, locks, and processing queues. The application team reviews logs, recent deployments, and performance metrics. Networks investigate latency, packet loss, and link utilization. Security checks events that may indicate blocks or attacks, while the Service Desk keeps receiving new tickets and managers follow, under growing pressure, the impact on the business.

None of these teams is following a wrong path. In fact, all of them are doing exactly what they should be doing. The problem is that each one observes the incident through a different window.

Each group has its own tools, specific indicators, distinct languages, and priorities naturally aligned with its responsibilities. Individually, all these views make sense. Collectively, however, they rarely produce an immediate understanding of what is happening.

It is like trying to assemble a puzzle in which each person has only a few pieces and nobody knows the complete picture. The greater the number of pieces, the greater the effort needed to find out how they connect.

Perhaps that is precisely why major incidents tend to mobilize so many people at the same time. Not because all of them are needed to carry out the technical solution, but because someone needs to bring different interpretations together until a single explanation starts to make sense.

The real accelerator of MTTR

For many years, we got used to associating more mature operations with more automated operations. That relationship remains true, but perhaps it is no longer enough to explain why some organizations manage to respond to incidents far more efficiently than others.

When we look at companies recognized for operational excellence, it is common to imagine that the difference lies in more sophisticated technologies or significantly more experienced teams. Those factors certainly have an influence. However, when analyzing how these organizations conduct their incident response processes, a less obvious but probably more important pattern emerges.

They do not stand out only for the speed with which they execute technical procedures. They stand out, above all, for the speed with which they build a shared understanding of the incident.

This distinction is subtle, but it completely changes how we interpret MTTR.

Imagine two organizations with similar architectures, equivalent tools, and equally qualified teams. A practically identical incident occurs in both. At the end of the investigation, both reach the same technical conclusion and apply exactly the same fix. Even so, one restores the service in forty minutes, while the other takes two hours.

What explains this difference?

The answer is hardly in the last minutes of the incident, when the solution was finally applied. Most of the time, it will be concentrated in the first hour, when the teams were still trying to find out what the problem they needed to solve actually was.

This perception helps explain why some organizations manage to continuously reduce their operational indicators even without radical changes in infrastructure. The gain does not necessarily come from faster execution of technical procedures. It results from reducing the time spent eliminating incorrect hypotheses, correlating scattered information, understanding dependency relationships, and building a common view of the incident before making a decision.

It is like arriving at an unfamiliar intersection. Someone who knows the way spends only the time needed to reach the destination. Someone still trying to find out which road to take may spend much more time stopped than actually driving.

In IT operations, something similar happens.

Recovery itself usually represents only the final stage of a much larger process. Before it there is an invisible path, made up of investigation, interpretation, validation of evidence, elimination of hypotheses, and decision making. The faster this path is completed, the lower MTTR tends to be.

This change of perspective also helps explain why so many organizations continue to face difficulties even after significantly increasing their investments in observability.

Observability represents an extraordinary advance. It expands our ability to understand the behavior of systems, allowing us to analyze not only that something failed, but also how that failure evolved over time. However, seeing more does not automatically mean understanding more.

A modern environment can produce millions of metrics, billions of logs, and thousands of traces every day. None of this data, on its own, answers the question that mobilizes the entire operation during an incident: what is the most likely explanation for everything that is being observed?

It is precisely at this point that a new layer of operational maturity begins to emerge.

For many years we invested to expand our capacity to collect information. Today, the challenge becomes reducing the effort needed to connect that information and turn it into understanding.

This may be the main change that modern operations are going through. The problem stopped being a lack of visibility. It became an excess of fragmentation.

When each tool presents only a part of reality, the responsibility of connecting all those parts ends up falling on people. They are the ones who need to compare timestamps, identify dependency relationships, correlate events, interpret seemingly independent behaviors, and build a narrative able to explain the incident.

The more distributed the environment, the greater this cognitive effort tends to be.

And the greater this effort, the greater the invisible time that precedes any corrective action tends to be.

Perhaps that is precisely why so many organizations believe they are facing a speed problem when, in reality, they face a problem of understanding.

There is no shortage of competent professionals. There is no shortage of tools. There is no shortage of data.

What is often missing is a mechanism able to reduce the interval between the moment the first signals appear and the moment the operation can understand, with confidence, what is really happening.

It is in this interval that MTTR starts to be decided.

Long before recovery. Long before technical execution. Long before the first corrective action. It is there that mature operations begin to set themselves apart from the rest.

Understand first. Resolve later.

Throughout this series of articles, we have seen how IT operations have gone through a deep transformation.

In the first article, we discussed why so many teams keep putting out fires even when surrounded by increasingly sophisticated tools.

In the second, we reflected on the paradox of monitoring practically everything and still living with a growing volume of alerts.

This article adds one more piece to the same reflection by showing that reducing MTTR depends, above all, on the ability to quickly understand what is really happening.

Perhaps MTTR was never just a recovery metric.

More and more, it becomes an indicator of an organization’s ability to build understanding in the face of an incident.

This difference seems small. In practice, it completely changes the priorities of the operation.

If the biggest consumption of time happens before technical execution, reducing MTTR stops meaning only automating procedures or speeding up fixes. It starts to mean reducing the time needed to build understanding, connect scattered information, and give teams the conditions to make better decisions in the shortest possible interval.

In other words, the real accelerator of MTTR is not executing faster. It is understanding earlier.

In the coming years, we will keep producing ever larger volumes of operational data. We will keep expanding our monitoring and observability capacity. We will keep automating processes that today still depend on human intervention.

None of this, however, will eliminate the main challenge of modern operations: turning information into understanding.

Because, in the end, incidents were never just technical problems. They are, above all, problems of understanding.

That is precisely why the next big transformations in IT operations will probably not be related to the ability to generate more data, but to the ability to help teams understand, in a few seconds, what today takes long minutes and, in some cases, hours to discover.

It is exactly at this point that Artificial Intelligence begins to stop being merely a technological promise and take on a strategic role in modern operations.

But that is already the topic of the next article.

Food for thought

Modern operations do not suffer from a lack of data. They suffer from the difficulty of turning data into understanding.

The more distributed technology environments become, the greater the challenge of connecting information, identifying cause and effect relationships, and quickly understanding what is really happening.

Perhaps this is precisely the next frontier in the evolution of IT operations: not producing more information, but giving teams the context needed to make better decisions, faster and with more confidence.

Because reducing MTTR was never just a matter of restoring services faster. It is, above all, a matter of understanding incidents before they turn into impacts on the business.

About the author

Paulo Florêncio de Menezes is an entrepreneur, partner, and co-founder of companies such as Target Solutions, BiPTT, and ARGUS.

He holds a degree in Data Processing from the Federal University of Amazonas (UFAM), an MBA in Business Management from Fundação Dom Cabral (FDC), and a Professional Master’s in Economics from Fundação Getulio Vargas (FGV).

In his Master’s research, he developed the IBEOP-IA (Brazilian Index of Potential Occupational Exposure to Artificial Intelligence), investigating the potential exposure of Brazilian occupations to AI capabilities by combining occupational data, artificial intelligence patents, and natural language processing techniques.