AIOps
Your operation monitors everything. Why so many alerts?
In the article Why do so many IT operations keep putting out fires?, I shared a reflection on a recurring problem in IT operations. Over the years, we significantly expanded our ability to monitor environments, collect data, and track the behavior of infrastructure, applications, and services. Even so, many teams still struggle to turn all that information into fast, safe decisions.
This problem becomes even more evident when a relevant incident occurs.
Within a few minutes, different tools begin to record signs of what is happening. The infrastructure shows degradation. An application responds more slowly. The database increases its processing time. A network component loses performance. Users begin to report instability. New tickets are opened and, almost at the same time, alerts appear on different platforms.
Each of these pieces of information may be correct.
The problem is that they arrive in isolation.
Monitoring tools were developed to observe specific parts of the environment. Some track servers and network equipment. Others analyze applications, databases, cloud services, logs, or digital experience. Each platform offers a valuable view, but one limited to the universe it can see.
When a failure crosses several layers of the operation, these fragmented views start to generate a large number of events related to the same problem.
That is how an alert storm forms.
How an alert storm forms
A single incident can produce dozens or even hundreds of notifications. The server reports high resource utilization. The application records an increase in response time. The database points to slow connections. The digital experience tool identifies degradation in the user journey. The service management system receives tickets from different areas.
None of these signals is necessarily false. Most of the time, they all describe real effects of the same incident.
The difficulty lies in quickly understanding how they relate to one another.
For many years, the main challenge of operations was expanding monitoring coverage. There were important components without adequate tracking, critical applications with little visibility, and failures that were only noticed after users started complaining.
That scenario led organizations to invest in new tools, expand metric collection, and create alerts for practically every relevant element of the environment.
It was a necessary evolution.
Today, however, in many operations, the problem is no longer the absence of information. It is the excess.
The greater the monitoring coverage, the greater the volume of events generated during a failure tends to be. More distributed environments, with applications that depend on several services, hybrid infrastructures, and third-party integrations, amplify this effect even more.
The operation starts to see more, but not always to understand better.
That difference is important.
Monitoring is not the same as understanding the incident
Monitoring means observing behaviors, identifying deviations, and recording events. Understanding an incident requires relating that information, reconstructing the sequence of events, and distinguishing the origin of the problem from its consequences.
In practice, a good part of the teams’ effort is still concentrated on this stage.
When several alerts appear at the same time, professionals from different areas start their own analyses. The infrastructure team checks servers. The applications team consults logs. The database team evaluates queries and connections. The network is analyzed. Support organizes the tickets received.
Each group tries to confirm whether the problem is under its responsibility.
This work is necessary, but it often happens in parallel and with little coordination. Since each team observes only part of the environment, the incident can look different depending on the point of view of whoever is analyzing it.
Meanwhile, downtime keeps increasing.
In some cases, the team manages to restore the service quickly because someone recognizes an already known pattern. In others, the investigation depends on the experience of specific professionals who know the environment deeply and can connect scattered information.
That knowledge is valuable, but it should not be the only mechanism capable of making sense of alerts.
When interpretation depends exclusively on memory and individual experience, the operation becomes vulnerable. Team changes, environment growth, or increased complexity can quickly reduce the ability to respond.
Alert fatigue and loss of trust
Excess alerts also produces another effect well known to operational teams: the gradual loss of trust in notifications.
When an alert fires very often and rarely requires concrete action, it starts to be treated as noise. Thresholds are adjusted to reduce occurrences. Notifications are muted. Some events stop receiving the same attention.
This behavior is understandable. No team can maintain the same level of concentration when faced with hundreds of messages with no clear priority.
The risk arises when, amid repetitive alerts, the very signal that should have been noticed earlier appears.
Alert fatigue does not happen because professionals stopped caring about the operation. It arises when the system demands constant attention without offering enough context to guide that attention.
This is a point that deserves care.
Many organizations try to solve the problem by adjusting thresholds, disabling notifications, or reducing the number of rules. These measures may be necessary, especially when there are inadequate configurations or alerts with no operational usefulness.
But reducing the volume of notifications does not, by itself, solve the difficulty of understanding incidents.
An operation can receive fewer alerts and still not know how they relate to one another.
Correlating alerts to see an incident
The most important advance occurs when the organization stops analyzing events only individually and starts treating them as parts of the same occurrence.
This requires a view that crosses tools.
Imagine, for example, that a failure in an authentication service affects several applications. Each application may generate its own alerts. Users may open different tickets. Infrastructure components may show changes in behavior. The team may initially interpret all of these signals as independent problems.
When the events are analyzed together, the scenario changes.
The proximity in time, the common dependency among the applications, and the sequence of symptoms help reveal that there is a single origin for several occurrences.
The volume of information remains the same. What changes is how it is organized.
Instead of dozens of alerts competing for the team’s attention, the operation starts to see a main incident, its impacts, and the related components.
This type of correlation represents a relevant change in operational maturity.
The question stops being only “which alert was generated?” and becomes “what is this set of alerts showing?”.
It seems like a small difference, but it completely changes the response process.
When the team receives context, it can prioritize better, involve the right areas, and avoid unnecessary investigations. Related tickets can be grouped. Symptoms stop being treated as independent causes. Communication between teams becomes more objective.
The operation starts to spend less time organizing information and more time acting on the real problem.
This evolution does not mean replacing existing monitoring tools.
On the contrary.
Specialized platforms remain essential because they produce the signals that make it possible to understand the environment. Metrics, logs, events, tickets, and experience data are important parts of the analysis.
What is missing, in many cases, is a layer capable of connecting that information.
This layer does not need to eliminate the logic already built over the years. It should leverage existing investments and give more meaning to what the tools already produce.
AIOps, correlation, and operational context
This is where AIOps approaches begin to gain relevance.
The value is not simply in applying artificial intelligence to a large volume of data. It is in combining rules, history, topology, relationships, and behavior patterns to reduce the fragmentation of the operation.
In some scenarios, correlation can be based on known relationships and objective criteria. In others, artificial intelligence models help identify similarities, recurrences, and patterns that are hard to notice manually.
The best result usually comes from combining these approaches.
Deterministic rules remain important when the relationship between events is known. Artificial intelligence adds analytical capability in situations where the environment is more dynamic or the patterns are less evident.
The goal is not to replace the teams’ judgment.
It is to allow that judgment to happen with better information.
This care is important because the debate about artificial intelligence often creates unrealistic expectations. No technology completely eliminates the need for operational knowledge. Critical environments continue to require professionals able to assess risks, validate hypotheses, and decide the best way to act.
The difference is that these professionals should not need to start every incident by manually organizing hundreds of alerts.
When technology can reduce noise and present events within a common context, the team starts to act at a more strategic level.
Instead of searching for the problem amid a large number of notifications, it can focus its experience on evaluating the most likely causes, the impacts, and the actions needed to restore the service.
This transformation also changes how operational efficiency should be measured.
For a long time, increasing the number of monitored items was considered an indicator of progress. In part, it still is. An operation without adequate coverage remains exposed to failures that can go unnoticed.
But the number of monitored components does not, on its own, reveal response capability.
It is possible to monitor thousands of elements and still depend on long meetings to understand an incident.
It is possible to maintain detailed dashboards and still investigate symptoms in isolation.
It is possible to receive alerts in real time and still discover too late which one really mattered.
That is why more mature operations start to look at other signals.
They evaluate how long the team takes to understand the impact of an event. They check how many notifications are associated with the same incident. They identify how many tickets were opened for symptoms of a single failure. They analyze how much effort was spent before locating the likely cause.
These measures help show whether monitoring is, in fact, supporting decision making or merely increasing the volume of information available.
Monitor better, not just more
In the end, the question is not about choosing between monitoring more or monitoring less.
It is about monitoring better.
This means ensuring adequate coverage, improving the quality of alerts, and, above all, creating mechanisms to relate the different signals produced by the environment.
An efficient operation is not one that stops receiving alerts.
It is one that can quickly understand which alerts belong to the same problem, which represent only consequences, and which really require a decision.
The alert storm is not just a volume problem.
It is a context problem.
And as environments become more distributed and interdependent, the ability to build that context stops being a desirable improvement and becomes an essential competency for any operation that intends to respond with speed and safety.
The next text in the series, “Detecting an incident is easy. The hard part is building the context to find the root cause.”, will deepen this discussion. The proposal is to analyze why symptoms tend to appear in places different from the origin of the failure and how this influences the time to diagnose and recover services.
Food for thought
When looking at your operation, it is worth analyzing whether alerts help organize the response to incidents or only increase the amount of information that needs to be interpreted.
How many notifications does a single problem usually generate?
Can teams quickly perceive that different events are related?
Do alerts indicate priority and impact or only record that something left the expected behavior?
Is the team’s time used to resolve the incident or to find out which signals really deserve attention?
The answers help identify whether the current challenge lies in monitoring coverage or in the ability to turn scattered events into a clear view of what is happening.
About the author
Paulo Florêncio de Menezes is an entrepreneur, partner, and co-founder of companies such as Target Solutions, BiPTT, and ARGUS.
He holds a degree in Data Processing from the Federal University of Amazonas (UFAM), an MBA in Business Management from Fundação Dom Cabral (FDC), and a Professional Master’s in Economics from Fundação Getulio Vargas (FGV).
In his Master’s research, he developed the IBEOP-IA (Brazilian Index of Potential Occupational Exposure to Artificial Intelligence), investigating the potential exposure of Brazilian occupations to AI capabilities by combining occupational data, artificial intelligence patents, and natural language processing techniques.