To improve the data experience, change the way you experience data

An Intellyx BrainBlog by Jason English | Part 1 of the Auguria Data Experience Series

Despite complaints of a softening job market in certain tech sectors, we are finding that security, platform and observability roles are still massively understaffed. There are more than half a million unfilled US jobs in cybersecurity alone in 2025, and 3.5 million globally, according to current reports. 

It seems we’ll never be able to train and hire enough great talent to effectively monitor our application estates for security and reliability, given the rate of growth of enterprise data that our systems generate.

A medium-sized enterprise might take in hundreds of millions of event logs in a single day, emanating from every gateway, cluster and node in a hybrid cloud infrastructure. Add in AI-driven application functionality, AI-generated code assets, and ‘autonomous’ agent traffic, and the whole estate can become ten times as chatty.

This results in a brutal data experience for analysts and operators, which we’ll never be able to hire our way out of as an industry. Perhaps it’s time to get ahead of the storm and change the way we experience data.

Caught in the storm of event data

Security pros and SREs are getting deluged with huge volumes of event data. These event data storms result in too many alerts and issues to process and prioritize. Teams wind up pulling together custom reports and participating in incessant incident calls and sev1 war rooms, which inevitably leads to employee burnout and turnover.

New high-volume data pipeline approaches are on the rise, accompanied by highly scalable back ends to receive them. But the data experience problem won’t be solved by pumping more logs and traces into our SIEM platforms and data lakes, so we can then sort through them to find out what mattered after the fact.

Even the best researchers want to give up and quit the data wrangling business entirely. Why try harder, if you are still unable to get to root causes before issues appear in production?

We need to abstract the data to make it usable in the here and now, so it’s easier for time-constrained experts to understand how that data will impact real-world outcomes.

Funnel all the data in, or filter it?

SOC and operations teams are only as effective as the data they run on. Organizations often pay premium vendor rates to ingest telemetry data for use within a SIEM or observability platform, including ingress, data processing, and storage costs. 

For critical business applications, we really might want to capture all of the data anyway. We’re not sure if any given log is not significant, therefore, we funnel it all in so there’s no chance we missed anything. Besides, slower cold storage is becoming cheaper (but not at the rate of data growth), so we can keep just about everything a little older in history, if we need it again. The most current data can be cached in expensive hot storage, in order to reduce search latency for recent anomalies.

As the data increases in volume, it slows down searches simply because there’s more of it in storage tiers, while also increasing operational expenses across the board.

So companies try another approach to bringing data into the pipe: applying pattern-matching algorithms and automation to attempt to process, identify, and filter incoming data to keep the most significant logs and traces, or only sample it at certain time intervals. Sure, the reduced data flow will significantly reduce the latency of searches and the resulting cost, but what if we miss something?

Does any of this squelching data wrangling even help analysts and engineers do their job better? It seems like we are looking at lots of water molecules, and missing the shape of the cloud.

A new way to experience data

Anyone with a basic grasp of science can look at a cloud and know it’s mostly made of air, but also immediately recognize it as a relevant grouping of water particles. It’s just like that with telemetry data. We’re missing the connections between seemingly unrelated data points that could provide us with meaning to understand emerging problems. 

When raw telemetry data comes in, it has a timestamp and a payload, but it has no valence at all. Similarly, a water molecule doesn’t know that it is attracted to other water molecules in a cloud, but it remains a part of the cloud until it happens to attract too many. Clouds are real, but they are also chaotic processes without a specific pattern.

Traditional SIEM treats logs as rows and time-series statistics. That paradigm won’t scale. Nor will tagging incoming data by keywords in an object store, nor will putting it into columns to search in a relational database.

Auguria takes a unique approach to this problem. Before incoming data is finished with its ETL cycle, they have a fast machine learning driven engine that recognizes the commonalities between logs, so they can be grouped together, sorted, or moved by thousands of relevance parameters.

While GenAI code assistants and support agents can be prompted to handle such tasks, an LLM isn’t specialized for this purpose. Auguria’s pre-load inference engine provides reflex-like recognition and association capabilities, and surfaces these meanings to users.

The Intellyx Take

Analysts and engineers need to immediately grasp the meaning of telemetry data flooding in from distributed cloud applications. Simply pattern-matching known CVEs and failure indicators won’t help them resolve emerging problems.

The emerging practice of DXM (data experience management) is all about changing our experience of working with data and improving our ability to act instead of react.

Instead of making your most valuable and constrained resources sift through data looking for evidence, let them look at data that is self-evidently relevant, because it is understood at a deeper level.

Next up in this series: Eric Newcomer digs deeper into how semantic understanding of telemetry data provides actionable intelligence to SOC and SRE work.

©2025 Intellyx B.V. Intellyx is editorially responsible for this document. At the time of writing, Auguria is an Intellyx customer. None of the other organizations mentioned here are Intellyx customers. No AI bots were used to write this content. Image Source: Adobe Image Express (Gemini 3.5)