The Future of Data Experience Management for Security and Observability

By Eric Newcomer, Principal Analyst, Intellyx

Security and observability data present different views of incidents and outages from the application and network sides. 

In a pointillist painting, the artist's careful placement of individual colored dots creates a coherent picture seen from a distance, which is one way to think about the potential result of combining these two different sources of data. 

For example, information about security incidents and alerts provides an external view from the network for a given IT environment. Observability tools provide an internal view from the application side. These two sides are often overlapping and complementary, and combined, represent a more coherent picture of an event. 

When the two sets of data points are combined into a single, coherent image, analysts get better information. It’s easier for them to spot the source of trouble or to more easily predict potential trouble. And they will have a better overall experience in diagnosing and remediating incidents.

Modern Data Challenges

Not too long ago, applications and network infrastructure were insufficiently instrumented. When something went wrong, there was often no data, or more often, incomplete data about what happened and why. 

It's far past the time when the challenge was to instrument everything. It's almost the other way around now. There's too much information gathered and stored. It's hard to see the forest for the trees, so much security and observability data is pouring in every day. 

The solution, however, isn't to go back to the old days and reduce the amount of alerting and logging data, but rather to find a better way to use and make sense of all that data. 

Fortunately, generative AI capabilities offer excellent mechanisms for summarizing, analyzing, pattern matching, and identifying what’s important and pertinent, and can provide a great next-level answer to this challenge.

A Combined View of Security and Observability

Security and observability monitoring tools have a lot in common. Security alerts typically originate outside the application, but they indicate what the application is doing that creates an incident. 

Much of the same information is available via observability tools. However, observability is primarily focused on capturing information about what’s happening inside an application. 

An enhanced or simplified combined view should ideally present existing data in a new light, identify additional or new patterns in the data, and more quickly surface a holistic view that better reflects reality than either single viewpoint could provide on its own.

Such a combination could surface things you might otherwise not see, such as:

• Patterns in the data, combining data points that otherwise are too small to pick out individually

• A constantly changing view that keeps pace with the constantly changing world, which helps forecast what the infrastructure and applications will do

• An improved data experience (DXM) for analysts, SOC leaders, platform, and observability teams that helps reduce cognitive load and burnout

How Does Auguria Help?

Auguria is a gen AI-based tool that optimizes security data, reduces SIEM costs, and minimizes alert noise by using a proprietary Security Knowledge Layer (SKL) to analyze, rank, and contextualize security and application telemetry, helping analysts identify threats faster.

The SKL combines data points from existing security and observability sources into a coherent picture that improves DXM and productivity for security analysts.

The SKL applies observability-style data analysis to traditional security telemetry. It functions as an AI-powered pipeline that treats security logs and events as continuous data streams for pattern analysis.

The SKL achieves this through:

• Data Optimization: Filtering out noise and prioritizing critical security data to reduce SIEM ingestion costs.

• An Explainability Graph: Visualizing threat data, allowing security teams to understand root causes and connect related, previously isolated events.

• Integration: Supporting integrations with any SIEM or data lake.

Security Platform Integration (EDR & XDR)

Auguria’s SKL inputs data from a variety of sources and applies its AI-based optimization, explainability graph, and data-wrangling techniques to it. 

The major security products for AI-driven noise reduction and prioritization include:

• CrowdStrike (Falcon®): For actionable incident response intelligence

• SentinelOne (Singularity™): For alert correlation

• Palo Alto Networks: For enriched firewall/network telemetry

• Microsoft Windows: For filtering high-volume event logs.

The major integrations with SIEM & data infrastructure products to lower data ingestion costs and bridge silos include:

• Splunk: A dedicated Auguria SKL App is available for Splunk Cloud/Enterprise.

• Snowflake & Databricks: Facilitate cost-effective, long-term storage and security, and data orchestration.

• AWS S3: Supports direct connection for storage and archiving.

Then the following interoperability standards enable data normalization and visualization:

• OCSF Schema: Automatically normalizes data from over 350 products into the Open Cybersecurity Schema Framework

• No-Code ETL: Provides a visual workflow editor for building data routes.

The SKL combines these security and observability data sources and applies the interoperability standards to produce the big picture outcome essential to delivering the next step in the evolution of better leveraging application and network instrumentation

The Intellyx Take

Collecting data on security incidents and application outages is no longer the problem. It’s about making sense of all the data without driving analysts crazy. 

Auguria leverages AI to abstract meaning from the data, or as we used to say, create meaningful information out of it all. 

They effectively operate as a layer above the noise, as it were. Carving out a catbird seat from which to more easily assemble all the data into a meaningful picture. Constructing a fire tower so you can see the forest. 

And they are not replacing existing systems, but rather taking the next logical step in the evolution of security and application monitoring. 

Filtering out the noise to produce a clear picture of what's going on helps everyone – security analysts can go home happy, confident they have the help they need to weed out the chaff. 

Management gets more out of its already expensive investment in Splunk and other tools. 

And organizations can improve the confidence with which they update their applications, less fearful of costly incidents, breaches, and outages. 

Copyright © Intellyx BV. Auguria is an Intellyx customer. Intellyx retains final editorial control of this article. No AI was used to write this article. Image source: Google Gemini.