Most AI Hallucinations Are Actually Infrastructure Failures in Disguise

You have worked across Splunk, Databricks, and now Cribl, giving you a rare view into enterprise data and observability over multiple technology cycles. What feels genuinely different about the observability challenge created by agentic AI?

In application observability, we’ve always dealt with complex, branching execution paths, tracing requests through microservices and correlating logs across distributed systems. In security operations, we’ve tackled high-volume, disparate data sources to detect threats across systems that, while complex, still behave in largely predictable ways. Agentic AI doesn’t reinvent either of those problems; it explodes them simultaneously.

Agents chain decisions, invoke external tools, and spawn sub-agents dynamically in ways that aren’t predetermined at write time. The execution graph isn’t just complex — it can be wildly different every time, and the volume of telemetry required to reconstruct what actually happened grows exponentially with each hop. Collecting all of the telemetry needed to make agentic systems observable has to be both easy to instrument and cost-effective to operate at scale. Otherwise enterprises will make the same tradeoff they always have and reduce visibility to control costs — exactly the wrong time to go blind.

Where are the biggest visibility gaps between what an AI agent is asked to do and what actually happens across the underlying systems?

Agents complete workflows that look successful on the surface while quietly making bad tool calls, hallucinating parameters, or drifting from the original goal. The consequences often surface hours later in another system.

The disconnect usually starts at interpretation: slight ambiguity in the original instruction gets resolved differently depending on what the agent retrieves from memory, what tools are available at that moment, or how a prior step shaped its context. You also have to instrument the downstream systems the agent depends on, because an agent can make a perfectly well-formed database call and still return a wrong answer if the underlying table was never refreshed because an ETL job failed silently upstream. The agent did exactly what it was told; the data it trusted was just stale.

Why are traditional observability tools often insufficient for monitoring agentic AI systems?

First, they were built around rigid, proprietary schemas and predictable data shapes. Agentic traces don’t conform to those expectations — they are voluminous, highly variable in structure, and include data types like full prompt text, model reasoning chains, and tool call payloads that legacy tools weren’t designed to ingest without significant transformation work. Open standards like OpenTelemetry’s GenAI semantic conventions are emerging to address this, but many legacy platforms weren’t built to take advantage of them.

Second, the scale and cost equation breaks down fast. A single agent handling production workloads can generate more telemetry in an hour than an entire application stack produced in a day, and most traditional platforms charge by volume ingested and stored. That creates an impossible tradeoff — either instrument everything and face runaway costs, or reduce visibility into the systems where you need it most.

When an AI agent fails, how can organizations determine whether the root cause was the model, the retrieval layer, an API dependency, or a downstream infrastructure problem?

Isolating root cause in an agentic failure is a process of elimination across layers. It only works if your telemetry spans all of them consistently. Latency and error signals on the model spans point to the LLM itself. Retrieval spans tell you whether the context returned was relevant or stale. Tool call spans expose failed API dependencies or permission errors.

You also need telemetry from the downstream systems the agent depends on, because an agent can make a perfectly well-formed call and still produce a wrong answer if the data it trusted was never refreshed. No single signal tells the whole story, which is why correlating across all of those layers in one place is what separates teams that can debug agentic failures quickly from those that are still guessing.

What types of telemetry data become most important when enterprises are trying to understand agent behavior at scale?

The foundation is still MELT — metrics, events, logs, and traces — but the weight shifts considerably when you move into agentic systems. Metrics tell you something is wrong and logs give you the raw record of what happened, but neither one can reconstruct the chain of reasoning an agent followed to arrive at a wrong answer. That’s where traces become the most critical signal.

In a traditional distributed system, a trace shows you how a request moved through services. In an agentic system, it shows you how a model reasoned through a problem, which tools it called, what context it retrieved, where it branched, and what it decided at each step. Without that trace, you’re looking at an output with no way to explain how you got there.

Are enterprises underestimating operational failures like bad retrievals, broken integrations, or missing context, compared to the attention paid to hallucinations?

Most hallucinations in production aren’t model failures; they are operational failures in disguise. Bad retrieval returns stale or incomplete context, and the model fills the gap with something plausible-sounding. A broken integration means the agent never got the data it needed to reason correctly in the first place. Missing context from a failed ETL job looks like a hallucination but is really an infrastructure problem.

When teams can’t see the difference, they default to what they can control — usually the model itself. They iterate on system prompts, add guardrails, and refine evaluation frameworks. Those techniques have real value, but they are treating a symptom rather than the cause. In my experience, the highest-leverage investment is almost always data quality and freshness in the retrieval layer. A well-grounded agent with clean, current data will outperform a heavily prompt-engineered one working from stale or incomplete context every time.

What should enterprise teams log, trace, or monitor to maintain accountability without overwhelming themselves with noise?

The foundation is still MELT, but accountability for agentic systems requires layering an eval framework on top of traditional telemetry. Metrics, events, and logs surface infrastructure failures and broken integrations that can corrupt agent behavior before the model responds; traces give you the reasoning chain to understand why it arrived at a particular answer. But neither one tells you whether the answer was actually correct — and that’s where LLM-as-judge evaluation fills the gap. The same AI capabilities that power agents can be used to assess them, scoring outputs for accuracy, groundedness, and task completion at a scale no human review process can match.

The key to avoiding noise is recognizing that not every agent workflow carries the same risk. Teams should move away from uniform monitoring toward what the industry is starting to call human-on-the-loop governance: define behavioral baselines and confidence thresholds upfront, let agents operate within them, and reserve human intervention for exceptions.

What mistakes are you seeing enterprises make when they try to retrofit existing monitoring practices onto agentic AI deployments?

The most common mistake is reaching for existing premium observability platforms to collect agent traces without thinking through the cost implications first. Agentic traces are large, variable, and voluminous in ways that traditional application telemetry is not, and most premium platforms price on ingestion and storage. The bill can become a serious problem before teams even realize what’s happening — and the typical response is to reduce what they collect. The result is the worst of both worlds: high cost and incomplete visibility.

The second mistake is applying the wrong mental model. Traditional infrastructure monitoring asks “is the system up?” and raises alerts when it isn’t. That question is necessary but nowhere near sufficient for agentic systems, where the infrastructure can be perfectly healthy while the agent is quietly making poor decisions, retrieving stale data, or drifting from its original goal.

The third mistake is treating instrumentation as an afterthought. The teams that scale agent deployments most successfully instrument early, establish baselines while traffic is low, and build their evaluation framework before they need it rather than after something goes wrong.

Will infrastructure visibility become a prerequisite for enterprise AI adoption in the same way cloud observability became essential for large-scale cloud migration?

The parallel to the cloud is extremely similar. The most important thing I tell customers when they move infrastructure to the cloud is that they aren’t lifting and shifting — they are deploying a whole new architecture with new capabilities. All of it needs to be instrumented and monitored in ways previous tools were likely not set up for. The enterprises that skipped that step ran into cascading failures they couldn’t diagnose, spiraling costs they couldn’t explain, and outages they couldn’t prevent. Observability became the foundation of trust that made cloud scale possible, and the same pattern is playing out now with agentic AI, moving faster and with higher stakes.

Agents will operate within defined boundaries of trust, and those boundaries will expand over time as observability, policy, and evaluation infrastructure matures. The boundaries themselves will never disappear, and that is not a failure state. It is how every consequential autonomous system in the world already works.