Bogdan Rau Explains Why Analytics Agents Must Earn Trust Beyond Accuracy

An analytics agent returned a seemingly straightforward answer: 105,000 people across a group of clinic service areas could benefit from a diabetes management program. The arithmetic checked out. The population estimate did not.

The problem, Bogdan Rau explained at HealthDataCon 2026, was that the service areas overlapped. Some people had been counted more than once. Accounting for that overlap brought the estimate to approximately 81,000 in the example he presented.

That gap framed his session, “Right Answer, Wrong Decision: How Analytics Agents Earn Enterprise Trust.” For healthcare organizations bringing AI into analytics, Rau argued, technical accuracy is only the beginning. An answer also has to make sense in context and help someone act without being misled.

Rau, an epidemiologist by training whose experience spans healthcare policy, providers and payers, described a standard built around three questions: Is the answer correct? Is it coherent? Is it usable?

Correct math can answer the wrong question

The service-area example illustrates a distinction that can disappear behind a polished chart. Adding the figures for several clinics may be a valid calculation. Treating that sum as a count of distinct people requires another assumption: that the underlying populations do not overlap.

Someone familiar with how clinics define their service areas might recognize the problem immediately. An agent given tables and a request may not. Rau said the early agents in his example neither knew about the overlap nor asked about it.

His broader argument was that analytics should be understood as knowledge discovery. Summarizing existing material can be useful, but discovering a pattern that informs a new decision requires attention to the data, its meaning and the purpose of the analysis.

That distinction has a long research history. A foundational 1996 AI Magazine paper by Usama Fayyad, Gregory Piatetsky-Shapiro and Padhraic Smyth situated data mining within a broader knowledge-discovery process. Rau drew on that tradition to emphasize validity, novelty, usefulness and understandability.

In practical terms, the organization must establish what its question actually means before treating the resulting number as decision-ready. For a program expansion, the relevant population, geography, eligibility criteria and time period can matter as much as the calculation itself.

Approval is not meaningful oversight

Rau challenged a common reassurance about enterprise AI: that placing a human in the loop resolves the risk. His criticism focused on arrangements that require approval without giving the reviewer enough context, time or control to exercise judgment.

“I agree” has not necessarily meant “I understand,” he observed. Repeated approval prompts can become another administrative task, especially when incentives reward throughput and the person reviewing the output cannot readily assess its consequences.

That concern connects with anthropologist Madeleine Clare Elish’s work on crumple zones. Her 2019 paper describes how responsibility for failures in complex automated systems can fall on human operators whose control over those systems is limited.

The implication is not that human review has no value. It is that a signature or button click cannot establish that effective review occurred. Organizations need to specify what the person is expected to judge and whether the workflow makes that judgment possible.

For analytics, Rau proposed putting the “human in the right loop.” That means engaging the knowledge people bring from their working environment: which assumptions are unrealistic, which definitions are contested and which consequences a technically sound answer might overlook.

Give reviewers the ability to change outcomes

Rau organized meaningful oversight around four conditions: the capacity to judge, the cognitive space to do so, the authority to decide and the power to intervene. Each requires support beyond the individual reviewer. Together, they turn oversight from a nominal checkpoint into an opportunity to challenge an unsuitable result.

A July 2026 npj commentary by Davy van de Sande, Nicoleta Economou-Zavlanos and Michel E. van Genderen proposes those four interlocking conditions for medical AI oversight. The authors treat oversight as an organizational and technical capability that can be designed and assessed.

For Rau, that means developers must provide usable information, operational leaders must allow time for appraisal, and governance must support people who decline or escalate an output. A reviewer also needs a practical way to stop or redirect an action before its consequences spread.

He suggested adapting familiar healthcare communication practices. SBAR—situation, background, assessment and recommendation or request—offers a structure for presenting the context behind a proposed action. An agent could explain the problem, relevant background and assessment before asking for approval.

Capturing assumptions earlier is equally important. A clarifying question about the population or business objective may prevent a flawed analysis from advancing through several apparently successful steps. Approval becomes more meaningful when the reviewer understands what is about to happen and why their expertise is needed.

Test whether the answer is usable

Technical observability still matters. Analysts and developers need access to queries, tables, definitions and execution records. But Rau cautioned that showing a nontechnical leader the code does not necessarily help that person judge a recommendation.

Decision-makers need a traceable connection between source data, findings and proposed action. If an agent recommends expanding a senior program in particular ZIP codes, the reviewer should be able to inspect the population evidence and assumptions supporting that recommendation.

Rau highlighted two evaluation approaches. First, compare an agent’s narrative with the underlying numbers: do the data actually support the claims? Second, assess whether the chosen metrics fit the question. Measures such as emergency visits per 1,000 people can have different definitions across organizations and purposes.

These tests extend beyond checking that a query ran successfully. They ask whether the agent interpreted the result appropriately and whether the explanation contains enough evidence for its intended audience.

Governed data and agreed definitions provide an essential foundation, Rau said, but they cannot anticipate every user’s immediate objective. He advocated combining that foundation with explicit rules and code where appropriate, evaluation involving actual users, and participatory design before deployment.

The approach is consistent with NIST’s framework, which connects AI evaluation to context and calls for interpreting system outputs within that context. A useful evaluation therefore examines the decision environment as well as the model’s performance.

Earn autonomy one decision at a time

Rau compared increasing agent autonomy with graduated trust in medical training. The AAMC’s EPAs describe professional activities expected of learners entering residency. His analogy emphasized that independence should follow demonstrated capability in particular activities; the framework itself is not an AI validation standard.

Asked what would justify moving an agent from recommending an action to executing it, Rau said the decision’s importance and reversibility matter. Seeing the evidence behind a recommendation can increase confidence without establishing that the agent should be allowed to carry it out independently.

Asked about recovering from a bad decision, he emphasized reconstructing the context in which the mistake occurred. The relevant failure may involve assumptions shared between the person and the agent, rather than a software defect alone. He favored constraining the consequences of mistakes as organizations build experience.

His final test returned to the qualities of effective human analysts: understanding the business problem, recognizing the audience’s needs and knowing when more information is required. The best analysts, he said, could “read the room.” Enterprise trust depends on preserving that judgment as organizations automate more of the work that precedes a decision.