> ## Content Index
> Fetch the complete content index at: https://www.healthdatacon.io/llms.txt
> Use this file to discover other available public pages before exploring further.

# Clinicians should not become AI software validators, healthcare panel argues
- URL: https://www.healthdatacon.io/articles/clinical-ai-human-oversight-institutional-validation-hlth-2026/
- Published: 2026-11-17T21:50:00.000Z
- Updated: 2026-11-19T04:42:18.000Z
- Description: A debate over human oversight shifted responsibility toward institutions, local testing and continuous monitoring, while participants disagreed about how much risk is justified to improve access.
- Author: HDM Staff
- Tags: AI, Americas, HLTH

Putting a clinician in the loop is not a substitute for an institution validating and monitoring its AI, participants in a November 17 healthcare debate argued. Confronted with a hypothetical hospital that expects doctors to adjudicate sepsis alerts and help identify software failures, UCSF professor Julia Adler-Milstein said frontline clinicians should not be made responsible for validating the technology they use.

“Their job is to apply professional judgment to the set of information they have in front of them for the patient they're treating,” Adler-Milstein said. She placed responsibility for monitoring a tool’s performance on the institution deploying it. That moved the discussion from the reassurance of a human signature to whether the organization has built a workable system of oversight.

STAT health technology correspondent Mario Aguilar moderated [Human in the Loop or Human in the Way?](https://hlth.com/events/usa/agenda/2026/oxford-debate-human-in-the-loop-or-human-in-the-way?ref=healthdatacon.io) on the Ferrari Stage. Joining Adler-Milstein were GE HealthCare vice president of AI Danica Xiao, Galileo founder and CEO Dr. Tom X Lee, and ARPA-H chief medical officer Dr. Rafid Fadul. The participants tested different hypothetical clinical workflows, exposing disagreements about acceptable risk and strong concern that human review can become symbolic.

## Oversight has to be a capability, not an assumption

“Whether humans are good at oversight is an open question,” Adler-Milstein said near the beginning of the conversation. An expert intermediary can in principle identify where AI falls short, she explained, but that does not establish that a person can reliably perform the role in practice. The workload and type of output matter.

She drew a distinction between reviewing a narrowly focused output and checking a summary of more information than a human could reasonably reread. For the latter, requiring a person to sign off can promise more assurance than the workflow actually provides. A reviewer cannot verify omitted or distorted information simply by seeing a fluent summary.

Xiao made a related point about the meaning of a signature. “The clinician involvement should not be symbolic,” she said. A system that produces more material than a person has time to understand risks leaving that person as a liability shield, rather than giving them meaningful control. She argued that consequential, uncertain and irreversible decisions need evidence presented in a form the clinician can use.

## The comparator changes the risk calculation

Aguilar’s scenarios included psychiatric medication renewals, blood-pressure management, AI-directed urgent-care questioning and a sepsis prediction tool. These were discussion exercises, not announcements that the panel’s organizations had deployed those exact services. The differing examples helped show why a blanket judgment about autonomous AI can hide important clinical and operational distinctions.

Fadul argued that every medical decision carries some risk and that the comparison should include patients who cannot obtain care today. He pushed back against treating an ideal, universally available clinician as the baseline when some communities lack access. The benefit of a new service depends partly on what a patient would otherwise receive.

Adler-Milstein agreed that narrowing an existing access gap can justify considering new approaches, while saying a single standard of care remains the long-term goal. Xiao questioned whether directing underserved people toward AI while others retain human clinicians could reinforce a two-tier system. Their disagreement was about how to improve an unequal present without making inequality the permanent design.

Lee emphasized the specific setting and information available. “The clinical situation matters. The conditions matter, the data matter,” he said. A tool’s role in a care team, the consequence of an error and the patient’s alternatives all affect the assessment. A model’s performance in one scenario cannot settle the decision for a different workflow.

## Local validation and ongoing surveillance carry different work

The hypothetical sepsis deployment crystallized the distinction between clinical care and software assurance. Xiao called for local validation before use because a new patient population can change performance, followed by instrumentation for continuing monitoring. Neither a general clearance nor the presence of a clinician tells a hospital how well the product works for its own patients.

Fadul likewise argued that deployment has to be assessed in context. “Local validation is absolutely pertinent,” he said. His broader concern was that software can spread a flaw at the same scale that makes its benefits attractive. Observing the system after deployment is therefore part of the safety work, rather than an optional step after purchase.

He discussed ARPA-H’s [ADVOCATE program](https://arpa-h.gov/explore-funding/programs/advocate?ref=healthdatacon.io) and its planned supervisory layer for cardiovascular clinical AI. The program aims to develop and evaluate such systems, including monitoring their continuing behavior. That is a research and development effort, not proof that supervisory AI has already solved the problem across clinical settings.

The participants also acknowledged that monitoring broad conversational tools can be harder than monitoring a narrow prediction. Assessing whether an agent asked the right questions may lack a clear reference answer that experts consistently agree on. The difficulty of defining the outcome makes the evaluation problem more demanding, not less necessary.

## A second AI can share the first AI’s blind spots

Xiao supported the possibility of AI monitoring AI, while warning about correlated errors. If two systems were trained similarly, she said, they could make the same mistake and “form an echo chamber to agree on something incorrect.” A second model is not automatically an independent check merely because it is a separate process.

That raised a design question about how to obtain genuinely different evidence or perspectives. A supervisory model can add to an evaluation system, but the panel did not present it as a perfect replacement for institutional governance or clinical judgment. The proposed benefit needs to be tested against the failure modes it is intended to catch.

Fadul challenged the feasibility of expecting one clinician to review the output of systems operating at machine scale. “I'm judging the output of a thousand humans in a single day, which is absurd,” he said, describing the mismatch. His criticism was directed at an oversight model that expands the workload without expanding the reviewer’s capacity.

## The future role may span several workflows

Lee suggested that meaningful human oversight may eventually look different from a person sitting at a single gate in every transaction. “It's not necessarily one human in the middle,” he said, describing oversight across multiple similar loops with tools to validate actions. That could allow clinicians to focus on work where their involvement changes the outcome, while keeping accountability explicit.

He also noted the difficulty of judging quality in ambulatory care, where outcomes unfold over time and the available data can be incomplete. Access, quality and total cost are separate dimensions that both human-led and AI-assisted services struggle to measure well. The debate offered no simple score that combines them into a universal threshold for autonomy.

Aguilar closed without a single answer to how much human involvement every AI decision requires. The more useful distinction was between a person nominally assigned responsibility and a system that gives people the information, time and authority to exercise it. Hospitals considering clinical AI will need to specify both the clinician’s task and the institution’s validation work, rather than allowing the phrase human in the loop to stand in for either.