FDA wants to learn from clinical AI failures before the applications arrive
Richard Abramson and ARPA-H’s Haider Warraich described a joint effort to develop evidence methods alongside advanced AI, with simulation and real-world monitoring central to the work ahead.
The FDA wants a close view of how advanced clinical AI behaves while it is being developed, rather than waiting for a collection of finished products to reach its review teams. Dr. Richard Abramson, director of the agency’s Digital Health Center of Excellence, said collaboration with ARPA-H gives regulators a way to observe interactions with users, changing behavior and failure modes early enough to inform the evidence methods used to assess those systems.
“We don't want to wait until 30 Agentic AI systems show up at our door, and we have to figure out what to do with them,” Abramson said. He identified ARPA-H’s ADVOCATE cardiovascular-care program as an opportunity to learn from technology at the cutting edge. The objective is a more useful regulatory framework, not an assurance that participating products will receive authorization.
Abramson joined Dr. Haider Warraich, an ARPA-H program manager and cardiologist, in The Government’s Playbook for Clinical AI on November 17. Deloitte senior manager and research lead Anjalee Khemlani moderated the Ferrari Stage conversation. Their exchange put the development of regulatory science alongside the development of clinical software, rather than treating evidence requirements as a problem to confront only at the end.
A funding program can build more than a product
Warraich described ARPA-H’s distinctive role as taking risks on work that might not happen without government investment. He said his team focuses on high-capability, higher-risk AI that would be regulated as medical technology. That means thinking about implementation, evaluation and regulation as part of the same development problem.
“It's not just that we're making an investment, we're basically creating a future marketplace,” Warraich said. In his account, ADVOCATE brings technology developers together with health systems, regulators, evaluation teams and conversations with payers. Those groups have different responsibilities, but a product’s technical progress alone cannot resolve all the conditions needed for clinical use.
The program’s September award announcement describes a four-year effort to develop patient-facing cardiovascular AI, supervisory AI and a deployment plan. Its stated goal is an FDA-authorized system. The award is support for research and development toward that goal, not evidence that an autonomous clinical system has already met it.
Warraich said he often speaks with FDA colleagues at the beginning of an idea to understand the bottlenecks a program might address. The partnership can help identify where new regulatory-science tools are needed while teams build the underlying technology. That is different from funding a company in isolation and assuming the rest of the system will adapt later.
Simulation is a research priority, not a replacement already in place
One of Warraich’s clearest priorities was simulation. He described the possibility of a digital health system in which developers could examine clinical AI across scenarios before exposing patients to those interactions. He questioned whether repeating expensive clinical studies would be the ideal long-term approach for software that changes frequently, while saying clinical studies remain important for the first generation of technologies.
“The space that I'm really interested in moving forward is really around simulation,” he said. That was a direction for future work, rather than a declaration that simulations can now substitute for clinical evidence. The panel did not announce a validated simulator, a new evidentiary standard or a waiver of existing requirements.
His argument was about making evaluations more informative as well as less burdensome. A test environment could help researchers probe how an agent responds to unusual circumstances and distinguish the model’s behavior from the surrounding software. Whether those tests predict real-world performance remains a question the work itself must answer.
At the end of the discussion, Warraich said lessons from ADVOCATE should be shared beyond the teams receiving funding. He listed issues such as edge cases, in-silico data and separating deterministic systems from generative components. He said the goal is to disseminate those lessons so others can understand the pathway being developed.
Authorization helps establish trust, but does not finish the job
Khemlani pressed both officials on public trust in technology and government. Abramson argued that innovation creates value only when it is adopted, and adoption depends on trust. He said the regulator needs to be neutral, objective and clear about the science used to evaluate products.
“We need local governance,” Abramson said, alongside independent evaluation and continuing surveillance. He described FDA authorization as one important part of the foundation, while rejecting the idea that trust comes entirely from government. That division matters for hospitals: receiving an authorized product does not eliminate the operational responsibility to understand how it performs in a particular setting.
He also encouraged companies to engage with review teams early, especially when proposing a novel technology or an unfamiliar way to generate evidence. Discussions about real-world or synthetic data can help developers avoid spending time and money collecting evidence that will not answer the relevant questions. Early engagement offers a chance to improve a development plan; it does not predetermine a review decision.
The FDA’s generative-AI discussion paper illustrates that distinction. It asks for input on risk assessment, premarket evaluation and monitoring, and explicitly says it is not draft or final guidance and does not implement policy changes. Abramson presented discussion papers and early meetings as ways to develop better approaches with outside expertise.
Whose interests should a clinical agent serve?
Warraich identified alignment as another difficult problem. An agent optimized simply to satisfy a patient’s immediate request could give poor clinical advice, he said, using the example of a patient seeking antibiotics for a viral infection. Healthcare requires reconciling the interests and knowledge of patients, caregivers, clinicians, health systems and payers.
He also urged development for older people who are often missing from training data or may have sensory and cognitive difficulties. He said he wanted technology a 95-year-old could use, explaining that “those are the people who need clinical AI technologies the most.” The point was a design requirement, not merely a larger potential market: technology intended to extend specialty care has to be usable by people with the greatest need for that care.
Both speakers resisted the suggestion that progress in software alone solves the unknowns of biology. Abramson argued for computational methods working iteratively with traditional science. Warraich said the quality and purpose of future health datasets could matter as much as model advances, particularly if researchers want systems to move beyond correlation toward causal understanding.
The practical message for developers was to begin evidence planning while the technology is still taking shape. For health systems, the discussion pointed to a role in co-development, evaluation and monitoring rather than waiting for a regulator to resolve every implementation question. ADVOCATE’s significance in this account lies partly in what it can teach the field about that shared work, even before any participating system reaches a final regulatory decision.