Meltem Singh Offers Four Tests for Moving Healthcare AI Beyond Pilots

Meltem Singh urged healthcare leaders to make the expansion of an AI pilot contingent on four dimensions: organizational value, adoption feasibility, implementation complexity and risk. A technically successful demonstration, she told HealthDataCon 2026, still needs evidence that the organization can use the intelligence reliably and turn it into a meaningful result.

Singh, eFACHDM, senior vice president of AI strategy and enablement at Jarrard Inc., presented “Beyond the AI Pilot: Turning Healthcare Intelligence Into Operational Impact” during the Oct. 15 conference. Her framework connected investment decisions to the work surrounding a model: who acts, when information arrives, how exceptions are handled and what happens after special project support ends.

“Can this organization use this intelligence reliably enough to change an outcome?” she asked. The question placed responsibility on the organization as a whole. Her examples illustrated an evaluation approach; they were not presented as measured results from a named deployment.

The four dimensions interact. A valuable opportunity may demand more change than a team can absorb, while a technically simple capability can carry substantial risk when it influences a consequential decision. Singh framed the assessment as a way to surface those trade-offs before enthusiasm becomes a large commitment.

Define the decision and its owner

Singh began with organizational value. Before selecting a capability, leaders need to identify the decision it should improve, the person responsible for that decision and the point at which new information can affect action. An accurate recommendation arriving after a decision has been made has limited operational usefulness.

Her referral-intake example made the distinction concrete. An AI-generated summary is an output. Value emerges when a coordinator uses it to identify missing documentation, route a referral appropriately or prepare a case for review. The business case depends on that changed work and the result it produces.

“What behavior changes and what outcome should follow?” Singh asked. She also challenged claims that a tool would save time: leaders should explain how the released time would be used and which result would improve. A faster task may create capacity, but the organization still has to decide where that capacity goes.

That emphasis is consistent with AHRQ’s evaluation toolkit, which begins with project goals, stakeholder priorities and feasible measures. Its examples include nurses’ time spent on direct patient care and the proportion of alerts that lead to an intended action. These are ways to examine what technology changes in practice.

Test the work a pilot may be hiding

Singh described pilots as learning environments with protections that may disappear in production. Data can be cleaner, the scope narrower and a highly motivated champion more available. A project team may resolve problems manually, while users tolerate limitations because they expect them to be fixed before wider deployment.

“Pilots are actually extremely valuable from a learning perspective,” she said. Their value depends partly on being explicit about the conditions being tested and those the pilot temporarily avoids. Routine use exposes an organization to incomplete or delayed data, existing systems, competing responsibilities and ordinary staffing constraints.

She identified four transitions: controlled data to live data, a dedicated champion to shared ownership, a demonstration workflow to everyday work, and project attention to ongoing operations. Each transition creates questions that technical accuracy alone cannot answer. Who handles an unfamiliar document? Who takes over when the champion is unavailable? Which team supports users after launch?

The DECIDE-AI guideline provides relevant context for clinical decision-support systems. It emphasizes early evaluation in live clinical settings, including safety, clinical performance and human factors. That reporting guidance supports examining use in context; it does not validate Singh’s framework or establish that any particular product is ready to scale.

Treat adoption as evidence about the workflow

Adoption feasibility was Singh’s second dimension. A tool embedded in an existing task creates a different burden from one that changes roles, decision rights, timing or coordination across departments. A substantial change may be worthwhile, she said, but it requires the sponsorship and implementation capacity to support it.

“Training can explain how to use a tool,” she said. It cannot, in her account, resolve missing capacity, unclear ownership or a poor fit with the work. Leaders therefore need to observe where users hesitate, verify an output, repeat a step or encounter a conflict between an AI recommendation and professional judgment.

AHRQ’s workflow toolkit similarly treats clinical and administrative workflow as central to health IT implementation. Singh extended that concern to incentives and responsibilities: people need both a practical way and a reason to use the capability within the work they are accountable for doing.

During questions, an attendee asked how to interpret ignored recommendations and overrides. Singh emphasized understanding why people behave as they do and tailoring the response accordingly. The practical implication is that low use needs investigation. It can reflect a learning need, an unsuitable workflow or a legitimate concern about the output; usage counts alone cannot distinguish those explanations.

Match implementation and controls to the use

Implementation complexity includes the less visible work behind a model and interface. Singh pointed to data access, integration, exception handling, provenance, monitoring, support and governance. A pilot that depends on manual assistance requires a plan for who will continue that work and with what capacity.

Responsibilities may be distributed across model, data, workflow and governance teams. Those teams must coordinate their decisions, she said, particularly when a solution crosses facilities or systems. Two products with similar outputs can impose substantially different operational burdens on the same organization.

Risk, the fourth dimension, depends on the setting and authority assigned to the capability. “A summary that supports a human review carries a very different risk profile from a summary that automatically determines the next action,” Singh said. She included clinical, operational, privacy, security and reputational risks in that assessment.

Her argument was for controls proportionate to consequences. Excessive process around a low-consequence task can create friction, while insufficient oversight of a consequential patient-facing use can leave material risks unaddressed. The voluntary NIST framework likewise places risk management across the AI lifecycle and uses contextual understanding to inform decisions about whether to proceed. Singh’s four dimensions were a discussion structure, not a numerical certification or substitute for applicable requirements.

Write the conditions for scaling, redesigning or stopping

To make the assessment actionable, Singh proposed a “pilot contract” covering the hypothesis, required evidence, accountable owner and exit criteria. It would document what the organization needed to learn and the conditions for expanding, redesigning or stopping the initiative.

“We are not doing this to create more paperwork,” she said. The purpose was to reduce uncertainty about the operating model. A controlled pilot should make workflow ownership, review controls, exception handling and the response to uncertainty explicit before the organization treats a successful demonstration as permission to expand.

In an audience exercise, Singh asked participants to assess an initiative across the four dimensions and mark confidence in each answer. The weakest answer identified a question needing further work, potentially involving clinical, operational, compliance or technical expertise. It did not automatically disqualify the initiative.

Asked when to stop a technically capable system, she returned to the intended outcome. If a pilot designed to reduce oncology waiting times fails to demonstrate that improvement, leaders should consider whether workflow redesign could address the gap or whether the initiative no longer serves its purpose. The example was hypothetical, but the decision rule was clear: expansion depends on evidence relevant to the original goal.

Singh closed with a proposed 30-day starting sequence: clarify the decision, test the operating case and write the pilot contract with the relevant stakeholders. “You have to create stakeholder alignment,” she said. That work gives a team a shared basis for deciding what to fund, what to change and when to stop.