AI

Nigam Shah puts a price on verifying AI’s work

A discharge summary can cost less to generate than to check for omissions. Stanford, Google and Abridge panelists examined what hospitals should count before signing a contract.

Author
HDM Staff
Save
Share
Four participants on the HLTH USA Main Stage.
The Tokenomics panel at HLTH USA. Photo credit: HLTH USA 2026.

    Generating a hospital discharge summary with AI can cost about 13 cents, Stanford physician and data scientist Nigam Shah told a Las Vegas audience on Nov. 16. Checking whether the summary introduced false information cost another six cents in his example. Checking whether it left out anything important cost 72 cents. The most expensive step was finding out whether a cheap summary had missed something that belonged in it.

    Shah was speaking on the Main Stage with Google global healthcare solutions director Aashima Gupta and Abridge chief commercial officer Brian Wilson. Jay Rughani, a partner at Andreessen Horowitz, moderated. He asked what a hospital should pay for as AI use spreads, from individual units of model usage to a completed job or a clinical outcome. Shah’s figures gave the discussion a concrete place to start. A low price for generating a document can coexist with a much higher cost for getting one a care team can use.

    Count the attempts that do not finish the work

    Shah said buyers should watch the “cost per successful completion.” For a prior authorization, he gave a separate example in which a model run cost about $2 but a completed authorization cost about $8 once failed attempts were counted. Those were illustrations presented by Shah. They were not bids from vendors or measurements that can be applied to every hospital. The difference is important because the process could fail for reasons that vary by organization, including what information the agent receives and how the evidence is checked.

    Shah’s team describes a related problem in its HealthAdminBench preprint, which tests computer-use agents in simulated health administration environments. A model that performs an individual step well may still struggle to complete a task spanning several systems. A purchaser looking at a low cost per attempt could miss that difference. Even if one step gets cheaper, total spending can remain high when several steps need retries or someone has to finish what the software started.

    The discharge summary illustrates a different source of expense. A reviewer looking for invented details can compare claims in the draft against the source. Someone looking for omissions must also look for what the draft should have said and did not. One kind of review begins with the text the model produced. The other starts with the clinical information the draft is meant to carry forward. Shah’s point was that savings on the first pass do little for the total cost if the verification step stays expensive.

    Keep the evidence needed to judge a model

    Aashima Gupta speaking on the Main Stage.
    Aashima Gupta speaks during the panel. Photo credit: HLTH USA 2026.

    Gupta said health systems should be able to “swap the model” as prices and capabilities change. She urged them to “own your evals,” meaning the cases and expected results they use to judge a system. That gives the organization a way to compare a replacement with the system it already uses. The exercise is most informative if those cases reflect the work the hospital is actually trying to do and if someone can see which kinds of cases fail.

    Shah made a similar point about control over context and review. “Only you can tell” whether the response is good enough for the job, he said. The organization has to decide what clinical or operational information the model should get, how staff will grade its output and what to monitor after deployment. For a discharge summary, for instance, a system might consistently produce fluent text while missing the details staff need for a particular kind of handoff. Local cases and subsequent monitoring can reveal that pattern.

    Gupta said buyers also need a way to see spending across deployments. She described deciding when to use a lower-cost model, when to turn a task over to a person and what budgets to set. Hundreds of agents can each follow their own instructions and still leave finance and technology leaders with no useful overall view. Shared controls would let leaders see where the work is being sent and which decisions carry more cost, even when different departments use different tools.

    Decide what the contract is paying for

    Wilson called jobs “bundles of tasks.” He told hospitals to “pay for jobs to be done,” describing Abridge’s approach as tying the price to an agreed job and giving health systems more predictable costs. He said a contract based on the price of each query can become hard to forecast when many clinicians use it and each job draws on multiple calls. His example started with an assumed workforce and number of queries per clinician. It illustrated what can happen as use grows rather than showing the budget of a specific customer.

    That leaves a question about the definition of a job. In the discharge example, delivery might mean that a draft has appeared in the chart. It might also mean that the material has been compared with the source and someone responsible for the handoff can use it. When a contract prices the first step, the hospital still has to account for the second. When it prices the whole job, the buyer and supplier need a shared understanding of when the work is complete and who pays when it has to be repeated.

    Gupta said Google supports per-seat and consumption-based pricing. She described how a seat might make sense for teams that use a tool frequently while usage-based pricing might be a better fit for departments with lower use. Either choice still leaves the organization to compare what it pays with the volume of acceptable work the system produces. Rughani’s broader question about paying for outcomes goes further. The buyer would have to define the outcome and decide how much of it can reasonably be attributed to the tool rather than to everything else in the workflow.

    The figures Shah brought to the panel make those choices easier to test. A hospital can ask what it costs to produce a draft, what it costs to check it and what staff must do when the first attempt falls short. A purchasing team could track the same measures before and after a model change, using the same set of cases and the same definition of an acceptable result. Those numbers tell purchasers where savings could actually come from. They also tell them what a vendor's price leaves the hospital to do itself.