The Best Clinical AI Agent Finishes Fewer Than Half Its Tasks

The best AI agent yet tested inside a working electronic health record environment completed 46 percent of physician tasks on its first attempt, and succeeded on only 28 percent of tasks across three consecutive runs (PhysicianBench, 2026, limited evidence, preprint not yet peer reviewed). A second suite built by Microsoft Research put its strongest agent at roughly 42 percent across 54 clinical tasks (HealthAgentBench, 2026, limited evidence, preprint). These are the numbers that matter for anyone deciding whether to put an agent into a clinical workflow, and they are not the numbers that show up in vendor decks.

The gap is not a knowledge gap. Current models pass medical licensing examinations comfortably. The gap is between knowing medicine and finishing a clinical task without supervision, and for the first time it is being measured directly.

Benchmarks stopped asking questions and started handing over the keys

Until recently, almost all healthcare AI evaluation was question answering. A systematic review of 519 studies published between January 2022 and February 2024 found that 84.2 percent focused on question answering, 44.5 percent on medical examination questions, and only 5 percent used real patient care data. Accuracy was the primary measure in 95.4 percent of studies, while deployment considerations appeared in 4.6 percent and calibration or uncertainty in 1.2 percent (Bedi et al., JAMA, 2025, strong evidence, systematic review).

That style tells you almost nothing about an agent. An agent does not answer a question. It reads a chart, decides what it still needs to know, queries a system, interprets what comes back, places an order, and writes the note. Failure can occur at any step, and a correct diagnosis reached after an incorrect order is still a failed task. Three benchmarks now test that sequence by putting agents inside executable environments rather than prompt windows.

MedAgentBench built 300 physician-written tasks against a FHIR-compliant EHR containing 100 patient profiles and over 700,000 data elements. Its best model at the time, Claude 3.5 Sonnet v2, reached a 69.67 percent success rate (NEJM AI, 2025, moderate evidence, peer-reviewed benchmark). Performance fell sharply on tasks requiring three or more steps.

PhysicianBench raised the difficulty. Its 100 tasks were adapted from real primary care to specialist consultations across 21 specialties, broken into 670 gradeable checkpoints, and each required an average of 27 tool calls. Of twelve agents evaluated, the best reached 46.3 percent pass@1 and the strongest open-source model 18.7 percent.

HealthAgentBench went wider, spanning 54 tasks across seven environments and five data types, from tabular records to gigapixel pathology slides. Ten frontier agents were run three times each. The best scored about 42 percent, the weakest 16 percent.

The consistency number is the one to negotiate on

Pass@1 tells you an agent can do the job. It does not tell you the agent will do the job. PhysicianBench reported both, and the difference is where the argument for deployment weakens. The best agent's 46.3 percent first-attempt success dropped to 28.0 percent when the requirement was success on all three independent runs of the same task.

That is a spread of 18 percentage points, meaning close to 40 percent of the agent's apparent successes were not reproducible. Nothing about the task changed between runs. The agent took a different path and failed.

Clinical operations do not tolerate that distribution. A laboratory analyzer correct on 46 percent of first passes and reproducible on 28 percent would not be commissioned. Any evaluation reporting only a single-run figure is reporting the more flattering half of the result, and buyers should ask for the other half by name.

Agents fail on grounding more than on medicine

PhysicianBench classified every failed checkpoint. Clinical reasoning accounted for roughly 52 percent of failures, action execution about 19 percent, documentation about 18 percent, and data retrieval about 12 percent. On its face that looks like a medicine problem.

The authors' own analysis points elsewhere. The leading model's advantage was attributed to more consistent and thorough exploration of the record, producing better grounded conclusions, rather than to superior clinical knowledge. The reasoning failed because it was built on an incomplete picture of the patient.

HealthAgentBench found the same pattern from another direction. On a data quality auditing task spanning eight tables and more than 800,000 rows, no agent exceeded a 42 percent success rate. When the instructions disclosed which table to inspect, recall improved for every agent, with the strongest approaching near-complete recovery. The constraint was search, not judgment.

This distinction matters commercially. If agents failed on medical knowledge, the fix would be a better model and a buyer could wait for it. If they fail because they do not look in the right places, the fix is environment design, tool scaffolding, and constrained task boundaries, which is engineering work the deploying organization has to do itself.

A follow-up team supports that reading. They rebuilt the agent layer around MedAgentBench without changing the model class, adding structured planning prompts, purpose-built tools, and a memory of prior failures, and lifted success rates substantially on the original task set (Pacific Symposium on Biocomputing, 2026, emerging evidence, single benchmark study). The scaffolding moved the number more than the model did.

Splitting the work beats letting the agent handle it whole

The most useful architectural finding in this literature is also the least discussed. HealthAgentBench ran the same set of injected data errors two ways, once as a single combined task and once as three separate focused passes whose results were pooled. Every agent recovered more errors when the work was decomposed. Compositional load, not difficulty, drove the failures.

That result cuts against the direction most agent marketing points. The instinct is to ask for one agent that handles the whole workflow. The measured behavior favors several narrow agents with defined inputs, defined outputs, and a defined handoff, because each one operates inside a search space it can actually cover.

These scores do not transfer, and neither do the environments

Three cautions before anyone uses these figures in a business case.

The benchmarks are not comparable to each other. Different tasks, different success thresholds, different models, different dates. MedAgentBench's higher score reflects easier tasks, not better agents. Reading the three as a trend line would be wrong.

Two of the three are preprints. They have not cleared peer review, and their numbers may move. AgentClinic, which has been peer reviewed, demonstrates how fragile these measurements can be. It found that agent performance varied substantially depending on which model was used to simulate the patient, meaning the benchmark's own construction influenced the score (npj Digital Medicine, 2026, moderate evidence, peer-reviewed).

Most importantly, none of this is clinical outcome evidence. No randomized trial has shown that a clinical AI agent improves patient outcomes. Benchmark performance is a necessary precondition for deployment, not a substitute for evaluating it.

Every one of these environments assumes infrastructure most facilities do not have

All three executable benchmarks run against structured, digitized, standards-compliant health records. PhysicianBench and MedAgentBench both use FHIR APIs. HealthAgentBench assumes queryable relational EHR tables, DICOM-scale imaging, and containerized compute.

That describes a well-resourced hospital, and it quietly encodes a definition of the problem. An agent scoring 46 percent against a FHIR server has been measured on a task that does not exist in a facility running on paper registers, a spreadsheet, or an intermittent connection.

A group of clinical informaticians makes this case in peer-reviewed form, arguing that healthcare AI evaluation must be local, meaning grounded in the specific deployment context, and task-specific, meaning aligned to the intended clinical use rather than to a general capability score (PLOS Digital Health, 2026, peer-reviewed framework, not empirical). Curely does not have a verified figure for record digitization across the facilities it serves and will not publish one it cannot source. The structural point stands without it. A benchmark built on infrastructure a hospital lacks cannot say what an agent will do there.

Cost carries the same problem. HealthAgentBench reported mean spend of 0.60 to 4.80 US dollars per task attempt, with agents taking 14 to 24 minutes each. At those rates, an agent covering a meaningful share of a district hospital's daily workload becomes a budget line before it becomes a clinical asset.

How Curely reads this

Curely builds specialized agents for African and lower-resource health systems, and this evidence shapes the design rather than validating it. The following describes design intent. Curely has not published deployment outcome data and does not present these choices as proven.

Three decisions follow directly from the benchmark results. Agent scope is kept narrow, with each agent given a bounded task and an explicit handoff, because decomposition measurably outperforms monolithic autonomy. Retrieval is treated as a first-class engineering problem rather than something the model handles, because the failure analysis points at grounding rather than clinical knowledge. And evaluation is designed against the record systems that actually exist in target facilities, including partial and paper-derived records, because a FHIR benchmark score does not describe that environment.

The reasoning behind this architecture is set out further in AI Clinical Assistance

What to ask before an agent goes anywhere near a patient

Ask for the consistency figure alongside the success rate, and across how many independent runs it was measured. Ask which environment produced the score and how closely it resembles the record system in the building. Ask for the failure breakdown, because an agent that fails on retrieval is a different risk from one that fails on execution. Ask the cost per completed task at the volume proposed. And ask whether any of it was measured on real patient data, given that only 5 percent of the published evaluation literature has been.

An agent that finishes fewer than half its tasks is not useless. It is a system that needs a human at the end of every sequence, a narrow remit, and an environment engineered around its known failure modes. That is a workable deployment, and a very different purchase from the one usually described.