Healthcare AI benchmarks

525 clinician tasks with physician-written rubrics; results retain evaluator, grader and length-adjustment details.

Partial source review · Updated through the September 28 Sonnet 5.5 card, September 22 OpenAI correction and GPT-6 appendix. All source numbers checked. Evaluators use different graders; the Anthropic Astra reproduction is separate. Grok scores are numeric-source verified but protocol details are incomplete.

The 1,000 HealthBench conversations frontier models failed most, still graded on the original physician rubrics.

Source review · Checked primary model reports, applied September 22 Astra correction, added Sol/Luna and Baichuan-report results. Historical raw results and current adjusted results are explicitly labeled and are not a matched comparison.

Realistic multi-turn health conversations graded on physician-written rubrics for accuracy, completeness, and communication.

Source review · Updated through September 28 Sonnet 5.5 and September 22 OpenAI correction. Fixed Fable/Mythos alias and Opus 5 raw/adjusted display. Raw older results remain labeled; evaluator protocols differ.

257 evidence-grounded tasks across eight preventive medicine subjects, graded blind by independent model families.

Source review · Replaced the prior 89-task incretin ranking with the official main ranking for 257 subject-suite tasks, snapshot September 10. All sixteen scores and confidence intervals checked against the source table.

A composite of clinical benchmarks spanning diagnostic and management reasoning, safety, multimodal imaging, and agentic capability.

Source review · Official source still reports August 15, 2026; all eight displayed general-ranking scores match. This is a fresh source check of the existing snapshot, not a new model evaluation.

Holistic clinical evaluation across 121 tasks in a clinician-validated taxonomy, ranked by mean win rate.

Source review · Official v5.0.0 home-page scores match all ten indexed models and still state May 14, 2026. No later result is implied by the September source check.

How often, and how severely, model consultation recommendations contain potentially harmful errors.

Source review · Checked the official v2 technical board and added five omitted published systems/models. Seventeen of nineteen rows are indexed, with RAG systems labeled separately in settings. Board remains in preview. Newly indexed scores show a September observation date with measurement date left unknown.

Agent harnesses completing realistic terminal-based healthcare tasks built from real clinical artifacts.

Source review · All 12 official harness results checked; added the omitted Codex GPT 5.3 row (22%). Current leader remains Claude Code (Opus 5), 55%. Source does not publish a new run date.

Long-horizon healthcare operations workflows for agents: prior authorization, utilization management, and care management.

Source review · All 45 submissions inspected; retained all 44 numeric all-domain results and excluded one PA-only result. No newer scores found on the official board.

ICD-10-CM coding of whole hospital stays from discharge summaries and notes, checked against professional coders.

Source review · All 102 scored configurations checked against the first-party embedded table; source parameters captured per row. Updated 2026-09-26; individual run dates unavailable.

Quality and compliance of SOAP notes generated from clinical visits, scored against documentation rubrics.

Source review · All 104 scored configurations checked against the first-party embedded table; source parameters captured per row. Updated 2026-09-26; individual run dates unavailable.

Expert-level multiple-choice questions over clinical images across 17 specialties.

Partial source review · Meta, Qwen3.5 and Google score tables verified; all published Gemma 4 and Qwen3.5 comparison columns included. Six Qwen3.7/3.8 blog values remain historical and partial because primary pages are empty. No cross-vendor protocol equivalence implied.

A healthcare-weighted composite of six evaluations, covering medical knowledge, long records, knowledge work, reasoning and tools.

Partial source review · Verified the current six-evaluation methodology and 25 numeric model rows from first-party embedded chart data. This is partial coverage of 77 available variants; individual evaluation dates are undisclosed. Older incompatible index rows were removed.

Agents carrying out long-horizon physician workflows inside real EHR systems, verified by execution against those systems.

Source review · Verified all 12 original-paper rows plus 9 vendor-reported model/effort rows from today’s Sonnet 5.5 system card, pp. 137–139. Same public task set; vendor grader/harness cohort explicitly distinguished.

Agentic clinical reasoning over MIMIC-IV records through SQL and Python, at patient and population level.

Source review · Tables 3 and 10 verified: 18 configurations, including both benchmark-specific SFT variants. Current arXiv version remains v1 dated 2026-06-22.

Women's health scenarios graded on a 23-criterion rubric for clinical accuracy, safety, equity, and guideline adherence.

Source review · All 22 model scores in Table 3 checked against v2, revised 2026-07-23. Historical March 2026 evaluation set retained; no live refresh claimed.

Computer-use agents completing healthcare administration workflows: prior authorizations, denial appeals, and DME ordering.

Source review · All seven Figure 3(a) task-success rates verified against v1 and official project page. The paper remains dated 2026-04-10; source checks are not new evaluations.