# Health Evals: MedPIC-Bench, reviewed 2026-09-28 Independent Arcophos analysis of arXiv:2608.03028v1, published 2026-08-04. Source: https://arxiv.org/html/2608.03028v1, Table 2. All results are percentages; the GF–CF gap is percentage points. The source review is not a new model measurement. Actual measurement dates and detailed inference settings were not available. Configuration: Each question independently, zero-shot, common prompt requests brief rationale and option letters in an answer field. Linked cases presented separately. Exact set match scoring. Text inputs for all models. Open models via PyTorch/Transformers/SGLang on two 80GB A800 GPUs; proprietary models via APIs. Denominators: 467 total questions; 284 guideline-following (GF); 183 counterfactual (CF); 71 activation; 68 deactivation; 89 paper-reported linked pairs. Pair accuracy requires both linked cases correct. The public dataset lacks explicit pair links, so we do not reconstruct pair scores. Activation and deactivation exclude 44 other counterfactual questions. A smaller GF–CF gap alone does not establish robustness. | Model | Overall | GF | CF | GF–CF gap | Activation | Deactivation | Pair | |---|---|---|---|---|---|---|---| | HealthGPT-Pro-8B | 42.6 | 51.8 | 28.4 | 23.3 | 42.3 | 20.6 | 5.6 | | MedGemma-4B | 39.6 | 45.4 | 30.6 | 14.8 | 46.5 | 10.3 | 5.6 | | Lingshu-7B | 43.3 | 50.7 | 31.7 | 19.0 | 39.4 | 32.4 | 6.7 | | Hulu-Med-7B | 42.0 | 48.2 | 32.2 | 16.0 | 36.6 | 38.2 | 14.6 | | HealthGPT-Pro-4B | 36.0 | 37.7 | 33.3 | 4.3 | 36.6 | 41.2 | 11.2 | | HuatuoGPT-o1-8B | 44.3 | 50.0 | 35.5 | 14.5 | 49.3 | 20.6 | 13.5 | | Fleming-R1-7B | 50.7 | 59.9 | 36.6 | 23.2 | 53.5 | 29.4 | 12.4 | | Baichuan-M2-32B | 47.8 | 54.6 | 37.2 | 17.4 | 49.3 | 30.9 | 11.2 | | HuatuoGPT-o1-70B | 55.2 | 63.7 | 42.1 | 21.7 | 63.4 | 23.5 | 14.6 | | Fleming-R1-32B | 60.6 | 69.0 | 47.5 | 21.5 | 59.2 | 44.1 | 20.2 | | MedGemma-27B-Text | 61.2 | 67.6 | 51.4 | 16.2 | 76.1 | 32.4 | 23.6 | | Lingshu-32B | 59.5 | 63.7 | 53.0 | 10.7 | 60.6 | 54.4 | 25.8 | | Gemma-3-12B | 39.6 | 44.0 | 32.8 | 11.2 | 57.7 | 7.4 | 3.4 | | Gemma-3-27B | 40.3 | 44.7 | 33.3 | 11.4 | 53.5 | 10.3 | 5.6 | | Llama-3.1-8B | 40.0 | 44.4 | 33.3 | 11.0 | 54.9 | 10.3 | 5.6 | | Llama-3.1-70B | 55.7 | 61.3 | 47.0 | 14.3 | 63.4 | 32.4 | 16.9 | | GPT-OSS-20B | 59.1 | 66.5 | 47.5 | 19.0 | 52.1 | 48.5 | 24.7 | | Qwen3.5-9B | 63.8 | 73.9 | 48.1 | 25.9 | 54.9 | 47.1 | 24.7 | | Qwen3.5-35B-A3B | 68.5 | 77.5 | 54.6 | 22.8 | 70.4 | 45.6 | 30.3 | | Qwen3.5-27B | 69.4 | 78.2 | 55.7 | 22.4 | 67.6 | 51.5 | 31.5 | | GPT-OSS-120B | 70.7 | 79.6 | 56.8 | 22.7 | 62.0 | 55.9 | 30.3 | | Gemini-2.5-Pro | 54.4 | 60.6 | 44.8 | 15.8 | 63.4 | 27.9 | 20.2 | | Claude-Sonnet-4.6 | 64.2 | 74.6 | 48.1 | 26.6 | 62.0 | 35.3 | 21.3 | | GPT-5.2 | 68.1 | 78.2 | 52.5 | 25.7 | 63.4 | 42.6 | 29.2 | | DeepSeek-V4-Pro | 71.7 | 81.3 | 56.8 | 24.5 | 67.6 | 50.0 | 32.6 | | Qwen3.5-Plus | 72.4 | 81.7 | 57.9 | 23.8 | 66.2 | 55.9 | 32.6 | | GPT-5 | 75.6 | 83.5 | 63.4 | 20.1 | 70.4 | 61.8 | 38.2 | | Gemini-3.1-Pro | 80.7 | 87.7 | 69.9 | 17.7 | 74.6 | 77.9 | 48.3 | Dataset: https://huggingface.co/datasets/TIM0927/MedPIC-Bench, revision 9ef6db4f13865b14fc2e6be3f94dcfaf3a0cf983. Coverage counts and provenance: https://healthevals.com/data/medpic.json. Result CSV: https://healthevals.com/data/medpic.csv. Methodology: https://healthevals.com/medpic-methodology. # Broader Health Evals index, 2026-09-28 17 healthcare AI benchmarks are included in this curated snapshot. Each section below gives one benchmark's published results with source citations. Review notes identify scale conversions and verification gaps. Scores are never comparable across benchmarks. Source: https://healthevals.com ## HealthBench Professional 525 tasks that physicians picked out of 15,079 real workplace AI conversations, spanning care consults, clinical documentation, and medical research, each judged on a rubric physicians wrote for it. Published by OpenAI (2026-04-22); 525 physician-authored tasks; scale 0 to 1, higher is better. Source: Published model evaluation reports (mixed), https://deploymentsafety.openai.com/gpt-6-astra/healthbench. Full leaderboard: https://healthbenchprofessional.com 29 indexed rows; source board size: 29. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | GPT-6 Astra (Anthropic run) | OpenAI | 0.703 | 2026-09 | Claude Sonnet 5.5 System Card (system_card, Section 8.15, pp. 137–138; text above Figure 8.15.B: max-effort Astra 70.3 adjusted and 74.0 raw) | | 2 | Claude Sonnet 5.5 | Anthropic | 0.692 | 2026-09 | Claude Sonnet 5.5 System Card (system_card, Section 8.15.2; pp. 137–139, Figure 8.15.B (max effort)) | | 3 | Claude Opus 5.5 | Anthropic | 0.656 | 2026-09 | Claude Opus 5.5 System Card (system_card, Section 8.15.2; pp. 213–214, Figures 8.15.1.A and 8.15.2.A) | | 4 | GPT-6 Astra | OpenAI | 0.647 | 2026-09 | GPT-6 Astra System Card — September 22 revision (system_card, Section 11.4.1, Table 29; HealthBench Professional length-adjusted row, GPT-6 Astra column; value 64.7 (68.2, 3185)) | | 5 | Claude Fable 5 | Anthropic | 0.633 | 2026-09 | Claude Fable 5.1 and Claude Mythos 5.1 System Card (system_card, p. 199, Figure 8.17.2.A; Claude Fable 5, adjusted bar 63.3%; also p. 167 Table 8.1.A) | | 6 | Claude Fable 5.1 | Anthropic | 0.621 | 2026-09 | Claude Fable 5.1 and Claude Mythos 5.1 System Card (system_card, p. 199, sec. 8.17.2 (same figure printed as 62.1% in Table 8.1.A, p. 167)) | | 7 | GPT-6 Sol | OpenAI | 0.608 | 2026-09 | GPT-6 Astra System Card — September 22 revision (system_card, Section 11.4.1, Table 29; HealthBench Professional length-adjusted row, GPT-6 Sol column; value 60.8 (59.5, 1573)) | | 8 | GPT-6 Luna | OpenAI | 0.608 | 2026-09 | GPT-6 Astra System Card — September 22 revision (system_card, Section 11.4.1, Table 29; HealthBench Professional length-adjusted row, GPT-6 Luna column; value 60.8 (61.2, 2119)) | | 9 | GPT-5.6 Sol | OpenAI | 0.605 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.6-SOL) | | 10 | Claude Opus 5 | Anthropic | 0.598 | 2026-07 | System Card: Claude Opus 5 (system_card, p. 189, section 8.15.2 HealthBench Professional results; also Table 8.1.A p. 152 ('HealthBench Professional 59.8 ...')) | | 11 | Muse Spark 1.1 | Meta | 0.593 | 2026-07 | Muse Spark 1.1 Evaluation Report (model_card, p. 101, Figure 44 'General capability benchmark results' (image), row HealthBench Professional, column Muse Spark 1.1; protocol p. 104 (printed 103): HealthBench Pro comprises 525 evaluation data points graded by rubrics. We use GPT-5.4 with low reasoning effort as the grader and report the length-normalized rubric score as done in their paper.) | | 12 | Claude Sonnet 5 | Anthropic | 0.578 | 2026-06 | System Card: Claude Sonnet 5 (system_card, p. 115, Table 8.1.A, row HealthBench Professional, column Claude Sonnet 5; Figure 8.12.2.A p. 139) | | 13 | GPT-5.6 Terra | OpenAI | 0.577 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.6-TERRA) | | 14 | Claude Opus 4.8 | Anthropic | 0.574 | 2026-06 | Claude Sonnet 5 System Card (system_card, p. 139, Figure 8.12.2.A; Opus 4.8 bar 57.4%) | | 15 | Grok 4.7 | xAI | 0.567 | 2026-09 | Introducing Grok 4.7 (launch_post, Model Improvements table, Clinical reasoning / HealthBench Professional row, Grok 4.7 xhigh column) | | 16 | GPT-5.6 Luna | OpenAI | 0.557 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.6-LUNA) | | 17 | Muse Spark | Meta | 0.541 | 2026-07 | Muse Spark 1.1 Evaluation Report (model_card, p. 101, Figure 44 (image), row HealthBench Professional, column Muse Spark; protocol p. 104 (printed 103)) | | 18 | GPT-5.6 Sol (August) | OpenAI | 0.540 | 2026-08 | GPT-5.6 - August Updates (system card addendum) (system_card, p. 11, section 5.1 HealthBench, table "Reported as length-adjusted score (unadjusted, mean response length in characters)", column GPT-5.6 Sol (August)) | | 19 | Claude Opus 4.7 | Anthropic | 0.519 | 2026-05 | System Card: Claude Opus 4.8 (system_card, p. 228, section 8.14.1 HealthBench Professional; Figure 8.14.A p. 229) | | 20 | GPT-5.5 | OpenAI | 0.518 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.5) | | 21 | Grok 4.6 | xAI | 0.485 | 2026-09 | Introducing Grok 4.7 (launch_post, Model Improvements table, Clinical reasoning / HealthBench Professional row, Grok 4.6 high column) | | 22 | GPT-5.4 | OpenAI | 0.481 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.4) | | 23 | GPT-5 | OpenAI | 0.462 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5) | | 24 | GPT-5.2 | OpenAI | 0.459 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.2) | | 25 | Claude Sonnet 4.6 | Anthropic | 0.442 | 2026-06 | System Card: Claude Sonnet 5 (system_card, p. 115, Table 8.1.A, row HealthBench Professional, column Claude Sonnet 4.6) | | 26 | GPT-5.6 Luna (August) | OpenAI | 0.441 | 2026-08 | GPT-5.6 - August Updates (system card addendum) (system_card, p. 11, section 5.1 HealthBench, table "Reported as length-adjusted score (unadjusted, mean response length in characters)", column GPT-5.6 Luna (August)) | | 27 | GPT-5.1 | OpenAI | 0.396 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.1) | | 28 | GPT-5.5 Instant | OpenAI | 0.384 | 2026-05 | GPT-5.5 Instant System Card (system_card, Section 4.1 HealthBench, Table 5 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.5 INSTANT) | | 29 | MAI-Thinking-1 | Microsoft | 0.350 | 2026-08 | MAI-Thinking-1: Building a Hill-Climbing Machine (model_card, p. 54, Table 12 'Post-trained model evaluation results on various public benchmarks', Health group, column HealthBench Prof.; protocol Appendix K.6 p. 106: HealthBench Professional introduces a length penalty for the primary metric, to correct for a well-observed correlation between lengthy responses and artificially increased LLM-grader scores. For all reported scores, we use the standard GPT-5.4 grader and rubrics provided by OpenAI.) | All displayed values use a 0–1 scale (source percentages divided by 100). Most are length-adjusted, but grader models, safeguards and evaluator protocols differ. Anthropic’s Astra reproduction (Opus 4.8 grader) is a separate row from OpenAI’s own run. Grok’s release table does not fully document its grading protocol. A score-source check is not an independent rerun. Source review: 2026-09-28; partial. Updated through the September 28 Sonnet 5.5 card, September 22 OpenAI correction and GPT-6 appendix. All source numbers checked. Evaluators use different graders; the Anthropic Astra reproduction is separate. Grok scores are numeric-source verified but protocol details are incomplete. ## HealthBench Hard The bottom fifth of HealthBench: 1,000 conversations where frontier models failed most at the May 2025 release, still graded on the original physician-written rubrics. Published by OpenAI (2025-05-12); 1,000 conversations; scale 0 to 1, higher is better. Source: healthbenchhard.ai (mixed), https://healthbenchhard.ai. Full leaderboard: https://healthbenchhard.ai 20 indexed rows; source board size: 20. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | Baichuan-M3 | Baichuan | 0.444 | 2026-02 | Baichuan-M3 Technical Report (paper, p. 23 (PDF page index 22), section 4.2.1 and Figure 7; HealthBench Hard) | | 2 | Muse Spark | Meta | 0.428 | 2026-04 | Introducing Muse Spark: Scaling Towards Personal Superintelligence (launch_post, Launch-post benchmark table image, HEALTH section, row HealthBench Hard, column Muse Spark Thinking; identical table in the Eval Methodology PDF p. 5; protocol p. 2: HealthBench Hard: This is a subset of OpenAI's HealthBench benchmark, containing 1000 prompts. We used the same implementation as in the OpenAI’s official simple-evals repo, with GPT-4.1-genai as the LLM-as-judge model.) | | 3 | GPT-5.2-High (Baichuan run) | OpenAI | 0.420 | 2026-02 | Baichuan-M3 Technical Report (paper, p. 23 (PDF page index 22), section 4.2.1 and Figure 7; HealthBench Hard) | | 4 | GPT-6 Astra | OpenAI | 0.366 | 2026-09 | GPT-6 Astra System Card — September 22 revision (system_card, Section 11.4.1, Table 29; HealthBench Hard length-adjusted row, GPT-6 Astra column; value 36.6 (34.2, 1697)) | | 5 | GPT-5 | OpenAI | 0.347 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5) | | 6 | GPT-5.2 | OpenAI | 0.343 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.2) | | 7 | GPT-5.6 Sol | OpenAI | 0.331 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.6-SOL) | | 8 | GPT-5.6 Terra | OpenAI | 0.327 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.6-TERRA) | | 9 | GPT-5.6 Luna | OpenAI | 0.320 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.6-LUNA) | | 10 | GPT-5.5 | OpenAI | 0.315 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.5) | | 11 | GPT-5.6 Sol (August) | OpenAI | 0.314 | 2026-08 | GPT-5.6 - August Updates (system card addendum) (system_card, p. 11, section 5.1 HealthBench, table "Reported as length-adjusted score (unadjusted, mean response length in characters)", column GPT-5.6 Sol (August)) | | 12 | GPT-6 Luna | OpenAI | 0.314 | 2026-09 | GPT-6 Astra System Card — September 22 revision (system_card, Section 11.4.1, Table 29; HealthBench Hard length-adjusted row, GPT-6 Luna column; value 31.4 (25.4, 1241)) | | 13 | GPT-6 Sol | OpenAI | 0.301 | 2026-09 | GPT-6 Astra System Card — September 22 revision (system_card, Section 11.4.1, Table 29; HealthBench Hard length-adjusted row, GPT-6 Sol column; value 30.1 (22.1, 974)) | | 14 | GPT OSS 120B | OpenAI | 0.300 | 2025-08 | gpt-oss-120b & gpt-oss-20b Model Card (model_card, Section 2, Table 3: Evaluations across multiple benchmarks and reasoning levels, row HealthBench Hard, column gpt-oss-120b high) | | 15 | GPT-5.4 | OpenAI | 0.291 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.4) | | 16 | GPT-5.6 Luna (August) | OpenAI | 0.287 | 2026-08 | GPT-5.6 - August Updates (system card addendum) (system_card, p. 11, section 5.1 HealthBench, table "Reported as length-adjusted score (unadjusted, mean response length in characters)", column GPT-5.6 Luna (August)) | | 17 | GPT-5.3 Chat | OpenAI | 0.259 | 2026-03 | GPT-5.3 Instant System Card (system_card, Section 4.1 HealthBench, Table 3: HealthBench, row Hard, column GPT-5.3-INSTANT) | | 18 | GPT-5.1 | OpenAI | 0.254 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.1) | | 19 | GPT-5.5 Instant | OpenAI | 0.229 | 2026-05 | GPT-5.5 Instant System Card (system_card, Section 4.1 HealthBench, Table 5 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.5 INSTANT) | | 20 | GPT OSS 20B | OpenAI | 0.108 | 2025-08 | gpt-oss-120b & gpt-oss-20b Model Card (model_card, Section 2, Table 3: Evaluations across multiple benchmarks and reasoning levels, row HealthBench Hard, column gpt-oss-20b high) | The 0–1 display divides source percentages by 100. OpenAI’s newer rows are length-adjusted; Meta, Baichuan, gpt-oss and GPT-5.3 launch rows report raw scores. These protocols are not directly interchangeable. The September 22 Astra correction and GPT-6 Sol/Luna appendix are included. Source review: 2026-09-28; verified. Checked primary model reports, applied September 22 Astra correction, added Sol/Luna and Baichuan-report results. Historical raw results and current adjusted results are explicitly labeled and are not a matched comparison. ## HealthBench 5,000 realistic multi-turn health conversations graded against physician-written rubrics (48,562 criteria) covering accuracy, completeness, context awareness, communication, and instruction following. OpenAI now also reports a length-adjusted variant that penalizes verbosity. Published by OpenAI (2025-05); 5,000 conversations; scale 0-100 rubric-point percentage (some sites display 0-1), higher better; length-adjusted and unadjusted variants. Source: Published model evaluation reports (mixed), https://deploymentsafety.openai.com/gpt-6-astra/healthbench 26 indexed rows; source board size: 26. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | Claude Sonnet 5.5 | Anthropic | 65.4 | 2026-09 | Claude Sonnet 5.5 System Card (system_card, Section 8.15.1; pp. 137–139, Figure 8.15.B (max effort)) | | 2 | Baichuan-M3 | Baichuan | 65.1 | 2026-02 | Baichuan-M3 Technical Report (paper, p. 23, section 4.2.1 HealthBench-Main; Table on p. 25 (Model / HealthBench Score)) | | 3 | GPT-5.2-High | OpenAI | 63.3 | 2026-02 | Baichuan-M3 Technical Report (paper, p. 25, HealthBench-Hallu table (Model / HealthBench Score column); also p. 23 prose) | | 4 | Claude Opus 5.5 | Anthropic | 60.6 | 2026-09 | Claude Opus 5.5 System Card (system_card, Section 8.15.1; pp. 213–214, Figures 8.15.1.A and 8.15.2.A) | | 5 | Claude Fable 5 | Anthropic | 60.4 | 2026-09 | Claude Fable 5.1 and Claude Mythos 5.1 System Card (system_card, p. 198, Figure 8.17.1.A; Claude Fable 5, adjusted bar 60.4%) | | 6 | Claude Fable 5.1 | Anthropic | 60% | 2026-09 | Claude Fable 5.1 and Claude Mythos 5.1 System Card (system_card, p. 198, sec. 8.17.1 / Figure 8.17.1.A) | | 7 | Claude Opus 4.8 | Anthropic | 59.3 | 2026-06 | Claude Fable 5 and Claude Mythos 5 System Card (system_card, p. 252, Table 8.1.A, row HealthBench, column Opus 4.8) | | 8 | Claude Sonnet 5 | Anthropic | 58.7% | 2026-06 | System Card: Claude Sonnet 5 (system_card, p. 138, section 8.12.1 HealthBench results, Figure 8.12.1.A bar label (no prose or table number)) | | 9 | GPT-6 Astra | OpenAI | 58.3 | 2026-09 | GPT-6 Astra System Card — September 22 revision (system_card, Section 11.4.1, Table 29; HealthBench length-adjusted row, GPT-6 Astra column; value 58.3 (56.9, 1760)) | | 10 | Claude Opus 5 | Anthropic | 57.8 | 2026-07 | System Card: Claude Opus 5 (system_card, p. 188, section 8.15.1 and Figure 8.15.1.A; adjusted 57.8%, raw 67.1%) | | 11 | GPT-5 | OpenAI | 57.7 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1, Table 6, HealthBench length-adjusted row, GPT-5 column) | | 12 | GPT OSS 120B | OpenAI | 57.6 | 2025-08 | gpt-oss-120b & gpt-oss-20b Model Card (model_card, Section 2, Table 3: Evaluations across multiple benchmarks and reasoning levels, row HealthBench, column gpt-oss-120b high) | | 13 | GPT-5.6 Sol | OpenAI | 57.0 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.6-SOL) | | 14 | GPT-5.6 Terra | OpenAI | 57.0 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.6-TERRA) | | 15 | GPT-5.2 | OpenAI | 56.8 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1, Table 6, HealthBench length-adjusted row, GPT-5.2 column) | | 16 | GPT-5.5 | OpenAI | 56.5 | 2026-04 | GPT-5.5 System Card (system_card, Section 5 Health, Table 7 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.5) | | 17 | GPT-5.6 Luna | OpenAI | 55.8 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1 HealthBench, Table 6 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.6-LUNA) | | 18 | GPT-5.6 Sol (August) | OpenAI | 55.0 | 2026-08 | GPT-5.6 - August Updates (system card addendum) (system_card, p. 11, section 5.1 HealthBench, table "Reported as length-adjusted score (unadjusted, mean response length in characters)", column GPT-5.6 Sol (August)) | | 19 | GPT-6 Luna | OpenAI | 54.5 | 2026-09 | GPT-6 Astra System Card — September 22 revision (system_card, Section 11.4.1, Table 29; HealthBench length-adjusted row, GPT-6 Luna column; value 54.5 (50, 1255)) | | 20 | GPT-5.3 Chat | OpenAI | 54.1% | 2026-03 | GPT-5.3 Instant System Card (system_card, Section 4.1 HealthBench, Table 3: HealthBench, row HealthBench, column GPT-5.3-INSTANT) | | 21 | GPT-5.4 | OpenAI | 54.0 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1, Table 6, HealthBench length-adjusted row, GPT-5.4 column) | | 22 | GPT-5.6 Luna (August) | OpenAI | 53.3 | 2026-08 | GPT-5.6 - August Updates (system card addendum) (system_card, p. 11, section 5.1 HealthBench, table "Reported as length-adjusted score (unadjusted, mean response length in characters)", column GPT-5.6 Luna (August)) | | 23 | GPT-6 Sol | OpenAI | 53.2 | 2026-09 | GPT-6 Astra System Card — September 22 revision (system_card, Section 11.4.1, Table 29; HealthBench length-adjusted row, GPT-6 Sol column; value 53.2 (47.1, 977)) | | 24 | GPT-5.5 Instant | OpenAI | 51.4 | 2026-05 | GPT-5.5 Instant System Card (system_card, Section 4.1 HealthBench, Table 5 (reported as length-adjusted score (unadjusted, mean response length in characters)), column GPT-5.5 INSTANT) | | 25 | GPT-5.1 | OpenAI | 50.9 | 2026-06 | GPT-5.6 System Card (system_card, Section 5.1, Table 6, HealthBench length-adjusted row, GPT-5.1 column) | | 26 | GPT OSS 20B | OpenAI | 42.5 | 2025-08 | gpt-oss-120b & gpt-oss-20b Model Card (model_card, Section 2, Table 3: Evaluations across multiple benchmarks and reasoning levels, row HealthBench, column gpt-oss-20b high) | Rows mix documented length-adjusted and raw scores and are not a controlled cross-model comparison. Raw Baichuan, gpt-oss and GPT-5.3 launch results remain labeled in their row settings. New Claude rows use Anthropic’s Opus 4.8 grader; OpenAI reports its own evaluations. The September 22 Astra correction is applied. Use HealthBench Professional for a newer clinician task set. Source review: 2026-09-28; verified. Updated through September 28 Sonnet 5.5 and September 22 OpenAI correction. Fixed Fable/Mythos alias and Opus 5 raw/adjusted display. Raw older results remain labeled; evaluator protocols differ. ## Health Optimization Bench Questions in eight areas of preventive and optimization medicine, grounded in primary evidence and scored against task-specific rubrics. The current main ranking covers 257 released tasks, separate from the 89-task incretin therapeutics evidence suite. Published by Arcophos (2026-08); 257 tasks across eight subject suites; scale 0-100 rubric credit, higher better. Source: healthoptimizationbench.com (independent-run), https://healthoptimizationbench.com/sources. Full leaderboard: https://healthoptimizationbench.com 16 indexed rows; source board size: 16. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | Claude Fable 5 | Anthropic | 70.9 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; Claude Fable 5; tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.709, n=257) | | 2 | Claude Opus 5 | Anthropic | 69.3 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; Claude Opus 5; tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.693, n=257) | | 3 | Grok 4.6 | xAI | 66.8 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; Grok 4.6; tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.668, n=257) | | 4 | GPT-5.6 Sol (max) | OpenAI | 66.6 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; GPT-5.6 Sol (max); tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.666, n=257) | | 5 | GPT-5.6 Sol (high) | OpenAI | 64.6 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; GPT-5.6 Sol (high); tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.646, n=257) | | 6 | Kimi K3 | Moonshot AI | 59.9 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; Kimi K3; tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.599, n=257) | | 7 | Muse Spark | Meta | 57.2 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; Muse Spark; tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.572, n=257) | | 8 | Claude Fable 5.1 | Anthropic | 47.3 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; Claude Fable 5.1; tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.473, n=257) | | 9 | Gemini 3.6 | Google | 39.7 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; Gemini 3.6; tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.397, n=257) | | 10 | Inkling | Thinking Machines | 35.6 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; Inkling; tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.356, n=257) | | 11 | Claude Sonnet 5 | Anthropic | 34.6 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; Claude Sonnet 5; tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.346, n=257) | | 12 | GLM 5.2 | Zhipu | 20.7 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; GLM 5.2; tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.207, n=257) | | 13 | MiniMax M3 | MiniMax | 18.3 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; MiniMax M3; tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.183, n=257) | | 14 | MAI Thinking | Microsoft AI | 17.5 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; MAI Thinking; tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.175, n=257) | | 15 | Mistral Medium 3.5 | Mistral | 9.2 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; Mistral Medium 3.5; tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.092, n=257) | | 16 | Nemotron 3.5 Lightning | NVIDIA | 4.9 | 2026-09 | Health Optimization Bench — subject suites results (arcophos_run, Subject suites table; Nemotron 3.5 Lightning; tasksets/mb2ev/analysis.json snapshot 2026-09-10; mean 0.049, n=257) | The main ranking moved from the 89-task incretin suite to 257 subject-suite tasks. This page shows only the 257-task September 10 snapshot; the two task sets must not be pooled. Three grader families vote, with split decisions escalated to a fourth, and the authoring family excluded. Fable 5.1 safeguards declined 97 tasks; those receive no credit. Source review: 2026-09-28; verified. Replaced the prior 89-task incretin ranking with the official main ranking for 257 subject-suite tasks, snapshot September 10. All sixteen scores and confidence intervals checked against the source table. ## MAST (Medical AI Superintelligence Test) Composite score across curated clinical benchmarks spanning diagnostic reasoning, management reasoning, safety, multimodal images, multimodal radiology, and agentic capability. Components: First Do NOHARM v2, SCT-Bench, MedAgentBench v2, PhysicianBench, ReXrank Mini, CPC-Bench. Published by ARISE AI Research Network (multi-institutional) (2026-08); composite of 6 component benchmarks; 11 models; scale percentage composite, higher better. Source: MAST: Medical AI Superintelligence Test leaderboard (General board) (independent-run), https://arise-ai.org/mast 8 indexed rows; source board size: 11. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | GPT-5.6 Sol | OpenAI | 60.2% | 2026-08 | MAST: Medical AI Superintelligence Test leaderboard (General board) (official_leaderboard, arise-ai.org/mast, 'Which AI can you trust for medical questions?' General tab, composite score table (8 of 11 models shown; 'Last updated August 15, 2026')) | | 2 | Kimi K3 | Moonshot AI | 60.1% | 2026-08 | MAST: Medical AI Superintelligence Test leaderboard (General board) (official_leaderboard, arise-ai.org/mast, 'Which AI can you trust for medical questions?' General tab, composite score table (8 of 11 models shown; 'Last updated August 15, 2026')) | | 3 | Gemini 3.6 Flash | Google | 59.3% | 2026-08 | MAST: Medical AI Superintelligence Test leaderboard (General board) (official_leaderboard, arise-ai.org/mast, 'Which AI can you trust for medical questions?' General tab, composite score table (8 of 11 models shown; 'Last updated August 15, 2026')) | | 4 | Gemini 3.1 Pro | Google | 58.9% | 2026-08 | MAST: Medical AI Superintelligence Test leaderboard (General board) (official_leaderboard, arise-ai.org/mast, 'Which AI can you trust for medical questions?' General tab, composite score table (8 of 11 models shown; 'Last updated August 15, 2026')) | | 5 | Qwen3.5 397B A17B | Alibaba | 57.9% | 2026-08 | MAST: Medical AI Superintelligence Test leaderboard (General board) (official_leaderboard, arise-ai.org/mast, 'Which AI can you trust for medical questions?' General tab, composite score table (8 of 11 models shown; 'Last updated August 15, 2026')) | | 6 | Claude Opus 5 | Anthropic | 57.1% | 2026-08 | MAST: Medical AI Superintelligence Test leaderboard (General board) (official_leaderboard, arise-ai.org/mast, 'Which AI can you trust for medical questions?' General tab, composite score table (8 of 11 models shown; 'Last updated August 15, 2026')) | | 7 | Claude Sonnet 5 | Anthropic | 56.6% | 2026-08 | MAST: Medical AI Superintelligence Test leaderboard (General board) (official_leaderboard, arise-ai.org/mast, 'Which AI can you trust for medical questions?' General tab, composite score table (8 of 11 models shown; 'Last updated August 15, 2026')) | | 8 | Grok 4.3 | xAI | 53.7% | 2026-08 | MAST: Medical AI Superintelligence Test leaderboard (General board) (official_leaderboard, arise-ai.org/mast, 'Which AI can you trust for medical questions?' General tab, composite score table (8 of 11 models shown; 'Last updated August 15, 2026')) | The board is marked as a preview and was last updated August 15, 2026; component-level breakdowns are published only for First, Do NOHARM v2. Scores may move before the full release. The official overview still states August 15, 2026. Eight displayed general-ranking rows are indexed here, from eleven models on that view. MAST is in preview and scores may change. Source review: 2026-09-28; verified. Official source still reports August 15, 2026; all eight displayed general-ranking scores match. This is a fresh source check of the existing snapshot, not a new model evaluation. ## MedHELM Holistic evaluation of LLMs on 121 clinical tasks across 5 categories and 22 subcategories (31 datasets) in a clinician-validated taxonomy; ranked by mean win rate. Published by Stanford CRFM / HAI and multi-institution collaborators (2025-02); 121 tasks / 31 datasets; scale mean win rate 0-1, higher better. Source: MedHELM leaderboard (medhelm.org), v5.0.0 (official-leaderboard), https://medhelm.org/ 10 indexed rows; source board size: 11. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | Gemini 3.1 Pro (Preview) | Google | 0.652 | 2026-05 | MedHELM leaderboard (medhelm.org), v5.0.0 (official_leaderboard, medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')) | | 2 | Gemini 3.5 Flash | Google | 0.642 | 2026-05 | MedHELM leaderboard (medhelm.org), v5.0.0 (official_leaderboard, medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')) | | 3 | Muse Spark (2026-04-08) | Meta | 0.621 | 2026-05 | MedHELM leaderboard (medhelm.org), v5.0.0 (official_leaderboard, medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')) | | 4 | GPT-5.4 mini | OpenAI | 0.552 | 2026-05 | MedHELM leaderboard (medhelm.org), v5.0.0 (official_leaderboard, medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')) | | 5 | GPT-5.4 (2026-03-05) | OpenAI | 0.538 | 2026-05 | MedHELM leaderboard (medhelm.org), v5.0.0 (official_leaderboard, medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')) | | 6 | Gemini 2.5 Pro | Google | 0.529 | 2026-05 | MedHELM leaderboard (medhelm.org), v5.0.0 (official_leaderboard, medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')) | | 7 | DeepSeek R1 | DeepSeek | 0.485 | 2026-05 | MedHELM leaderboard (medhelm.org), v5.0.0 (official_leaderboard, medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')) | | 8 | Claude 4.6 Opus | Anthropic | 0.456 | 2026-05 | MedHELM leaderboard (medhelm.org), v5.0.0 (official_leaderboard, medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')) | | 9 | Claude 3.7 Sonnet | Anthropic | 0.45 | 2026-05 | MedHELM leaderboard (medhelm.org), v5.0.0 (official_leaderboard, medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')) | | 10 | Gemini 2.0 Flash | Google | 0.342 | 2026-05 | MedHELM leaderboard (medhelm.org), v5.0.0 (official_leaderboard, medhelm.org home, 'Current leaders Mean win rate v5.0.0' table ('10 of 11 models · Updated 14 May 2026')) | Version 5.0.0, last updated May 14, 2026, run by the Stanford-led maintainers on a roughly quarterly cadence. No Claude 5 family or GPT-5.6 rows yet. Mean win rate is relative to the evaluated cohort, so scores shift whenever the model set changes. The current official home-page ranking is v5.0.0, updated May 14, 2026; ten of eleven models are displayed. September 28 is a source check, not a new evaluation. Source review: 2026-09-28; verified. Official v5.0.0 home-page scores match all ten indexed models and still state May 14, 2026. No later result is implied by the September source check. ## First, Do NOHARM (v2) Frequency and severity of potentially harmful errors in LLM-generated medical consultation recommendations (Numerous Options Harm Assessment for Risk in Medicine); primary-care-to-specialist consults. Published by Stanford/Harvard-led consortium (50+ researchers incl. 29 board-certified physicians); hosted by ARISE (2025-12); 1,100 consultation cases, 10 specialties, 12,747 expert annotations on 4,249 management options; scale percentage safety score, higher better. Source: MAST technical leaderboard (First Do NOHARM v2 and per-benchmark results) (official-leaderboard), https://arise-ai.org/mast/technical 17 indexed rows; source board size: 19. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | LiSA 2.5 | AMBOSS | 86.2 | 2026-09 | ARISE MAST technical leaderboard (official_leaderboard, First Do NOHARM v2 overall leaderboard; LiSA 2.5; score 86.2%) | | 2 | Doximity Ask 6.1 | Doximity | 84.5 | 2026-09 | ARISE MAST technical leaderboard (official_leaderboard, First Do NOHARM v2 overall leaderboard; Doximity Ask 6.1; score 84.5%) | | 3 | OpenEvidence | OpenEvidence | 80.0 | 2026-09 | ARISE MAST technical leaderboard (official_leaderboard, First Do NOHARM v2 overall leaderboard; OpenEvidence; score 80%) | | 4 | Muse Spark 1.1 | Meta | 79.7% | 2026-08 | MAST technical leaderboard (First Do NOHARM v2 and per-benchmark results) (official_leaderboard, arise-ai.org/mast/technical, 'First Do NOHARM v2 overall metric across 19 models' ranking (Latest Flagships view)) | | 5 | Glass 5.6 Max | Glass Health | 79.7 | 2026-09 | ARISE MAST technical leaderboard (official_leaderboard, First Do NOHARM v2 overall leaderboard; Glass 5.6 Max; score 79.7%) | | 6 | Claude Opus 5 | Anthropic | 74.6% | 2026-08 | MAST technical leaderboard (First Do NOHARM v2 and per-benchmark results) (official_leaderboard, arise-ai.org/mast/technical, 'First Do NOHARM v2 overall metric across 19 models' ranking (Latest Flagships view), plus Model Leaderboard SAFETY column) | | 7 | Kimi K3 | Moonshot AI | 74.0% | 2026-08 | MAST technical leaderboard (First Do NOHARM v2 and per-benchmark results) (official_leaderboard, arise-ai.org/mast/technical, 'First Do NOHARM v2 overall metric across 19 models' ranking (Latest Flagships view), plus Model Leaderboard SAFETY column) | | 8 | GPT-5.6 Sol | OpenAI | 70.1% | 2026-08 | MAST technical leaderboard (First Do NOHARM v2 and per-benchmark results) (official_leaderboard, arise-ai.org/mast/technical, 'First Do NOHARM v2 overall metric across 19 models' ranking (Latest Flagships view), plus Model Leaderboard SAFETY column) | | 9 | GPT-5.5 | OpenAI | 70.0% | 2026-08 | MAST technical leaderboard (First Do NOHARM v2 and per-benchmark results) (official_leaderboard, arise-ai.org/mast/technical, 'First Do NOHARM v2 overall metric across 19 models' ranking (Latest Flagships view), plus Model Leaderboard SAFETY column) | | 10 | GPT-5 | OpenAI | 68.6% | 2026-08 | MAST technical leaderboard (First Do NOHARM v2 and per-benchmark results) (official_leaderboard, arise-ai.org/mast/technical, Model Leaderboard (Top 10 shown), row 10, SAFETY column) | | 11 | Claude Fable 5 | Anthropic | 65.0% | 2026-08 | MAST technical leaderboard (First Do NOHARM v2 and per-benchmark results) (official_leaderboard, arise-ai.org/mast/technical, 'First Do NOHARM v2 overall metric across 19 models' ranking (Latest Flagships view)) | | 12 | Gemini 3.1 Pro | Google | 62.6% | 2026-08 | MAST technical leaderboard (First Do NOHARM v2 and per-benchmark results) (official_leaderboard, arise-ai.org/mast/technical, 'First Do NOHARM v2 overall metric across 19 models' ranking (Latest Flagships view), plus Model Leaderboard SAFETY column) | | 13 | Gemini 2.5 Pro | Google | 61.9% | 2026-08 | MAST technical leaderboard (First Do NOHARM v2 and per-benchmark results) (official_leaderboard, arise-ai.org/mast/technical, 'First Do NOHARM v2 overall metric across 19 models' ranking (Latest Flagships view)) | | 14 | Qwen3.5 397B A17B | Alibaba | 61.1% | 2026-08 | MAST technical leaderboard (First Do NOHARM v2 and per-benchmark results) (official_leaderboard, arise-ai.org/mast/technical, 'First Do NOHARM v2 overall metric across 19 models' ranking (Latest Flagships view)) | | 15 | Kimi K2.6 | Moonshot AI | 59.1% | 2026-08 | MAST technical leaderboard (First Do NOHARM v2 and per-benchmark results) (official_leaderboard, arise-ai.org/mast/technical, 'First Do NOHARM v2 overall metric across 19 models' ranking (Latest Flagships view)) | | 16 | GLM 5.1 | Z.ai | 57.9 | 2026-09 | ARISE MAST technical leaderboard (official_leaderboard, First Do NOHARM v2 overall leaderboard; GLM 5.1; score 57.9%) | | 17 | DeepSeek R1 | DeepSeek | 55.8% | 2026-08 | MAST technical leaderboard (First Do NOHARM v2 and per-benchmark results) (official_leaderboard, arise-ai.org/mast/technical, 'First Do NOHARM v2 overall metric across 19 models' ranking (Latest Flagships view)) | Paper: arXiv 2512.01241. The v1 study found potential for severe harm in up to 24.6 percent of directly applied recommendations, with errors of omission behind more than 80 percent of the severe cases. Official technical leaderboard checked September 28, 2026; retained August dates on existing rows; newly indexed rows have no claimed measurement date. Includes labeled RAG clinical systems and base models, which have different tool access. The board is in preview. Seventeen of nineteen overall rows are indexed. Source review: 2026-09-28; verified. Checked the official v2 technical board and added five omitted published systems/models. Seventeen of nineteen rows are indexed, with RAG systems labeled separately in settings. Board remains in preview. Newly indexed scores show a September observation date with measurement date left unknown. ## HealthAgentBench Agentic task success in realistic terminal-based healthcare environments built from real clinical artifacts; evaluates agent harnesses (Claude Code, Codex, Copilot) end to end, not bare models. Published by Microsoft Research (2026-07); 54 tasks across 7 environments; 162 trials (3 attempts per task); scale mean task success rate, 0-100%, higher better; cost per task also reported. Source: HealthAgentBench leaderboard (independent-run), https://microsoft.github.io/HealthAgentBench/ 12 indexed rows; source board size: 12. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | Claude Code (Opus 5) | Anthropic | 55% | 2026-07 | HealthAgentBench leaderboard (official_leaderboard, Leaderboard table (homepage), rank 1, Success Rate column) | | 2 | Codex (GPT-5.6-sol) | OpenAI | 45% | 2026-07 | HealthAgentBench leaderboard (official_leaderboard, Leaderboard table (homepage), rank 2, Success Rate column) | | 3 | Codex (GPT 5.5) | OpenAI | 42% | 2026-07 | HealthAgentBench leaderboard (official_leaderboard, Leaderboard table (homepage), rank 3, Success Rate column) | | 4 | Copilot (Opus 4.8) | Microsoft/Anthropic | 36% | 2026-07 | HealthAgentBench leaderboard (official_leaderboard, Leaderboard table (homepage), rank 4, Success Rate column) | | 5 | Copilot (GPT 5.5) | Microsoft/OpenAI | 35% | 2026-07 | HealthAgentBench leaderboard (official_leaderboard, Leaderboard table (homepage), rank 5, Success Rate column) | | 6 | Claude Code (Opus 4.8) | Anthropic | 32% | 2026-07 | HealthAgentBench leaderboard (official_leaderboard, Leaderboard table (homepage), rank 6, Success Rate column) | | 7 | Codex (GPT 5.4) | OpenAI | 28% | 2026-07 | HealthAgentBench leaderboard (official_leaderboard, Leaderboard table (homepage), rank 7, Success Rate column) | | 8 | Claude Code (Opus 4.7) | Anthropic | 27% | 2026-07 | HealthAgentBench leaderboard (official_leaderboard, Leaderboard table (homepage), rank 8, Success Rate column) | | 9 | Codex (GPT 5.3) | OpenAI | 22% | 2026-07 | HealthAgentBench leaderboard (official_leaderboard, Homepage leaderboard, rank 9, success rate and cost/task) | | 10 | Claude Code (Opus 4.6) | Anthropic | 19% | 2026-07 | HealthAgentBench leaderboard (official_leaderboard, Leaderboard table (homepage), rank 10, Success Rate column) | | 11 | Claude Code (Sonnet 4.6) | Anthropic | 17% | 2026-07 | HealthAgentBench leaderboard (official_leaderboard, Leaderboard table (homepage), rank 11, Success Rate column) | | 12 | Codex (GPT 5.4 Mini) | OpenAI | 16% | 2026-07 | HealthAgentBench leaderboard (official_leaderboard, Leaderboard table (homepage), rank 12, Success Rate column) | Paper: arXiv 2606.31179. The rows are agent harnesses rather than bare models, and the board is run by Microsoft Research; Copilot, Microsoft's own harness, does not top it. Source review: 2026-09-28; verified. All 12 official harness results checked; added the omitted Codex GPT 5.3 row (22%). Current leader remains Claude Code (Opus 5), 55%. Source does not publish a new run date. ## CHI-Bench Long-horizon US healthcare operations workflows for agents (prior authorization, utilization management, care management), 60-80 step tasks across 4-6 stages, judged by deterministic unit tests plus an LLM judge for evidence grounding, consent, and cross-stage consistency. Published by actAVA.ai (2026-05); 75 workflows (25 per domain), 21 healthcare applications, 200+ MCP tools; scale pass@1 with binary 0/1 reward, higher better. Source: CHI-Bench leaderboard (actAVA) (mixed), https://actava.ai/benchmarks/leaderboards 44 indexed rows; source board size: 45. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | erius + claude-opus-5 | Humana (harness) / Anthropic (model) | 54.7% | 2026-07-26 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 1, Accuracy column; submission date 2026-07-26) | | 2 | erius + claude-opus-4-8 | Humana (harness) / Anthropic (model) | 37.3% | 2026-06-05 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 2, Accuracy column; submission date 2026-06-05) | | 3 | claude-code + claude-opus-5 | Anthropic | 37.3% | 2026-07-24 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 3, Accuracy column; submission date 2026-07-24) | | 4 | claude-code + claude-opus-4-8 | Anthropic | 33.3% | 2026-05-28 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 4, Accuracy column; submission date 2026-05-28) | | 5 | claude-code + claude-opus-4-6 | Anthropic | 28.0% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 5, Accuracy column; submission date 2026-05-01) | | 6 | claude-code + claude-sonnet-4-6 | Anthropic | 26.2% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 6, Accuracy column; submission date 2026-05-01) | | 7 | codex + gpt-5.6-sol | OpenAI | 25.3% | 2026-07-24 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 7, Accuracy column; submission date 2026-07-24) | | 8 | openai-agents + kimi-k3 | Moonshot AI | 25.3% | 2026-07-24 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 8, Accuracy column; submission date 2026-07-24) | | 9 | claude-code + claude-opus-4-7 | Anthropic | 24.4% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 9, Accuracy column; submission date 2026-05-01) | | 10 | claude-code + claude-fable-5 | Anthropic | 24.0% | 2026-07-22 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 10, Accuracy column; submission date 2026-07-22) | | 11 | hermes + MedGuard | MedGuard | 22.7% | 2026-07-06 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 11, Accuracy column; submission date 2026-07-06) | | 12 | codex + gpt-5.5 | OpenAI | 20.9% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 12, Accuracy column; submission date 2026-05-01) | | 13 | claude-code + claude-sonnet-5 | Anthropic | 20.0% | 2026-07-06 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 13, Accuracy column; submission date 2026-07-06) | | 14 | openai-agents + glm-5.1 | Zhipu AI | 18.7% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 14, Accuracy column; submission date 2026-05-01) | | 15 | hermes + glm-5.1 | Zhipu AI | 18.7% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 15, Accuracy column; submission date 2026-05-01) | | 16 | openai-agents + glm-5.2 | Zhipu AI | 18.7% | 2026-07-06 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 16, Accuracy column; submission date 2026-07-06) | | 17 | openclaw + claude-opus-4-7 | Anthropic | 17.3% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 17, Accuracy column; submission date 2026-05-01) | | 18 | openclaw + glm-5.1 | Zhipu AI | 16.9% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 18, Accuracy column; submission date 2026-05-01) | | 19 | hermes + qwen-3.6-max | Alibaba | 16.4% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 19, Accuracy column; submission date 2026-05-01) | | 20 | codex + gpt-5.4 | OpenAI | 16.0% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 20, Accuracy column; submission date 2026-05-01) | | 21 | openai-agents + qwen-3.6-max | Alibaba | 15.6% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 21, Accuracy column; submission date 2026-05-01) | | 22 | hermes + kimi-k2.6 | Moonshot AI | 15.6% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 22, Accuracy column; submission date 2026-05-01) | | 23 | openai-agents + kimi-k2.6 | Moonshot AI | 15.1% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 23, Accuracy column; submission date 2026-05-01) | | 24 | openai-agents + deepseek-v4-pro | DeepSeek | 14.2% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 24, Accuracy column; submission date 2026-05-01) | | 25 | hermes + deepseek-v4-pro | DeepSeek | 13.8% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 25, Accuracy column; submission date 2026-05-01) | | 26 | codex + gpt-5.6-terra | OpenAI | 13.3% | 2026-07-24 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 26, Accuracy column; submission date 2026-07-24) | | 27 | codex + gpt-5.6-luna | OpenAI | 13.3% | 2026-07-24 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 27, Accuracy column; submission date 2026-07-24) | | 28 | gemini-cli + gemini-3-flash | Google | 12.5% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 28, Accuracy column; submission date 2026-05-01) | | 29 | openclaw + deepseek-v4-pro | DeepSeek | 11.1% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 29, Accuracy column; submission date 2026-05-01) | | 30 | deepagents + glm-5.1 | Zhipu AI | 11.1% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 30, Accuracy column; submission date 2026-05-01) | | 31 | deepagents + deepseek-v4-pro | DeepSeek | 10.7% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 31, Accuracy column; submission date 2026-05-01) | | 32 | openclaw + kimi-k2.6 | Moonshot AI | 10.2% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 32, Accuracy column; submission date 2026-05-01) | | 33 | deepagents + qwen-3.6-max | Alibaba | 9.3% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 33, Accuracy column; submission date 2026-05-01) | | 34 | codex + gpt-5.4-mini | OpenAI | 8.4% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 34, Accuracy column; submission date 2026-05-01) | | 35 | openai-agents + TML Inkling 256K | Thinking Machines | 8.0% | 2026-07-24 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 35, Accuracy column; submission date 2026-07-24) | | 36 | gemini-cli + gemini-3.1-pro | Google | 7.1% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 36, Accuracy column; submission date 2026-05-01) | | 37 | claude-code + claude-haiku-4-5 | Anthropic | 6.2% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 37, Accuracy column; submission date 2026-05-01) | | 38 | openai-agents + grok-4.3 | SpaceX AI | 5.8% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 38, Accuracy column; submission date 2026-05-01) | | 39 | openclaw + qwen-3.6-max | Alibaba | 4.9% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 39, Accuracy column; submission date 2026-05-01) | | 40 | hermes + grok-4.3 | SpaceX AI | 4.4% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 40, Accuracy column; submission date 2026-05-01) | | 41 | deepagents + kimi-k2.6 | Moonshot AI | 3.1% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 41, Accuracy column; submission date 2026-05-01) | | 42 | deepagents + grok-4.3 | SpaceX AI | 2.2% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 42, Accuracy column; submission date 2026-05-01) | | 43 | openclaw + grok-4.3 | SpaceX AI | 0.4% | 2026-05-01 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 43, Accuracy column; submission date 2026-05-01) | | 44 | openai-agents + Nemotron 3 Ultra 256K | NVIDIA | 0.0% | 2026-07-24 | CHI-Bench leaderboard (actAVA) (official_leaderboard, All Domains leaderboard, rank 44, Accuracy column; submission date 2026-07-24) | Official board updated 2026-08-12: 45 submitted harness configurations, 44 with all-domain accuracy. The PA-only MedArise submission has no all-domain score and is excluded here. Row dates are submission dates; run dates are not published. Community submissions and author-run baselines share the automated workspace judge. Source review: 2026-09-28; verified. All 45 submissions inspected; retained all 44 numeric all-domain results and excluded one PA-only result. No newer scores found on the official board. ## MedCode (Vals AI) ICD-10-CM diagnosis coding for entire hospital stays: models assign primary and secondary codes from discharge summaries plus progress/consult notes; ground truth double-annotated by certified professional coders. Published by Vals AI (dataset with Protege) (2026-02); 2,755 patient records; scale percentage accuracy 0-100, higher better. Source: Vals AI MedCode leaderboard (independent-run), https://www.vals.ai/benchmarks/medcode 102 indexed rows; source board size: 102. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | Claude Opus 5 | Anthropic | 63.57% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-5"].accuracy; Overall leaderboard rank 1) | | 2 | Gemini 3.1 Pro Preview (02/26) | Google | 59.06% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3.1-pro-preview"].accuracy; Overall leaderboard rank 2) | | 3 | Claude Fable 5 | Anthropic | 56.07% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-fable-5"].accuracy; Overall leaderboard rank 3) | | 4 | Gemini 3 Flash (12/25) | Google | 55.92% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3-flash-preview"].accuracy; Overall leaderboard rank 4) | | 5 | Gemini 3.5 Flash | Google | 55.83% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3.5-flash"].accuracy; Overall leaderboard rank 5) | | 6 | Claude Opus 4.7 | Anthropic | 54.86% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-7"].accuracy; Overall leaderboard rank 6) | | 7 | Claude Fable 5.1 | Anthropic | 53.51% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-fable-5-1"].accuracy; Overall leaderboard rank 7) | | 8 | Gemini 3.7 Flash | Google | 53.39% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3.7-flash"].accuracy; Overall leaderboard rank 8) | | 9 | Claude Opus 4.8 | Anthropic | 53.22% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-8"].accuracy; Overall leaderboard rank 9) | | 10 | Gemini 3.6 Flash | Google | 53.15% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3.6-flash"].accuracy; Overall leaderboard rank 10) | | 11 | Claude Sonnet 5.5 | Anthropic | 52.92% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-sonnet-5-5"].accuracy; Overall leaderboard rank 11) | | 12 | GPT 5.1 | OpenAI | 52.73% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.1-2025-11-13"].accuracy; Overall leaderboard rank 12) | | 13 | Gemini 3 Pro (11/25) | Google | 52.20% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3-pro-preview"].accuracy; Overall leaderboard rank 13) | | 14 | Muse Spark | Meta | 51.31% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["meta/muse_spark"].accuracy; Overall leaderboard rank 14) | | 15 | Gemini 2.5 Pro | Google | 50.59% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-pro"].accuracy; Overall leaderboard rank 15) | | 16 | Claude Opus 5.5 | Anthropic | 49.80% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-5-5"].accuracy; Overall leaderboard rank 16) | | 17 | GPT 5.2 | OpenAI | 49.75% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.2-2025-12-11"].accuracy; Overall leaderboard rank 17) | | 18 | GPT 5 | OpenAI | 49.63% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5-2025-08-07"].accuracy; Overall leaderboard rank 18) | | 19 | Grok 4.7 | SpaceXAI | 49.55% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4.7"].accuracy; Overall leaderboard rank 19) | | 20 | Muse Spark 1.2 | Meta | 49.35% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["meta/muse_spark_1_2"].accuracy; Overall leaderboard rank 20) | | 21 | Claude Opus 4.5 (Thinking) | Anthropic | 49.16% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-5-20251101-thinking"].accuracy; Overall leaderboard rank 21) | | 22 | Claude Opus 4.6 (Thinking) | Anthropic | 49.13% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-6-thinking"].accuracy; Overall leaderboard rank 22) | | 23 | GPT 5.5 | OpenAI | 49.10% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.5"].accuracy; Overall leaderboard rank 23) | | 24 | Kimi K3 | Moonshot AI | 48.88% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["kimi/kimi-k3"].accuracy; Overall leaderboard rank 24) | | 25 | GPT-6 Astra | OpenAI | 48.49% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-6-astra"].accuracy; Overall leaderboard rank 25) | | 26 | Claude Opus 4.6 (Nonthinking) | Anthropic | 48.24% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-6"].accuracy; Overall leaderboard rank 26) | | 27 | Gemini 3.8 Flash | Google | 48.13% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3.8-flash"].accuracy; Overall leaderboard rank 27) | | 28 | Gemini 3.1 Flash Lite Preview | Google | 47.60% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3.1-flash-lite-preview"].accuracy; Overall leaderboard rank 28) | | 29 | Claude Sonnet 5 | Anthropic | 47.54% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-sonnet-5"].accuracy; Overall leaderboard rank 29) | | 30 | o3 | OpenAI | 47.29% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/o3-2025-04-16"].accuracy; Overall leaderboard rank 30) | | 31 | Claude Opus 4.1 (Thinking) | Anthropic | 47.23% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-1-20250805-thinking"].accuracy; Overall leaderboard rank 31) | | 32 | GPT-6 Sol | OpenAI | 47.07% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-6-sol"].accuracy; Overall leaderboard rank 32) | | 33 | MiniMax-M3 | MiniMax | 46.29% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["minimax/MiniMax-M3"].accuracy; Overall leaderboard rank 33) | | 34 | Claude Opus 4.5 (Nonthinking) | Anthropic | 45.17% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-5-20251101"].accuracy; Overall leaderboard rank 34) | | 35 | MiMo V2.6 Pro | Xiaomi | 44.97% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["xiaomi/mimo-v2.6-pro"].accuracy; Overall leaderboard rank 35) | | 36 | Grok 4.6 | SpaceXAI | 44.71% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4.6"].accuracy; Overall leaderboard rank 36) | | 37 | GPT-6 Luna | OpenAI | 44.69% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-6-luna"].accuracy; Overall leaderboard rank 37) | | 38 | Claude Sonnet 4.5 (Thinking) | Anthropic | 44.13% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-sonnet-4-5-20250929-thinking"].accuracy; Overall leaderboard rank 38) | | 39 | GPT-5.6 Sol | OpenAI | 43.97% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.6-sol"].accuracy; Overall leaderboard rank 39) | | 40 | Gemini 3.5 Flash Lite | Google | 43.49% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3.5-flash-lite"].accuracy; Overall leaderboard rank 40) | | 41 | GPT-5.6 Terra | OpenAI | 43.41% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.6-terra"].accuracy; Overall leaderboard rank 41) | | 42 | Grok 4.5 | SpaceXAI | 43.29% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4.5"].accuracy; Overall leaderboard rank 42) | | 43 | Hy4 Preview | Tencent | 43.25% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["tencent/hy4-preview"].accuracy; Overall leaderboard rank 43) | | 44 | GPT 5 Mini | OpenAI | 43.05% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5-mini-2025-08-07"].accuracy; Overall leaderboard rank 44) | | 45 | GLM 5.3 | zAI | 42.86% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["zai/glm-5.3"].accuracy; Overall leaderboard rank 45) | | 46 | DeepSeek V4 Pro 0813 | DeepSeek | 42.47% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["deepseek/deepseek-v4-pro-0813"].accuracy; Overall leaderboard rank 46) | | 47 | GPT-5.6 Luna | OpenAI | 42.39% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.6-luna"].accuracy; Overall leaderboard rank 47) | | 48 | GLM 5.1 | zAI | 41.60% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["zai/glm-5.1"].accuracy; Overall leaderboard rank 48) | | 49 | DeepSeek V4 Flash 0731 | DeepSeek | 41.41% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["deepseek/deepseek-v4-flash-0731"].accuracy; Overall leaderboard rank 49) | | 50 | Claude Opus 4.1 (Nonthinking) | Anthropic | 41.37% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-1-20250805"].accuracy; Overall leaderboard rank 50) | | 51 | GPT 5.4 (xhigh) | OpenAI | 41.29% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.4-2026-03-05"].accuracy; Overall leaderboard rank 51) | | 52 | Inkling | Thinking Machines | 41.19% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["thinkingmachines/inkling"].accuracy; Overall leaderboard rank 52) | | 53 | DeepSeek V4.1 Flash | DeepSeek | 41.17% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["deepseek/deepseek-v4.1-flash"].accuracy; Overall leaderboard rank 53) | | 54 | MiMo V2.6 Flash | Xiaomi | 41.06% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["xiaomi/mimo-v2.6-flash"].accuracy; Overall leaderboard rank 54) | | 55 | GPT 5.4 Nano | OpenAI | 41.03% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.4-nano-2026-03-17"].accuracy; Overall leaderboard rank 55) | | 56 | GLM 5.2 | zAI | 40.77% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["zai/glm-5.2"].accuracy; Overall leaderboard rank 56) | | 57 | Qwen 3.8 Max | Alibaba | 40.67% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["alibaba/qwen3.8-max"].accuracy; Overall leaderboard rank 57) | | 58 | Claude Sonnet 4.5 (Nonthinking) | Anthropic | 40.57% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-sonnet-4-5-20250929"].accuracy; Overall leaderboard rank 58) | | 59 | Gemini 2.5 Flash Preview (9/25) (Nonthinking) | Google | 40.54% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-flash-preview-09-2025"].accuracy; Overall leaderboard rank 59) | | 60 | DeepSeek V4 | DeepSeek | 40.45% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["deepseek/deepseek-v4-pro"].accuracy; Overall leaderboard rank 60) | | 61 | Gemini 2.5 Flash (7/17) (Thinking) | Google | 40.36% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-flash-thinking"].accuracy; Overall leaderboard rank 61) | | 62 | Gemini 2.5 Flash Preview (9/25) (Thinking) | Google | 40.33% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-flash-preview-09-2025-thinking"].accuracy; Overall leaderboard rank 62) | | 63 | Kimi K2.6 | Moonshot AI | 40.14% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["kimi/kimi-k2.6"].accuracy; Overall leaderboard rank 63) | | 64 | Kimi K2.5 | Moonshot AI | 39.32% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["kimi/kimi-k2.5-thinking"].accuracy; Overall leaderboard rank 64) | | 65 | Qwen 3.7 Max | Alibaba | 38.75% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["alibaba/qwen3.7-max"].accuracy; Overall leaderboard rank 65) | | 66 | Nemotron 3 Ultra | NVIDIA | 38.62% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["nvidia/nemotron-3-ultra-550b-a55b"].accuracy; Overall leaderboard rank 66) | | 67 | Gemini 2.5 Flash (7/17) (Nonthinking) | Google | 38.42% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-flash"].accuracy; Overall leaderboard rank 67) | | 68 | Grok 4 | SpaceXAI | 38.08% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4-0709"].accuracy; Overall leaderboard rank 68) | | 69 | Grok 4.3 | SpaceXAI | 38.07% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4.3"].accuracy; Overall leaderboard rank 69) | | 70 | Inkling Small | Thinking Machines | 37.89% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["thinkingmachines/inkling-small"].accuracy; Overall leaderboard rank 70) | | 71 | Grok 4 Fast (Reasoning) | SpaceXAI | 37.38% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4-fast-reasoning"].accuracy; Overall leaderboard rank 71) | | 72 | Qwen 3.6 Plus | Alibaba | 36.89% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["alibaba/qwen3.6-plus"].accuracy; Overall leaderboard rank 72) | | 73 | Llama 4 Maverick | Meta | 36.51% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["fireworks/llama4-maverick-instruct-basic"].accuracy; Overall leaderboard rank 73) | | 74 | Claude Sonnet 4 (Thinking) | Anthropic | 34.96% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-sonnet-4-20250514-thinking"].accuracy; Overall leaderboard rank 74) | | 75 | MiniMax-M2.7 | MiniMax | 34.44% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["minimax/MiniMax-M2.7"].accuracy; Overall leaderboard rank 75) | | 76 | Gemini 2.5 Flash Lite (9/25) (Thinking) | Google | 34.19% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-flash-lite-preview-09-2025-thinking"].accuracy; Overall leaderboard rank 76) | | 77 | MiniMax-M2.1 | MiniMax | 34.08% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["minimax/MiniMax-M2.1"].accuracy; Overall leaderboard rank 77) | | 78 | Claude Sonnet 4 (Nonthinking) | Anthropic | 33.94% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-sonnet-4-20250514"].accuracy; Overall leaderboard rank 78) | | 79 | o4 Mini | OpenAI | 33.79% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/o4-mini-2025-04-16"].accuracy; Overall leaderboard rank 79) | | 80 | Mistral Medium 3.5 | Mistral | 33.75% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["mistralai/mistral-medium-3.5"].accuracy; Overall leaderboard rank 80) | | 81 | Qwen 3.5 Flash | Alibaba | 33.00% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["alibaba/qwen3.5-flash"].accuracy; Overall leaderboard rank 81) | | 82 | GLM 4.7 | zAI | 32.77% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["zai/glm-4.7"].accuracy; Overall leaderboard rank 82) | | 83 | Claude Haiku 4.5 (Thinking) | Anthropic | 32.68% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-haiku-4-5-20251001-thinking"].accuracy; Overall leaderboard rank 83) | | 84 | MiMo V2.5 Pro | Xiaomi | 32.48% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["xiaomi/mimo-v2.5-pro"].accuracy; Overall leaderboard rank 84) | | 85 | Ling 3.0 Flash | Ant Group | 32.27% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["ant/ling-3.0-flash-2607"].accuracy; Overall leaderboard rank 85) | | 86 | Grok 4.20 (Reasoning) | SpaceXAI | 32.16% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4.20-0309-reasoning"].accuracy; Overall leaderboard rank 86) | | 87 | MiMo V2.5 | Xiaomi | 31.89% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["xiaomi/mimo-v2.5"].accuracy; Overall leaderboard rank 87) | | 88 | Qwen 3 VL Plus | Alibaba | 31.65% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["alibaba/qwen3-vl-plus-2025-09-23"].accuracy; Overall leaderboard rank 88) | | 89 | Qwen 3 Max Thinking | Alibaba | 31.37% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["alibaba/qwen3-max-2026-01-23"].accuracy; Overall leaderboard rank 89) | | 90 | Mercury 2.5 | Inception | 31.33% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["inception/mercury-2.5"].accuracy; Overall leaderboard rank 90) | | 91 | GPT 5 Nano | OpenAI | 30.44% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5-nano-2025-08-07"].accuracy; Overall leaderboard rank 91) | | 92 | Grok 4 Fast (Non-Reasoning) | SpaceXAI | 30.04% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4-fast-non-reasoning"].accuracy; Overall leaderboard rank 92) | | 93 | Ling 3.0 Flash Fin | Ant Group | 29.30% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["ant/ling-3.0-flash-af-rc3"].accuracy; Overall leaderboard rank 93) | | 94 | Qwen 3.8 27B | Alibaba | 28.70% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["alibaba/qwen3.8-27b"].accuracy; Overall leaderboard rank 94) | | 95 | Grok 4.1 Fast Non-Reasoning | SpaceXAI | 28.35% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4-1-fast-non-reasoning"].accuracy; Overall leaderboard rank 95) | | 96 | Grok 4.1 Fast (Reasoning) | SpaceXAI | 28.08% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4-1-fast-reasoning"].accuracy; Overall leaderboard rank 96) | | 97 | Gemini 2.5 Flash Lite (Nonthinking) | Google | 27.11% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-flash-lite"].accuracy; Overall leaderboard rank 97) | | 98 | Gemini 2.5 Flash Lite (9/25) (Nonthinking) | Google | 27.08% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-flash-lite-preview-09-2025"].accuracy; Overall leaderboard rank 98) | | 99 | Llama 4 Scout | Meta | 23.31% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["together/meta-llama/Llama-4-Scout-17B-16E-Instruct"].accuracy; Overall leaderboard rank 99) | | 100 | Laguna M.1 | Poolside | 23.11% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["poolside/laguna-m.1"].accuracy; Overall leaderboard rank 100) | | 101 | Laguna XS.2 | Poolside | 21.25% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["poolside/laguna-xs.2"].accuracy; Overall leaderboard rank 101) | | 102 | Command A+ | Cohere | 19.72% | 2026-09-26 | Vals AI MedCode leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["cohere/command-a-plus-05-2026"].accuracy; Overall leaderboard rank 102) | Vals AI runs this benchmark. Full Overall data contains 102 scored model configurations in the 2026-09-26 snapshot, including older and reasoning variants. Scores use the published percent scale; numeric values retain source precision and displayed values round to two decimals. Snapshot update dates are not model measurement dates. Source review: 2026-09-28; verified. All 102 scored configurations checked against the first-party embedded table; source parameters captured per row. Updated 2026-09-26; individual run dates unavailable. ## MedScribe (Vals AI) Clinical documentation support: quality of SOAP notes generated from clinical visits, scored against rubrics for documentation quality and compliance. Published by Vals AI (dataset with Protege) (2026-02); 100 rubric-scored SOAP-note cases; scale percentage accuracy 0-100, higher better. Source: Vals AI MedScribe leaderboard (independent-run), https://www.vals.ai/benchmarks/medscribe 104 indexed rows; source board size: 104. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | Claude Opus 5.5 | Anthropic | 91.43% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-5-5"].accuracy; Overall leaderboard rank 1) | | 2 | Claude Fable 5.1 | Anthropic | 91.29% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-fable-5-1"].accuracy; Overall leaderboard rank 2) | | 3 | Claude Sonnet 5.5 | Anthropic | 91.10% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-sonnet-5-5"].accuracy; Overall leaderboard rank 3) | | 4 | Claude Opus 5 | Anthropic | 90.98% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-5"].accuracy; Overall leaderboard rank 4) | | 5 | Muse Spark 1.2 | Meta | 90.06% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["meta/muse_spark_1_2"].accuracy; Overall leaderboard rank 5) | | 6 | Grok 4.7 | SpaceXAI | 89.38% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4.7"].accuracy; Overall leaderboard rank 6) | | 7 | GLM 5.3 Flash | zAI | 88.94% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["zai/glm-5.3-flash"].accuracy; Overall leaderboard rank 7) | | 8 | Muse Spark 1.1 | Meta | 88.89% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["meta/muse_spark_1_1"].accuracy; Overall leaderboard rank 8) | | 9 | GLM 5.3 | zAI | 88.81% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["zai/glm-5.3"].accuracy; Overall leaderboard rank 9) | | 10 | Claude Fable 5 | Anthropic | 88.52% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-fable-5"].accuracy; Overall leaderboard rank 10) | | 11 | MiMo V2.6 Pro | Xiaomi | 88.31% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["xiaomi/mimo-v2.6-pro"].accuracy; Overall leaderboard rank 11) | | 12 | GPT 5.1 | OpenAI | 88.09% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.1-2025-11-13"].accuracy; Overall leaderboard rank 12) | | 13 | Kimi K3 | Moonshot AI | 87.96% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["kimi/kimi-k3"].accuracy; Overall leaderboard rank 13) | | 14 | GPT-6 Astra | OpenAI | 87.91% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-6-astra"].accuracy; Overall leaderboard rank 14) | | 15 | MiniMax-M3 | MiniMax | 87.25% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["minimax/MiniMax-M3"].accuracy; Overall leaderboard rank 15) | | 16 | Grok 4.5 | SpaceXAI | 86.88% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4.5"].accuracy; Overall leaderboard rank 16) | | 17 | GPT 5.5 | OpenAI | 86.87% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.5"].accuracy; Overall leaderboard rank 17) | | 18 | Claude Opus 4.6 (Nonthinking) | Anthropic | 86.74% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-6"].accuracy; Overall leaderboard rank 18) | | 19 | Grok 4.6 | xAI | 86.53% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4.6"].accuracy; Overall leaderboard rank 19) | | 20 | Claude Opus 4.6 (Thinking) | Anthropic | 86.13% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-6-thinking"].accuracy; Overall leaderboard rank 20) | | 21 | Muse Spark | Meta | 85.90% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["meta/muse_spark"].accuracy; Overall leaderboard rank 21) | | 22 | Claude Opus 4.8 | Anthropic | 85.75% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-8"].accuracy; Overall leaderboard rank 22) | | 23 | DeepSeek V4.1 Flash | DeepSeek | 85.50% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["deepseek/deepseek-v4.1-flash"].accuracy; Overall leaderboard rank 23) | | 24 | Inkling | Thinking Machines | 85.41% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["thinkingmachines/inkling"].accuracy; Overall leaderboard rank 24) | | 25 | Claude Opus 4.5 (Thinking) | Anthropic | 85.32% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-5-20251101-thinking"].accuracy; Overall leaderboard rank 25) | | 26 | MiMo V2.6 Flash | Xiaomi | 85.28% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["xiaomi/mimo-v2.6-flash"].accuracy; Overall leaderboard rank 26) | | 27 | GPT-5.6 Sol | OpenAI | 85.23% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.6-sol"].accuracy; Overall leaderboard rank 27) | | 28 | Claude Haiku 4.5 (Thinking) | Anthropic | 85.23% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-haiku-4-5-20251001-thinking"].accuracy; Overall leaderboard rank 28) | | 29 | Qwen 3.8 Max | Alibaba | 84.95% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["alibaba/qwen3.8-max"].accuracy; Overall leaderboard rank 29) | | 30 | Claude Sonnet 4.5 (Nonthinking) | Anthropic | 84.52% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-sonnet-4-5-20250929"].accuracy; Overall leaderboard rank 30) | | 31 | Gemini 3.8 Flash | Google | 84.50% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3.8-flash"].accuracy; Overall leaderboard rank 31) | | 32 | GPT-5.6 Luna | OpenAI | 84.39% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.6-luna"].accuracy; Overall leaderboard rank 32) | | 33 | GPT 5.2 | OpenAI | 84.39% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.2-2025-12-11"].accuracy; Overall leaderboard rank 33) | | 34 | Inkling Small | Thinking Machines | 84.11% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["thinkingmachines/inkling-small"].accuracy; Overall leaderboard rank 34) | | 35 | Claude Sonnet 4.5 (Thinking) | Anthropic | 84.10% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-sonnet-4-5-20250929-thinking"].accuracy; Overall leaderboard rank 35) | | 36 | Gemini 3.7 Flash | Google | 83.94% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3.7-flash"].accuracy; Overall leaderboard rank 36) | | 37 | Qwen 3.8 27B | Alibaba | 83.85% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["alibaba/qwen3.8-27b"].accuracy; Overall leaderboard rank 37) | | 38 | MiMo V2.5 Pro | Xiaomi | 83.73% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["xiaomi/mimo-v2.5-pro"].accuracy; Overall leaderboard rank 38) | | 39 | GPT-6 Luna | OpenAI | 83.71% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-6-luna"].accuracy; Overall leaderboard rank 39) | | 40 | GPT 5 | OpenAI | 83.65% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5-2025-08-07"].accuracy; Overall leaderboard rank 40) | | 41 | Hy4 Preview | Tencent | 83.60% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["tencent/hy4-preview"].accuracy; Overall leaderboard rank 41) | | 42 | GLM 5.2 | zAI | 83.53% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["zai/glm-5.2"].accuracy; Overall leaderboard rank 42) | | 43 | Claude Opus 4.5 (Nonthinking) | Anthropic | 83.25% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-5-20251101"].accuracy; Overall leaderboard rank 43) | | 44 | Gemini 2.5 Flash (7/17) (Thinking) | Google | 82.98% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-flash-thinking"].accuracy; Overall leaderboard rank 44) | | 45 | Claude Opus 4.7 | Anthropic | 82.95% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-7"].accuracy; Overall leaderboard rank 45) | | 46 | Gemini 2.5 Flash (7/17) (Nonthinking) | Google | 82.87% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-flash"].accuracy; Overall leaderboard rank 46) | | 47 | GPT-5.6 Terra | OpenAI | 82.87% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.6-terra"].accuracy; Overall leaderboard rank 47) | | 48 | GPT-6 Sol | OpenAI | 82.03% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-6-sol"].accuracy; Overall leaderboard rank 48) | | 49 | Grok 4 Fast (Reasoning) | SpaceXAI | 81.63% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4-fast-reasoning"].accuracy; Overall leaderboard rank 49) | | 50 | Ling 3.0 Flash | Ant Group | 80.90% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["ant/ling-3.0-flash-2607"].accuracy; Overall leaderboard rank 50) | | 51 | MiniMax-M2.1 | MiniMax | 80.78% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["minimax/MiniMax-M2.1"].accuracy; Overall leaderboard rank 51) | | 52 | GPT 5 Mini | OpenAI | 80.58% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5-mini-2025-08-07"].accuracy; Overall leaderboard rank 52) | | 53 | DeepSeek V4 Flash 0731 | DeepSeek | 80.36% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["deepseek/deepseek-v4-flash-0731"].accuracy; Overall leaderboard rank 53) | | 54 | DeepSeek V4 Pro 0813 | DeepSeek | 80.17% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["deepseek/deepseek-v4-pro-0813"].accuracy; Overall leaderboard rank 54) | | 55 | MiniMax-M2.7 | MiniMax | 79.87% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["minimax/MiniMax-M2.7"].accuracy; Overall leaderboard rank 55) | | 56 | Grok 4 Fast (Non-Reasoning) | SpaceXAI | 79.72% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4-fast-non-reasoning"].accuracy; Overall leaderboard rank 56) | | 57 | Gemini 3.6 Flash | Google | 79.66% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3.6-flash"].accuracy; Overall leaderboard rank 57) | | 58 | Qwen 3.7 Max | Alibaba | 79.40% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["alibaba/qwen3.7-max"].accuracy; Overall leaderboard rank 58) | | 59 | Grok 4.1 Fast (Reasoning) | SpaceXAI | 78.73% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4-1-fast-reasoning"].accuracy; Overall leaderboard rank 59) | | 60 | Gemini 2.5 Flash Preview (9/25) (Thinking) | Google | 78.50% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-flash-preview-09-2025-thinking"].accuracy; Overall leaderboard rank 60) | | 61 | Grok 4 | SpaceXAI | 78.15% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4-0709"].accuracy; Overall leaderboard rank 61) | | 62 | Kimi K2.6 | Moonshot AI | 78.15% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["kimi/kimi-k2.6"].accuracy; Overall leaderboard rank 62) | | 63 | Gemini 2.5 Flash Preview (9/25) (Nonthinking) | Google | 77.95% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-flash-preview-09-2025"].accuracy; Overall leaderboard rank 63) | | 64 | GPT 5.4 (xhigh) | OpenAI | 77.55% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.4-2026-03-05"].accuracy; Overall leaderboard rank 64) | | 65 | Grok 4.1 Fast Non-Reasoning | SpaceXAI | 77.46% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4-1-fast-non-reasoning"].accuracy; Overall leaderboard rank 65) | | 66 | Qwen 3 VL Plus | Alibaba | 77.13% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["alibaba/qwen3-vl-plus-2025-09-23"].accuracy; Overall leaderboard rank 66) | | 67 | GPT 5.4 Nano | OpenAI | 77.09% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5.4-nano-2026-03-17"].accuracy; Overall leaderboard rank 67) | | 68 | Qwen 3.6 Plus | Alibaba | 76.96% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["alibaba/qwen3.6-plus"].accuracy; Overall leaderboard rank 68) | | 69 | o3 | OpenAI | 76.65% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/o3-2025-04-16"].accuracy; Overall leaderboard rank 69) | | 70 | Gemini 3.5 Flash | Google | 76.57% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3.5-flash"].accuracy; Overall leaderboard rank 70) | | 71 | Kimi K2.5 | Moonshot AI | 76.44% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["kimi/kimi-k2.5-thinking"].accuracy; Overall leaderboard rank 71) | | 72 | Gemini 3.1 Pro Preview (02/26) | Google | 76.11% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3.1-pro-preview"].accuracy; Overall leaderboard rank 72) | | 73 | Claude Sonnet 5 | Anthropic | 76.05% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-sonnet-5"].accuracy; Overall leaderboard rank 73) | | 74 | Gemini 2.5 Flash Lite (9/25) (Nonthinking) | Google | 75.82% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-flash-lite-preview-09-2025"].accuracy; Overall leaderboard rank 74) | | 75 | Ling 3.0 Flash Fin | Ant Group | 75.59% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["ant/ling-3.0-flash-af-rc3"].accuracy; Overall leaderboard rank 75) | | 76 | DeepSeek V4 | DeepSeek | 75.14% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["deepseek/deepseek-v4-pro"].accuracy; Overall leaderboard rank 76) | | 77 | Grok 4.3 | SpaceXAI | 74.40% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4.3"].accuracy; Overall leaderboard rank 77) | | 78 | Claude Opus 4.1 (Thinking) | Anthropic | 73.90% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-1-20250805-thinking"].accuracy; Overall leaderboard rank 78) | | 79 | Gemini 2.5 Pro | Google | 73.55% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-pro"].accuracy; Overall leaderboard rank 79) | | 80 | GPT 5 Nano | OpenAI | 72.86% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/gpt-5-nano-2025-08-07"].accuracy; Overall leaderboard rank 80) | | 81 | Gemini 2.5 Flash Lite (Nonthinking) | Google | 72.83% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-flash-lite"].accuracy; Overall leaderboard rank 81) | | 82 | Qwen 3 Max Thinking | Alibaba | 72.71% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["alibaba/qwen3-max-2026-01-23"].accuracy; Overall leaderboard rank 82) | | 83 | Claude Sonnet 4 (Nonthinking) | Anthropic | 72.41% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-sonnet-4-20250514"].accuracy; Overall leaderboard rank 83) | | 84 | GLM 5.1 | zAI | 72.27% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["zai/glm-5.1"].accuracy; Overall leaderboard rank 84) | | 85 | MiMo V2.5 | Xiaomi | 72.15% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["xiaomi/mimo-v2.5"].accuracy; Overall leaderboard rank 85) | | 86 | Gemini 3 Pro (11/25) | Google | 72.04% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3-pro-preview"].accuracy; Overall leaderboard rank 86) | | 87 | Claude Opus 4.1 (Nonthinking) | Anthropic | 71.75% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-opus-4-1-20250805"].accuracy; Overall leaderboard rank 87) | | 88 | Gemini 3.5 Flash Lite | Google | 70.89% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3.5-flash-lite"].accuracy; Overall leaderboard rank 88) | | 89 | Qwen 3.5 Flash | Alibaba | 70.62% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["alibaba/qwen3.5-flash"].accuracy; Overall leaderboard rank 89) | | 90 | Gemini 3 Flash (12/25) | Google | 69.92% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3-flash-preview"].accuracy; Overall leaderboard rank 90) | | 91 | Claude Sonnet 4 (Thinking) | Anthropic | 69.35% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["anthropic/claude-sonnet-4-20250514-thinking"].accuracy; Overall leaderboard rank 91) | | 92 | o4 Mini | OpenAI | 69.14% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["openai/o4-mini-2025-04-16"].accuracy; Overall leaderboard rank 92) | | 93 | GLM 4.7 | zAI | 68.63% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["zai/glm-4.7"].accuracy; Overall leaderboard rank 93) | | 94 | Mistral Medium 3.5 | Mistral | 67.73% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["mistralai/mistral-medium-3.5"].accuracy; Overall leaderboard rank 94) | | 95 | Gemini 2.5 Flash Lite (9/25) (Thinking) | Google | 66.88% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-2.5-flash-lite-preview-09-2025-thinking"].accuracy; Overall leaderboard rank 95) | | 96 | Laguna M.1 | Poolside | 65.91% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["poolside/laguna-m.1"].accuracy; Overall leaderboard rank 96) | | 97 | Gemini 3.1 Flash Lite Preview | Google | 63.90% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["google/gemini-3.1-flash-lite-preview"].accuracy; Overall leaderboard rank 97) | | 98 | Grok 4.20 (Reasoning) | SpaceXAI | 63.41% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["grok/grok-4.20-0309-reasoning"].accuracy; Overall leaderboard rank 98) | | 99 | Laguna XS.2 | Poolside | 61.43% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["poolside/laguna-xs.2"].accuracy; Overall leaderboard rank 99) | | 100 | Command A+ | Cohere | 55.68% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["cohere/command-a-plus-05-2026"].accuracy; Overall leaderboard rank 100) | | 101 | Mercury 2.5 | Inception | 55.09% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["inception/mercury-2.5"].accuracy; Overall leaderboard rank 101) | | 102 | Llama 4 Maverick | Meta | 54.22% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["fireworks/llama4-maverick-instruct-basic"].accuracy; Overall leaderboard rank 102) | | 103 | Llama 4 Scout | Meta | 50.59% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["together/meta-llama/Llama-4-Scout-17B-16E-Instruct"].accuracy; Overall leaderboard rank 103) | | 104 | Nemotron 3.5 Lightning | NVIDIA | 4.27% | 2026-09-26 | Vals AI MedScribe leaderboard (official_leaderboard, Embedded BenchmarkView data, tasks.overall["fireworks/nemotron-lightning-3p5-30b-a3b"].accuracy; Overall leaderboard rank 104) | Vals AI runs this benchmark. Full Overall data contains 104 scored model configurations in the 2026-09-26 snapshot, including older and reasoning variants. Scores use the published percent scale; numeric values retain source precision and displayed values round to two decimals. Snapshot update dates are not model measurement dates. Source review: 2026-09-28; verified. All 104 scored configurations checked against the first-party embedded table; source parameters captured per row. Updated 2026-09-26; individual run dates unavailable. ## MedXpertQA (MM) Expert-level multimodal medical multiple-choice QA covering clinical images (X-ray, histology, dermatology, charts) across 17 specialties; MM subset of the 4,460-question MedXpertQA benchmark. Published by TsinghuaC3I (Tsinghua University) (2025-01); 2,000 multimodal questions (MM subset); scale percentage accuracy 0-100, higher better. Source: Introducing Muse Spark: Scaling Towards Personal Superintelligence (mixed), https://ai.meta.com/blog/introducing-muse-spark-msl/ 22 indexed rows; source board size: 22. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | GPT-5.6 Sol | OpenAI | 81.5 | null | Qwen3.8-Max: A New Bar for Coding and Cowork (launch_post, Full Benchmark Table, second (multimodal) table, row MedXpertQA-MM; columns Opus4.8 / Fable5 / Gemini3.1-Pro / GPT5.6-Sol / Qwen3.7-Plus / Qwen3.8-Max) | | 2 | Gemini 3.1 Pro | Google | 81.3% | 2026-04 | Introducing Muse Spark: Scaling Towards Personal Superintelligence (launch_post, Launch-post benchmark table image, HEALTH section, row MedXpertQA (MM), column Gemini 3.1 Pro High; the same table is page 5 of the Eval Methodology PDF; protocol p. 2: 'MedXpertQA Text/Multimodal: ... The multimodal variant contains 2,000 multimodal medical questions with clinical images (X-rays, histology, dermatology, etc.) and 5 answer choices (A-E). For grading, we use gpt-oss-120b to parse the predicted answer letter from free-form text.') | | 3 | Qwen3.8 Max | Alibaba | 80.4% | 2026-08 | Qwen3.8-Max: A New Bar for Coding and Cowork (launch_post, Multimodal Benchmarks table, Multimodal Reasoning section, row MedXpertQA-MM, column Qwen3.8-Max; header row: | | Opus4.8 | Fable5 | Gemini3.1-Pro | GPT5.6-Sol | Qwen3.7-Plus | Qwen3.8-Max |) | | 4 | Claude Fable 5 | Anthropic | 80.0 | null | Qwen3.8-Max: A New Bar for Coding and Cowork (launch_post, Full Benchmark Table, second (multimodal) table, row MedXpertQA-MM; columns Opus4.8 / Fable5 / Gemini3.1-Pro / GPT5.6-Sol / Qwen3.7-Plus / Qwen3.8-Max) | | 5 | Muse Spark | Meta | 78.4% | 2026-04 | Introducing Muse Spark: Scaling Towards Personal Superintelligence (launch_post, Launch-post benchmark table image, HEALTH section, row MedXpertQA (MM), column Muse Spark Thinking; the same table is page 5 of the Eval Methodology PDF; protocol p. 2: 'MedXpertQA Text/Multimodal: ... The multimodal variant contains 2,000 multimodal medical questions with clinical images (X-rays, histology, dermatology, etc.) and 5 answer choices (A-E). For grading, we use gpt-oss-120b to parse the predicted answer letter from free-form text.') | | 6 | GPT-5.4 | OpenAI | 77.1% | 2026-04 | Introducing Muse Spark: Scaling Towards Personal Superintelligence (launch_post, Launch-post benchmark table image, HEALTH section, row MedXpertQA (MM), column GPT 5.4 Xhigh; the same table is page 5 of the Eval Methodology PDF; protocol p. 2: 'MedXpertQA Text/Multimodal: ... The multimodal variant contains 2,000 multimodal medical questions with clinical images (X-rays, histology, dermatology, etc.) and 5 answer choices (A-E). For grading, we use gpt-oss-120b to parse the predicted answer letter from free-form text.') | | 7 | Gemini 3 Pro | Google | 76.0% | | Qwen/Qwen3.5-397B-A17B model card (model_card, Benchmark Results > Vision Language > Medical VQA, row MedXpertQA-MM; columns GPT5.2 / Claude 4.5 Opus / Gemini-3 Pro / Qwen3-VL-235B-A22B / K2.5-1T-A32B / Qwen3.5-397B-A17B) | | 8 | GPT-5.2 | OpenAI | 73.3 | null | Qwen/Qwen3.5-397B-A17B model card (model_card, Benchmark Results > Vision Language > Medical VQA, row MedXpertQA-MM; columns GPT5.2 / Claude 4.5 Opus / Gemini-3 Pro / Qwen3-VL-235B-A22B / K2.5-1T-A32B / Qwen3.5-397B-A17B) | | 9 | Claude Opus 4.8 | Anthropic | 71.7 | null | Qwen3.8-Max: A New Bar for Coding and Cowork (launch_post, Full Benchmark Table, second (multimodal) table, row MedXpertQA-MM; columns Opus4.8 / Fable5 / Gemini3.1-Pro / GPT5.6-Sol / Qwen3.7-Plus / Qwen3.8-Max) | | 10 | Qwen3.7 Plus | Alibaba | 71.0% | 2026-05 | Qwen3.7-Plus: Multimodal Agent Intelligence (launch_post, Multimodal Benchmarks table, Multimodal Reasoning section, row MedXpertQA-MM, column Qwen3.7-Plus; header row: | | GPT-5.4 (xhigh) | Opus-4.6 Max | Gemini-3.1 Pro | Qwen3.6-Plus | Qwen3.7-Plus |) | | 11 | Qwen3.5 397B A17B | Alibaba | 70.0 | null | Qwen/Qwen3.5-397B-A17B model card (model_card, Benchmark Results > Vision Language > Medical VQA, row MedXpertQA-MM; columns GPT5.2 / Claude 4.5 Opus / Gemini-3 Pro / Qwen3-VL-235B-A22B / K2.5-1T-A32B / Qwen3.5-397B-A17B) | | 12 | Qwen3.6 Plus | Alibaba | 68.7 | null | Qwen3.7-Plus: Multimodal Agent Intelligence (launch_post, Multimodal Benchmarks table, row MedXpertQA-MM; columns GPT-5.4 (xhigh) / Opus-4.6 Max / Gemini-3.1 Pro / Qwen3.6-Plus / Qwen3.7-Plus) | | 13 | Grok 4.20 | xAI | 65.8% | 2026-04 | Introducing Muse Spark: Scaling Towards Personal Superintelligence (launch_post, Launch-post benchmark table image, HEALTH section, row MedXpertQA (MM), column Grok 4.2 Reasoning; the same table is page 5 of the Eval Methodology PDF; protocol p. 2: 'MedXpertQA Text/Multimodal: ... The multimodal variant contains 2,000 multimodal medical questions with clinical images (X-rays, histology, dermatology, etc.) and 5 answer choices (A-E). For grading, we use gpt-oss-120b to parse the predicted answer letter from free-form text.') | | 14 | Kimi K2.5 | Moonshot AI | 65.3 | null | Qwen/Qwen3.5-397B-A17B model card (model_card, Benchmark Results > Vision Language > Medical VQA, row MedXpertQA-MM; columns GPT5.2 / Claude 4.5 Opus / Gemini-3 Pro / Qwen3-VL-235B-A22B / K2.5-1T-A32B / Qwen3.5-397B-A17B) | | 15 | Claude Opus 4.6 | Anthropic | 64.8% | 2026-04 | Introducing Muse Spark: Scaling Towards Personal Superintelligence (launch_post, Launch-post benchmark table image, HEALTH section, row MedXpertQA (MM), column Opus 4.6 Max; the same table is page 5 of the Eval Methodology PDF; protocol p. 2: 'MedXpertQA Text/Multimodal: ... The multimodal variant contains 2,000 multimodal medical questions with clinical images (X-rays, histology, dermatology, etc.) and 5 answer choices (A-E). For grading, we use gpt-oss-120b to parse the predicted answer letter from free-form text.') | | 16 | Claude Opus 4.5 | Anthropic | 63.6% | | Qwen/Qwen3.5-397B-A17B model card (model_card, Benchmark Results > Vision Language > Medical VQA, row MedXpertQA-MM; columns GPT5.2 / Claude 4.5 Opus / Gemini-3 Pro / Qwen3-VL-235B-A22B / K2.5-1T-A32B / Qwen3.5-397B-A17B) | | 17 | Gemma 4 31B | Google | 61.3% | | Gemma 4 model card (model_card, Evaluation Results table, MedXPertQA MM row, Gemma 4 31B column) | | 18 | Gemma 4 26B A4B | Google | 58.1% | | Gemma 4 model card (model_card, Evaluation Results table, MedXPertQA MM row, Gemma 4 26B A4B column) | | 19 | Gemma 4 12B | Google | 48.7% | 2026-04 | Gemma 4 model card (model_card, Benchmark Results table, Vision section, row MedXPertQA MM, column Gemma 4 12B Unified; header row: | | Gemma 4 31B | Gemma 4 26B A4B | Gemma 4 12B Unified | Gemma 4 E4B | Gemma 4 E2B | Gemma 3 27B (no think) |) | | 20 | Qwen3-VL-235B-A22B | Alibaba | 47.6% | | Qwen/Qwen3.5-397B-A17B model card (model_card, Benchmark Results > Vision Language > Medical VQA, row MedXpertQA-MM; columns GPT5.2 / Claude 4.5 Opus / Gemini-3 Pro / Qwen3-VL-235B-A22B / K2.5-1T-A32B / Qwen3.5-397B-A17B) | | 21 | Gemma 4 E4B | Google | 28.7% | | Gemma 4 model card (model_card, Evaluation Results table, MedXPertQA MM row, Gemma 4 E4B column) | | 22 | Gemma 4 E2B | Google | 23.5% | | Gemma 4 model card (model_card, Evaluation Results table, MedXPertQA MM row, Gemma 4 E2B column) | Assembled table, not a single board. Rows come from several vendors' own launch documents: Meta's Muse Spark evaluation (whose values live only in a rendered table image and cover Meta's own model plus four competitors it re-ran or quoted), Alibaba's Qwen3.5, 3.7 and 3.8 posts (Alibaba's own runs of its models and of competitors), and Google's Gemma 4 model card. Protocols are not known to match across rows; each row's source line says which document it came from and who ran it. benchlm.ai presents a subset as one mirrored view while citing only Meta's methodology. Checked 2026-09-28: Meta image table visually verified from its methodology PDF; Qwen3.5 and Google model-card tables verified. Six Qwen3.7/3.8 blog rows retain their historical retrieval dates and partial confidence because those primary pages currently return no article content. Source review: 2026-09-28; partial. Meta, Qwen3.5 and Google score tables verified; all published Gemma 4 and Qwen3.5 comparison columns included. Six Qwen3.7/3.8 blog values remain historical and partial because primary pages are empty. No cross-vendor protocol equivalence implied. ## Artificial Analysis Healthcare & Medical Index Artificial Analysis independently evaluates a weighted healthcare composite: knowledge 30%, agentic knowledge work 25%, long-context reasoning 15%, non-hallucination 10%, reasoning 10%, and agentic tool use 10%. Components include AA-Omniscience, GDPval-AA v2.1, AA-Briefcase v1.1, MLCR-AA, Humanity’s Last Exam, and AutomationBench-AA. Published by Artificial Analysis (2026-08); Six-evaluation composite; 25 default-selected models indexed; scale index score, higher better. Source: Best AI for Healthcare & Medical: LLM Leaderboard (independent-run), https://artificialanalysis.ai/models/capabilities/healthcare-and-medical 25 indexed rows; source board size: 77. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) | Anthropic | 61 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=claude-opus-5-5, weightedIndex=60.5359874288733; round to nearest whole point) | | 2 | Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) | Anthropic | 58 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=claude-fable-5-1, weightedIndex=58.0046680298741; round to nearest whole point) | | 3 | Claude Opus 5 (Adaptive Reasoning, Max Effort) | Anthropic | 53 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=claude-opus-5, weightedIndex=53.3389883833501; round to nearest whole point) | | 4 | GPT-6 Astra (max) | OpenAI | 52 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=gpt-6-astra, weightedIndex=51.6668965494132; round to nearest whole point) | | 5 | Muse Spark 1.3 (max) | Meta | 50 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=muse-spark-1-3, weightedIndex=49.6912519301107; round to nearest whole point) | | 6 | Grok 4.7 (xhigh) | SpaceXAI | 47 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=grok-4-7, weightedIndex=47.1903068758764; round to nearest whole point) | | 7 | GLM-5.3 (max) | Z AI | 47 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=glm-5-3, weightedIndex=46.6708125611712; round to nearest whole point) | | 8 | GPT-5.6 Sol (max) | OpenAI | 45 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=gpt-5-6-sol, weightedIndex=45.4043575131731; round to nearest whole point) | | 9 | Kimi K3 (max) | Kimi | 45 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=kimi-k3, weightedIndex=45.0434943845881; round to nearest whole point) | | 10 | GLM 5.3 Flash | Z AI | 45 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=glm-5-3-flash, weightedIndex=44.8426192521491; round to nearest whole point) | | 11 | GPT-6 Sol (max) | OpenAI | 43 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=gpt-6-sol, weightedIndex=43.4880669048048; round to nearest whole point) | | 12 | MiMo-V2.6-Pro | Xiaomi | 42 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=mimo-v2-6-pro, weightedIndex=41.8204296860305; round to nearest whole point) | | 13 | Gemini 3.8 Flash (high) | Google | 42 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=gemini-3-8-flash, weightedIndex=41.7830626872104; round to nearest whole point) | | 14 | Qwen3.8 Max (0902) | Alibaba | 41 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=qwen3-8-max, weightedIndex=41.4279151085327; round to nearest whole point) | | 15 | Step 5 Preview | StepFun | 41 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=step-5, weightedIndex=40.8171769208551; round to nearest whole point) | | 16 | DeepSeek V4.1 Flash (Reasoning, Max Effort) | DeepSeek | 41 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=deepseek-v4-1-flash, weightedIndex=40.6388737813166; round to nearest whole point) | | 17 | GPT-6 Luna (max) | OpenAI | 37 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=gpt-6-luna, weightedIndex=36.7252603853009; round to nearest whole point) | | 18 | GPT-5.6 Luna (max) | OpenAI | 36 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=gpt-5-6-luna, weightedIndex=35.740444759121; round to nearest whole point) | | 19 | Qwen3.8 27B (xhigh) | Alibaba | 34 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=qwen3-8-27b, weightedIndex=34.4310443115205; round to nearest whole point) | | 20 | MiniMax-M3 | MiniMax | 30 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=minimax-m3, weightedIndex=29.7042505129902; round to nearest whole point) | | 21 | Inkling (xhigh) | Thinking Machines | 25 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=inkling, weightedIndex=25.3933378261902; round to nearest whole point) | | 22 | Nemotron 3 Ultra 550B A55B (Reasoning) | NVIDIA | 23 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=nvidia-nemotron-3-ultra-550b-a55b, weightedIndex=23.0046725383136; round to nearest whole point) | | 23 | Gemini 3.5 Flash-Lite | Google | 23 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=gemini-3-5-flash-lite, weightedIndex=23.074121756108; round to nearest whole point) | | 24 | Muse Glimmer (high) | Meta | 18 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=muse-glimmer, weightedIndex=17.5980321302316; round to nearest whole point) | | 25 | Mistral Medium 3.5 | Mistral | 14 | 2026-09 | Artificial Analysis Healthcare & Medical Index (official_leaderboard, Embedded chart data: capability=healthcareAndMedical, initialModels slug=mistral-medium-3-5, weightedIndex=13.5556785369912; round to nearest whole point) | Current component versions and weights were checked September 28, 2026. This snapshot indexes the 25 default-selected models with numeric data embedded in the official page; the page offers 77 model variants overall. Historical rows absent from that snapshot are omitted rather than mixed with the prior five-evaluation composition. Scores are rounded whole index points; underlying values are recorded in source locators. Individual model evaluation dates are not published. Source review: 2026-09-28; partial. Verified the current six-evaluation methodology and 25 numeric model rows from first-party embedded chart data. This is partial coverage of 77 available variants; individual evaluation dates are undisclosed. Older incompatible index rows were removed. ## PhysicianBench LLM agents on long-horizon composite physician workflows inside real EHR environments, with execution-grounded verification against actual EHR systems via standard commercial APIs. Published by Academic team (Ruoqi Liu, Imran Q. Mohiuddin et al., arXiv 2605.02240); also a MAST component (2026-05); 100 real-world clinical tasks, 21 specialties, 670 structured checkpoints (~27 tool calls per task); scale pass@1 success rate %, higher better (3 independent runs; Pass^3 also reported). Source: PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) (mixed), https://arxiv.org/pdf/2605.02240 21 indexed rows; source board size: 21. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | Claude Opus 5.5 (max) | Anthropic | 68.4% | 2026-09-28 | Claude Sonnet 5.5 System Card (system_card, pp. 137–139, Section 8.15.3, Figure 8.15.B, PhysicianBench pass@1) | | 2 | Claude Sonnet 5.5 (max) | Anthropic | 63.2% | 2026-09-28 | Claude Sonnet 5.5 System Card (system_card, pp. 137–139, Section 8.15.3, Figure 8.15.B, PhysicianBench pass@1) | | 3 | Claude Fable 5.1 (max) | Anthropic | 61.0% | 2026-09-28 | Claude Sonnet 5.5 System Card (system_card, pp. 137–139, Section 8.15.3, Figure 8.15.B, PhysicianBench pass@1) | | 4 | Claude Opus 5 (max) | Anthropic | 57.6% | 2026-09-28 | Claude Sonnet 5.5 System Card (system_card, pp. 137–139, Section 8.15.3, Figure 8.15.B, PhysicianBench pass@1) | | 5 | Claude Sonnet 5.5 (xhigh) | Anthropic | 56.4% | 2026-09-28 | Claude Sonnet 5.5 System Card (system_card, pp. 137–139, Section 8.15.3, Figure 8.15.B, PhysicianBench pass@1) | | 6 | Claude Sonnet 5.5 (high) | Anthropic | 47.6% | 2026-09-28 | Claude Sonnet 5.5 System Card (system_card, pp. 137–139, Section 8.15.3, Figure 8.15.B, PhysicianBench pass@1) | | 7 | GPT-5.5 | OpenAI | 46.3 ± 1.2 | 2026-05 | PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) (paper, p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)) | | 8 | Claude Sonnet 5 (max) | Anthropic | 37.4% | 2026-09-28 | Claude Sonnet 5.5 System Card (system_card, pp. 137–139, Section 8.15.3, Figure 8.15.B, PhysicianBench pass@1) | | 9 | Claude Opus 4.6 | Anthropic | 31.7 ± 2.3 | 2026-05 | PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) (paper, p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)) | | 10 | Claude Sonnet 5.5 (medium) | Anthropic | 30.0% | 2026-09-28 | Claude Sonnet 5.5 System Card (system_card, pp. 137–139, Section 8.15.3, Figure 8.15.B, PhysicianBench pass@1) | | 11 | Claude Opus 4.7 | Anthropic | 29.3 ± 2.5 | 2026-05 | PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) (paper, p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)) | | 12 | GPT-5.4 | OpenAI | 27.7 ± 1.5 | 2026-05 | PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) (paper, p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)) | | 13 | Claude Sonnet 5.5 (low) | Anthropic | 27.2% | 2026-09-28 | Claude Sonnet 5.5 System Card (system_card, pp. 137–139, Section 8.15.3, Figure 8.15.B, PhysicianBench pass@1) | | 14 | Claude Sonnet 4.6 | Anthropic | 23.0 ± 2.6 | 2026-05 | PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) (paper, p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)) | | 15 | DeepSeek V4-Pro | DeepSeek | 18.7 ± 2.9 | 2026-05 | PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) (paper, p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)) | | 16 | Kimi-K2.6 | Moonshot AI | 17.0 ± 2.6 | 2026-05 | PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) (paper, p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)) | | 17 | MiMo-v2.5-Pro | Xiaomi | 16.7 ± 4.0 | 2026-05 | PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) (paper, p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)) | | 18 | Qwen3.6-Plus | Alibaba | 13.7 ± 4.0 | 2026-05 | PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) (paper, p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)) | | 19 | MiniMax M2.7 | MiniMax | 8.7 ± 1.2 | 2026-05 | PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) (paper, p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)) | | 20 | Gemini Pro 3.1 | Google | 6.0 ± 1.0 | 2026-05 | PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) (paper, p. 8, Table 2 (columns Pass@1, Pass@3, Pass^3, #Turns)) | | 21 | Grok-4.20 | xAI | 5.3 ± 3.2 | 2026-05 | PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments (arXiv 2605.02240v1 PDF) (paper, p. 8, Table 2 (Proprietary Models block)) | Two evaluator cohorts are shown on the same public 100-task set: the May 2026 benchmark paper (12 models, shared FHIR loop, 3 runs), and Anthropic’s September 28 system card (9 model/effort configurations, shared vendor harness, Opus 5 rubric grader). The vendor cohort is not a replication of the paper protocol. Read configuration and reporting labels before comparing cohorts. Opus 5.5 leads Anthropic’s max-effort cohort at 68.4%; GPT-5.5 leads the paper cohort at 46.3%. Differences of 5–6 points are near the resolution of this 100-task benchmark. Source review: 2026-09-28; verified. Verified all 12 original-paper rows plus 9 vendor-reported model/effort rows from today’s Sonnet 5.5 system card, pp. 137–139. Same public task set; vendor grader/harness cohort explicitly distinguished. ## EHR-Complex Agentic clinical reasoning over MIMIC-IV EHR databases via SQL and Python across six clinical intents, at patient and population level with temporal evidence paths. Published by Academic team (Qiao et al., Ant Group-affiliated; arXiv 2606.23301) (2026-06); ~52,000 tasks (3,915-task test set) over 365K patients, 31 tables, 500M+ records; scale exact-match accuracy, 0-1, higher better. Source: EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (independent-run), https://arxiv.org/pdf/2606.23301 18 indexed rows; source board size: 18. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | GPT-5.4 (high reasoning) | OpenAI | 0.65 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 15, Table 10 (Strong commercial model results), Avg. column) | | 2 | Gemini 3.1 Pro | Google | 0.63 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 15, Table 10 (Strong commercial model results), Avg. column) | | 3 | Kimi-K2.5 | Moonshot AI | 0.62 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 6, Table 3 (Evaluation Results on the EHR-Complex Test Set), Avg. column) | | 4 | Qwen3.5-397B | Alibaba | 0.62 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 6, Table 3 (Evaluation Results on the EHR-Complex Test Set), Avg. column) | | 5 | DeepSeek-V3.2-Exp | DeepSeek | 0.59 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 6, Table 3, Avg. column) | | 6 | GPT-5.4 (low reasoning) | OpenAI | 0.58 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 15, Table 10 (Strong commercial model results), Avg. column) | | 7 | DeepSeek-V3.1 | DeepSeek | 0.56 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 6, Table 3, Avg. column) | | 8 | Qwen3-32B-SFT | Alibaba | 0.55 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 6, Table 3, Avg. column) | | 9 | Qwen3-235B | Alibaba | 0.53 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 6, Table 3, Avg. column) | | 10 | GPT-4.1 mini | OpenAI | 0.49 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 6, Table 3, Avg. column) | | 11 | GPT-4.1 | OpenAI | 0.47 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 6, Table 3, Avg. column) | | 12 | Qwen3-14B-SFT | Alibaba | 0.45 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 6, Table 3, Avg. column) | | 13 | Claude Sonnet 4.6 | Anthropic | 0.36 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 15, Table 10 (Strong commercial model results), Avg. column) | | 14 | Qwen3-32B | Alibaba | 0.36 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 6, Table 3, Avg. column) | | 15 | GPT-4o | OpenAI | 0.31 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 6, Table 3, Avg. column) | | 16 | Gemini 2.5 Pro | Google | 0.31 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 6, Table 3, Avg. column) | | 17 | Qwen3-14B | Alibaba | 0.30 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 6, Table 3, Avg. column) | | 18 | Qwen3-4B | Alibaba | 0.16 | 2026-06 | EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning (arXiv 2606.23301v1 PDF) (paper, p. 6, Table 3, Avg. column) | The paper reports 12 base models, 2 benchmark-specific SFT variants, and 4 commercial configurations used for human validation. All 18 are shown with configuration labels. The macro-average gives equal weight to 12 intent/scope columns; it is not the micro-average over all test cases. Rechecked against the June 2026 paper; no new runs implied. Source review: 2026-09-28; verified. Tables 3 and 10 verified: 18 configurations, including both benchmark-specific SFT variants. Current arXiv version remains v1 dated 2026-06-22. ## WHBench Women's health: 47 expert-crafted scenarios across 10 topics graded on a 23-criterion rubric for clinical accuracy, safety, equity, and guideline adherence; targets failure modes like outdated guidelines, unsafe omissions, dosing errors, equity blind spots. Published by Independent researchers (Maurya, Govindgari, Kumar) (2026-04); 47 scenarios / 3,100 scored responses across 22 models; scale mean normalized percentage 0-100, higher better. Source: WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (independent-run), https://arxiv.org/pdf/2604.00024v2 22 indexed rows; source board size: 22. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | Claude Opus 4.6 | Anthropic | 72.1% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 2 | Claude Sonnet 4.6 | Anthropic | 67.1% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 3 | GPT-5.4 | OpenAI | 66.8% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 4 | Gemini 3 Flash Preview | Google | 64.7% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 5 | OpenAI o3 | OpenAI | 63.6% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 6 | DeepSeek V3.2 | DeepSeek | 61.3% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 7 | Grok 3 | SpaceX AI | 60.7% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 8 | Mistral Large | Mistral AI | 60.2% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 9 | Grok 4 | SpaceX AI | 57.9% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 10 | DeepSeek-R1 | DeepSeek | 52.9% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 11 | GPT-4.1 | OpenAI | 51.8% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 12 | Grok 3 Mini | SpaceX AI | 50.0% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 13 | Gemini 2.5 Flash | Google | 49.5% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 14 | Claude Opus 4 | Anthropic | 49.1% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 15 | Claude Sonnet 4 | Anthropic | 48.1% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 16 | GPT-4o | OpenAI | 44.6% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 17 | Llama 4 Maverick | Meta | 42.1% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 18 | Nemotron 70B | NVIDIA | 39.3% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 19 | Llama 3.3 70B | Meta | 37.8% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 20 | Llama 3.1 405B | Meta | 36.1% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 21 | Gemini 2.5 Pro | Google | 35.3% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | | 22 | Llama 4 Scout | Meta | 35.2% | 2026-03 | WHBench: A Women's Health Benchmark for Evaluating Frontier LLMs with Expert-in-the-Loop Validation (arXiv 2604.00024v2) (paper, p. 5, Table 3 (WHBench v3.0 leaderboard)) | An academic study with expert validation rather than a live leaderboard; the model set was frozen in March 2026, before GPT-5.6 and the Claude 5 family shipped. Source review: 2026-09-28; verified. All 22 model scores in Table 3 checked against v2, revised 2026-07-23. Historical March 2026 evaluation set retained; no live refresh claimed. ## HealthAdminBench End-to-end task success of computer-use LLM agents on healthcare administration workflows: prior authorizations, denial appeals, and DME ordering; success requires completing every subtask in a task. Published by Kinetic Systems (with Stanford Hospital domain experts) (2026-04); 135 tasks / 1,698 rubric-scored subtasks; scale percentage end-to-end task success 0-100, higher better. Source: HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks (arXiv 2604.09937v1) (independent-run), https://arxiv.org/pdf/2604.09937 7 indexed rows; source board size: 7. Row numbers refer to this index. Configuration variants remain separate rows. | # | model | lab | score | as of | source | |---|---|---|---|---|---| | 1 | Claude Opus 4.6 (computer-use agent) | Anthropic | 36.3% | 2026-04 | HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks (arXiv 2604.09937v1) (paper, p. 8, Figure 3(a) Task Success Rate (bar labels and values, extracted in order)) | | 2 | GPT-5.4 (computer-use agent) | OpenAI | 26.7% | 2026-04 | HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks (arXiv 2604.09937v1) (paper, p. 8, Figure 3(a) Task Success Rate (bar labels and values, extracted in order)) | | 3 | Kimi K2.5 | Moonshot AI | 15.6% | 2026-04 | HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks (arXiv 2604.09937v1) (paper, p. 8, Figure 3(a) Task Success Rate (bar labels and values, extracted in order)) | | 4 | Claude Opus 4.6 (standardized harness) | Anthropic | 14.8% | 2026-04 | HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks (arXiv 2604.09937v1) (paper, p. 8, Figure 3(a) Task Success Rate (bar labels and values, extracted in order)) | | 5 | Qwen 3.5 | Alibaba | 13.3% | 2026-04 | HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks (arXiv 2604.09937v1) (paper, p. 8, Figure 3(a) Task Success Rate (bar labels and values, extracted in order)) | | 6 | Gemini 3.1 Pro | Google | 11.9% | 2026-04 | HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks (arXiv 2604.09937v1) (paper, p. 8, Figure 3(a) Task Success Rate (bar labels and values, extracted in order)) | | 7 | GPT-5.4 (standardized harness) | OpenAI | 5.9% | 2026-04 | HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks (arXiv 2604.09937v1) (paper, p. 8, Figure 3(a) Task Success Rate (bar labels and values, extracted in order)) | Seven agent configurations are compared using screenshots plus task description and portal guidance. Native computer-use agents and the standardized harness have separate rows. The paper also reports accessibility-tree settings, which must not be compared directly with these screenshot-only scores. No revised paper or newer scored configuration found on the official site. Source review: 2026-09-28; verified. All seven Figure 3(a) task-success rates verified against v1 and official project page. The paper remains dated 2026-04-10; source checks are not new evaluations. ## Reuse Index data: CC BY 4.0. Cite the index as: Health Evals index, 2026-09-28 snapshot. https://healthevals.com. Cite individual scores to the document named on their row; full citations with locators are at https://healthevals.com/sources. Machine-readable: https://healthevals.com/data/benchmarks.json