Evaluating Frontier LLMs for Healthcare with MedHELM: 40+ Clinical Scenarios and a New Q2 2026 Leaderboard

MedHELM is an open-source clinical benchmark that evaluates large language models (LLMs). The current MedHELM site describes it as covering 121 clinical tasks, 22 subcategories, 31 datasets, and 5 clinical categories, with evaluation spanning accuracy, calibration, robustness, and writing style across real clinical workflows. It builds on Stanford CRFM’s HELM framework and supports medical datasets such as PubMedQA, MedQA, MedMCQA, ACI-Bench, and DischargeMe.

Developed in collaboration with Stanford’s Center for Research on Foundation Models (CRFM), it is purpose-built for healthcare AI evaluation. The fully rebuilt Q2 2026 leaderboard evaluates 9 frontier models from Google, OpenAI, Anthropic, xAI, and DeepSeek. Benchmark methodology is held constant across all versions to enable direct cross-model comparison.

Why general benchmarks don’t reflect clinical performance

General-purpose LLM benchmarks such as MMLU, HumanEval, and GPQA were not designed for clinical use. They measure reasoning, coding ability, and academic knowledge across broad domains. None of them test a model’s ability to reason through a differential diagnosis, extract structured data from an unstructured clinical note, flag a potential drug interaction buried in a medication reconciliation, or handle a safety-sensitive patient communication scenario.

A model that leads on MMLU can fail in ways that matter in a clinical setting. The gap is not incidental. It reflects a fundamental mismatch between what those benchmarks measure and what clinical AI actually performs.

MedHELM closes that gap by grounding evaluation in real clinical workflows.

How MedHELM evaluates: scenarios, win-rate scoring, and cross-version comparability

MedHELM is built on the HELM (Holistic Evaluation of Language Models) framework from Stanford CRFM (Liang et al., “Holistic Evaluation of Language Models,” Transactions on Machine Learning Research, 2023; arxiv.org/abs/2211.09110), adapted specifically for healthcare. The evaluation pipeline processes each model through a standardized scenario set, applies consistent scoring criteria, and surfaces results on a public leaderboard.

Scenarios are the core unit of evaluation. Each scenario represents a distinct clinical task with a defined dataset, evaluation metric, and expected output format. The scenario library covers medical question-answering, clinical documentation, EHR-based reasoning, safety and bias testing, specialist referral judgment, and hallucination detection.

Scoring uses win rate as the primary comparative metric. Win rate measures how often a model outperforms a reference baseline across the full scenario set. It is stable across leaderboard versions even as models are added or retired, because it measures relative performance rather than absolute score against a shifting target.

Reproducibility is enforced by holding methodology, scoring criteria, and evaluation logic constant across all releases. When the leaderboard is updated with new models, existing scores are preserved unchanged. A model evaluated in v0.6 can be compared directly against one evaluated in v0.7 without adjustment.

What the 40+ MedHELM scenarios cover: medical QA, EHR documentation, safety testing, and expert reasoning

MedHELM v0.7 includes more than 40 clinical scenarios, organized by the type of clinical task they evaluate.

Medical question-answering

MedQA, MedMCQA, MedBullets, HeadQA, PubMedQA, MedicationQA. These scenarios test clinical knowledge across medical licensing exam-style questions, pharmacology, and published biomedical literature.

Expert-level clinical reasoning

MedXpertQA Text evaluates multi-step medical reasoning on complex questions that standard QA benchmarks don’t reach. The difference between MedQA and MedXpertQA Text is the difference between recall and reasoning: MedQA asks whether a model can identify the diagnostic criteria for sepsis; MedXpertQA Text presents a multi-part clinical vignette that requires integrating history, labs, and imaging findings, reasoning through competing diagnoses, and arriving at a defensible clinical conclusion. For teams evaluating models for complex clinical decision support, this is the scenario that surfaces the differentiation other benchmarks miss.

Clinical documentation and EHR tasks

ACI-Bench, EHR-SQL, MT-Samples (with procedure and replicate variants), MedDialog. These scenarios test the ability to extract structured information from unstructured clinical text, generate SQL queries against EHR schemas, and support clinical documentation workflows.

Discharge planning and care coordination

DischargeMe, MIMIC-BHC, MIMIC-RRS, NoteExtract, PatientInstruct. These scenarios evaluate model performance on handoff tasks that drive a significant share of clinical AI applications: summarizing a patient encounter, generating a discharge summary, and producing patient-facing instructions.

Specialist referral and clinical judgment

ClinicReferral, ENT-Referral, HospiceReferral, CDI-QA, BMT-Status. These scenarios evaluate appropriateness of specialist referrals and clinical status assessments across specific care settings.

Safety, bias, and hallucination detection

RaceBias, PrivacyDetection, ProxySender, MedHallu, CLEAR, N2C2-CT, ADHD-Behavior, MentalHealth, MedConfInfo. RaceBias evaluates whether model outputs vary by patient race across equivalent clinical presentations. PrivacyDetection tests appropriate handling of PHI in clinical text. MedHallu and CLEAR evaluate hallucination on clinical claims.

HealthBench

HealthBench Original and HealthBench Professional extend MedHELM’s coverage with scenarios designed for professional healthcare QA contexts. They are not yet included in the current MedHELM leaderboard. Once added, they will extend MedHELM’s coverage with scenarios designed for professional healthcare question-answering contexts.

v0.6 through v0.7: 3 clinical benchmarks added, OpenAI evaluation cost cut by 50%, and a 6-model leaderboard refresh

Two benchmark additions in v0.6 and v0.6.1 expanded MedHELM’s clinical scenario library. A leaderboard refresh in v0.7 brought the evaluation current for the Q2 2026 frontier model generation.

v0.6 added two benchmark integrations.

HealthBench Original (OpenAI, May 2025) is a 5,000-conversation benchmark scored against physician rubrics, covering realistic patient and clinician dialogue. HealthBench Professional (OpenAI, April 2026) extends it to three clinician task types: care consult, clinical writing and documentation, and medical research. Both map to the Patient Communication and Education category in the MedHELM taxonomy. Neither is currently on the leaderboard but can be run as part of any MedHELM evaluation.

SCT-Bench adds Script Concordance Testing to the scenario set. Based on McCoy et al. (medRxiv, February 2025), it measures how new clinical information shifts a model’s diagnostic and treatment hypotheses against expert panels of practicing clinicians. The integration covers 174 publicly released questions from the original 750-question set. SCT-Bench evaluates clinical reasoning under uncertainty, a dimension that standard QA benchmarks and medical licensing exam simulations do not capture.

v0.6 also added OpenAI Batch API support. Setting –batch-size submits evaluation requests through OpenAI’s Batch API, reducing cost by 50% on input and output tokens for any MedHELM evaluation run against OpenAI models.

v0.6.1 introduced MedXpertQA Text, described in the scenario library above. The release also resolved installation issues affecting optional dependencies.

v0.7 delivers the leaderboard refresh described in the next section. Benchmark methodology, evaluation framework, and scoring criteria are unchanged from prior versions, so scores from v0.6 compare directly to v0.7 results without adjustment.

Q2 2026 leaderboard: six new frontier models added, 6 legacy snapshots retired

The MedHELM leaderboard is maintained by the community and updated quarterly. The Q2 2026 refresh (v0.7) adds six new frontier models from Google, OpenAI, Anthropic, and xAI, and retires six legacy snapshots no longer in active deployment.

Q2 2026 rankings by mean win rate (new in v0.7):

Rankings shift across individual scenarios. Muse Spark leads on clinical documentation generation (ACI-Bench Jury Score: 4.969). GPT-5.4 mini tops the clinical error detection task (Medec FlagAcc: 0.722), outperforming Gemini 3.1 Pro (0.680) on that specific scenario despite trailing overall. Gemini 2.5 Pro leads on HeadQA exact-match (0.931) despite ranking sixth in the overall standings. These per-scenario differences are why the full breakdown matters for deployment decisions rather than overall rank alone.

Six legacy snapshots have been retired: GPT-5 (2025-08-07), GPT-5 mini (2025-08-07), o4-mini (2025-04-16), o3-mini (2025-01-31), Claude 3.5 Sonnet (20241022), and GPT-4o (2024-05-13). Previously published scores for these models remain valid for comparison purposes.

Per-scenario breakdowns across 24 out of 40+ MedHELM scenarios are available at leaderboard.medhelm.org.

What MedHELM results mean for healthcare AI deployment

A model that ranks highly on MedHELM has been evaluated on tasks that reflect real clinical use cases, using consistent methodology that makes results reproducible and comparable across model generations. That is meaningful signal, but it has specific limits that deployment teams need to understand.

What MedHELM scores support:

• Relative comparison of frontier models on clinical tasks, using consistent and reproducible methodology
• Evidence that a model handles a defined set of healthcare scenarios at a measured performance level
• A stable baseline for tracking model behavior across versions over time -> A reproducible benchmark framework that teams can use to re-run evaluations and inspect model behavior on specific tasks.
• Pre-release gating criteria in CI/CD pipelines for healthcare AI systems -> Input into model selection, regression testing, and pre-release review workflows for healthcare AI systems.

What MedHELM scores don’t replace:

• Institution-specific validation on your patient population and clinical workflows
• Red-teaming for your specific deployment context and failure modes
• Human expert review of model outputs in high-stakes clinical settings
• Regulatory compliance assessment under HIPAA, FDA 2024-D-4488, HHS HTI-1, or ACA 1557

MedHELM results provide reproducible, evidence-based performance data on a defined task set. They are one input into a governance decision. They document measured performance on a defined set of clinical scenarios; they don’t substitute for regulatory compliance assessment. Organizations deploying clinical AI should consult their compliance counsel regarding applicable regulations.

For organizations that need to operationalize MedHELM evaluation as part of a formal AI governance workflow, Pacific AI’s Gatekeeper runs MedHELM benchmarks as automated pre-release gates in CI/CD pipelines. Results feed into Governor for risk scoring, model card generation, and audit trail maintenance. Guardian monitors for performance drift against MedHELM baselines in deployed models. Running MedHELM inside Pacific AI turns a one-time evaluation into a repeatable, auditable governance process.

How to run MedHELM evaluations

MedHELM is open source. To install the current version:

uv pip install medhelm

With optional summarization dependencies:

uv pip install "medhelm[summarization]"

Scenarios: PubMedQA, MedCalc-Bench, MedicationQA, MedHallu.

Example (ACI-Bench; runs without extra data):

uv run medhelm-run \
    --run-entries "aci_bench:model=qwen/qwen2.5-7b-instruct,model_deployment=huggingface/qwen2.5-7b-instruct" \
    --suite med_summaries \
    --max-eval-instances 5
  uv run helm-summarize --suite med_summaries
  uv run helm-server --suite med_summaries

Resources:

• GitHub repository
• Live leaderboard
• MedHELM documentation

To integrate MedHELM evaluations into a pre-release gating and production monitoring workflow, request a Pacific AI demo.

FAQ

What is MedHELM?

MedHELM is an open-source benchmark for evaluating large language models on healthcare-specific tasks. Built on Stanford CRFM’s HELM framework and extended with more than 40 clinical scenarios, it covers medical QA, EHR documentation, clinical reasoning, safety testing, and hallucination detection. Pacific AI develops and maintains MedHELM in collaboration with Stanford CRFM.

How is MedHELM different from general LLM benchmarks like MMLU?

General benchmarks measure reasoning and knowledge across broad domains. MedHELM is designed specifically for clinical workflows. The scenarios reflect real healthcare tasks, and scoring is calibrated to what matters in clinical AI deployment. A model that leads on MMLU may not perform well on MedHELM, and the reverse is also true.

What is MedXpertQA Text, and why does it matter?

MedXpertQA Text evaluates expert-level medical reasoning on multi-step clinical questions that standard QA benchmarks don’t cover. It was added in v0.7 to extend MedHELM’s coverage beyond factual recall into the kind of complex clinical judgment that matters most for high-stakes AI applications. See the scenario library section above for details.

Why does MedHELM retire old model snapshots?

Retired models are snapshots of older API versions that are no longer supported or widely deployed. Retiring them keeps the leaderboard focused on currently available model generations. Methodology is unchanged across versions, so previously reported scores remain valid for comparison purposes.

Can I run MedHELM evaluations on my own models?

Yes. MedHELM is open source and supports evaluation of any model accessible via API or local deployment. See the GitHub repository for setup instructions and scenario documentation.

How does MedHELM integrate with the Pacific AI platform?

Pacific AI’s Gatekeeper runs MedHELM evaluations as automated pre-release gates in a CI/CD pipeline, with results feeding into Governor for risk scoring, model card generation, and audit trail maintenance. Guardian monitors for performance drift against MedHELM baselines in deployed models.

Does a strong MedHELM score mean a model is safe to deploy clinically?

No. MedHELM results provide reproducible performance data on a defined set of clinical tasks. They support governance decisions and pre-release gating, but they don’t substitute for institution-specific validation, clinical expert review, or regulatory compliance assessment. Organizations deploying clinical AI should consult their compliance counsel regarding applicable regulations.