HealthBench Professional is now available in Pacific AI Gatekeeper. It is an open benchmark from OpenAI built on 525 physician-authored tasks, and it measures one thing well: how usefully a model handles the work clinicians bring to a conversational assistant. Running it in Gatekeeper, gpt-5.4-mini scored 0.68, claude-opus-4.6 scored 0.60, and gemini-3.1-pro-preview scored 0.35 on usefulness and efficacy. Those three numbers describe one of the three principles a healthcare deployer has to answer for.
Four limits on what a HealthBench Professional score supports
MedHELM supplies a clinician-validated taxonomy of health AI tasks and a benchmark suite spanning them. HealthBench, the original, scores open-ended clinical dialogue against physician-written rubrics. A 2026 generation of agentic benchmarks including MedAgentBench, PhysicianBench, HealthAgentBench and HealthAdminBench measures whether a model can execute long-horizon clinical and administrative workflows against a live environment.
HealthBench Professional is one more instrument in that set, and a good one. It narrows the original HealthBench to clinician-only, professional-grade workflows: 525 tasks selected from 15,079 candidate conversations, authored by 190 physicians across 26 specialties and 50 countries, weighted toward harder cases through 3.5 times enrichment, and graded by a GPT-5.4 rubric grader rather than GPT-4.1.
Coverage is not guaranteed for any task. The 525 items spread across three categories and 26 specialties. That is roughly 175 items per category and roughly 20 per specialty. At 20 specialties, the 95% confidence interval on a score near the middle of the range is about plus or minus 22 points – useless in differentiating between the best and worst models. The headline number is a reasonable estimate of general clinician-facing usefulness. It does not support a claim about how a model performs on discharge summaries in nephrology. OpenAI publishes 95% confidence intervals as error bars on its use-case and specialty breakdowns, but those are marginal slices: there is no published interval for the intersection of a use case and a specialty, and only 10 of the 26 specialties are plotted at all.
The data contamination clock starts at publication. Every public benchmark degrades as it enters training corpora, and the effect is documented across the field. The original HealthBench has been public since May 2025. HealthBench Professional is newer, so the effect is smaller today than it will be in a year. A benchmark whose items are public cannot separate a model that reasons well from a model that has seen the items, and neither the benchmark nor any harness running it can tell you which you are looking at.
Scoring depends on a judge model. Physician-written rubrics are graded by GPT-5.4 with a length adjustment to reduce verbosity bias. The length adjustment is a real improvement and worth noting. The judge dependency is still a dependency: scores are comparable only across runs that used the same judge and the same configuration.
Its popularity is partly a function of its publisher. HealthBench and HealthBench Professional are widely run because OpenAI published them, and that is a reasonable basis for including a benchmark in a suite. It is not a basis for treating one benchmark as the answer.
The annotation process behind HealthBench Professional is stronger than most: every conversation and rubric involves three or more physicians, and difficult examples are independently verified by two physicians for realism and for whether the model genuinely erred. The 3.5 times enrichment of hard cases is a deliberate answer to the saturation problem that flattens exam-style benchmarks. The point is narrower: it is one input to one of three principles, and it is most useful when it is read that way.
525 tasks across three categories of clinician work
Each example comes from a physician testing ChatGPT for Clinicians. The dataset combines good-faith use across routine clinical, academic, administrative and research tasks with deliberate red-team conversations built to expose model weaknesses. Red-team examples make up 36% of the final benchmark.
Three categories map to how clinicians spend time with an assistant:
- Care consult. Clinical reasoning, diagnosis, treatment planning, differential diagnosis and patient management.
- Writing and documentation. Clinical notes, summaries, patient communication, coding and structured documentation.
- Medical research. Evidence synthesis, literature search, guideline interpretation and scientific reasoning.
In OpenAI’s own evaluation, GPT-5.4 inside ChatGPT for Clinicians scored highest, outperforming the base GPT-5.4 model and physician-written baseline responses on the same tasks. Increased reasoning effort improved performance across the board, with an average of 3.3 points from the lowest non-none to the highest reasoning setting. The full methodology is in OpenAI’s HealthBench Professional paper.
Six things that change when Gatekeeper runs it
The benchmark is open, so any team with an engineer and an API key can run it standalone. Six things change when it runs inside Gatekeeper.
- It reports into a model card alongside other benchmarks. Gatekeeper organises results around the three CHAI responsible AI principles: usefulness and efficacy, bias and fairness, and safety and reliability. HealthBench Professional populates part of the first. A deployer sees where it sits, what it covers, and what is still unmeasured, on the same page.
- Scores become comparable across runs and across models. Same harness, same judge configuration, same prompt handling, same date-stamped model version strings. A number produced in one team’s notebook and a number produced in another’s are not comparable; two numbers from the same Gatekeeper configuration are.
- You test your deployment, not a bare model. Most healthcare AI in production is a model behind a system prompt, a retrieval layer and a set of guardrails. Gatekeeper runs the benchmark against your own endpoint with your own configuration, so the score describes the thing you are shipping.
- Version drift becomes visible. Models update. A benchmark run once is a snapshot; the same benchmark run on every release is a control. Gatekeeper keeps the history, so a regression between two versions of your own deployment shows up as a regression rather than as an anecdote.
- Pre-release testing carries into production monitoring. Gatekeeper gates the release; Guardian monitors the deployed system for accuracy, bias, safety and drift. The same evaluation vocabulary spans both, so a pre-release finding and a production alert describe the same quantity.
- It runs in your own environment. Pacific AI deploys inside the customer’s own AWS or Azure environment. Clinical prompts and model outputs do not leave it.
Results: usefulness and efficacy on three frontier models
Pacific AI ran HealthBench Professional in Gatekeeper against three frontier models. Scores are on the benchmark’s own scale, graded by the GPT-5.4 rubric grader with the standard length adjustment.
Table 1. HealthBench Professional Comparison chart “Usefulness & efficacy” scores by model
| Model | Usefulness & efficacy score |
|---|---|
| ChatGPT-5.4-mini | 0.68 |
| Claude-opus-4.6 | 0.60 |
| Gemini-3.1-pro-preview | 0.35 |
Claude-opus-4.6 Model Results:
ChatGPT-5.4-mini Results:
Gemini-3.1-pro Model Results:
OpenAI publishes confidence intervals for its own runs, but the per-model score reported here is a single number, so a gap of a few points between two models should not be read as a ranking. These are also scores on one benchmark measuring one principle, which is the subject of the next section.
Bias and safety scores are absent by design
Bias, fairness, safety, and reliability are blank for this run because HealthBench Professional does not measure them: it was not built to. It measures how usefully a model handles clinician-facing tasks. It does not measure whether the model’s clinical decision changes when the patient’s race, insurance status, or immigration status changes. It does not measure whether the model refuses to help someone reach a record they have no basis to see. It does not measure whether its answer holds steady when the same note is reworded, translated or restructured.
Those are different constructs; they need different instruments. No amount of running HealthBench Professional produces them.
Pacific AI therefore treats a benchmark as one input to a principle rather than as an evaluation in itself. The other two columns are filled by benchmarks built for them. Pacific AI has suites in development covering bias and fairness through patient-attribute perturbation, cognitive bias in clinical reasoning, and coding integrity under irrelevant variation; and covering safety and reliability through refusal and over-refusal, confidentiality and access, health data security, and invariance under meaning-preserving rewording. Each is constructed rather than sampled from public sources, which is what keeps it out of the contamination path that public benchmarks sit in. We will publish those results as each suite completes expert review.
Until then, the honest reading of a HealthBench Professional score is that it answers one of three questions well and says nothing about the other two.
Reading a HealthBench Professional score before release
Run it, and read it as the one data point that it is. A model that scores well here is a model that handles clinician-facing tasks capably in the conditions the benchmark tests. That is useful information, yet not sufficient on its own for a release decision on a clinical system.
If you are evaluating a model for deployment, the practical sequence is to run HealthBench Professional against your own endpoint rather than a bare model, record the version string and the date, keep the run as a baseline for the next release, and treat the two empty columns as open questions rather than as passes.
HealthBench Professional is available now in Pacific AI Gatekeeper.
FAQ
What is HealthBench Professional?
An open benchmark from OpenAI that evaluates language models on 525 physician-authored tasks drawn from real clinician conversations with ChatGPT, rather than multiple-choice exam questions. It covers care consult, writing and documentation, and medical research.
How is it different from the original HealthBench?
The original, introduced in May 2025, covers 5,000 conversations built with 262 physicians across 60 countries and grades against five axes and seven themes spanning both layperson and clinician users. HealthBench Professional narrows to clinician-only workflows: 525 tasks from 15,079 candidates, built by 190 physicians across 26 specialties, weighted toward harder cases, and graded with a GPT-5.4 rubric grader.
Does a good HealthBench Professional score mean a model is safe to deploy in healthcare?
No. It measures usefulness and efficacy on clinician-facing tasks. It does not measure bias and fairness or safety and reliability, which are separate principles requiring separate instruments.
How reliable is a single HealthBench Professional score?
OpenAI reports 95% confidence intervals as error bars on its use-case and specialty breakdowns, but only for marginal slices, and only 10 of the 26 specialties are plotted. With 525 items across three categories and 26 specialties, per-specialty estimates rest on roughly 20 items each, which supports a general reading rather than a per-task claim. Treat small gaps between models as unresolved rather than as a ranking.
Is HealthBench Professional affected by data contamination?
Every public benchmark is exposed to it, and the exposure grows from the date of publication. HealthBench Professional is recent enough that the effect is smaller today than it will be later. This is one reason Pacific AI runs it alongside benchmarks constructed rather than sampled from public sources.
Why run HealthBench Professional in Gatekeeper rather than directly?
Results report into a model card alongside other benchmarks, scores stay comparable across runs and versions through a fixed harness and judge configuration, the benchmark runs against your own endpoint and system prompt rather than a bare model, version history is retained so drift is visible, findings carry into Guardian for production monitoring, and everything runs inside your own cloud environment.
When will the bias and safety columns be populated?
As each Pacific AI suite completes expert review. Those suites measure patient-attribute perturbation, cognitive bias, coding integrity, refusal and over-refusal, confidentiality and access, health data security, and invariance under rewording. We will publish results per suite as they are completed.
