Cognitive biases such as anchoring, availability, and framing are among the best-documented causes of medical error, implicated in up to 75% of diagnostic errors in internal medicine and up to 96% of diagnostic errors in emergency medicine. LLMs inherit the same weakness: BiasMedQA shows a 10 to 26 percent accuracy loss on USMLE when bias cues are added. The tendency to defer to a stated position has been shown to grow with model size and with the reinforcement learning used to align modern assistants, and prompt-based fixes do not close the gap. A model can top a medical exam benchmark and still change its answer when a colleague’s assertion, the order of findings, or a survival-versus-mortality framing is introduced. Testing for these biases is a regulatory necessity: rules such as the HHS HTI-1 transparency rule, the FDA’s lifecycle validation expectations, and ACA Section 1557’s bar on algorithmic discrimination hold the deploying organization responsible for validating a model under realistic conditions.
This session walks through the 17 benchmarks in Pacific AI’s healthcare-specific cognitive-bias test suite. Every case is fully synthetic and contains no protected health information, and each is a matched pair: two versions of the same realistic clinical note differ only in the cue. Scoring is deterministic and needs no LLM judge. Twelve benchmarks cover clinical decision making and five cover clinical coding:
Clinical decision-making (12): anchoring, availability, confirmation, order effects, social pressure, premature closure, frequency, base-rate neglect, therapeutic inertia, patient affect, framing, and defensive medicine.
Clinical coding (5): assertion over documentation, principal-diagnosis anchoring, common-code default, copy-forward inertia, and position sensitivity.
What you’ll learn:
- Why cognitive bias is a distinct failure mode from clinical accuracy and from demographic bias, and why a model can pass a knowledge benchmark and still fail here.
- How the 17 benchmarks are built: the matched-pair design, realistic synthetic notes, grounding in the diagnostic error literature, deterministic scoring, balancing across multiple medical dimensions, and clinician and certified-coder review.
- How the twelve decision biases and five coding biases map to real coding and billing exposure and to documented clinical error.
- How to run the suites continuously: as a pre-release CI/CD gate with Gatekeeper and as a production monitor with Guardian, so a model is re-checked on every release and as it drifts.
