UT Southwestern gave three AI models pairs of emergency patients with similar severity and asked which one would get worse. Then they changed one line in the chart. In some comparisons, nearly 10% of patients were wrongly pushed down the list, and the stigmatizing wording pushed harder than the neutral one.
The study in npj Digital Medicine, released by UT Southwestern on October 20, tested something every clinician has watched happen at the triage desk. Researchers took emergency department records from Stanford Medicine and Beth Israel Deaconess, paired patients who arrived with similar severity, and asked three open-source large language models which one was more likely to deteriorate.
On the clean records, all three models picked the higher-risk patient well above chance.
Then the team changed the description, not the patient. Two of the swaps they used: "Established patient with multiple chronic conditions" became "17 visits in the past 12 months." "Opioid use disorder, on maintenance therapy" became "active IV drug use, heroin, last use today." In some comparisons, nearly 10% of patients were wrongly downgraded once the stigmatizing detail went in. Even a neutral mention of the same history moved the models; the stigmatizing wording moved them further, by roughly 1 to 7 points depending on the model. How ED visits were described was the clearest case. Substance use results were mixed, and race, language and insurance showed no consistent harmful shift.
Anyone who has worked a busy department knows the human version. The frequent flyer waits. Sometimes the frequent flyer is the one who crashes.
Most conversations about AI bias go straight to training data. Audit it, rebalance it, ship the model.
This study points somewhere less comfortable. A model can be well built and still return a skewed answer, because it is reading the same chart a tired clinician wrote at 3 AM. The words carry a judgment, and the model inherits it.
So a tool validated on tidy research data can fail on your actual notes. Your documentation is part of the input. Almost nobody validates it.
The researchers found the effect varied by model. Their recommendation was blunt: test each tool against real patient outcomes before you use it. A vendor's general statement on bias tells you nothing about how its model behaves on your charts.
One pre-deployment check like this lands in The Sarah Matt Briefing every Tuesday. Get it by email.
It does not tell us whether flagging or rewriting stigmatizing language before the model reads it would close the gap. That is the obvious next experiment, and the authors call how clinical information is selected and worded for these models a safety-critical design choice. These were open-weight research models, not commercial triage products, so the result is a reason to test your vendor, not a verdict on it. Age shifted some decisions too, which the researchers attribute to real medical risk in older patients rather than bias.
My patients in charity care are the people with seventeen visits in twelve months. Most of them are not gaming anything. They have nowhere else to go. If a triage tool reads that history and moves them down the list, the cost lands on the patients with the least room for error.
A few teams are already auditing their documentation and testing on these populations before go-live. Most have not started.
Has anyone on your team searched your ED note templates for the words this study tested?
Source: Jarrett P, Jamieson A, et al., npj Digital Medicine, published online September 19, 2026; UT Southwestern Medical Center release, October 20, 2026.
Dr. Sarah Matt is a surgery-trained physician-executive currently practicing internal medicine (charity care). She advises health systems and digital health companies on clinical AI implementation, vendor contracting and operational strategy.
Weekly clinical reality on AI, governance, and what actually works inside health systems.