A hospital algorithm underrates Black patients' illness
On 25 October 2019 Science published an analysis of a widely used commercial algorithm that selects patients for extra care: at the same score Black patients were considerably sicker than White patients, and removing the gap would raise their share among those selected from 17.7% to 46.5%.
Why it matters
The bias lay not in data about race but in what the model predicts: health care costs instead of illness. Because less is spent on Black patients' care, a model accurate on cost turned out biased on illness.
The authors describe the algorithm as typical of the industry and affecting millions of patients. The paper's conclusion reaches beyond medicine: a convenient proxy label in place of the real target can be a source of bias in many tasks. Neither the abstract nor the full text (read in the University of California's copy) names the algorithm or its maker, and the record claims neither. The authors trace the bias to the algorithm predicting health care costs rather than illness; the manufacturer, in their words, independently replicated the analysis on its own data of 3,695,943 commercially insured patients and confirmed the result.