A blinded evaluation of MYCIN's advice

Eight independent specialists, not knowing the authorship, rated MYCIN's prescriptions for meningitis above those of five university physicians.

Why it matters

The first blinded comparison of a machine with experts in a practical domain, and the machine did not lose.

Ten meningitis cases were assessed; MYCIN's recommendations were rated acceptable 65 percent of the time against 42.5 to 62.5 percent for the faculty. The system never failed to cover a pathogen that needed covering. MYCIN still did not reach the clinic: liability questions, no integration with hospital systems and the need to enter data by hand all stood in the way.

Event record

Event date

September 1979

Timeline date

Event date

Verification

Sources gathered automatically · September 17, 2026

Lines

ID

evt-0135

Published in JAMA on 21 September 1979.

Sources

primary Antimicrobial selection by a computer: a blinded evaluation by infectious diseases experts
JAMA · Published September 1979
primary V. L. Yu, L. M. Fagan, S. W. Bennett, W. J. Clancey, A. C. Scott, J. F. Hannigan, R. L. Blum, B. G. Buchanan, S. N. Cohen, An Evaluation of MYCIN's Advice (chapter 31), in Buchanan and Shortliffe (eds.), Rule-Based Expert Systems, Addison-Wesley, 1984, pp. 589-596; edited from JAMA 242:1279-1282 (1979)
Addison-Wesley (author-hosted copy at shortliffe.net); JAMA, American Medical Association · Published 1984

Related events

Earlier

Extends MYCIN 1974
The evaluation is of the system described in the 1974 thesis.
- Antimicrobial selection by a computer: a blinded evaluation by infectious diseases experts · JAMA

Later

Related INTERNIST-I falls short of experts 1982links to this record
Both evaluations set an expert system beside physicians on real cases: MYCIN blinded, on ten meningitis cases, INTERNIST-I on nineteen published cases across internal medicine.
Contrasts BLEU: a formula scores translation quality 2002links to this record
MYCIN was checked by a blinded panel of specialists, the most expensive and slowest method available. BLEU measured how close a formula can come to such a panel.
- Bleu: a Method for Automatic Evaluation of Machine Translation · Association for Computational Linguistics
Related The FDA admits a diagnosis without a physician 2018links to this record
MYCIN beat physicians in a blinded evaluation but never reached the clinic; IDx-DR was admitted to it precisely as a decision with no clinician interpreting the image.