Benchmark · September 1979

A blinded evaluation of MYCIN's advice

Eight independent specialists, not knowing the authorship, rated MYCIN's prescriptions for meningitis above those of five university physicians.

Why it matters

The first blinded comparison of a machine with experts in a practical domain, and the machine did not lose.

Ten meningitis cases were assessed; MYCIN's recommendations were rated acceptable 65 percent of the time against 42.5 to 62.5 percent for the faculty. The system never failed to cover a pathogen that needed covering. MYCIN still did not reach the clinic: liability questions, no integration with hospital systems and the need to enter data by hand all stood in the way.

Event record

Event date

September 1979

Timeline date

Event date

Verification

Sources gathered automatically · September 17, 2026

Lines

ID

evt-0135

Published in JAMA on 21 September 1979.

Sources

primary Antimicrobial selection by a computer: a blinded evaluation by infectious diseases experts
JAMA · Published September 1979

Related events

Extends MYCIN
The evaluation is of the system described in the 1974 thesis.
Antimicrobial selection by a computer: a blinded evaluation by infectious diseases experts

Records that link to this one

Contrasts BLEU: a formula scores translation quality
MYCIN was checked by a blinded panel of specialists, the most expensive and slowest method available. BLEU measured how close a formula can come to such a panel.
Bleu: a Method for Automatic Evaluation of Machine Translation
Related INTERNIST-I falls short of experts
Both evaluations set an expert system beside physicians on real cases: MYCIN blinded, on ten meningitis cases, INTERNIST-I on nineteen published cases across internal medicine.
Related The FDA admits a diagnosis without a physician
MYCIN beat physicians in a blinded evaluation but never reached the clinic; IDx-DR was admitted to it precisely as a decision with no clinician interpreting the image.