AI supposedly outperforms doctors in US study, but experts urge caution

AI Summary

A new study by Harvard researchers indicates that AI is outperforming human doctors in emergency triage diagnoses, excelling in clinical reasoning tasks. However, experts caution that the AI's performance may not translate well in real-world applications due to challenges in clinical settings.

Artificial intelligence (AI) is better than humans at emergency triage diagnoses, a study has suggested.Researchers at Harvard University, Massachusetts, said their analysis showed that large language models (LLMs) powered by AI “have eclipsed most benchmarks of clinical reasoning.” But experts, including the researchers themselves, admitted that the AI system would struggle in the real world.Their study, published in the journal Science,1 evaluated how well OpenAI’s o1 series of LLMs could reason medically across six different experiments, including differential diagnoses and clinical reasoning, and then compared them with the way hundreds of human doctors assessed the same problems.The AI model was tested on established clinical vignettes, including New England Journal of Medicine case reports, management challenges, and probabilistic reasoning scenarios. The researchers also carried out a further blinded study using unstructured clinical data from 76 randomly selected emergency room cases from Beth Israel Deaconess Medical Center in Boston, Massachusetts, assessing how...