Medical AI has a measurement problem
Article excerpt
Nature, Published online: 28 July 2026; doi:10.1038/d41586-026-02125-z The development of two medical AI assistants highlights an unnerving challenge: as the technology races ahead, what is the best way to evaluate what works?
Arjun K. Manrai is in the Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts 02115, USA.
Access to this article via Institution of Civil Engineers Library is not available.
Evidence-based medicine rests on the assumption that the yardstick used to measure the effectiveness of a new drug or medical device is reliable. But this assumption is often shaky, especially in the context of the artificial-intelligence tools that are making their way into clinical practice. Two papers in Nature , one by Ferber et al . 1 and one by Liévin et al . 2 show how capable medical AI tools have become, but also demonstrate why that capability is increasingly difficult to assess and understand properly.
Prices may be subject to local taxes which are calculated during checkout
Feng, J. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2606.28960 (2026).
Begley, C. G. & Ioannidis, J. P. A. Circ. Res. 116 , 116, 126 (2015).
Brodeur, P. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2603.08448 (2026).
American Medical Association. Physician Survey on Augmented Intelligence . (American Medical Association; 2026).
Read the paper: Towards conversational artificial intelligence for disease management
Read the paper: Towards autonomous medical artificial intelligence agents
People are turning to AI chatbots to plug gaps in health information
When physicians and AI work together, who is accountable? How to lay out medical liability
All living things emit a faint glow. Could this light be useful?
Personalized gene therapy relieves severe epilepsy in two boys
How does Make America Healthy Again hold up against scientific scrutiny?
Conversational AI: align commercial incentives with public interests
Scientific Director (f/m/x) at the Gregor Mendel Institute of Molecular Plant Biology (GmbH) of the Austrian Academy of Sciences (OeAW), Vienna.
Field or Position: Postdoctoral Research Fellow in Solid-State Battery Field Work Description We conduct research on solid-state batteries utilizin...
The Sharifi Lab seeks postdocs to study endocrine and metabolic mechanisms in GU cancers, driving translational discovery to therapies
Job Title: Executive Publisher, Springer Journals Location: New York or Philadelphia (Hybrid Working Model) Application Deadline: August 24, 2026 ...