How AI can fail orthopedics 

Advertisement

Two University of California San Francisco orthopedic surgeons have identified two distinct ways large language models fail clinicians, even though today’s models can outperform a good fifth-year resident on standardized tests, according to a review published Nov. 1 in the Journal of the American Academy of Orthopaedic Surgeons.

Researchers Alan Dang, MD, and Alexis Dang, MD, of the University of California San Francisco and the San Francisco VA Health Center, set out to explain the science behind tools like OpenAI’s ChatGPT and Google’s MedGemma in enough depth for surgeons to critically evaluate AI outputs, design research and bring AI tools into clinical practice and resident education. 

Using a Schatzker VI tibial plateau fracture and an L4 pedicle screw sizing question as examples, the authors split AI errors into two classes: 

  1. Retrieval failures: The model generates incorrect content because its training data lacked or did not sufficiently reinforce the correct information. These hallucinations are the easiest to identify by checking against known references. 
  2. Reasoning failures: Even after a tool pulls in accurate information, the generation step still predicts the next token and can produce statistically likely but factually incorrect characterizations of the retrieved content. 

The review also flags that LLMs fail at providing accurate citations and that current performance benchmarks reward correct guesses without penalizing guessing. Retrieval-augmented generation, which assistants like Microsoft Copilot and Google Gemini use to blend LLMs with live information retrieval, is the main mechanism the authors described for reducing hallucinations. 

The authors concluded that LLMs have moved from novelty to practical tool for patient and resident education and clinical summaries, and that surgeons should evaluate AI output with the same rigor they apply to selection bias in clinical studies.

Advertisement

Next Up in Orthopedic

Advertisement