Evaluation of ChatGPT-Generated Simplifications of Shoulder MRI Reports: Assessments by Orthopedic Surgeons and Non-Healthcare Volunteers.
Authors
Affiliations (5)
Affiliations (5)
- Department of Orthopedics and Traumatology, Faculty of Medicine, Akdeniz University, Antalya 07070, Türkiye.
- Department of Orthopedics and Traumatology, Konya City Hospital, Konya 42020, Türkiye.
- Department of Orthopedics and Traumatology, Karapinar State Hospital, Konya 42400, Türkiye.
- Department of Orthopedics and Traumatology, Necip Fazıl City Hospital, Turkish Ministry of Health, Kahramanmaraş 46050, Türkiye.
- Department of Medical Informatics, Institute of Health Sciences, Akdeniz University, Antalya 07070, Türkiye.
Abstract
<b>Background/Objectives:</b> To evaluate ChatGPT-generated (GPT-5) simplifications of shoulder MRI reports from the perspectives of orthopedic surgeons and non-healthcare volunteers. <b>Methods</b>: In this single-center retrospective study, 100 de-identified shoulder MRI reports issued between June 2024 and June 2025 were simplified with ChatGPT in February 2026 using a standardized prompt. Between 10 March and 10 April 2026, three orthopedic surgeons independently rated each simplified text against the original report for accuracy, completeness, understandability, and misleading potential on a 5-point Likert scale (higher scores indicating better ratings). In a second arm, 100 adult non-healthcare volunteers each reviewed one simplified report and completed a six-item questionnaire on perceived clarity, comprehensibility, usefulness, and appropriateness. <b>Results</b>: Median expert scores were 5.0 in all domains, and all three raters assigned scores ≥ 4 in 86-98% of reports. Gwet's AC2 ranged from 0.93 to 0.98, whereas Krippendorff's α ranged from -0.02 to 0.05. Among volunteers, agreement ranged from 47% for understanding report terms to 80% for the appropriateness of the text. Health literacy correlated positively with most items (r = 0.329-0.429, <i>p</i> ≤ 0.001), and age correlated negatively with understanding report terms (r = -0.352, <i>p</i> < 0.001). <b>Conclusions</b>: The simplified reports were rated favorably by orthopedic surgeons and were perceived as useful by non-healthcare volunteers. However, the study was limited by its single-center design, the use of a single model and prompt, the absence of a comparison with the original reports in the volunteer arm, and the reliance on self-reported rather than objectively tested comprehension; therefore, it does not demonstrate improved patient comprehension or clinical safety. Such texts should supplement, not replace, the original report and physician consultation.