AI feedback for clinician assessment of umbilical and middle cerebral artery Doppler ultrasound: a multicenter randomized trial.

Authors

Abstract

Assessment accuracy for umbilical artery (UA) and middle cerebral artery (MCA) Doppler ultrasound assessment depends on operator expertise. Artificial intelligence (AI) based feedback may improve performance, but its effect on assessment accuracy and the role of clinical experience remain uncertain. To determine whether AI feedback improves assessment accuracy for standardized image-based umbilical artery and middle cerebral artery Doppler vignettes among novice and experienced clinicians. This multicenter randomized clinical trial was conducted at five Danish hospitals. Ninety-six clinicians from obstetrics and gynecology departments, including residents, gynecologists, and obstetricians, were randomized 1:1 to AI-generated visual and textual feedback or no AI feedback. Each clinician evaluated 40 standardized image-based Doppler vignettes containing pre-existing static UA and MCA Doppler images. In total, 1,920 unique Doppler vignettes were evaluated across 48 clinician pairs matched by obstetric ultrasound experience. For color Doppler vignettes, clinicians indicated intended gate placement; for spectral Doppler vignettes, they judged waveform technical adequacy. Both are essential components of a technically valid Doppler measurement. No real-time scanning was performed. Intervention clinicians viewed the unannotated image alongside artificial intelligence feedback on gate placement, anatomical suitability, insonation angle, and spectral waveform quality, then entered their own response on the unannotated image. Accuracy was determined by expert consensus using predefined International Society of Ultrasound in Obstetrics and Gynecology-based criteria. AI feedback was classified as correct or incorrect according to whether it agreed with expert consensus. Expert consensus was based on two fetal medicine sonographers with >10 years of obstetric ultrasound experience, with disagreements resolved by a fetal medicine consultant. Clinicians were classified as novices (<1 year) or experienced (≥1 year) according to duration of obstetric ultrasound experience, irrespective of prior UA or MCA Doppler experience. Secondary analyses examined responses when clinicians followed or did not follow AI feedback, performance with correct and incorrect feedback, and AI-alone performance. Randomized-group differences were compared at clinician level; secondary repeated-vignette analyses used generalized estimating equations. Overall assessment accuracy was higher with AI feedback than without (75.0% vs 62.3%; mean difference, 12.7%; 95% CI, 8.2-17.1; p < 0.001). Among novices (n=36; 18/group), accuracy was 70.0% with feedback versus 57.9% without (mean difference: 12.1%, 95% CI 2.4-21.7; p = 0.005). Among experienced clinicians (n=60; 30 /group), accuracy was 78.0% versus 65.0% (mean difference: 13.0%; 95% CI: 5.5-20.5; p < 0.001). Novices were more likely than experienced clinicians to follow incorrect AI feedback (22.3% vs 15.8%; P = .01). AI alone outperformed novices without feedback, whereas experienced clinicians with artificial intelligence feedback outperformed artificial intelligence alone. AI feedback improved assessment accuracy for standardized umbilical artery and middle cerebral artery Doppler vignettes among novice and experienced clinicians. Clinical experience remained important when AI feedback was incorrect, supporting further evaluation of human-artificial intelligence collaboration during real-time Doppler acquisition.

Tags

Topics