A Hybrid Human-AI Workflow for Ultrasound Report Quality Control: A Multicenter Performance and Efficiency Evaluation.
Authors
Affiliations (6)
Affiliations (6)
- Department of Diagnostic and Therapeutic Ultrasonography, Tianjin Medical University Cancer Institute and Hospital, National Clinical Research Center for Cancer; Tianjin's Clinical Research Center for Cancer; Key Laboratory of Cancer Prevention and Therapy, Tianjin, China (X.L., J.Z., X.W., W.H., D.L., J.W., and X.W.).
- Department of Ultrasonic Diagnosis, The 983th Hospital of Joint Logistic Support Force of PLA, Tianjin, China (H.H.).
- Tianjin Medical University Cancer Institute and Hospital, National Clinical Research Center for Cancer; Tianjin's Clinical Research Center for Cancer; Key Laboratory of Cancer Prevention and Therapy, Tianjin, China (C.L., R.S., Y.L., and J.S.); Department of Radiology, Tianjin Central Hospital of Obstetrics and Gynecology, Tianjin, China (X.L.).
- Tianjin Medical University Cancer Institute and Hospital, National Clinical Research Center for Cancer; Tianjin's Clinical Research Center for Cancer; Key Laboratory of Cancer Prevention and Therapy, Tianjin, China (C.L., R.S., Y.L., and J.S.).
- Department of Radiology, The Second Hospital of Tianjin Medical University, Tianjin, China (X.W.).
- Department of Diagnostic and Therapeutic Ultrasonography, Tianjin Medical University Cancer Institute and Hospital, National Clinical Research Center for Cancer; Tianjin's Clinical Research Center for Cancer; Key Laboratory of Cancer Prevention and Therapy, Tianjin, China (X.L., J.Z., X.W., W.H., D.L., J.W., and X.W.). Electronic address: [email protected].
Abstract
This study evaluated a hybrid human-AI workflow for ultrasound report quality assurance using multicenter retrospective data. 786 anonymized ultrasound reports covering 22 examination categories were collected from four hospitals. Reports containing naturally occurring or implanted errors were matched across arms. GPT-4o and DeepSeek-v3 performed initial screening before review by six human reviewers (HRs). HR with blinded expert over-reading (HR-ex) served as control. Accuracy, false-positive (FP) and false-negative (FN) rates, and review time were compared. Supplementary validation compared HR and DeepSeek-assisted review using 1680 consecutive reports without implanted errors. Hybrid workflow (HYB) accuracy was comparable to HR (4o-HYB 80.2%, v3-HYB 82.5%, HR 82.3%) but lower than HR-ex (87.3%). Both HYBs showed higher FN rates than HR-ex (4o-HYB 34.2% and v3-HYB 27.3% vs HR-ex 16.9%, both p < 0.001). Severity assessment did not indicate preferential omission of clinically significant errors. Human verification reduced LLM-generated FNs and FPs. Median review time decreased substantially from 51.0 s (HR-ex) to 11.0 s (4o-HYB) and 12.0 s (v3-HYB). In supplementary validation (error prevalence, 9.6%), v3.2-HYB and HR showed FN rates of 22.6% and 24.3%, FP rates of 7.7% and 10.0%, and median review times of 15.4 and 24.3 s, respectively. The LLM-assisted HYB achieved accuracy comparable to single-physician review while substantially reducing audit time. These findings require prospective real-world validation before routine clinical implementation.