Segment Anything Model-Guided Dual-Branch Swin Transformer with Cross-Attention Fusion for Knee Osteoarthritis Severity Grading on X-ray Images.
Authors
Affiliations (2)
Affiliations (2)
- School of Computer Science and Engineering, Nanjing University of Science and Technology, Xiaolingwei Rd 200, Xuanwu District, Nanjing, Jiangsu Province, Nanjing, 210094, China.
- Institute of Automation, Chinese Academy of Sciences, 95 Zhongguancun East Road, Haidian District, Beijing, Beijing, 100190, China.
Abstract
We investigate a Segment Anything Model (SAM)-guided dual-branch Swin Transformer for knee osteoarthritis severity grading from X-ray images. A threshold-assisted SAM pipeline generates bone-region masks without task-specific SAM fine-tuning. Separate Swin-B branches process the image and mask, and two unidirectional cross-attention modules integrate their features. In the reported five-class evaluation, the model achieves 78.80\% accuracy, a macro-averaged one-vs-rest AUC of 0.9588, and quadratic weighted kappa of 0.8780. The corresponding three-class accuracy is 79.58\%. Expanded comparisons include KOA-specific methods and ablations of mask generation, weight sharing, and fusion configuration. These results support the usefulness of combining image context with mask-derived structural information within the evaluated setting.