Dual-level knowledge distillation from vision transformer to Swin transformer for fine-grained brain tumor classification: A 44-class MRI benchmark.

Authors

Abstract

Accurate subtype classification of brain tumors from MRI requires simultaneous discrimination across histological category and imaging modality. We propose a dual-level knowledge distillation (DLKD) framework that transfers logit-level soft targets (temperature <i>T</i> = 4.0, weight α = 0.6) and feature-level representations via a learnable projector from a frozen vision transformer-Base teacher (86.6 M parameters) to a compact Swin-Tiny student (28.3 M parameters), evaluated on a 44-class brain tumor MRI benchmark (4479 images) spanning 15 histological subtypes across three imaging modalities. Teacher and student are trained under an identical protocol matched augmentation, class-balancing (oversampling to 255 images/class), and optimization on an 80/10/10 stratified split. Averaged across five random seeds, the proposed student achieves 97.9% ± 0.2% test accuracy and 0.978 ± 0.003 macro F1, compared with 95.25% ± 0.3% for the teacher, 95.8% ± 0.3% for a no-distillation student baseline, 96.4% ± 0.2% for logit-only distillation, and 96.5% ± 0.3% for feature-only distillation. Each distillation component individually improves over baseline (+0.6 pp. and +0.7 pp. respectively), and their combination yields the largest gain (+2.1 pp. over baseline; +2.65 pp. over the teacher), indicating the two objectives are complementary rather than redundant. Four widely used convolutional neural network baselines trained under the same protocol EfficientNet-B0 (94.4%), ResNet-50 (93.5%), DenseNet-121 (92.8%), and MobileNetV3-Large (91.6%) all trail the distilled student. We report four threats to validity: (1) patient identifiers are unavailable in the source data, so cross-modality leakage of the same patient's T1, T1C+, and T2 scans across splits cannot be excluded; (2) the Kaggle-sourced dataset may contain acquisition artifacts or shortcuts that inflate performance across all models; (3) the student's gains over the teacher and baseline, while consistent with typical knowledge-distillation results, still warrant external replication; (4) the smaller 10% test split leaves some classes with as few as 5-7 test images, making their per-class F1 unreliable. This work establishes a knowledge-distillation benchmark on the 44-class taxonomy and provides methodological infrastructure for future patient-stratified studies; the reported numbers should not be interpreted as clinical-grade performance until these threats are addressed.

Tags

Topics