NeuroPlast: a learnable activation function evaluated under knowledge distillation for medical image classification.

Authors

Abstract

Deep neural networks for medical image classification rely almost exclusively on fixed activation functions such as ReLU. We introduce NeuroPlast, a parametric activation function whose four differentiable components (a shifted sigmoid modeling NMDA-type voltage gating, a Gaussian plateau inspired by AMPA receptor scaling, an excitatory rectifier, and an inhibitory leak) are combined through six learnable parameters. Two mixing strategies are evaluated: a <i>static</i> variant with fixed learned weights, and a <i>metaplastic</i> variant whose mixing coefficients adapt per sample via a lightweight squeeze-excite gate conditioned on channel statistics. NeuroPlast is embedded within NADN Ultra, a 13.5 M-parameter residual convolutional neural network (CNN) with convolutional block attention modules (CBAM), trained entirely from scratch through a two-phase knowledge distillation (KD) pipeline. The teacher is a fine-tuned EfficientNet-B0; the student combines logit-level KD with optional feature-level alignment losses. Across four medical imaging benchmarks (Brain Tumor MRI, 7,200 images, 4 classes; Chest X-ray Pneumonia, 5,856 images, 2 classes; Skin Cancer HAM10000, 10,015 images, 7 classes; COVID-19 Radiography, 10,848 images, 2 classes), evaluated under five-fold stratified cross-validation with 95% confidence intervals, the static-KD variant reaches 99.4-99.5% accuracy on COVID-19 X-ray across three seeds, on par with pretrained EfficientNet-B0 (99.18%, within replication noise) and above ResNet-18 (98.69%). It closes 65% of the accuracy gap on Brain Tumor MRI (98.29% vs. 99.13%), reaches 92.5% on Chest X-ray under a uniform class-balancing rule, and 79.5% on Skin Cancer under lesion-grouped cross-validation. The metaplastic variant achieves 99.28% on COVID-19 X-ray, still above pretrained baselines, but does not consistently outperform the static version, a negative result analyzed through ablation experiments. On three tabular medical datasets spanning three orders of magnitude in sample size (569 to 253,680), NeuroPlast matches five established activations within ±1.7 percentage points; at 253 K samples all activations converge within 0.09 pp. Our findings indicate that knowledge distillation is the primary enabler for from-scratch architectures to approach pretrained-level performance on medical images; under an identical distillation pipeline, NeuroPlast adds a small but consistent gain over ReLU, GELU, Swish, Mish, and PReLU, leading on all four imaging benchmarks by margins below one point.

Tags

Topics