Reconstructing shared visual experiences from human brain activity across individuals.
Authors
Affiliations (6)
Affiliations (6)
- School of Information Science and Engineering, Lanzhou University, Lanzhou, China.
- State Key Laboratory of Brain-Machine Intelligence, Zhejiang University, Hangzhou, China.
- School of Computing and Data Science, The University of Hong Kong, Hong Kong Special Administrative Region of China.
- Institute of Cyberspace Security, Sun Yat-sen University, Shenzhen, China.
- School of Information Science and Engineering, Lanzhou University, Lanzhou, China. Electronic address: [email protected].
- School of Information Science and Engineering, Lanzhou University, Lanzhou, China. Electronic address: [email protected].
Abstract
Reconstructing visual experiences from brain activity promises to strengthen brain-computer interfaces and our fundamental understanding of perception. However, current deep learning approaches for functional magnetic resonance imaging (fMRI)-based image synthesis are often person-specific, requiring substantial data to adapt to new individuals, thus limiting their scalability and translational potential. Here, we present MindShow, a unified generative framework for shared-subject fMRI-to-image reconstruction under a cohort-level training setting. The core of MindShow is a Hierarchically-Conditioned Mixture-of-Experts (HiCo-MoE) encoder that disentangles population-shared latent representations from subject-specific neural characteristics, enabling data-efficient target-subject adaptation under limited calibration data. These representations are then processed by our Gated Perceiver Bottleneck (GPB), a gated Perceiver-style tokenization interface that resolves multi-scale representational misalignment by adaptively mapping the fMRI features into distinct, fixed-size image and text latent tokens. To improve semantic and structural consistency, we introduce a multi-granular optimal transport loss (MOT-Align), which regularizes sample- and token-level distributional alignment between brain-derived features and the latent space of a pretrained vision-language model. When guided by these aligned embeddings, a frozen diffusion model synthesizes images that aim to preserve the semantic content and coarse layout of the perceived content. MindShow improves high-level reconstruction metrics while maintaining competitive structural fidelity, representing a methodological step toward scalable shared-subject neural decoding. All implementation code is available on GitHub: https://github.com/AI-NMI/MindShow.