论文网址：BrainRAM: Cross-Modality Retrieval-Augmented Image Reconstruction from Human Brain Activity | Proceedings of the 32nd ACM International Conference on Multimedia

论文代码：https://github.com/HQ406/BrainRAM 

英文是纯手打的！论文原文的summarizing and paraphrasing。可能会出现难以避免的拼写错误和语法错误，若有发现欢迎评论指正！文章偏向于笔记，谨慎食用

目录

1. 心得

2. 论文逐段精读

2.1. Abstract

2.2. Introduction

2.3. Related Works

2.3.1. Visual stimuli reconstruction

2.3.2. Retrieval-Augmented Generation

2.4. Method

2.4.1. Overview

2.4.2. Stage I: Brain-visual-linguistic Consistency

2.4.3. Stage II: Retrieval-Augmentation

2.5. Experiments

2.5.1. Dataset

2.5.2. Implementation Details

2.5.3. Evaluation Metrics

2.6. Results

2.6.1. Reconstruction Results

2.6.2. Retrieval Results

2.6.3. Ablation Studies

2.7. Conclusion

1. 心得

（1）用检索真的好咩？会不会有点fMRI解码的违背初心

（2）而且也要求有文本这个模态诶，也不是自己生成来辅助的

2. 论文逐段精读

2.1. Abstract

        ①Challenges in brain activity decoding: scarse large-scale dataset and measurement noise

2.2. Introduction

        ①⭐fMRI temporarily reflect BOLD, while it does not contains any past kownledge

        ②Only relying on fMRI causes inconsistency of semantic and structure（是咩？我感觉还好？）:

        ③Individuals with rich knowledge and experience probably describe objects more detailed

        ④Difference between BrainRAM and other works:

2.3. Related Works

2.3.1. Visual stimuli reconstruction

        ①Lists relevant reconstruction works, and points out their poor results

2.3.2. Retrieval-Augmented Generation

        ①Retrival will provide higher reconstruction quality

2.4. Method

2.4.1. Overview

        ①Overall pipeline:

2.4.2. Stage I: Brain-visual-linguistic Consistency

        ①Triplet set: \Omega=\{S_{i},V_{i},T_{i}\}_{i=1}^{n}, where S_i \in \mathbb{R}^{1\times N} denotes fMRI \beta signal, V_i \in\mathbb{R}^{H\times W\times3} denotes stimuli and T is caption

        ②Raw image and text are processed to 257 \times 768 and 77 \times 768 by image encoder \mathcal{E}_{img}\left ( \cdot \right ) and \mathcal{E}_{txt}\left ( \cdot \right ) (the last hidden layer, instead of the final output)

        ③fMRI data is fed to MLP and, block and diffusion prior

        ④Diffusion step:

\mathbf{x}_t=\sqrt{\alpha_t}\mathbf{x}_0+\sqrt{1-\alpha_t}\epsilon_t

        ⑤The inverse diffusion U-Net contains cross diffusion:

\text{CrossAttention }(Q,K,V)=\mathrm{softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)V,\\Q=W_{Q}^{(i)}\varphi_{i}\left(\mathbf{x}_{t}\right),K=W_{K}^{(i)}\Phi_{proj}(S),V=W_{V}^{(i)}\Phi_{proj}(S),

        ⑥InfoNCE loss:

\mathcal{L}_{\mathrm{InfoNCE}}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp{(p_i\cdot c_i/\tau)}}{\sum_{j=1}^{B}\exp{(p_i\cdot c_j/\tau)}}

where p is the output of diffusion, c denotes aligned target, \tau is temperature hyperparameter

        ⑦Data augmentation (MixCo) of two samples S_i and S_k:

S_{mix_{i,k}}=\lambda_{i}\cdot S_{i}+(1-\lambda_{i})\cdot S_{k}

with MixCo loss:

\begin{aligned} \mathcal{L}_{\mathrm{MixCo}} & =-\sum_{i=1}^n\left[\lambda_i\cdot\log\frac{\exp\left(p_i^*\cdot c_i/\tau\right)}{\sum_{j=0}^K\exp\left(p_i^*\cdot c_j/\tau\right)}\right] \\ & +(1-\lambda_i)\cdot\log\frac{\exp\left(p_i^*\cdot c_k/\tau\right)}{\sum_{j=0}^K\exp\left(p_i^*\cdot c_j/\tau\right)} \end{aligned}

        ⑧The total loss:

\mathcal{L}_{\mathrm{total}1}=\gamma\mathcal{L}_{\mathrm{InfoNCE~}}+(1-\gamma)\mathcal{L}_{\mathrm{MSE~}}+\beta\mathcal{L}_{\mathrm{MixCo}}

2.4.3. Stage II: Retrieval-Augmentation

        ①For query p (the output of diffusion block), to search K embeddings \{c_i\}_{i=1}^K:

u_{c_{i}}=\text{ CrossAttention }(Q_{p},K_{c_{i}},V_{c_{i}}),\\u_{p}=\text{ SelfAttention }(Q_{p},K_{p},V_{p}),\\\mathbf{U}=\left[u_{p},u_{c_{1}},u_{c_{2}},\ldots,u_{c_{K}}\right]^{T},

where u_{p},u_{c_{i}}\in\mathbb{R}^{T\times D},\mathrm{U}\in\mathbb{R}^{(K+1)\times T\times D}

        ②Average features:

\bar{u}=\frac{1}{K+1}(u_p+\sum_i^Ku_{c_i})

and then extract features \mathbf{Z}\in\mathbb{R}^{(K+1)\times T} by MLP \mathcal{F}_i\left ( \cdot \right ):

\mathbf{Z}= \begin{bmatrix} \mathcal{F}_1(\bar{u}) \\ \mathcal{F}_2(\bar{u}) \\ \cdots \\ \mathcal{F}_{K+1}(\bar{u}) \end{bmatrix},\quad i=1,2,\ldots,K+1

futher applied Softmax on the first dimension of \mathbf{Z}:

\tilde{\mathbf{Z}}=\mathrm{softmax}(\mathbf{Z})

        ③The final output q\in\mathbb{R}^{T\times D} of RAM:

q_{ij}=\sum_k^{K+1}\sum_i^T\tilde{\mathbf{Z}}_{ki}\cdot\mathbf{U}_{kij},\quad j=1,2,\ldots,D

        ④Loss:

\mathcal{L}_{\mathrm{total}2}=\gamma\mathcal{L}_{\mathrm{InfoNCE}}+(1-\gamma)\mathcal{L}_{\mathrm{MSE}}

2.5. Experiments

2.5.1. Dataset

        ①Dataset: NSD

        ②Subject: 1, 2, 5, 7

2.5.2. Implementation Details

（1）Stage I

        ①Epoch: 180

        ②Batch size: 32

        ③Learning rate: 2.5e-4

        ④Optimizer: AdamW

        ⑤Loss term weight: \gamma =0.8, \beta =0.3

（2）Stage II

        ①Retrieve set: LAION-2B-en

        ②Reconstruction sample number: 4

        ③Retrieval number: 4

        ④Epoch: 120

        ⑤Batch size: 256

        ⑥Learning rate: 2.5e-4

        ⑦Optimizer: AdamW

        ⑧Loss term: \gamma =0.2

2.5.3. Evaluation Metrics

        ~

2.6. Results

2.6.1. Reconstruction Results

        ①Quantitative results:

        ②Qualitative results:

2.6.2. Retrieval Results

        ①The performance of retrieval in NSD pairs:

        ②Retrieval in LAION-2B-en:

2.6.3. Ablation Studies

        ①Module ablation:

        ②Comparison of RAM:

2.7. Conclusion

        ~

# 人工智能 # 机器学习 # 深度学习 # python # 计算机视觉 # 神经网络 # pytorch
Logo
科技创新，汇聚千年文化底蕴，共筑璀璨首都未来之梦！

欢迎加入北京社区

更多推荐

cover

AI写的代码越来越多，程序员真正的风险变了？

avatar 北京城市开发者社区
cover

数据市场VS自己写爬虫：什么时候该买什么时候该抓

avatar 北京城市开发者社区
cover

“智在灵玑”释放信号：智能体开始拼产业链协同

avatar 北京城市开发者社区