文献类型：期刊文献

英文题名：Learnable Retrieval Enhanced Visual-Text Alignment and Fusion for Radiology Report Generation

作者：Zhou, Qin[1,2]; Liang, Guoyan[3,4]; Li, Xindi[3,4]; Chen, Jingyuan[3]; Wang, Zhe[1,2]; Yao, Chang[3,4]; Wu, Sai[3,4]

机构：[1] Ecust, Department of Computer Science and Engineering, China; [2] Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, China; [3] Zhejiang University, Hangzhou, China; [4] Hangzhou High-Tech Zone [Binjiang] Institute of Blockchain and Data Security, China

年份：2025

起止页码：22529

外文期刊名：Proceedings of the IEEE International Conference on Computer Vision

收录：EI(收录号：20262821076831)

语种：英文

摘要：Automated radiology report generation is essential for improving diagnostic efficiency and reducing the workload of medical professionals. However, existing methods face significant challenges, such as disease class imbalance and insufficient cross-modal fusion. To address these issues, we propose the learnable Retrieval Enhanced Visual-Text Alignment and Fusion (REVTAF) framework, which effectively tackles both class imbalance and visual-text fusion in report generation. REVTAF incorporates two core components: (1) a Learnable Retrieval Enhancer (LRE) that utilizes semantic hierarchies from hyperbolic space and intra-batch context through a ranking-based metric. LRE adaptively retrieves the most relevant reference reports, enhancing image representations, particularly for underrepresented (tail) class inputs; and (2) a fine-grained visual-text alignment and fusion strategy that ensures consistency across multi-source cross-attention maps for precise alignment. This component further employs an optimal transport-based cross-attention mechanism to dynamically integrate task-relevant textual knowledge for improved report generation. By combining adaptive retrieval with multi-source alignment and fusion, REVTAF achieves finegrained visual-text integration under weak image-report level supervision while effectively mitigating data imbalance issues. The experiments demonstrate that REVTAF outperforms state-of-the-art methods, achieving an average improvement of 7.4% on the MIMIC-CXR dataset and 2.9% on the IU X-Ray dataset. Comparisons with mainstream multimodal LLMs (e.g., GPT-series models), further highlight its superiority in radiology report generation11https://github.com/banbooliang/REVTAF-RRG. ? 2025 IEEE.