文献类型：期刊文献

英文题名：Cross on Cross Attention: Deep Fusion Transformer for Image Captioning

作者：Zhang, Jing[1];Xie, Yingshuai[1];Ding, Weichao[1];Wang, Zhe[1]

机构：[1]East China Univ Sci & Technol, Dept Comp Sci & Engn, Shanghai 200237, Peoples R China

年份：2023

卷号：33

期号：8

起止页码：4257

外文期刊名：IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY

收录：;EI(收录号：20231013683698);WOS:【SCI-EXPANDED(收录号:WOS:001045167400055)】；

基金：& nbsp;This work was supported by the Natural Science Foundation of Shanghai "Research on image sentiment analysis and expression based on human vision and cognitive psychology" under Grant 22ZR1418400.

语种：英文

外文关键词：Image captioning; deep fusion transformer; global cross encoder; cross on cross attention

摘要：Numerous studies have shown that in-depth mining of correlations between multi-modal features can help improve the accuracy of cross-modal data analysis tasks. However, the current image description methods based on the encoder-decoder framework only carry out the interaction and fusion of multi-modal features in the encoding stage or the decoding stage, which cannot effectively alleviate the semantic gap. In this paper, we propose a Deep Fusion Transformer (DFT) for image captioning to provide a deep multi-feature and multi-modal information fusion strategy throughout the encoding to decoding process. We propose a novel global cross encoder to align different types of visual features, which can effectively compensate for the differences between features and incorporate each other's strengths. In the decoder, a novel cross on cross attention is proposed to realize hierarchical cross-modal data analysis, extending complex cross-modal reasoning capabilities through the multi-level interaction of visual and semantic features. Extensive experiments conducted on the MSCOCO dataset prove that our proposed DFT can achieve excellent performance and outperform state-of-the-art methods. The code is available at https://github.com/weimingboya/DFT.