文献类型：期刊文献

英文题名：Saccade Inspired Attentive Visual Patch Transformer for Image Sentiment Analysis

作者：Zhang, Jing[1]; Liu, Jiangpei[1]; Zhang, Xinzhou[1]; Sun, Han[1]; Wang, Zhe[1]

机构：[1] Department of Computer Science and Engineering, East China University of Science and Technology, Shanghai, China

年份：2024

外文期刊名：SSRN

收录：EI(收录号：20240027222)

语种：英文

外文关键词：Behavioral research - Encoding (symbols) - Eye movements - Image analysis - Signal encoding

摘要：The generation of image-evoked emotion is usually regarded as a transient process in the image sentiment analysis. However, according to the saccade mechanism of the human visual system, the evoked emotion generated during the saccade process changes over time and attention. Based on this, we propose an Attentive Visual Patch Transformer (AVPT), using visual attention sequence to represent the sentiment context of images and predict the possible distribution of sentiment. In AVPT, the spatial structure in the form of patches can be reconstructed and reorganized by visual attention shift sequentially. Simultaneously, the temporal characteristics of attention shift are introduced to the relative position encoding, and merged in a self-attention manner to form a spatial-temporal process, which works similarly to the human visual system. Specifically, we propose a sequence attention shift module to simulate the saccade process, which obtains sequence attention and reduces the computational effort by group attentive convolutional gate recurrent unit. Then, a spatial-temporal correlation encoder module is proposed to encode temporal attention with spatial visual features and obtain the sequential visual features of saccade. Finally, a selfattention fusion module is used to extract the correlation hidden in the relative encoding features. Our proposed AVPT achieves excellent performance on visual sentiment distribution prediction and is comparable to state-of-the-art methods, as demonstrated by extensive experiments on the Flickr LDL and Twitter LDL datasets. ? 2024, The Authors. All rights reserved.