文献类型：期刊文献

英文题名：Graph-Based Object Semantic Refinement for Visual Emotion Recognition

作者：Zhang, Jing[1];Liu, Xinyu[1];Wang, Zhe[1];Yang, Hai[1]

机构：[1]East China Univ Sci & Technol, Dept Comp Sci & Engn, Shanghai 200237, Peoples R China

年份：2022

卷号：32

期号：5

起止页码：3036

外文期刊名：IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY

收录：;EI(收录号：20213010683091);WOS:【SCI-EXPANDED(收录号:WOS:000790830300045)】；

基金：This work was supported by the National Natural Science Foundation of China under Grant 61806078.

语种：英文

外文关键词：Visualization; Semantics; Feature extraction; Emotion recognition; Analytical models; Predictive models; Neural networks; Object semantics; graph based object semantic refinement model; graph convolutional networks; visual emotion recognition

摘要：The rich semantic information contained in images is an important clue to explore visual emotions. Therefore, exploring the correlation between visual emotion and the semantic relationship of objects, and extracting more effective semantic features through explicit or implicit modeling is very important for visual emotion analysis. In this paper, a novel Graph-based Object Semantic Refinement (GOSR) model is proposed to extract multi-level semantic features for visual emotion classification, in which graph structures is used to represent the object semantics and their position relationships of an image, and Graph Convolutional Networks (GCN) is used to refine object information by the aggregating neighbor object with their position relationships. The different convolutional layer's features from GCN are further fused by Gated Recurrent Units (GRU) networks to achieve high-level semantic features. Then a framework with two branches to leverage visual and semantic information for visual sentiment analysis is proposed, which uses convolutional neural networks to extract visual features from images, and collaborates with semantic features from GOSR model to achieve better emotion recognition results. Besides, for alleviating the potentially unreasonable predictions and promote models collaboration, a novel tendency loss function based on the correlations among emotion labels is proposed to adjust the output activation value other than the target label. Extensive experiments on four widely used benchmark datasets show that our proposed method can achieve competitive performance and outperform most of the state-of-the-art methods on visual emotion recognition.