文献类型：期刊文献

英文题名：Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review

作者：Zhang, Jianhua[1];Yin, Zhong[2];Chen, Peng[3];Nichele, Stefano[1]

机构：[1]Oslo Metropolitan Univ, Dept Comp Sci, Oslo, Norway;[2]Univ Shanghai Sci & Technol, Dept Control Sci & Engn, Shanghai, Peoples R China;[3]East China Univ Sci & Technol, Sch Informat Sci & Engn, Shanghai, Peoples R China

年份：2020

卷号：59

起止页码：103

外文期刊名：INFORMATION FUSION

收录：;EI(收录号：20200608148097);WOS:【SSCI(收录号:WOS:000523567600007)，SCI-EXPANDED(收录号:WOS:000523567600007)】；

基金：This work was supported in part by OsloMet Faculty TKD Lighthouse Project [grant no. 201369-100]. Z. Yin's work was funded by the National Natural Science Foundation of China [grant no. 61703277] and the Shanghai Sailing Program [grant no. 17YF1427000]. We gratefully acknowledge the support from Dr. Salvador Garcia, University of Granada, Spain. We would also like to thank the anonymous reviewers for their insightful and constructive comments and suggestions, which helped to improve this paper.

语种：英文

外文关键词：Emotion recognition; Affective computing; Physiological signals; Feature dimensionality reduction; Data fusion; Machine learning; Deep learning

摘要：In recent years, the rapid advances in machine learning (ML) and information fusion has made it possible to endow machines/computers with the ability of emotion understanding, recognition, and analysis. Emotion recognition has attracted increasingly intense interest from researchers from diverse fields. Human emotions can be recognized from facial expressions, speech, behavior (gesture/posture) or physiological signals. However, the first three methods can be ineffective since humans may involuntarily or deliberately conceal their real emotions (so-called social masking). The use of physiological signals can lead to more objective and reliable emotion recognition. Compared with peripheral neurophysiological signals, electroencephalogram (EEG) signals respond to fluctuations of affective states more sensitively and in real time and thus can provide useful features of emotional states. Therefore, various EEG-based emotion recognition techniques have been developed recently. In this paper, the emotion recognition methods based on multi-channel EEG signals as well as multi-modal physiological signals are reviewed. According to the standard pipeline for emotion recognition, we review different feature extraction (e.g., wavelet transform and nonlinear dynamics), feature reduction, and ML classifier design methods (e.g., k-nearest neighbor (KNN), naive Bayesian (NB), support vector machine (SVM) and random forest (RF)). Furthermore, the EEG rhythms that are highly correlated with emotions are analyzed and the correlation between different brain areas and emotions is discussed. Finally, we compare different ML and deep learning algorithms for emotion recognition and suggest several open problems and future research directions in this exciting and fast-growing area of AI.