文献类型：期刊文献

英文题名：A multimodal transformer to fuse images and metadata for skin disease classification

作者：Cai, Gan[1];Zhu, Yu[1];Wu, Yue[1];Jiang, Xiaoben[1];Ye, Jiongyao[1];Yang, Dawei[2,3]

机构：[1]East China Univ Sci & Technol, Sch Informat Sci & Engn, Shanghai 200237, Peoples R China;[2]Fudan Univ, Zhongshan Hosp, Dept Pulm & Crit Care Med, Shanghai 200032, Peoples R China;[3]Shanghai Engn Res Ctr Internet Things Resp Med, Shanghai 200032, Peoples R China

年份：2023

卷号：39

期号：7

起止页码：2781

外文期刊名：VISUAL COMPUTER

收录：;EI(收录号：20221912072317);WOS:【SCI-EXPANDED(收录号:WOS:000791065400002)】；

基金：This research is supported in part by Science and Technology Commission of Shanghai Municipality (20DZ2254400, 21DZ2200600), National Scientific Foundation of China (82170110), Zhongshan Hospital Clinical Research Foundation(2019ZSGG15), and Shanghai Pujiang Program (20PJ1402400).

语种：英文

外文关键词：Skin disease; Deep learning; Transformer; Multimodal fusion; Attention

摘要：Skin disease cases are rising in prevalence, and the diagnosis of skin diseases is always a challenging task in the clinic. Utilizing deep learning to diagnose skin diseases could help to meet these challenges. In this study, a novel neural network is proposed for the classification of skin diseases. Since the datasets for the research consist of skin disease images and clinical metadata, we propose a novel multimodal Transformer, which consists of two encoders for both images and metadata and one decoder to fuse the multimodal information. In the proposed network, a suitable Vision Transformer (ViT) model is utilized as the backbone to extract image deep features. As for metadata, they are regarded as labels and a new Soft Label Encoder (SLE) is designed to embed them. Furthermore, in the decoder part, a novel Mutual Attention (MA) block is proposed to better fuse image features and metadata features. To evaluate the model's effectiveness, extensive experiments have been conducted on the private skin disease dataset and the benchmark dataset ISIC 2018. Compared with state-of-the-art methods, the proposed model shows better performance and represents an advancement in skin disease diagnosis.