文献类型：期刊文献

中文题名：基于域适应互增强的多模态图像语义分割

英文题名：Multi-modal image semantic segmentation based on domain adaptation and mutual enhancement

作者：蓝鑫[1];谷小婧[1]

机构：[1]华东理工大学信息科学与工程学院,上海200237

年份：2022

卷号：43

期号：9

起止页码：2584

中文期刊名：计算机工程与设计

外文期刊名：Computer Engineering and Design

收录：CSTPCD;;北大核心:【北大核心2020】；

基金：国家自然科学基金面上基金项目(61973122)。

语种：中文

中文关键词：语义分割;多模态;红外图像;域适应;特征对齐

外文关键词：semantic segmentation;multi-modal;infrared image;domain adaptation;feature alignment

摘要：为解决可见光/红外(RGB-IR)双模态图像语义分割任务中模态间特征对不齐的问题,提出一种基于域适应互增强的多模态语义分割方法,在语义分割任务上辅助以可见光图像和红外图像相互之间的域适应转化来对齐不同模态的特征。针对模态内信息对不齐的问题,提出一种多级特征聚合对齐模块来聚合及对齐不同层级的特征。通过在两个夜间街景RGB-IR双模态语义分割数据集上进行相关实验,验证所提方法达到了当前最优性能,设计了大量消融实验验证了模型各部分的有效性。
To address the problem of misalignment of features in RGB-IR image semantic segmentation,a multi-modal semantic segmentation algorithm based on domain adaptation and mutual enhancement was proposed.The semantic segmentation task and domain adaptation between RGB and IR images were combined to let the segmentation model learn the features of RGB and IR simultaneously.The information alignment and mutual enhancement was achieved by domain adaptation.The feature aggregation and alignment module was proposed to aggregate and align the intra-modality features.Experiments were conducted on two private scene understanding datasets which were captured at night.The proposed method achieves the best performance.The ablation studies verify the feasibility of the model and the effectiveness of the algorithm.