文献类型：会议论文

英文题名：In-Batch Negatives' Enhanced Self-Supervised Learning

作者：Tang, Yixin[1];Cheng, Hua[1];Fang, Yiquan[1];Pan, Yiming[1]

机构：[1]East China Univ Sci & Technol, Sch Informat Sci & Engn, Shanghai, Peoples R China

会议论文集：34th IEEE International Conference on Tools with Artificial Intelligence (ICTAI)

会议日期：OCT 31-NOV 02, 2022

会议地点：ELECTR NETWORK

语种：英文

外文关键词：Self-Supervised Learning; InfoNCE; Decision boundary

摘要：In contrastive self-supervised learning, such as SimCLR and MOCO, negative samples play an important role on the model's robustness. Instead of increasing the negatives' number, we promote the in-batch negatives' quality by adjusting their embedding. In-Batch Negatives' Enhanced Self-Supervised Learning (IBN-SSL) focus on negatives' quality by an importance-weighted algorithm and an online boundary. The importance-weighted negatives' quality algorithm decreases the negatives' relative locations from positive in projection space, which pushes the model to learn to distinguish positives from harder negatives. And the online boundary compresses negatives' vector space and keep negatives farther from positive. A better representation model can be leaned by IBN-SSL, and experiments show that it outperforms both RoBERTa and SimCLR in text classification task and similar sentence pair task.