文献类型：期刊文献

英文题名：AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model

作者：Yan, Yongyu[1]; Xue, Kui[2]; Shi, Xiaoming[2]; Ye, Qi[1]; Liu, Jingping[1]; Ruan, Tong[1]

机构：[1] East China University of Science and Technology, School of Information Science and Engineering, Shanghai, China; [2] Shanghai Artificial Intelligence Laboratory, Shanghai, China

年份：2023

起止页码：953

外文期刊名：Proceedings - 2023 2023 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2023

收录：EI(收录号：20240715560022)

语种：英文

摘要：Continual pretraining is a popular way of building a domain-specific pretrained language model from a general-domain language model. In spite of its high efficiency, continual pretraining suffers from catastrophic forgetting, which may harm the model's performance in downstream tasks. To alleviate the issue, in this paper, we propose a continual pretraining method for the BERT-based model, named Attention-FFN Adapter. Its main idea is to introduce a small number of attention heads and hidden units inside each self-attention layer and feed-forward network. Furthermore, we train a domain-specific language model named AF Adapter based RoBERTa for the Chinese biomedical domain. In experiments, models are applied to downstream tasks for evaluation. The results demonstrate that with only about 17% of model parameters trained, AF Adapter achieves 0.6%, 2% gain in performance on average, compared to strong baselines. Further experimental results show that our method alleviates the catastrophic forgetting problem by 11% compared to the fine-tuning method. Code is available at https://github.com/yanyongyu/AF-Adapter. ? 2023 IEEE.