文献类型：期刊文献

英文题名：Simple, Efficient, and Scalable Structure-Aware Adapter Boosts Protein Language Models

作者：Tan, Yang[1,2,3];Li, Mingchen[1,2,3];Zhou, Bingxin[3,4];Zhong, Bozitao[3,4];Zheng, Lirong[4,5,6];Tan, Pan[2,4];Zhou, Ziyi[3,4];Yu, Huiqun[1];Fan, Guisheng[1];Hong, Liang[2,3,4,7]

机构：[1]East China Univ Sci & Technol, Sch Informat Sci & Engn, Shanghai 200237, Peoples R China;[2]Shanghai Artificial Intelligence Lab, Shanghai 200232, Peoples R China;[3]Shanghai Natl Ctr Appl Math, SJTU Ctr, Shanghai 200240, Peoples R China;[4]Shanghai Jiao Tong Univ, Inst Nat Sci, Shanghai 200240, Peoples R China;[5]Univ Michigan, Med Sch, Dept Cell & Dev Biol, Ann Arbor, MI 48104 USA;[6]Univ Michigan, Michigan Neurosci Inst, Med Sch, Ann Arbor, MI 48104 USA;[7]Shanghai Jiao Tong Univ, Zhangjiang Inst Adv Study, Shanghai 200240, Peoples R China

年份：2024

卷号：64

期号：16

起止页码：6338

外文期刊名：JOURNAL OF CHEMICAL INFORMATION AND MODELING

收录：;EI(收录号：20240188659);WOS:【SCI-EXPANDED(收录号:WOS:001286276000001)】；

基金：This work was supported by Research Programme of National Engineering Laboratory for Big Data Distribution and Exchange Technologies, Shanghai Municipal Special Fund for Promoting High Quality Development(No. 2021-GYHLW-01007), the National Natural Science Foundation of China (11974239, 12104295), the Innovation Program of Shanghai Municipal Education Commission (2019-01-07-00-02-E00076), Shanghai Jiao Tong University Scientific and Technological Innovation Funds (21x010200843), the Student Innovation Center at Shanghai Jiao Tong University, and Shanghai Artificial Intelligence Laboratory.

语种：英文

外文关键词：Benchmarking - Computational linguistics - Embeddings - Natural language processing systems

摘要：Fine-tuning pretrained protein language models (PLMs) has emerged as a prominent strategy for enhancing downstream prediction tasks, often outperforming traditional supervised learning approaches. As a widely applied powerful technique in natural language processing, employing parameter-efficient fine-tuning techniques could potentially enhance the performance of PLMs. However, the direct transfer to life science tasks is nontrivial due to the different training strategies and data forms. To address this gap, we introduce SES-Adapter, a simple, efficient, and scalable adapter method for enhancing the representation learning of PLMs. SES-Adapter incorporates PLM embeddings with structural sequence embeddings to create structure-aware representations. We show that the proposed method is compatible with different PLM architectures and across diverse tasks. Extensive evaluations are conducted on 2 types of folding structures with notable quality differences, 9 state-of-the-art baselines, and 9 benchmark data sets across distinct downstream tasks. Results show that compared to vanilla PLMs, SES-Adapter improves downstream task performance by a maximum of 11% and an average of 3%, with significantly accelerated convergence speed by a maximum of 1034% and an average of 362%, the training efficiency is also improved by approximately 2 times. Moreover, positive optimization is observed even with low-quality predicted structures. The source code for SES-Adapter is available at https://github.com/tyang816/SES-Adapter.