文献类型：期刊文献

英文题名：PETA: Evaluating the Impact of Protein Transfer Learning with Sub-word Tokenization on Downstream Applications

作者：Tan, Yang[1,2]; Li, Mingchen[1,2]; Tan, Pan[2,3]; Zhou, Ziyi[3]; Yu, Huiqun[1]; Fan, Guisheng[1]; Hong, Liang[2,3]

机构：[1] School of Information Science and Engineering, East China University of Science and Technology, Shanghai, 200240, China; [2] Shanghai Artificial Intelligence Laboratory, Shanghai, 200240, China; [3] Shanghai National Center for Applied Mathematics, SJTU Center, Institute of Natural Sciences, Shanghai Jiao Tong University, China

年份：2023

外文期刊名：arXiv

收录：EI(收录号：20230391440)

语种：英文

外文关键词：Amino acids - Benchmarking - Classification (of information) - Computational linguistics - Learning systems - Natural language processing systems - Transfer learning

摘要：Large protein language models are adept at capturing the underlying evolutionary information in primary structures, offering significant practical value for protein engineering. Compared to natural language models, protein amino acid sequences have a smaller data volume and a limited combinatorial space. Choosing an appropriate vocabulary size to optimize the pre-trained model is a pivotal issue. Moreover, despite the wealth of benchmarks and studies in the natural language community, there remains a lack of a comprehensive benchmark for systematically evaluating protein language model quality. Given these challenges, PETA trained language models with 14 different vocabulary sizes under three tokenization methods. It conducted thousands of tests on 33 diverse downstream datasets to assess the models' transfer learning capabilities, incorporating two classification heads and three random seeds to mitigate potential biases. Extensive experiments indicate that vocabulary sizes between 50 and 200 optimize the model, whereas sizes exceeding 800 detrimentally affect the model's representational performance. Our code, model weights and datasets are available at https://github.com/ginnm/ProteinPretraining. ? 2023, CC BY.