文献类型：期刊文献

英文题名：PETA: evaluating the impact of protein transfer learning with sub-word tokenization on downstream applications

作者：Tan, Yang[1,2,3,4,5];Li, Mingchen[1,2,3,4,5];Zhou, Ziyi[2,3];Tan, Pan[2,3,4];Yu, Huiqun[1];Fan, Guisheng[1];Hong, Liang[2,3,4,5]

机构：[1]East China Univ Sci & Technol, Sch Informat Sci & Engn, Shanghai 200237, Peoples R China;[2]Shanghai Jiao Tong Univ, Shanghai Natl Ctr Appl Math SJTU Ctr, Shanghai 200240, Peoples R China;[3]Shanghai Jiao Tong Univ, Inst Nat Sci, Shanghai 200240, Peoples R China;[4]Shanghai Artificial Intelligence Lab, Shanghai 200240, Peoples R China;[5]Shanghai Jiao Tong Univ, Chongqing Artificial Intelligence Res Inst, Chongqing 200240, Peoples R China

年份：2024

卷号：16

期号：1

外文期刊名：JOURNAL OF CHEMINFORMATICS

收录：;WOS:【SCI-EXPANDED(收录号:WOS:001282761800002)】；

基金：This work was supported by Research Programme of National Engineering Laboratory for Big Data Distribution and Exchange Technologies, Shanghai Municipal Special Fund for Promoting High Quality Development (No. 2021-GYHLW-01007), the National Natural Science Foundation of China (11974239, 12104295), the Innovation Program of Shanghai Municipal Education Commission (2019-01-07-00-02-E00076), Shanghai Jiao Tong University Scientific and Technological Innovation Funds (21X010200843), the Student Innovation Center at Shanghai Jiao Tong University, and Shanghai Artificial Intelligence Laboratory.

语种：英文

外文关键词：Protein language model; Protein tokenization; Vocabulary size; Evaluation benchmark

摘要：Protein language models (PLMs) play a dominant role in protein representation learning. Most existing PLMs regard proteins as sequences of 20 natural amino acids. The problem with this representation method is that it simply divides the protein sequence into sequences of individual amino acids, ignoring the fact that certain residues often occur together. Therefore, it is inappropriate to view amino acids as isolated tokens. Instead, the PLMs should recognize the frequently occurring combinations of amino acids as a single token. In this study, we use the byte-pair-encoding algorithm and unigram to construct advanced residue vocabularies for protein sequence tokenization, and we have shown that PLMs pre-trained using these advanced vocabularies exhibit superior performance on downstream tasks when compared to those trained with simple vocabularies. Furthermore, we introduce PETA, a comprehensive benchmark for systematically evaluating PLMs. We find that vocabularies comprising 50 and 200 elements achieve optimal performance. Our code, model weights, and datasets are available at https://github.com/ginnm/ProteinPretraining. Scientific contributionThis study introduces advanced protein sequence tokenization analysis, leveraging the byte-pair-encoding algorithm and unigram. By recognizing frequently occurring combinations of amino acids as single tokens, our proposed method enhances the performance of PLMs on downstream tasks. Additionally, we present PETA, a new comprehensive benchmark for the systematic evaluation of PLMs, demonstrating that vocabularies of 50 and 200 elements offer optimal performance.