文献类型：期刊文献

英文题名：Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation

作者：Gu, Zhouhong[1]; Zhu, Xiaoxuan[1]; Ye, Haoning[1]; Zhang, Lin[1]; Wang, Jianchen[1]; Zhu, Yixin[1]; Jiang, Sihang[1]; Xiong, Zhuozhi[1]; Li, Zihan[1]; Wu, Weijie[1]; He, Qianyu[1]; Xu, Rui[1]; Huang, Wenhao[1]; Liu, Jingping[2]; Wang, Zili[3]; Wang, Shusen[3]; Zheng, Weiguo[4]; Feng, Hongwei[1]; Xiao, Yanghua[1,5]

机构：[1] Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University, China; [2] School of Information Science and Engineering, East China University of Science and Technology, China; [3] Xiaohongshu Inc, China; [4] School of Data Science, Fudan University, China; [5] Fudan-Aishu Cognitive Intelligence Joint Research Center, China

年份：2024

卷号：38

期号：16

起止页码：18099

外文期刊名：Proceedings of the AAAI Conference on Artificial Intelligence

收录：EI(收录号：20241515880809)

语种：英文

外文关键词：Computer games

摘要：New Natural Langauge Process (NLP) benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present Xiezhi, the most comprehensive evaluation suite designed to assess holistic domain knowledge. Xiezhi comprises multiple-choice questions across 516 diverse disciplines ranging from 13 different subjects with 249,587 questions and accompanied by Xiezhi-Specialty with 14,041 questions and Xiezhi-Interdiscipline with 10,746 questions. We conduct evaluation of the 47 cutting-edge LLMs on Xiezhi. Results indicate that LLMs exceed average performance of humans in science, engineering, agronomy, medicine, and art, but fall short in economics, jurisprudence, pedagogy, literature, history, and management. All the evaluation code and data are open sourced in https://github.com/MikeGu721/XiezhiBenchmark Copyright ? 2024, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.