文献类型：期刊文献

中文题名：MedBench:A Comprehensive,Standardized,and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models

作者：Mianxin Liu[1];Weiguo Hu[2];Jinru Ding[1];Jie Xu[1];Xiaoyang Li[2];Lifeng Zhu[2];Zhian Bai[2];Xiaoming Shi[1];Benyou Wang[3];Haitao Song[4];Pengfei Liu[5];Xiaofan Zhang[6];Shanshan Wang[7];Kang Li[8];Haofen Wang[9];Tong Ruan[10];Xuanjing Huang[11];Xin Sun[12];Shaoting Zhang[1]

机构：[1]Shanghai Artificial Intelligence Laboratory,Shanghai 200232,China;[2]Ruijin Hospital Affiliated to Shanghai Jiao Tong University School of Medicine,Shanghai 200025,China;[3]Chinese University of Hong Kong,Shenzhen 518172,China;[4]Shanghai Artificial Intelligence Research Institute,Shanghai 200240,and also with Shanghai Jiao Tong University,Shanghai 200240,China;[5]School of Electronic Information and Electrical Engineering,Shanghai Jiao Tong University,Shanghai 200240,China;[6]Qing Yuan Research Institute,Shanghai Jiao Tong University,Shanghai 200240,China;[7]Shenzhen Institutes of Advanced Technology,Chinese Academy of Sciences,Shenzhen 518055,China;[8]West China Hospital,Sichuan University,Chengdu 610041,China;[9]School of Design and Innovation,Tongji University,Shanghai 200092,China;[10]Department of Computer Science and Technology,East China University of Science and Technology,Shanghai 200237,China;[11]School of Computer Science,Fudan University,Shanghai 200433,China;[12]Xinhua Hospital Affiliated to Shanghai Jiaotong University School of Medicine,Shanghai 200092,China

年份：2024

卷号：7

期号：4

起止页码：1116

中文期刊名：Big Data Mining and Analytics

外文期刊名：大数据挖掘与分析(英文)

收录：CSCD:【CSCD2023_2024】；

基金：supported by the National Key R&D Program of China(Nos.2022ZD0160705 and 2022ZD0160704);the Three-year Action Program of Shanghai Municipality for Strengthening the Construction of Public Health System(No.GWVI-11.1-49);the Health Industry National Intelligent Social Governance Experiment Base(Shanghai)——Medical Artificial Intelligence Scenario Application Case Study and Social Experiment Survey,and Shanghai Artificial Intelligence Laboratory.

语种：英文

中文关键词：Medical Large Language Model(MLLM);benchmark;platform;open-source

摘要：Ensuring the general efficacy and benefit for human beings from medical Large Language Models(LLM)before real-world deployment is crucial.However,a widely accepted and accessible evaluation process for medical LLM,especially in the Chinese context,remains to be established.In this work,we introduce“MedBench”,a comprehensive,standardized,and reliable benchmarking system for Chinese medical LLM.First,MedBench assembles the currently largest evaluation dataset(300901 questions)to cover 43 clinical specialties,and performs multi-faceted evaluation on medical LLM.Second,MedBench provides a standardized and fully automatic cloud-based evaluation infrastructure,with physical separations between question and ground truth.Third,MedBench implements dynamic evaluation mechanisms to prevent shortcut learning and answer memorization.Applying MedBench to popular general and medical LLMs,we observe unbiased,reproducible evaluation results largely aligning with medical professionals’perspectives.This study establishes a significant foundation for preparing the practical applications of Chinese medical LLMs.