文献类型：期刊文献

英文题名：MSDiagnosis: A benchmark and framework for evaluating large language models in multi-step clinical diagnosis

作者：Hou, Ruihui[1];Chen, Shencheng[1];Fan, Yongqi[1];Yu, Guangya[1];Zhu, Lifeng[2];Sun, Jing[2];Liu, Jingping[1];Ruan, Tong[1]

机构：[1]East China Univ Sci & Technol, Sch Informat Sci & Engn, Shanghai 200237, Peoples R China;[2]Shanghai Jiao Tong Univ, Ruijin Hosp, Sch Med, Shanghai 200025, Peoples R China

年份：2025

卷号：330

外文期刊名：KNOWLEDGE-BASED SYSTEMS

收录：;EI(收录号：20254019278684);WOS:【SCI-EXPANDED(收录号:WOS:001589482700007)】；

基金：This work was supported in part by the Shanghai Pilot Program for Basic Research under Grant 22TQ1400100-20 and in part by the Shang-hai Natural Science Foundation Project under Grant 25ZR1402116.

语种：英文

外文关键词：Multi-step clinical diagnosis; Large language model; Prompting strategy

摘要：Clinical diagnosis is critical in clinical decision-making, typically requiring a continuous and evolving process that includes primary, differential, and final diagnoses. However, most existing clinical diagnostic tasks are single-step processes, which do not align with the complex multi-step diagnostic procedures found in real clinical scenarios. In this paper, we propose MSDiagnosis, a Chinese multi-step clinical diagnostic benchmark consisting of 2225 cases from 12 departments, covering primary, differential, and final diagnosis tasks. Conventional approaches often rely on large language models (LLMs) to perform these tasks sequentially, which can lead to error propagation. To address this, we propose a two-stage diagnostic framework consisting of a forward inference module and a backward reasoning and refinement module. This framework is applied at each diagnostic stage to effectively mitigate error propagation across steps. The forward module retrieves similar cases to assist the LLM in generating an initial diagnosis. In the backward inference and refinement module, we first perform backward inference to infer the diagnostic criteria associated with the initially identified potential diseases. These criteria are then compared with the patient's records to identify and eliminate possible misdiagnoses. Finally, the diagnostic conclusion is further refined and confirmed. Based on the MSDiagnosis, we evaluate medical LLMs (e.g., OpenBioLLM, PULSE, and Apollo2), general LLMs (e.g., DeepSeek-V3, OpenAI-O1, and GLM4), and our proposed framework. Experimental results show that our framework achieves state-of-the-art performance, demonstrating its effectiveness in multi-step diagnostic tasks. We also provide a detailed analysis and suggest future research directions for this task. Our code and data are publicly available at https://github.com/nlper-hou/MSDiagnosis.