文献类型：期刊文献

英文题名：CR-LLM: A Dataset and Optimization for Concept Reasoning of Large Language Models

作者：Li, Nianqi[1]; Liu, Jingping[2]; Jiang, Sihang[1]; Jiang, Haiyun[1]; Xiao, Yanghua[1]; Liang, Jiaqing[3]; Liang, Zujie[4]; Wei, Feng[4]; Chen, Jinglei[4]; Hao, Zhenghong[4]; Han, Bing[4]

机构：[1] Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University, China; [2] School of Information Science and Engineering, East China University of Science and Technology, China; [3] School of Data Science, Fudan University, China; [4] MYbank, Ant Group, China

年份：2024

起止页码：13737

外文期刊名：Proceedings of the Annual Meeting of the Association for Computational Linguistics

收录：EI(收录号：20244017142878)

语种：英文

外文关键词：Computational linguistics - Modeling languages

摘要：Concept reasoning is an important capability for models to understand the world. However, the existing datasets, such as concept extraction and concept generation, suffer from modeledge leakage and context leakage. To address these limitations, we construct a dataset of concept reasoning for large language models (CR-LLM) with modeledge leakage prevention and context leakage prevention, which consists of 2,167 samples and covers different concept types. In addition, we propose a hybrid reasoning method, consisting of inductive reasoning, deductive reasoning and a controller. This method allows large language models to adaptively select the optimal reasoning method for each input sample. Finally, we conduct extensive experiments on CR-LLM using different models and methods. The results show that existing large language models and reasoning methods perform sub-optimally in the concept reasoning task. In contrast, our proposed method significantly improves the capabilities, achieving a 7% increase in accuracy compared to CoT and demonstrating better granularity. We release CR-LLM and code at https://github.com/Nianqi-Li/Concept-Reasoning-for-LLMs. ? 2024 Association for Computational Linguistics.