arxiv_cl 95% Match Research paper RAG system developers,LLM researchers,AI benchmark creators,Information retrieval specialists 3 weeks ago

RAGCap-Bench: Benchmarking Capabilities of LLMs in Agentic Retrieval Augmented Generation Systems

large-language-models › evaluation

📄 Abstract

Abstract: Retrieval-Augmented Generation (RAG) mitigates key limitations of Large Language Models (LLMs)-such as factual errors, outdated knowledge, and hallucinations-by dynamically retrieving external information. Recent work extends this paradigm through agentic RAG systems, where LLMs act as agents to iteratively plan, retrieve, and reason over complex queries. However, these systems still struggle with challenging multi-hop questions, and their intermediate reasoning capabilities remain underexplored. To address this, we propose RAGCap-Bench, a capability-oriented benchmark for fine-grained evaluation of intermediate tasks in agentic RAG workflows. We analyze outputs from state-of-the-art systems to identify common tasks and the core capabilities required for their execution, then construct a taxonomy of typical LLM errors to design targeted evaluation questions. Experiments show that "slow-thinking" models with stronger RAGCap performance achieve better end-to-end results, underscoring the benchmark's validity and the importance of enhancing these intermediate capabilities.

Authors (4)

Jingru Lin

Chen Zhang

Stephen Y. Liu

Haizhou Li

Submitted

October 15, 2025

arXiv Category

cs.CL

arXiv PDF

Key Contributions

Introduces RAGCap-Bench, a capability-oriented benchmark for fine-grained evaluation of intermediate tasks in agentic Retrieval-Augmented Generation (RAG) systems. It analyzes common tasks, identifies required LLM capabilities, and constructs a taxonomy of errors to design targeted evaluation questions, finding that 'slow-thinking' models with stronger RAGCap performance achieve better end-to-end results.

Business Value

Enables developers to build more reliable and accurate RAG systems by identifying and addressing specific capability gaps, leading to better information retrieval and knowledge synthesis applications.

Paper Metadata

Innovation Type

Benchmark and evaluation framework for agentic RAG

Deployment Feasibility

High for evaluation and benchmarking purposes.

Limitations Addressed

Struggles of agentic RAG systems with challenging multi-hop questions,Underexplored intermediate reasoning capabilities in RAG,Lack of detailed benchmarks for RAG system components

Performance Gains

Identifies that models with stronger RAGCap performance achieve better end-to-end results, highlighting the importance of intermediate capabilities.

Technical Tags

Retrieval-Augmented Generation (RAG)agentic RAGLLM capabilitiesbenchmarkingmulti-hop questionsintermediate reasoningRAGCap-BenchLLM errors

Research Topics

Retrieval-Augmented GenerationLLM EvaluationAgentic AIKnowledge Integration

Methods & Architectures

RAGCap-Bench benchmarkCapability-oriented evaluationFine-grained task analysisTaxonomy of LLM errorsAnalysis of state-of-the-art systems Large Language Models (LLMs)Retrieval-Augmented Generation (RAG) systems

Applications & Tasks

Question Answering Information Retrieval Knowledge Management AI Agents Challenges in agentic RAG systems (multi-hop questions, intermediate reasoning)Lack of capability-oriented benchmarks for RAG Evaluating intermediate tasks in agentic RAG workflowsBenchmarking LLM capabilities in RAG systemsIdentifying common LLM errors in RAG

Datasets & Benchmarks

Benchmarks

RAGCap-Bench

Performance on intermediate RAG tasksError analysisCorrelation between RAGCap performance and end-to-end results

Related Fields

Information RetrievalNatural Language ProcessingMachine LearningAI AgentsKnowledge Representation

Keywords

RAGRetrieval-Augmented GenerationLLMagentic AIbenchmarkevaluationcapabilitiesreasoningmulti-hopRAGCap-Benchinformation retrieval

Academic Context

#Retrieval-Augmented Generation#LLM Evaluation#Agentic AI#Knowledge Integration

Commercial Potential

Potential Products

Benchmarking tools for RAG systemsFrameworks for evaluating LLM capabilities in RAG

Target Industries

TechnologySearch EnginesKnowledge ManagementEnterprise AI

Use Case Examples

Evaluating and improving LLMs for complex question answering systemsDeveloping more robust AI agents that can reason over external knowledge

Competitive Edge

Provides a novel, capability-focused benchmark specifically designed for agentic RAG systems, addressing a gap in current evaluation methodologies.

Market Opportunity

Growing market for LLM evaluation tools and RAG solutions.

Revenue Models

Licensing of benchmarkconsulting services.

Resource Requirements

Compute Needs

High (for running RAG systems and evaluation)

Data Requirements

Benchmark dataset (RAGCap-Bench), access to RAG systems.

Deployment Constraints

Complexity of agentic RAG systems, defining and measuring intermediate capabilities accurately.

Scalability

The benchmark design is scalable to new tasks and capabilities.

Regulatory Considerations

None directly.

Production Readiness

Maturity Level

Research/Benchmark

Time to Market

N/A (benchmark focus)

Licensing

Likely open source for benchmark code

Patent Potential

Low (benchmark methodology)

View Full Paper Back to Papers