arxiv_cv 95% Match Research Paper AI researchers,Medical professionals,AI ethicists,System designers 3 days ago

Learning to Seek Evidence: A Verifiable Reasoning Agent with Causal Faithfulness Analysis

ai-safety › interpretability

📄 Abstract

Abstract: Explanations for AI models in high-stakes domains like medicine often lack verifiability, which can hinder trust. To address this, we propose an interactive agent that produces explanations through an auditable sequence of actions. The agent learns a policy to strategically seek external visual evidence to support its diagnostic reasoning. This policy is optimized using reinforcement learning, resulting in a model that is both efficient and generalizable. Our experiments show that this action-based reasoning process significantly improves calibrated accuracy, reducing the Brier score by 18\% compared to a non-interactive baseline. To validate the faithfulness of the agent's explanations, we introduce a causal intervention method. By masking the visual evidence the agent chooses to use, we observe a measurable degradation in its performance ($\Delta$Brier=+0.029), confirming that the evidence is integral to its decision-making process. Our work provides a practical framework for building AI systems with verifiable and faithful reasoning capabilities.

Authors (4)

Yuhang Huang

Zekai Lin

Fan Zhong

Lei Liu

Submitted

November 3, 2025

arXiv Category

cs.AI

arXiv PDF

Key Contributions

This paper introduces an interactive agent that generates verifiable explanations for AI models in high-stakes domains by strategically seeking external visual evidence. This approach, optimized via reinforcement learning, significantly improves calibrated accuracy and provides a causal intervention method to validate the faithfulness of the agent's reasoning, addressing the critical need for trust and auditability in AI decision-making.

Business Value

Enhances trust and reliability in AI systems used in critical applications like healthcare, leading to better decision-making and potentially reducing errors and liability.

Paper Metadata

Innovation Type

Novel Framework/Methodology

Deployment Feasibility

Moderate. Requires integration with external data sources and a user interface for interaction, but the core RL policy is deployable.

Limitations Addressed

Lack of verifiability in AI explanations,Hinderance of trust in AI systems,Inefficient diagnostic reasoning

Performance Gains

18% reduction in Brier score compared to a non-interactive baseline; +0.029 Brier score degradation upon evidence masking, confirming faithfulness.

Technical Tags

reinforcement learningcausal inferenceinteractive agentvisual evidence seekingauditable explanationscalibrated accuracybrier scorepolicy optimizationgeneralizabilitycausal faithfulness

Research Topics

Explainable AI (XAI)Trustworthy AIMedical DiagnosisInteractive Machine LearningCausal Reasoning

Methods & Architectures

Reinforcement LearningCausal InterventionPolicy OptimizationInteractive Agent Design Policy Network

Applications & Tasks

Medicine High-stakes decision making Lack of verifiability in AI explanationsHinderance of trust in AI systemsImproving diagnostic accuracyValidating explanation faithfulness Diagnostic reasoningEvidence seekingExplanation generation

Related Fields

Explainable AICausal InferenceMachine LearningMedical InformaticsHuman-Computer Interaction

Keywords

Explainable AIVerifiable ReasoningCausal FaithfulnessInteractive AgentMedical DiagnosisReinforcement LearningEvidence SeekingTrustworthy AIAuditable ExplanationsDecision Support Systems

Academic Context

#Explainable AI (XAI)#Trustworthy AI#Medical Diagnosis#Interactive Machine Learning#Causal Reasoning

Commercial Potential

Potential Products

AI diagnostic assistantsTrustworthy AI platformsInteractive decision support tools

Target Industries

HealthcareFinanceLegal

Use Case Examples

AI-assisted medical diagnosis with verifiable reasoningFinancial risk assessment with auditable explanations

Competitive Edge

Offers a novel approach to verifiable explanations by integrating interactive evidence seeking and causal validation, differentiating from static explanation methods.

Market Opportunity

Growing market for AI in healthcare, increasing demand for trustworthy and explainable AI solutions.

Revenue Models

SaaS for AI diagnostic platformslicensing of the technology to medical device manufacturers.

Resource Requirements

Compute Needs

Moderate to high, depending on the complexity of the RL training and the scale of the visual evidence.

Data Requirements

Requires labeled medical imaging data with associated diagnostic outcomes and potentially access to external visual evidence sources.

Deployment Constraints

Integration with existing clinical workflows, data privacy regulations (HIPAA), and user acceptance of interactive AI.

Scalability

Scalability depends on the efficiency of the RL policy and the ability to access and process external evidence in real-time.

Regulatory Considerations

HIPAA compliance for medical dataFDA regulations for medical devices.

Production Readiness

Maturity Level

Research/Prototype

Time to Market

2-5 years for robust clinical deployment.

Patent Potential

Moderate, for the novel interactive agent design and causal faithfulness validation method.

View Full Paper Back to Papers