Can LLMs subtract numbers?

📄 Abstract

Abstract: We present a systematic study of subtraction in large language models (LLMs). While prior benchmarks emphasize addition and multiplication, subtraction has received comparatively little attention despite being structurally distinct as a non-commutative operation. We evaluate eight pretrained LLMs spanning four families on addition and subtraction problems. Our experiments reveal that subtraction accuracy lags behind addition by a wide margin. We find that the errors for ($a-b$) are concentrated in cases where ($a

Key Contributions

This paper provides a systematic study of subtraction capabilities in LLMs, revealing a significant performance gap compared to addition. It identifies common error patterns, such as omitting negative signs, and investigates the effectiveness of few-shot learning and instruction tuning in improving subtraction accuracy.

Business Value

Highlights critical limitations in current LLMs for tasks requiring precise arithmetic, guiding development towards more robust and reliable models for applications involving calculations.

Paper Metadata

Innovation Type

Empirical Study and Analysis

Deployment Feasibility

N/A (research paper focused on evaluation)

Limitations Addressed

Addresses the under-explored area of subtraction in LLMs and identifies specific failure modes and potential mitigation strategies.

Performance Gains

Modest gains from few-shot prompting; instruction tuning's effectiveness is suggested but not fully detailed.

Technical Tags

subtractionlarge language modelsarithmeticnon-commutative operationsfew-shot learninginstruction tuningprobing analysisLLM evaluation

Research Topics

LLM Arithmetic CapabilitiesReasoning in LLMsModel EvaluationLLM LimitationsFew-Shot Learning

Methods & Architectures

Systematic evaluationcomparison of addition vs. subtractionprobing analysesfew-shot promptinginstruction-tuning

Applications & Tasks

Natural Language Processing Artificial Intelligence Research Arithmetic AccuracyReasoning DeficienciesLLM Evaluation Evaluating subtraction capabilities of LLMsIdentifying sources of arithmetic errorsImproving LLM arithmetic performance

Related Fields

Natural Language ProcessingMachine LearningArtificial IntelligenceCognitive Science

Keywords

large language modelsLLMsubtractionarithmeticreasoningevaluationfew-shot learninginstruction tuningnon-commutativeaccuracymodel limitationsprobing

Academic Context

#LLM Arithmetic Capabilities#Reasoning in LLMs#Model Evaluation#LLM Limitations#Few-Shot Learning

Commercial Potential

Target Industries

TechnologySoftware Development

Use Case Examples

Assessing LLM suitability for financial calculationsDeveloping educational tools that rely on accurate arithmetic

Competitive Edge

Provides a critical benchmark and analysis of LLM performance on a specific, often overlooked, arithmetic operation.

Market Opportunity

Growing market for LLM evaluation tools and benchmarks.

Revenue Models

N/A

Resource Requirements

Compute Needs

Moderate compute for running evaluations across multiple LLMs and datasets.

Data Requirements

Requires carefully constructed datasets of subtraction problems.

Scalability

The evaluation methodology is scalable to different LLMs and problem sets.

Production Readiness

Maturity Level

Research

Time to Market

N/A

View Full Paper Back to Papers