arxiv_cl 95% Match Research Paper AI Researchers,ML Engineers,NLP Practitioners,Financial Technology Developers 19 hours ago

Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance

large-language-models › training-methods

📄 Abstract

Abstract: While LLMs excel at general tasks, they struggle in specialized domains like finance, requiring diverse skills in domain knowledge, mathematical reasoning, and multilingual processing. Merging domain-specific Continual Pre-training (CPT) "experts" offers a practical alternative to costly and unstable multi-skill training. However, unlike established Supervised Fine-Tuning (SFT) model-based merging, CPT model merging remains largely unexplored. We address this gap by creating financial LLMs from experts in finance, math, and Japanese. We propose a three-stage evaluation focusing on knowledge recovery, complementarity, and emergence, and assess three merging methods (Task Arithmetic, TIES, and DARE-TIES) on a comprehensive financial benchmark curated from 18 tasks across 8 established datasets. Results show that merging an expert with its base model recovers general knowledge lost during CPT, while merging experts improves performance and can yield emergent cross-domain skills. Among the methods, Task Arithmetic performs strongly but is hyperparameter-sensitive, whereas TIES is more robust. Our findings also suggest that while model similarity correlates with merging success, emergent skills depend on more complex factors. This work presents the first foundational analysis of CPT model merging, establishing a principled framework and providing clear guidance for building multi-skill LLMs from existing assets.

Key Contributions

This paper pioneers the exploration of merging Continual Pre-training (CPT) models for creating domain-specialized LLMs, specifically in finance. It proposes a novel three-stage evaluation framework and assesses existing merging techniques (Task Arithmetic, TIES, DARE-TIES), demonstrating that merging CPT experts can recover lost general knowledge and yield emergent cross-domain skills, offering a practical alternative to multi-skill training.

Business Value

Enables the creation of highly capable, domain-specific LLMs for finance more efficiently, reducing training costs and improving performance on specialized tasks.

Paper Metadata

Innovation Type

Methodological Innovation

Deployment Feasibility

Feasible, as it builds upon existing LLM architectures and merging techniques, requiring expertise in CPT and model merging.

Limitations Addressed

Addresses the challenge of adapting general LLMs to specialized domains like finance, which require diverse skills, by proposing and evaluating CPT model merging as a more efficient alternative to multi-skill training.

Performance Gains

Merging experts improves performance on financial tasks and can lead to emergent cross-domain skills; merging with the base model recovers general knowledge.

Technical Tags

LLM MergingContinual Pretraining (CPT)Domain AdaptationFinancial LLMsModel Merging TechniquesTask ArithmeticTIESDARE-TIESKnowledge RecoveryEmergent Skills

Research Topics

Large Language ModelsModel Compression and MergingDomain SpecializationTransfer LearningNatural Language Processing

Methods & Architectures

Continual Pretraining (CPT)Model Merging (Task Arithmetic, TIES, DARE-TIES)Three-stage Evaluation (Knowledge Recovery, Complementarity, Emergence)Fine-tuning Transformer-based LLMsCPT Experts

Applications & Tasks

Finance Natural Language Processing Machine Learning Research Domain SpecializationModel MergingKnowledge TransferMulti-skill Learning Creating domain-specific LLMsMerging CPT modelsEvaluating merging strategies

Datasets & Benchmarks

Datasets

Financial benchmark (18 tasks, 8 datasets)

Benchmarks

Performance on financial tasks • Knowledge recovery metrics • Complementarity metrics • Emergence metrics

Task PerformanceKnowledge Recovery ScoreComplementarity ScoreEmergence Score

Related Fields

Large Language ModelsMachine LearningNatural Language ProcessingTransfer LearningModel Compression

Keywords

LLM MergingContinual PretrainingDomain AdaptationFinanceModel MergingTask ArithmeticTIESDARE-TIESKnowledge RecoveryEmergent SkillsSpecialized LLMsCPTNLP

Academic Context

#Large Language Models#Model Compression and Merging#Domain Specialization#Transfer Learning#Natural Language Processing

Technology Stack

Frameworks & Libraries

Hugging Face Transformers

Programming Languages

Python

ML Infrastructure

Distributed Training Frameworks

Commercial Potential

Potential Products

Domain-specific LLM APIs for financeCustomizable LLM solutions for financial institutions

Target Industries

FinanceFinTechInvestment BankingInsuranceFinancial Consulting

Use Case Examples

Developing LLMs for financial report analysis.Creating chatbots for financial advice.Automating financial document summarization and Q&A.

Competitive Edge

Offers a novel and effective approach to creating specialized LLMs, potentially outperforming general-purpose LLMs or models trained from scratch on limited domain data.

Market Opportunity

Rapidly growing market for AI solutions in finance.

Revenue Models

Licensing of specialized LLMsAPI accesscustom model development services.

Resource Requirements

Compute Needs

High, requires significant GPU resources for CPT and model merging.

Data Requirements

Large datasets for CPT in finance, math, and Japanese; benchmark datasets for evaluation.

Deployment Constraints

Computational cost of training and merging, potential for catastrophic forgetting if not managed properly.

Scalability

Scalable with distributed computing infrastructure.

Regulatory Considerations

Data privacy in financeCompliance with financial regulations

Production Readiness

Maturity Level

Research/Development

Time to Market

18-36 months for robust financial LLM products.

View Full Paper Back to Papers