arxiv_cl 95% Match Research Paper ML Engineers,AI Researchers,NLP Practitioners,Deep Learning Engineers 3 weeks ago

OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning

large-language-models › training-methods

📄 Abstract

Abstract: Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large language models but suffers from catastrophic forgetting when learned updates interfere with the dominant singular directions that encode essential pre-trained knowledge. We propose Orthogonal Projection LoRA (OPLoRA), a theoretically grounded approach that prevents this interference through double-sided orthogonal projections. By decomposing frozen weights via SVD, OPLoRA constrains LoRA updates to lie entirely within the orthogonal complement of the top-$k$ singular subspace using projections $P_L = I - U_k U_k^\top$ and $P_R = I - V_k V_k^\top$. We prove that this construction exactly preserves the top-$k$ singular triples, providing mathematical guarantees for knowledge retention. To quantify subspace interference, we introduce $\rho_k$, a metric measuring update alignment with dominant directions. Extensive experiments across commonsense reasoning, mathematics, and code generation demonstrate that OPLoRA significantly reduces forgetting while maintaining competitive task-specific performance on LLaMA-2 7B and Qwen2.5 7B, establishing orthogonal projection as an effective mechanism for knowledge preservation in parameter-efficient fine-tuning.

Authors (2)

Yifeng Xiong

Xiaohui Xie

Submitted

October 14, 2025

arXiv Category

cs.CL

arXiv PDF

Key Contributions

Proposes Orthogonal Projection LoRA (OPLoRA), a theoretically grounded method to prevent catastrophic forgetting during parameter-efficient fine-tuning. OPLoRA uses double-sided orthogonal projections based on SVD to constrain LoRA updates, mathematically guaranteeing preservation of top-k singular triples and thus essential pre-trained knowledge. Experiments show OPLoRA significantly improves performance across reasoning, math, and code generation tasks.

Business Value

Enables more stable and reliable fine-tuning of LLMs for specific tasks, reducing the risk of performance degradation and preserving valuable pre-trained capabilities, leading to more robust and efficient model adaptation.

Paper Metadata

Innovation Type

Algorithmic Improvement

Deployment Feasibility

High, as it's a modification to the LoRA fine-tuning process.

Limitations Addressed

Catastrophic forgetting in LoRA,Interference of LoRA updates with dominant singular directions,Loss of essential pre-trained knowledge,Lack of theoretical guarantees for knowledge preservation

Performance Gains

Significantly improves performance across commonsense reasoning, mathematics, and code generation tasks compared to standard LoRA.

Technical Tags

Parameter-Efficient Fine-Tuning (PEFT)Low-Rank Adaptation (LoRA)Catastrophic ForgettingOrthogonal Projection LoRA (OPLoRA)Singular Value Decomposition (SVD)Knowledge RetentionCommonsense ReasoningMathematicsCode GenerationSubspace Interference

Research Topics

Preventing catastrophic forgetting in PEFTImproving knowledge retention during LLM fine-tuningDeveloping theoretically grounded PEFT methodsQuantifying interference in parameter updates

Methods & Architectures

Orthogonal Projection LoRA (OPLoRA)Singular Value Decomposition (SVD)Double-sided Orthogonal ProjectionsDecomposition of Frozen WeightsMetric for Subspace Interference ($\rho_k$) Large Language Models (LLMs)LoRA Adapters

Applications & Tasks

Natural Language Processing Machine Learning AI Model Training Catastrophic forgetting in LoRAInterference between learned updates and pre-trained knowledgeDegradation of performance on downstream tasksLack of theoretical guarantees for knowledge retention Commonsense ReasoningMathematicsCode GenerationFine-tuning LLMs

Related Fields

Natural Language ProcessingMachine LearningDeep LearningOptimizationLinear Algebra

Keywords

OPLoRALoRAParameter-Efficient Fine-TuningPEFTCatastrophic ForgettingKnowledge RetentionSVDOrthogonal ProjectionLLM Fine-tuningCommonsense ReasoningMathematicsCode Generation

Academic Context

#Preventing catastrophic forgetting in PEFT#Improving knowledge retention during LLM fine-tuning#Developing theoretically grounded PEFT methods#Quantifying interference in parameter updates

Commercial Potential

Potential Products

Improved LLM fine-tuning librariesSpecialized PEFT modules

Target Industries

TechnologyAI DevelopmentSoftware Engineering

Use Case Examples

Fine-tuning a large language model for medical diagnosis without losing its general language understanding capabilities.Adapting an LLM for legal document analysis while maintaining its proficiency in creative writing.

Competitive Edge

Offers a theoretically grounded solution to catastrophic forgetting in LoRA, providing mathematical guarantees for knowledge retention, which is a significant improvement over empirical solutions.

Market Opportunity

Large and growing market for efficient LLM fine-tuning solutions.

Revenue Models

Licensing of the OPLoRA algorithmintegration into ML platforms.

Resource Requirements

Compute Needs

Moderate, as it's a fine-tuning technique, but potentially less compute-intensive than full fine-tuning.

Data Requirements

Requires datasets for the target tasks (commonsense reasoning, math, code generation).

Deployment Constraints

Requires integration into existing PEFT frameworks.

Scalability

Designed to be scalable with LLM size and complexity.

Production Readiness

Maturity Level

Research

Time to Market

6-12 months

Patent Potential

Moderate, for the OPLoRA algorithm and its theoretical underpinnings.

View Full Paper Back to Papers