arxiv_ai 95% Match Research Paper ML Engineers,AI Researchers,Developers deploying LLMs,Mobile AI Developers 1 week ago

zFLoRA: Zero-Latency Fused Low-Rank Adapters

large-language-models › model-architecture

📄 Abstract

Abstract: Large language models (LLMs) are increasingly deployed with task-specific adapters catering to multiple downstream applications. In such a scenario, the additional compute associated with these apparently insignificant number of adapter parameters (typically less than 1% of the base model) turns out to be disproportionately significant during inference time (upto 2.5x times that of the base model). In this paper, we propose a new zero-latency fused low-rank adapter (zFLoRA) that introduces zero or negligible latency overhead on top of the base model. Experimental results on LLMs of size 1B, 3B and 7B show that zFLoRA compares favorably against the popular supervised fine-tuning benchmarks including low-rank adapters (LoRA) as well as full fine-tuning (FFT). Experiments are conducted on 18 different tasks across three different categories namely commonsense reasoning, math reasoning and summary-dialogue. Latency measurements made on NPU (Samsung Galaxy S25+) as well as GPU (NVIDIA H100) platforms show that the proposed zFLoRA adapters introduce zero to negligible latency overhead.

Authors (4)

Dhananjaya Gowda

Seoha Song

Harshith Goka

Junhyun Lee

Submitted

October 28, 2025

arXiv Category

cs.CL

arXiv PDF

Key Contributions

This paper introduces zFLoRA, a novel zero-latency fused low-rank adapter technique that significantly reduces or eliminates the inference latency overhead typically associated with task-specific adapters in LLMs. Experiments on various LLM sizes and tasks demonstrate that zFLoRA achieves comparable performance to existing methods while offering substantial latency improvements.

Business Value

Enables faster and more cost-effective deployment of LLMs for real-time applications, especially on edge devices, improving user experience and reducing operational costs.

Paper Metadata

Innovation Type

Novel adapter fusion technique

Deployment Feasibility

High, as it focuses on optimizing inference for existing LLMs and is designed for practical deployment on devices like smartphones.

Limitations Addressed

The significant inference latency overhead introduced by traditional low-rank adapters (LoRA) and other fine-tuning methods when applied to LLMs for multiple downstream tasks.

Performance Gains

Zero or negligible latency overhead compared to base model, while maintaining performance comparable to LoRA and FFT.

Technical Tags

Low-Rank Adapters (LoRA)Zero-Latency AdaptersFused AdaptersLLM InferenceParameter Efficient Fine-Tuning (PEFT)Latency OverheadDownstream ApplicationsCommonsense ReasoningMath ReasoningSummarizationDialogue SystemsNPUGPU

Research Topics

Efficient LLM AdaptationReducing Inference LatencyParameter-Efficient Fine-TuningLLM Deployment OptimizationTask-Specific LLM Adaptation

Methods & Architectures

Adapter FusionLow-Rank DecompositionLatency MeasurementBenchmarking on Downstream Tasks Large Language Models (LLMs)Low-Rank Adapters (LoRA)

Applications & Tasks

Natural Language Processing Edge AI Mobile Computing LLM Deployment High inference latency introduced by task-specific adaptersBalancing model performance with computational costEnabling efficient deployment of LLMs for multiple tasks Commonsense reasoningMath reasoningSummarizationDialogue generationGeneral LLM inference

Datasets & Benchmarks

Benchmarks

Commonsense reasoning benchmarks • Math reasoning benchmarks • Summarization benchmarks • Dialogue system benchmarks

LatencyAccuracyPerformance on downstream tasks

Related Fields

Machine LearningNatural Language ProcessingModel CompressionEfficient AIEdge Computing

Keywords

LLMLoRAAdapterInference LatencyPEFTZero-LatencyFused AdaptersEdge AIModel OptimizationDownstream TasksNPUGPU

Academic Context

#Efficient LLM Adaptation#Reducing Inference Latency#Parameter-Efficient Fine-Tuning#LLM Deployment Optimization#Task-Specific LLM Adaptation

Companies & Organizations

Companies Mentioned

Samsung

Technology Stack

Frameworks & Libraries

PyTorchTensorFlow

Programming Languages

Python

ML Infrastructure

NPUGPU

Commercial Potential

Potential Products

Optimized LLM inference engines for mobile devicesSDKs for deploying efficient LLMs

Target Industries

Mobile TechnologyConsumer ElectronicsSoftware DevelopmentAI Services

Use Case Examples

Real-time AI assistants on smartphonesOn-device language translationEfficient LLM-powered applications

Competitive Edge

Offers a significant improvement in inference latency over existing adapter methods like LoRA, without sacrificing performance.

Market Opportunity

Massive market for mobile AI applications,Increasing demand for on-device LLM capabilities

Resource Requirements

Compute Needs

Optimized for inference on devices with NPUs and GPUs.

Data Requirements

Standard datasets for commonsense reasoning, math, summarization, and dialogue.

Deployment Constraints

Requires hardware support for efficient adapter fusion (e.g., specific NPU/GPU capabilities).

Scalability

Designed for efficient scaling across different LLM sizes (1B, 3B, 7B) and multiple tasks.

Production Readiness

Maturity Level

Research

Time to Market

Short to medium, depending on hardware integration and software support.

View Full Paper Back to Papers