Redirecting to original paper in 30 seconds...
Click below to go immediately or wait for automatic redirect
📄 Abstract
Abstract: Quantization is the key method for reducing inference latency, power and
memory footprint of generative AI models. However, accuracy often degrades
sharply when activations are quantized below eight bits. Recent work suggests
that invertible linear transformations (e.g. rotations) can aid quantization,
by reparameterizing feature channels and weights. In this paper, we propose
\textit{Sequence Transformation and Mixed Precision} (STaMP) quantization, a
novel strategy that applies linear transformations along the \textit{sequence}
dimension to exploit the strong local correlation in language and visual data.
By keeping a small number of tokens in each intermediate activation at higher
precision, we can maintain model accuracy at lower (average) activations
bit-widths. We evaluate STaMP on recent LVM and LLM architectures,
demonstrating that it significantly improves low bit width activation
quantization and complements established activation and weight quantization
methods including recent feature transformations.
Authors (5)
Marco Federici
Riccardo Del Chiaro
Boris van Breugel
Paul Whatmough
Markus Nagel
Submitted
October 30, 2025
Key Contributions
Proposes STaMP quantization, a novel strategy that applies linear transformations along the sequence dimension for low-precision activation quantization. It maintains accuracy at lower bit-widths by keeping a small number of tokens at higher precision, significantly improving efficiency for LLMs and LVMs.
Business Value
Enables the deployment of large generative AI models on resource-constrained devices and reduces operational costs for cloud-based inference, making advanced AI more accessible and efficient.