Redirecting to original paper in 30 seconds...
Click below to go immediately or wait for automatic redirect
📄 Abstract
Abstract: While Transformer self-attention offers strong parallelism, the Key-Value
(KV) cache grows linearly with sequence length and becomes a bottleneck for
inference efficiency. Multi-head latent attention was recently developed to
compress the KV cache into a low-rank latent space. This paper proposes
Multi-head Temporal Latent Attention (MTLA), which further reduces the KV cache
size along the temporal dimension, greatly lowering the memory footprint of
self-attention inference. MTLA employs a hyper-network to dynamically merge
temporally adjacent KV cache vectors. To address the mismatch between the
compressed KV cache and processed sequence lengths, a stride-aware causal mask
is proposed to ensure efficient parallel training and consistency with
inference behaviour. Experiments across tasks, including speech translation,
speech recognition, speech understanding and text summarisation, demonstrate
that MTLA achieves competitive performance compared to standard Multi-Head
Attention (MHA), while greatly improving inference speed and GPU memory usage.
For example, on a English-German speech translation task, MTLA achieves a 5.3x
speedup and a reduction in GPU memory usage by a factor of 8.3 compared to MHA,
while maintaining translation quality.
Authors (2)
Keqi Deng
Philip C. Woodland
Key Contributions
This paper introduces Multi-head Temporal Latent Attention (MTLA) to significantly reduce the KV cache size in self-attention mechanisms by compressing it along the temporal dimension. This innovation addresses the memory bottleneck in efficient inference, enabling faster and more memory-efficient processing of long sequences, particularly in speech-related tasks.
Business Value
Reduces computational costs and latency for AI models processing sequential data, making applications like real-time speech translation and transcription more feasible and scalable in production environments.