arxiv_ai 92% Match Research Paper Speech Researchers,NLP Engineers,Machine Translation Experts,TTS Developers 3 weeks ago

StressTransfer: Stress-Aware Speech-to-Speech Translation with Emphasis Preservation

speech-audio › text-to-speech

📄 Abstract

Abstract: We propose a stress-aware speech-to-speech translation (S2ST) system that preserves word-level emphasis by leveraging LLMs for cross-lingual emphasis conversion. Our method translates source-language stress into target-language tags that guide a controllable TTS model. To overcome data scarcity, we developed a pipeline to automatically generate aligned training data and introduce the "LLM-as-Judge" for evaluation. Experiments show our approach substantially outperforms baselines in preserving emphasis while maintaining comparable translation quality, speaker intent, and naturalness. Our work highlights the importance of prosody in translation and provides an effective, data-efficient solution for preserving paralinguistic cues in S2ST.

Authors (3)

Xi Chen

Yuchen Song

Satoshi Nakamura

Submitted

October 15, 2025

arXiv Category

cs.CL

arXiv PDF

Key Contributions

This paper proposes a novel stress-aware speech-to-speech translation (S2ST) system that preserves word-level emphasis by leveraging LLMs for cross-lingual emphasis conversion. It introduces a data generation pipeline and 'LLM-as-Judge' evaluation to overcome data scarcity, achieving superior emphasis preservation.

Business Value

Enhances the expressiveness and emotional nuance of translated speech, improving cross-cultural communication and user experience in applications like virtual assistants and international calls.

Paper Metadata

Innovation Type

Algorithmic Improvement

Deployment Feasibility

Moderate. Requires integration of LLMs and TTS models, potentially with high computational costs for real-time processing.

Limitations Addressed

The loss of word-level emphasis and prosody in traditional speech translation systems, and the challenge of data scarcity for training such systems.

Performance Gains

Substantially outperforms baselines in preserving emphasis while maintaining comparable translation quality, speaker intent, and naturalness.

Technical Tags

speech-to-speech translationstress-aware translationemphasis preservationcross-lingual emphasis conversionLLMscontrollable TTSdata augmentationLLM-as-Judgeprosody

Research Topics

Speech TranslationProsody ModelingNatural Language ProcessingSpeech SynthesisMachine Learning for Speech

Methods & Architectures

Stress-Aware Speech-to-Speech TranslationCross-lingual Emphasis ConversionLLM-based Tag GenerationControllable Text-to-Speech (TTS)Automated Data Generation PipelineLLM-as-Judge Evaluation Large Language Models (LLMs)Controllable Text-to-Speech (TTS) models

Applications & Tasks

Communication Accessibility Entertainment Global Business Speech TranslationProsody PreservationData Scarcity Speech-to-Speech Translation with Emphasis PreservationCross-lingual Prosody Transfer

Related Fields

Speech ProcessingNatural Language ProcessingMachine TranslationText-to-Speech SynthesisComputational Linguistics

Keywords

speech translationS2STemphasisstressprosodyLLMTTScross-lingualdata augmentationLLM-as-Judgespeech synthesisnaturalnessspeaker intent

Academic Context

#Speech Translation#Prosody Modeling#Natural Language Processing#Speech Synthesis#Machine Learning for Speech

Commercial Potential

Potential Products

Real-time Speech Translation Services with Emotional NuanceDubbing Tools for MediaVoice Assistants with Enhanced Expressiveness

Target Industries

TelecommunicationsMedia and EntertainmentCustomer ServiceGlobal Business

Use Case Examples

Translating a spoken presentation while preserving the speaker's original emphasis and tone.Enabling more natural and engaging conversations between people speaking different languages.

Competitive Edge

Addresses the critical issue of prosody preservation in S2ST, offering a data-efficient solution that outperforms existing methods in emphasis transfer.

Market Opportunity

Growing market for real-time translation and AI-powered communication tools.

Revenue Models

API access for translation serviceslicensing of the S2ST technology.

Resource Requirements

Compute Needs

High, especially for real-time LLM inference and controllable TTS.

Data Requirements

Requires parallel speech data and potentially annotated emphasis information, though the paper proposes methods to generate this.

Deployment Constraints

Latency for real-time translation.,Computational resources for LLM and TTS models.,Quality of emphasis tags generated by LLM.

Scalability

Scalability depends on the efficiency of the LLM and TTS components and the data generation pipeline.

Production Readiness

Maturity Level

Research

Time to Market

2-4 years for robust commercial deployment.

Patent Potential

Moderate, for the novel emphasis conversion mechanism and data generation pipeline.

View Full Paper Back to Papers