Browse State-of-the-Art › Speech-to-Speech Translation
Speech-to-Speech Translation
39 papers with code · 3 benchmarks · 5 datasets archive 2025-07-28
Speech-to-speech translation (S2ST) consists on translating speech from one language to speech in another language. This can be done with a cascade of automatic speech recognition (ASR), text-to-text machine translation (MT), and text-to-speech (TTS) synthesis sub-systems, which is text-centric. Recently, works on S2ST without relying on intermediate text representation is emerging.
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
3 leaderboard tables shown for this task, 3 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| TAT (8 rows) | Hokkien→En (Two-pass decoding) | Speech-to-speech translation for a real-world unwritten language | code | — | Compare |
| FLEURS X-eng (7 rows) | GenTranslateV2 | GenTranslate: Large Language Models are Generative Multilingual... | code | — | Compare |
| CVSS (2 rows) | SeamlessM4T Large | SeamlessM4T: Massively Multilingual & Multimodal Machine Translation | code | Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
5 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 117 · with a code link: 39 · where Syntology ran a sample: 12 (10 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (12 of 117 tagged: 10 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 117, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
6 Dec 2022 15 repositories listed Syntology community repositories only · 49 ran (of which 0 constructed an object rather than computing a result; 36 with no instrument failure: 4 honoured, 1 violated, 31 with no contract checked; 13 where Syntology's instrument failed) · 10 unverified (of 59 harvested samples) · 21 pointer-only (licence)We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet.
-
7 Sep 2022 6 repositories listed Syntology 12 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 2 violated, 8 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 15 harvested samples) · 4 pointer-only (licence)We introduce AudioLM, a framework for high-quality audio generation with long-term consistency.
-
22 Aug 2023 4 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)What does it take to create the Babel Fish, a tool that can help individuals translate speech between any two languages?
-
4 Jul 2024 3 repositories listed Syntology official (archive's flag): 9 ran · 10 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 0 unverified (of 10 harvested samples)This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs).
-
24 May 2023 2 repositories listedWe first pretrain a model on large-scale monolingual speech data, finetune it with a small amount of parallel speech data (20-60 hours), and lastly train with an unsupervised backtranslation objective.
-
21 May 2025 1 repository listedWe propose the unit language to overcome the two modeling challenges.
-
22 Apr 2025 1 repository listedThis paper explores the idea of using phonemes as a textual representation within a conventional multilingual simultaneous speech-to-speech translation pipeline, as opposed to the traditional reliance on text-based…
-
5 Feb 2025 1 repository listedTo do so, we introduce a weakly-supervised method that leverages the perplexity of an off-the-shelf text translation system to identify optimal delays on a per-word basis and create aligned synthetic data.
-
17 Sep 2024 1 repository listedSpeech Emotion Recognition (SER) is a crucial component in developing general-purpose AI agents capable of natural human-computer interaction.
-
17 Jul 2024 1 repository listedThis paper introduces EmoCtrl-TTS, an emotion-controllable zero-shot TTS that can generate highly emotional speech with NVs for any speaker.
-
11 Jun 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)Simultaneous translation models play a crucial role in facilitating communication.
-
11 Jun 2024 1 repository listedDirect speech-to-speech translation (S2ST) has achieved impressive translation quality, but it often faces the challenge of slow decoding due to the considerable length of speech sequences.
-
5 Jun 2024 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples)Simultaneous speech-to-speech translation (Simul-S2ST, a. k. a streaming speech translation) outputs target speech while receiving streaming speech inputs, which is critical for real-time communication.
-
28 May 2024 1 repository listedThere is a rising interest and trend in research towards directly translating speech from one language to another, known as end-to-end speech-to-speech translation.
-
22 May 2024 1 repository listedNon-autoregressive Transformers (NATs) are recently applied in direct speech-to-speech translation systems, which convert speech across different languages without intermediate text data.
-
10 Feb 2024 1 repository listedLeveraging the rich linguistic knowledge and strong reasoning abilities of LLMs, our new paradigm can integrate the rich information in N-best candidates to generate a higher-quality translation result.
-
21 Dec 2023 1 repository listedWe introduce EmphAssess, a prosodic benchmark designed to evaluate the capability of speech-to-speech models to encode and reproduce prosodic emphasis.
-
5 Dec 2023 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples) · 6 pointer-only (licence)To mitigate the problem of the absence of a parallel AV2AV translation dataset, we propose to train our spoken language translation system with the audio-only dataset of A2A.
-
11 Oct 2023 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 7 harvested samples) · 7 pointer-only (licence)However, due to the presence of linguistic and acoustic diversity, the target speech follows a complex multimodal distribution, posing challenges to achieving both high-quality translations and fast decoding speeds for…
-
3 Aug 2023 1 repository listedBy setting both the inputs and outputs of our learning problem as speech units, we propose to train an encoder-decoder model in a many-to-many spoken language translation setting, namely Unit-to-Unit Translation (UTUT).
-
9 Jul 2023 1 repository listedSpeech-to-speech translation systems today do not adequately support use for dialog purposes.
-
1 Jun 2023 1 repository listedRecent work in speech-to-speech translation (S2ST) has focused primarily on offline settings, where the full input utterance is available before any output is given.
-
10 Apr 2023 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 1 pointer-only (licence)ESPnet-ST-v2 is a revamp of the open-source ESPnet-ST toolkit necessitated by the broadening interests of the spoken language translation community.
-
7 Mar 2023 1 repository listed Syntology 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples)We propose a cross-lingual neural codec language model, VALL-E X, for cross-lingual speech synthesis.
-
16 Dec 2022 1 repository listedIn this paper, we propose a text-free evaluation metric for end-to-end S2ST, named BLASER, to avoid the dependency on ASR systems.
-
15 Dec 2022 1 repository listedWe enhance the model performance by subword prediction in the first-pass decoder, advanced two-pass decoder architecture design and search strategy, and better training regularization.
-
18 Nov 2022 1 repository listedTo support machine learning of cross-language prosodic mappings and other ways to improve speech-to-speech translation, we present a protocol for collecting closely matched pairs of utterances across languages, a…
-
31 Oct 2022 1 repository listedHowever, direct S2ST suffers from the data scarcity problem because the corpora from speech of the source language to speech of the target language are very rare.
-
21 Oct 2022 1 repository listedIn this paper, we introduce a new and simple method for comparing speech utterances without relying on text transcripts.
-
25 May 2022 1 repository listedSpecifically, a sequence of discrete representations derived in a self-supervised manner are predicted from the model and passed to a vocoder for speech reconstruction, while still facing the following challenges: 1)…
-
18 May 2022 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 1 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 5 pointer-only (licence)
-
1 May 2022 1 repository listed
-
22 Apr 2022 1 repository listed
-
11 Jan 2022 1 repository listed
-
1 Dec 2021 1 repository listed
-
12 Jul 2021 1 repository listed
-
21 Apr 2020 1 repository listed
-
1 Mar 2020 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
12 Apr 2019 1 repository listed
-
Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMs12 Jun 2025 0 repositories listed
-
S2ST-Omni: An Efficient and Scalable Multilingual Speech-to-Speech Translation Framework via Seamless Speech-Text Alignment and Streaming Speech Generation11 Jun 2025 0 repositories listed
-
Phi-Omni-ST: A multimodal language model for direct speech-to-speech translation4 Jun 2025 0 repositories listed
-
Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbing27 May 2025 0 repositories listed
-
Language translation, and change of accent for speech-to-speech task using diffusion model4 May 2025 0 repositories listed
-
SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation22 Apr 2025 0 repositories listed
-
Direct Speech to Speech Translation: A Review3 Mar 2025 0 repositories listed
-
Connecting Voices: LoReSpeech as a Low-Resource Speech Parallel Corpus25 Feb 2025 0 repositories listed
-
Balancing Speech Understanding and Generation Using Continual Pre-training for Codec-based Speech LLM24 Feb 2025 0 repositories listed
-
Speech to Speech Translation with Translatotron: A State of the Art Review9 Feb 2025 0 repositories listed
-
A Unit-based System and Dataset for Expressive Direct Speech-to-Speech Translation1 Feb 2025 0 repositories listed
-
Improving Lip-synchrony in Direct Audio-Visual Speech-to-Speech Translation21 Dec 2024 0 repositories listed
-
Direct Speech-to-Speech Neural Machine Translation: A Survey13 Nov 2024 0 repositories listed
-
Findings of the IWSLT 2024 Evaluation Campaign7 Nov 2024 0 repositories listed
-
Phonology-Guided Speech-to-Speech Translation for African Languages30 Oct 2024 0 repositories listed
-
Textless Streaming Speech-to-Speech Translation using Semantic Speech Tokens4 Oct 2024 0 repositories listed
-
What does it take to get state of the art in simultaneous speech-to-speech translation?2 Sep 2024 0 repositories listed
-
PolySinger: Singing-Voice to Singing-Voice Translation from English to Japanese19 Jul 2024 0 repositories listed
-
Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems18 Jul 2024 0 repositories listed
-
Analyzing Speech Unit Selection for Textless Speech-to-Speech Translation8 Jul 2024 0 repositories listed
-
NAIST Simultaneous Speech Translation System for IWSLT 202430 Jun 2024 0 repositories listed
-
Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translation14 Jun 2024 0 repositories listed
-
Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?11 Jun 2024 0 repositories listed
-
Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing4 Jun 2024 0 repositories listed
-
SimulTron: On-Device Simultaneous Speech to Speech Translation4 Jun 2024 0 repositories listed
-
Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation4 Jun 2024 0 repositories listed
-
SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thought30 May 2024 0 repositories listed
-
CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning23 May 2024 0 repositories listed
-
MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation19 Mar 2024 0 repositories listed
-
Direct Punjabi to English speech translation using discrete units25 Feb 2024 0 repositories listed
-
A Case Study on Filtering for End-to-End Speech Translation2 Feb 2024 0 repositories listed
-
TranSentence: Speech-to-speech Translation via Language-agnostic Sentence-level Speech Encoding without Language-parallel Data17 Jan 2024 0 repositories listed
-
TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation23 Dec 2023 0 repositories listed
-
DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation26 Oct 2023 0 repositories listed
-
Enhancing expressivity transfer in textless speech-to-speech translation11 Oct 2023 0 repositories listed
-
Direct Text to Speech Translation System using Acoustic Units14 Sep 2023 0 repositories listed
-
Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer14 Sep 2023 0 repositories listed
-
Multilingual Speech-to-Speech Translation into Multiple Target Languages17 Jul 2023 0 repositories listed
-
AudioPaLM: A Large Language Model That Can Speak and Listen22 Jun 2023 0 repositories listed
-
PolyVoice: Language Models for Speech to Speech Translation5 Jun 2023 0 repositories listed
-
Translatotron 3: Speech to Speech Translation with Monolingual Data27 May 2023 0 repositories listed
-
AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation24 May 2023 0 repositories listed
-
i-Code Studio: A Configurable and Composable Framework for Integrative AI23 May 2023 0 repositories listed
-
Duplex Diffusion Models Improve Speech-to-Speech Translation22 May 2023 0 repositories listed
-
Enhancing Speech-to-Speech Translation with Multiple TTS Targets10 Apr 2023 0 repositories listed
-
A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation25 Jan 2023 0 repositories listed
-
Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Features12 Dec 2022 0 repositories listed
-
Speech-to-Speech Translation For A Real-world Unwritten Language11 Nov 2022 0 repositories listed
-
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations8 Nov 2022 0 repositories listed
-
Textless Direct Speech-to-Speech Translation with Discrete Speech Representation31 Oct 2022 0 repositories listed
-
Improving Speech-to-Speech Translation Through Unlabeled Text26 Oct 2022 0 repositories listed
-
Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling30 Sep 2022 0 repositories listed
-
SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation17 May 2022 0 repositories listed
-
Findings of the IWSLT 2022 Evaluation Campaign1 May 2022 0 repositories listed
-
MLLP-VRAIN UPV systems for the IWSLT 2022 Simultaneous Speech Translation and Speech-to-Speech Translation tasks1 May 2022 0 repositories listed
-
The HW-TSC’s Speech to Speech Translation System for IWSLT 2022 Evaluation1 May 2022 0 repositories listed
-
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation6 Apr 2022 0 repositories listed
-
Prosodic Alignment for off-screen automatic dubbing6 Apr 2022 0 repositories listed
-
Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation24 Mar 2022 0 repositories listed
-
Evaluating MT Systems: A Theoretical Framework11 Feb 2022 0 repositories listed
-
Textless Speech-to-Speech Translation on Real Data15 Dec 2021 0 repositories listed
Syntology lines on 12 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections