{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/seamless-multilingual-expressive-and","title":"Seamless: Multilingual Expressive and Streaming Speech Translation","arxiv_id":"2312.05187","date":"2023-12-08","proceeding":null,"authors":["Seamless Communication","Loïc Barrault","Yu-An Chung","Mariano Coria Meglioli","David Dale","Ning Dong","Mark Duppenthaler","Paul-Ambroise Duquenne","Brian Ellis","Hady Elsahar","Justin Haaheim","John Hoffman","Min-Jae Hwang","Hirofumi Inaguma","Christopher Klaiber","Ilia Kulikov","Pengwei Li","Daniel Licht","Jean Maillard","Ruslan Mavlyutov","Alice Rakotoarison","Kaushik Ram Sadagopan","Abinesh Ramakrishnan","Tuan Tran","Guillaume Wenzek","Yilin Yang","Ethan Ye","Ivan Evtimov","Pierre Fernandez","Cynthia Gao","Prangthip Hansanti","Elahe Kalbassi","Amanda Kallet","Artyom Kozhevnikov","Gabriel Mejia Gonzalez","Robin San Roman","Christophe Touret","Corinne Wong","Carleigh Wood","Bokai Yu","Pierre Andrews","Can Balioglu","Peng-Jen Chen","Marta R. Costa-jussà","Maha Elbayad","Hongyu Gong","Francisco Guzmán","Kevin Heffernan","Somya Jain","Justine Kao","Ann Lee","Xutai Ma","Alex Mourachko","Benjamin Peloquin","Juan Pino","Sravya Popuri","Christophe Ropers","Safiyyah Saleem","Holger Schwenk","Anna Sun","Paden Tomasello","Changhan Wang","Jeff Wang","Skyler Wang","Mary Williamson"],"abstract":"Large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model-SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. SeamlessM4T v2 provides the foundation on which our next two models are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one's voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. The contributions to this work are publicly released and accessible at https://github.com/facebookresearch/seamless_communication","url_abs":"https://arxiv.org/abs/2312.05187v1","url_pdf":"https://arxiv.org/pdf/2312.05187v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"seamless-multilingual-expressive-and","repo_url":"https://github.com/facebookresearch/seamless_communication","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"multimodal-machine-translation","task_name":"Multimodal Machine Translation"},{"task_slug":"red-teaming","task_name":"Red Teaming"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"automatic-speech-translation","task_name":"automatic-speech-translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.05187","atlas_url":"https://app.syntology.ai/?focus=2312.05187","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2312.05187"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2312.05187\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2312.05187"},"call":"get_concepts_for_paper(arxiv_id=\"2312.05187\")"}],"by_repo":[{"repo":"facebookresearch/seamless_communication","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2308.11596"],"harvested_for_calls":[{"arxiv_id":"2308.11596","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2308.11596"},"call":"get_harvested_code_for_paper(arxiv_id=\"2308.11596\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"fcf5efbeca184fa018048dd5683061d31628d62eda64c80606969e31c9b2328d","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}