{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/revise-self-supervised-speech-resynthesis","title":"ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement","arxiv_id":"2212.11377","date":"2022-12-21","proceeding":null,"authors":["Wei-Ning Hsu","Tal Remez","Bowen Shi","Jacob Donley","Yossi Adi"],"abstract":"Prior works on improving speech quality with visual input typically study each type of auditory distortion separately (e.g., separation, inpainting, video-to-speech) and present tailored algorithms. This paper proposes to unify these subjects and study Generalized Speech Enhancement, where the goal is not to reconstruct the exact reference clean signal, but to focus on improving certain aspects of speech. In particular, this paper concerns intelligibility, quality, and video synchronization. We cast the problem as audio-visual speech resynthesis, which is composed of two steps: pseudo audio-visual speech recognition (P-AVSR) and pseudo text-to-speech synthesis (P-TTS). P-AVSR and P-TTS are connected by discrete units derived from a self-supervised speech model. Moreover, we utilize self-supervised audio-visual speech model to initialize P-AVSR. The proposed model is coined ReVISE. ReVISE is the first high-quality model for in-the-wild video-to-speech synthesis and achieves superior performance on all LRS3 audio-visual enhancement tasks with a single model. To demonstrates its applicability in the real world, ReVISE is also evaluated on EasyCom, an audio-visual benchmark collected under challenging acoustic conditions with only 1.6 hours of training data. Similarly, ReVISE greatly suppresses noise and improves quality. Project page: https://wnhsu.github.io/ReVISE.","url_abs":"https://arxiv.org/abs/2212.11377v1","url_pdf":"https://arxiv.org/pdf/2212.11377v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"audio-visual-speech-recognition","task_name":"Audio-Visual Speech Recognition"},{"task_slug":"resynthesis","task_name":"Resynthesis"},{"task_slug":"speech-enhancement","task_name":"Speech Enhancement"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"speech-synthesis","task_name":"Speech Synthesis"},{"task_slug":"text-to-speech","task_name":"Text to Speech"},{"task_slug":"text-to-speech-synthesis","task_name":"Text-To-Speech Synthesis"},{"task_slug":"video-synchronization","task_name":"Video Synchronization"},{"task_slug":"visual-speech-recognition","task_name":"Visual Speech Recognition"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"},{"task_slug":"text-to-speech-1","task_name":"text-to-speech"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/speech-enhancement-on-easycom","task":"Speech Enhancement","dataset":"EasyCom","model":"ReVISE (ch2)","rank_in_archive_order":2,"of":6,"metrics":{"Audio Quality MOS":"4.19"},"uses_additional_data":false},{"leaderboard":"/sota/speech-enhancement-on-easycom","task":"Speech Enhancement","dataset":"EasyCom","model":"ReVISE (bf)","rank_in_archive_order":3,"of":6,"metrics":{"Audio Quality MOS":"4.11"},"uses_additional_data":false},{"leaderboard":"/sota/speech-enhancement-on-easycom","task":"Speech Enhancement","dataset":"EasyCom","model":"Demucs (ch2)","rank_in_archive_order":4,"of":6,"metrics":{"Audio Quality MOS":"2.95"},"uses_additional_data":false},{"leaderboard":"/sota/speech-enhancement-on-easycom","task":"Speech Enhancement","dataset":"EasyCom","model":"Demucs (bf)","rank_in_archive_order":5,"of":6,"metrics":{"Audio Quality MOS":"2.39"},"uses_additional_data":false},{"leaderboard":"/sota/speech-recognition-on-easycom","task":"Speech Recognition","dataset":"EasyCom","model":"ReVISE (bf)","rank_in_archive_order":1,"of":5,"metrics":{"WER (%)":"52.1"},"uses_additional_data":false},{"leaderboard":"/sota/speech-recognition-on-easycom","task":"Speech Recognition","dataset":"EasyCom","model":"ReVISE (ch2)","rank_in_archive_order":2,"of":5,"metrics":{"WER (%)":"55.0"},"uses_additional_data":false},{"leaderboard":"/sota/speech-recognition-on-easycom","task":"Speech Recognition","dataset":"EasyCom","model":"Demucs (bf)","rank_in_archive_order":4,"of":5,"metrics":{"WER (%)":"69.8"},"uses_additional_data":false},{"leaderboard":"/sota/speech-recognition-on-easycom","task":"Speech Recognition","dataset":"EasyCom","model":"Demucs (ch2)","rank_in_archive_order":5,"of":5,"metrics":{"WER (%)":"86.8"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2212.11377","atlas_url":"https://app.syntology.ai/?focus=2212.11377","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2212.11377"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2212.11377\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2212.11377"},"call":"get_concepts_for_paper(arxiv_id=\"2212.11377\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"f2aeba67c38db6de4bf83dfa954126fc5fb0872616f7f55dbbc0b5e25cb646ef","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}