{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/face-landmark-based-speaker-independent-audio","title":"Face Landmark-based Speaker-Independent Audio-Visual Speech Enhancement in Multi-Talker Environments","arxiv_id":"1811.02480","date":"2018-11-06","proceeding":null,"authors":["Giovanni Morrone","Luca Pasa","Vadim Tikhanoff","Sonia Bergamaschi","Luciano Fadiga","Leonardo Badino"],"abstract":"In this paper, we address the problem of enhancing the speech of a speaker of interest in a cocktail party scenario when visual information of the speaker of interest is available. Contrary to most previous studies, we do not learn visual features on the typically small audio-visual datasets, but use an already available face landmark detector (trained on a separate image dataset). The landmarks are used by LSTM-based models to generate time-frequency masks which are applied to the acoustic mixed-speech spectrogram. Results show that: (i) landmark motion features are very effective features for this task, (ii) similarly to previous work, reconstruction of the target speaker's spectrogram mediated by masking is significantly more accurate than direct spectrogram reconstruction, and (iii) the best masks depend on both motion landmark features and the input mixed-speech spectrogram. To the best of our knowledge, our proposed models are the first models trained and evaluated on the limited size GRID and TCD-TIMIT datasets, that achieve speaker-independent speech enhancement in a multi-talker setting.","url_abs":"https://arxiv.org/abs/1811.02480v3","url_pdf":"https://arxiv.org/pdf/1811.02480v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"face-landmark-based-speaker-independent-audio","repo_url":"https://github.com/dr-pato/audio_visual_speech_enhancement","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"speech-enhancement","task_name":"Speech Enhancement"},{"task_slug":"speech-separation","task_name":"Speech Separation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/speech-enhancement-on-grid-corpus-mixed","task":"Speech Enhancement","dataset":"GRID corpus (mixed-speech)","model":"Audio-Visual concat-ref","rank_in_archive_order":1,"of":1,"metrics":{"PESQ":"2.70"},"uses_additional_data":false},{"leaderboard":"/sota/speech-enhancement-on-tcd-timit-corpus-mixed","task":"Speech Enhancement","dataset":"TCD-TIMIT corpus (mixed-speech)","model":"Audio-Visual concat-ref","rank_in_archive_order":1,"of":1,"metrics":{"PESQ":"3.03"},"uses_additional_data":false},{"leaderboard":"/sota/speech-separation-on-grid-corpus-mixed-speech","task":"Speech Separation","dataset":"GRID corpus (mixed-speech)","model":"Audio-Visual concat-ref","rank_in_archive_order":2,"of":2,"metrics":{"SDR":"8.05"},"uses_additional_data":false},{"leaderboard":"/sota/speech-separation-on-tcd-timit-corpus-mixed","task":"Speech Separation","dataset":"TCD-TIMIT corpus (mixed-speech)","model":"Audio-Visual concat-ref","rank_in_archive_order":1,"of":1,"metrics":{"SDR":"10.55"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1811.02480","atlas_url":"https://app.syntology.ai/?focus=1811.02480","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1811.02480"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1811.02480\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1811.02480"},"call":"get_concepts_for_paper(arxiv_id=\"1811.02480\")"}],"by_repo":[{"repo":"dr-pato/audio_visual_speech_enhancement","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"2e32f336b5cfa83d192b38aa93414782f8684a0c511110249f1ab4dde25078c3","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}