{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/3d-convolutional-neural-networks-for-cross","title":"3D Convolutional Neural Networks for Cross Audio-Visual Matching Recognition","arxiv_id":"1706.05739","date":"2017-06-18","proceeding":null,"authors":["Amirsina Torfi","Seyed Mehdi Iranmanesh","Nasser M. Nasrabadi","Jeremy Dawson"],"abstract":"Audio-visual recognition (AVR) has been considered as a solution for speech\nrecognition tasks when the audio is corrupted, as well as a visual recognition\nmethod used for speaker verification in multi-speaker scenarios. The approach\nof AVR systems is to leverage the extracted information from one modality to\nimprove the recognition ability of the other modality by complementing the\nmissing information. The essential problem is to find the correspondence\nbetween the audio and visual streams, which is the goal of this work. We\npropose the use of a coupled 3D Convolutional Neural Network (3D-CNN)\narchitecture that can map both modalities into a representation space to\nevaluate the correspondence of audio-visual streams using the learned\nmultimodal features. The proposed architecture will incorporate both spatial\nand temporal information jointly to effectively find the correlation between\ntemporal information for different modalities. By using a relatively small\nnetwork architecture and much smaller dataset for training, our proposed method\nsurpasses the performance of the existing similar methods for audio-visual\nmatching which use 3D CNNs for feature representation. We also demonstrate that\nan effective pair selection method can significantly increase the performance.\nThe proposed method achieves relative improvements over 20% on the Equal Error\nRate (EER) and over 7% on the Average Precision (AP) in comparison to the\nstate-of-the-art method.","url_abs":"http://arxiv.org/abs/1706.05739v5","url_pdf":"http://arxiv.org/pdf/1706.05739v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"3d-convolutional-neural-networks-for-cross","repo_url":"https://github.com/astorfi/lip-reading-deeplearning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"3d-convolutional-neural-networks-for-cross","repo_url":"https://github.com/sergeyrachev/neuon","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"speaker-verification","task_name":"Speaker Verification"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1706.05739","atlas_url":"https://app.syntology.ai/?focus=1706.05739","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1706.05739"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1706.05739\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1706.05739"},"call":"get_concepts_for_paper(arxiv_id=\"1706.05739\")"}],"by_repo":[{"repo":"astorfi/lip-reading-deeplearning","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"sergeyrachev/neuon","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"e5e3abbeea876c4fb0a015641afe4391a2f5d23ff2e462a6758eb48d716e7d36","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}