{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/avinet-diving-deep-into-audio-visual-saliency","title":"ViNet: Pushing the limits of Visual Modality for Audio-Visual Saliency Prediction","arxiv_id":"2012.06170","date":"2020-12-11","proceeding":null,"authors":["Samyak Jain","Pradeep Yarlagadda","Shreyank Jyoti","Shyamgopal Karthik","Ramanathan Subramanian","Vineet Gandhi"],"abstract":"We propose the ViNet architecture for audio-visual saliency prediction. ViNet is a fully convolutional encoder-decoder architecture. The encoder uses visual features from a network trained for action recognition, and the decoder infers a saliency map via trilinear interpolation and 3D convolutions, combining features from multiple hierarchies. The overall architecture of ViNet is conceptually simple; it is causal and runs in real-time (60 fps). ViNet does not use audio as input and still outperforms the state-of-the-art audio-visual saliency prediction models on nine different datasets (three visual-only and six audio-visual datasets). ViNet also surpasses human performance on the CC, SIM and AUC metrics for the AVE dataset, and to our knowledge, it is the first network to do so. We also explore a variation of ViNet architecture by augmenting audio features into the decoder. To our surprise, upon sufficient training, the network becomes agnostic to the input audio and provides the same output irrespective of the input. Interestingly, we also observe similar behaviour in the previous state-of-the-art models \\cite{tsiami2020stavis} for audio-visual saliency prediction. Our findings contrast with previous works on deep learning-based audio-visual saliency prediction, suggesting a clear avenue for future explorations incorporating audio in a more effective manner. The code and pre-trained models are available at https://github.com/samyak0210/ViNet.","url_abs":"https://arxiv.org/abs/2012.06170v3","url_pdf":"https://arxiv.org/pdf/2012.06170v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"avinet-diving-deep-into-audio-visual-saliency","repo_url":"https://github.com/samyak0210/ViNet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"saliency-prediction","task_name":"Saliency Prediction"},{"task_slug":"video-saliency-detection","task_name":"Video Saliency Detection"},{"task_slug":"video-saliency-prediction","task_name":"Video Saliency Prediction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-saliency-detection-on-dhf1k","task":"Video Saliency Detection","dataset":"DHF1K","model":"ViNet","rank_in_archive_order":1,"of":3,"metrics":{"AUC-J":"0.908","CC":"0.51","NSS":"2.87","s-AUC":"0.728"},"uses_additional_data":false},{"leaderboard":"/sota/video-saliency-detection-on-diem","task":"Video Saliency Detection","dataset":"DIEM","model":"AViNet","rank_in_archive_order":1,"of":1,"metrics":{"CC":"0.632"},"uses_additional_data":false},{"leaderboard":"/sota/video-saliency-detection-on-hollywood2","task":"Video Saliency Detection","dataset":"Hollywood2","model":"ViNet","rank_in_archive_order":1,"of":1,"metrics":{"CC":"0.693"},"uses_additional_data":false},{"leaderboard":"/sota/video-saliency-detection-on-msu-video","task":"Video Saliency Detection","dataset":"MSU Video Saliency Prediction","model":"ViNet (dave)","rank_in_archive_order":1,"of":14,"metrics":{"AUC-J":"0.864","CC":"0.733","FPS":"1.10","KLDiv":"0.497","NSS":"2.13","SIM":"0.627"},"uses_additional_data":false},{"leaderboard":"/sota/video-saliency-detection-on-ucfsports","task":"Video Saliency Detection","dataset":"UCFSports","model":"ViNet","rank_in_archive_order":1,"of":1,"metrics":{"CC":"0.673"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2012.06170","atlas_url":"https://app.syntology.ai/?focus=2012.06170","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2012.06170"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2012.06170\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2012.06170"},"call":"get_concepts_for_paper(arxiv_id=\"2012.06170\")"}],"by_repo":[{"repo":"samyak0210/ViNet","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"7e4f60e446fc200f76aeccc47dbabf430f2377a6ec6da52351a8649cb2290c75","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}