{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/valor-vision-audio-language-omni-perception","title":"VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset","arxiv_id":"2304.08345","date":"2023-04-17","proceeding":null,"authors":["Jing Liu","Sihan Chen","Xingjian He","Longteng Guo","Xinxin Zhu","Weining Wang","Jinhui Tang"],"abstract":"In this paper, we propose a Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multi-modal understanding and generation. Different from widely-studied vision-language pretraining models, VALOR jointly models relationships of vision, audio and language in an end-to-end manner. It contains three separate encoders for single modality representations, and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain VALOR model, including Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language and audio to the same common space, building vision-language, audio-language and audiovisual-language alignment simultaneously. MGC learns how to generate text tokens in conditions of vision, audio or their both. To promote vision-audio-language pretraining research, we construct a large-scale high-quality tri-modality dataset named VALOR-1M, which contains 1M audiable videos with human annotated audiovisual captions. Extensive experiments show that VALOR can learn strong multimodal correlations and be generalized to various downstream tasks (e.g., retrieval, captioning and question answering), with different input modalities (e.g., vision-language, audio-language and audiovisual-language). VALOR achieves new state-of-the-art performances on series of public cross-modality benchmarks. Code and data are available at project page https://casia-iva-group.github.io/projects/VALOR.","url_abs":"https://arxiv.org/abs/2304.08345v2","url_pdf":"https://arxiv.org/pdf/2304.08345v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"valor-vision-audio-language-omni-perception","repo_url":"https://github.com/TXH-mercury/VALOR","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"audio-captioning","task_name":"Audio captioning"},{"task_slug":"audio-video-question-answering-avqa","task_name":"Audio-Video Question Answering (AVQA)"},{"task_slug":null,"task_name":"Audio-Visual Captioning"},{"task_slug":"audio-visual-question-answering-avqa","task_name":"Audio-Visual Question Answering (AVQA)"},{"task_slug":"audio-visual-question-answering","task_name":"Audio-visual Question Answering"},{"task_slug":"conditional-text-generation","task_name":"Conditional Text Generation"},{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"tgif-frame","task_name":"TGIF-Frame"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"text-to-audio-retrieval","task_name":"Text to Audio Retrieval"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"zero-shot-text-to-audio-retrieval","task_name":"Zero-shot Text to Audio Retrieval"},{"task_slug":null,"task_name":"text-to-audiovisual retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/audio-captioning-on-audiocaps","task":"Audio captioning","dataset":"AudioCaps","model":"VALOR","rank_in_archive_order":18,"of":18,"metrics":{"BLEU-4":"0.270","CIDEr":"0.741","METEOR":"0.231","ROUGE-L":"0.494"},"uses_additional_data":true},{"leaderboard":"/sota/audio-captioning-on-clotho","task":"Audio captioning","dataset":"Clotho","model":"VALOR","rank_in_archive_order":10,"of":11,"metrics":{"BLEU-4":"16.2","CIDEr":"0.423","METEOR":"17.4","ROUGE-L":"38.2"},"uses_additional_data":true},{"leaderboard":"/sota/audio-visual-question-answering-on-music-avqa","task":"Audio-visual Question Answering","dataset":"MUSIC-AVQA","model":"VALOR","rank_in_archive_order":3,"of":6,"metrics":{"Acc":"78.9"},"uses_additional_data":true},{"leaderboard":"/sota/cross-modal-retrieval-on-coco-2014","task":"Cross-Modal Retrieval","dataset":"COCO 2014","model":"VALOR","rank_in_archive_order":14,"of":36,"metrics":{"Text-to-image R@1":"61.4","Text-to-image R@10":"90.9","Text-to-image R@5":"84.4"},"uses_additional_data":true},{"leaderboard":"/sota/image-captioning-on-coco-captions","task":"Image Captioning","dataset":"COCO Captions","model":"VALOR","rank_in_archive_order":36,"of":41,"metrics":{"CIDER":"152.5","SPICE":"25.7"},"uses_additional_data":true},{"leaderboard":"/sota/text-to-audio-retrieval-on-audiocaps","task":"Text to Audio Retrieval","dataset":"AudioCaps","model":"VALOR","rank_in_archive_order":4,"of":11,"metrics":{"R@1":"40.1","R@10":"83.1","R@5":"73.9"},"uses_additional_data":true},{"leaderboard":"/sota/text-to-audio-retrieval-on-clotho","task":"Text to Audio Retrieval","dataset":"Clotho","model":"VALOR","rank_in_archive_order":6,"of":12,"metrics":{"R@1":"17.5","R@10":"55.3","R@5":"42.7"},"uses_additional_data":true},{"leaderboard":"/sota/video-captioning-on-msr-vtt-1","task":"Video Captioning","dataset":"MSR-VTT","model":"VALOR","rank_in_archive_order":6,"of":24,"metrics":{"BLEU-4":"54.4","CIDEr":"74.0","METEOR":"32.9","ROUGE-L":"68.0"},"uses_additional_data":true},{"leaderboard":"/sota/video-captioning-on-msvd-1","task":"Video Captioning","dataset":"MSVD","model":"VALOR","rank_in_archive_order":3,"of":16,"metrics":{"BLEU-4":"80.7","CIDEr":"178.5","METEOR":"51.0","ROUGE-L":"87.9"},"uses_additional_data":true},{"leaderboard":"/sota/video-captioning-on-vatex-1","task":"Video Captioning","dataset":"VATEX","model":"VALOR","rank_in_archive_order":1,"of":10,"metrics":{"BLEU-4":"45.6","CIDEr":"95.8","METEOR":"29.4","ROUGE-L":"57.4"},"uses_additional_data":true},{"leaderboard":"/sota/video-question-answering-on-activitynet-qa","task":"Video Question Answering","dataset":"ActivityNet-QA","model":"VALOR","rank_in_archive_order":9,"of":36,"metrics":{"Accuracy":"48.6"},"uses_additional_data":true},{"leaderboard":"/sota/video-question-answering-on-msrvtt-qa","task":"Video Question Answering","dataset":"MSRVTT-QA","model":"VALOR","rank_in_archive_order":3,"of":14,"metrics":{"Accuracy":"49.2"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-activitynet","task":"Video Retrieval","dataset":"ActivityNet","model":"VALOR","rank_in_archive_order":3,"of":31,"metrics":{"text-to-video R@1":"70.1","text-to-video R@10":"95.3","text-to-video R@5":"90.8"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-didemo","task":"Video Retrieval","dataset":"DiDeMo","model":"VALOR","rank_in_archive_order":7,"of":40,"metrics":{"text-to-video R@1":"61.5","text-to-video R@10":"90.4","text-to-video R@5":"85.3"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"VALOR","rank_in_archive_order":7,"of":38,"metrics":{"text-to-video R@1":"34.2","text-to-video R@10":"64.1","text-to-video R@5":"56.0"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msr-vtt","task":"Video Retrieval","dataset":"MSR-VTT","model":"VALOR","rank_in_archive_order":4,"of":40,"metrics":{"text-to-video R@1":"59.9","text-to-video R@10":"89.6","text-to-video R@5":"83.5"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-vatex","task":"Video Retrieval","dataset":"VATEX","model":"VALOR","rank_in_archive_order":3,"of":13,"metrics":{"text-to-video R@1":"78.5","text-to-video R@10":"98.7","text-to-video R@5":"97.1"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-msvd-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSVD-QA","model":"VALOR","rank_in_archive_order":4,"of":36,"metrics":{"Accuracy":"0.60"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-vqa-v2-test-dev","task":"Visual Question Answering (VQA)","dataset":"VQA v2 test-dev","model":"VALOR","rank_in_archive_order":15,"of":56,"metrics":{"Accuracy":"78.46"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-vqa-v2-test-std","task":"Visual Question Answering (VQA)","dataset":"VQA v2 test-std","model":"VALOR","rank_in_archive_order":9,"of":38,"metrics":{"overall":"78.62"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2304.08345","atlas_url":"https://app.syntology.ai/?focus=2304.08345","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2304.08345"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2304.08345\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2304.08345"},"call":"get_concepts_for_paper(arxiv_id=\"2304.08345\")"}],"by_repo":[{"repo":"TXH-mercury/VALOR","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2305.18500"],"harvested_for_calls":[{"arxiv_id":"2305.18500","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2305.18500"},"call":"get_harvested_code_for_paper(arxiv_id=\"2305.18500\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"f66971ef0eb3f4cebb10e9b34dc9e4bbde13816e71ad95b4f9fdda84372c988c","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}