Papers › VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset

VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset

17 Apr 2023arXiv:2304.08345archive 2025-07-28

Jing Liu, Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang

In this paper, we propose a Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multi-modal understanding and generation. Different from widely-studied vision-language pretraining models, VALOR jointly models relationships of vision, audio and language in an end-to-end manner. It contains three separate encoders for single modality representations, and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain VALOR model, including Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language and audio to the same common space, building vision-language, audio-language and audiovisual-language alignment simultaneously. MGC learns how to generate text tokens in conditions of vision, audio or their both. To promote vision-audio-language pretraining research, we construct a large-scale high-quality tri-modality dataset named VALOR-1M, which contains 1M audiable videos with human annotated audiovisual captions. Extensive experiments show that VALOR can learn strong multimodal correlations and be generalized to various downstream tasks (e.g., retrieval, captioning and question answering), with different input modalities (e.g., vision-language, audio-language and audiovisual-language). VALOR achieves new state-of-the-art performances on series of public cross-modality benchmarks. Code and data are available at project page https://casia-iva-group.github.io/projects/VALOR.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

TXH-mercury/VALOR officialmentioned on GitHubpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset (arXiv:2305.18500; that paper's own run record: 35 ran (of which 4 constructed an object rather than computing a result; 25 with no instrument failure: 4 honoured, 1 violated, 20 with no contract checked; 10 where Syntology's instrument failed) · 7 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Audio captioningAudio-Video Question Answering (AVQA)Audio-Visual Question Answering (AVQA)Audio-visual Question AnsweringConditional Text GenerationCross-Modal RetrievalDecoderImage CaptioningQuestion AnsweringRetrievalTGIF-FrameText GenerationText to Audio RetrievalVideo CaptioningVideo Question AnsweringVideo RetrievalVisual Question Answering (VQA)Zero-shot Text to Audio Retrieval

2 archive task tags without a task page not shown.

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Audio captioning AudioCaps VALOR BLEU-4 0.270 #18 of 18 Archive leaderboard report
Audio captioning AudioCaps VALOR CIDEr 0.741 #18 of 18 Archive leaderboard report
Audio captioning AudioCaps VALOR METEOR 0.231 #18 of 18 Archive leaderboard report
Audio captioning AudioCaps VALOR ROUGE-L 0.494 #18 of 18 Archive leaderboard report
Audio captioning Clotho VALOR BLEU-4 16.2 #10 of 11 Archive leaderboard report
Audio captioning Clotho VALOR CIDEr 0.423 #10 of 11 Archive leaderboard report
Audio captioning Clotho VALOR METEOR 17.4 #10 of 11 Archive leaderboard report
Audio captioning Clotho VALOR ROUGE-L 38.2 #10 of 11 Archive leaderboard report
Audio-visual Question Answering MUSIC-AVQA VALOR Acc 78.9 #3 of 6 Archive leaderboard report
Cross-Modal Retrieval COCO 2014 VALOR Text-to-image R@1 61.4 #14 of 36 Archive leaderboard report
Cross-Modal Retrieval COCO 2014 VALOR Text-to-image R@10 90.9 #14 of 36 Archive leaderboard report
Cross-Modal Retrieval COCO 2014 VALOR Text-to-image R@5 84.4 #14 of 36 Archive leaderboard report
Image Captioning COCO Captions VALOR CIDER 152.5 #36 of 41 Archive leaderboard report
Image Captioning COCO Captions VALOR SPICE 25.7 #36 of 41 Archive leaderboard report
Text to Audio Retrieval AudioCaps VALOR R@1 40.1 #4 of 11 Archive leaderboard report
Text to Audio Retrieval AudioCaps VALOR R@10 83.1 #4 of 11 Archive leaderboard report
Text to Audio Retrieval AudioCaps VALOR R@5 73.9 #4 of 11 Archive leaderboard report
Text to Audio Retrieval Clotho VALOR R@1 17.5 #6 of 12 Archive leaderboard report
Text to Audio Retrieval Clotho VALOR R@10 55.3 #6 of 12 Archive leaderboard report
Text to Audio Retrieval Clotho VALOR R@5 42.7 #6 of 12 Archive leaderboard report
Video Captioning MSR-VTT VALOR BLEU-4 54.4 #6 of 24 Archive leaderboard report
Video Captioning MSR-VTT VALOR CIDEr 74.0 #6 of 24 Archive leaderboard report
Video Captioning MSR-VTT VALOR METEOR 32.9 #6 of 24 Archive leaderboard report
Video Captioning MSR-VTT VALOR ROUGE-L 68.0 #6 of 24 Archive leaderboard report
Video Captioning MSVD VALOR BLEU-4 80.7 #3 of 16 Archive leaderboard report
Video Captioning MSVD VALOR CIDEr 178.5 #3 of 16 Archive leaderboard report
Video Captioning MSVD VALOR METEOR 51.0 #3 of 16 Archive leaderboard report
Video Captioning MSVD VALOR ROUGE-L 87.9 #3 of 16 Archive leaderboard report
Video Captioning VATEX VALOR BLEU-4 45.6 #1 of 10 Archive leaderboard report
Video Captioning VATEX VALOR CIDEr 95.8 #1 of 10 Archive leaderboard report
Video Captioning VATEX VALOR METEOR 29.4 #1 of 10 Archive leaderboard report
Video Captioning VATEX VALOR ROUGE-L 57.4 #1 of 10 Archive leaderboard report
Video Question Answering ActivityNet-QA VALOR Accuracy 48.6 #9 of 36 Archive leaderboard report
Video Question Answering MSRVTT-QA VALOR Accuracy 49.2 #3 of 14 Archive leaderboard report
Video Retrieval ActivityNet VALOR text-to-video R@1 70.1 #3 of 31 Archive leaderboard report
Video Retrieval ActivityNet VALOR text-to-video R@10 95.3 #3 of 31 Archive leaderboard report
Video Retrieval ActivityNet VALOR text-to-video R@5 90.8 #3 of 31 Archive leaderboard report
Video Retrieval DiDeMo VALOR text-to-video R@1 61.5 #7 of 40 Archive leaderboard report
Video Retrieval DiDeMo VALOR text-to-video R@10 90.4 #7 of 40 Archive leaderboard report
Video Retrieval DiDeMo VALOR text-to-video R@5 85.3 #7 of 40 Archive leaderboard report
Video Retrieval LSMDC VALOR text-to-video R@1 34.2 #7 of 38 Archive leaderboard report
Video Retrieval LSMDC VALOR text-to-video R@10 64.1 #7 of 38 Archive leaderboard report
Video Retrieval LSMDC VALOR text-to-video R@5 56.0 #7 of 38 Archive leaderboard report
Video Retrieval MSR-VTT VALOR text-to-video R@1 59.9 #4 of 40 Archive leaderboard report
Video Retrieval MSR-VTT VALOR text-to-video R@10 89.6 #4 of 40 Archive leaderboard report
Video Retrieval MSR-VTT VALOR text-to-video R@5 83.5 #4 of 40 Archive leaderboard report
Video Retrieval VATEX VALOR text-to-video R@1 78.5 #3 of 13 Archive leaderboard report
Video Retrieval VATEX VALOR text-to-video R@10 98.7 #3 of 13 Archive leaderboard report
Video Retrieval VATEX VALOR text-to-video R@5 97.1 #3 of 13 Archive leaderboard report
Visual Question Answering (VQA) MSVD-QA VALOR Accuracy 0.60 #4 of 36 Archive leaderboard report
Visual Question Answering (VQA) VQA v2 test-dev VALOR Accuracy 78.46 #15 of 56 Archive leaderboard report
Visual Question Answering (VQA) VQA v2 test-std VALOR overall 78.62 #9 of 38 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections