{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/omnivl-one-foundation-model-for-image","title":"OmniVL:One Foundation Model for Image-Language and Video-Language Tasks","arxiv_id":"2209.07526","date":"2022-09-15","proceeding":null,"authors":["Junke Wang","Dongdong Chen","Zuxuan Wu","Chong Luo","Luowei Zhou","Yucheng Zhao","Yujia Xie","Ce Liu","Yu-Gang Jiang","Lu Yuan"],"abstract":"This paper presents OmniVL, a new foundation model to support both image-language and video-language tasks using one universal architecture. It adopts a unified transformer-based visual encoder for both image and video inputs, and thus can perform joint image-language and video-language pretraining. We demonstrate, for the first time, such a paradigm benefits both image and video tasks, as opposed to the conventional one-directional transfer (e.g., use image-language to help video-language). To this end, we propose a decoupled joint pretraining of image-language and video-language to effectively decompose the vision-language modeling into spatial and temporal dimensions and obtain performance boost on both image and video tasks. Moreover, we introduce a novel unified vision-language contrastive (UniVLC) loss to leverage image-text, video-text, image-label (e.g., image classification), video-label (e.g., video action recognition) data together, so that both supervised and noisily supervised pretraining data are utilized as much as possible. Without incurring extra task-specific adaptors, OmniVL can simultaneously support visual only tasks (e.g., image classification, video action recognition), cross-modal alignment tasks (e.g., image/video-text retrieval), and multi-modal understanding and generation tasks (e.g., image/video question answering, captioning). We evaluate OmniVL on a wide range of downstream tasks and achieve state-of-the-art or competitive results with similar model size and data scale.","url_abs":"https://arxiv.org/abs/2209.07526v2","url_pdf":"https://arxiv.org/pdf/2209.07526v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"video-text-retrieval","task_name":"Video-Text Retrieval"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"zero-shot-video-retrieval","task_name":"Zero-Shot Video Retrieval"},{"task_slug":"cross-modal-alignment","task_name":"cross-modal alignment"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"OmniVL","rank_in_archive_order":117,"of":207,"metrics":{"Acc@1":"79.1","Acc@5":"94.5"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"OmniVL","rank_in_archive_order":102,"of":123,"metrics":{"Top-1 Accuracy":"62.5","Top-5 Accuracy":"86.2"},"uses_additional_data":false},{"leaderboard":"/sota/cross-modal-retrieval-on-coco-2014","task":"Cross-Modal Retrieval","dataset":"COCO 2014","model":"OmniVL (14M)","rank_in_archive_order":7,"of":36,"metrics":{"Image-to-text R@1":"82.1","Image-to-text R@10":"98.1","Image-to-text R@5":"95.9","Text-to-image R@1":"64.8","Text-to-image R@10":"91.6","Text-to-image R@5":"86.1"},"uses_additional_data":true},{"leaderboard":"/sota/cross-modal-retrieval-on-flickr30k","task":"Cross-Modal Retrieval","dataset":"Flickr30k","model":"OmniVL (14M)","rank_in_archive_order":4,"of":27,"metrics":{"Image-to-text R@1":"97.3","Image-to-text R@10":"100","Image-to-text R@5":"99.9","Text-to-image R@1":"87.9","Text-to-image R@10":"99.1","Text-to-image R@5":"97.8"},"uses_additional_data":true},{"leaderboard":"/sota/image-captioning-on-nocaps-val-in-domain","task":"Image Captioning","dataset":"nocaps-val-in-domain","model":"OmniVL","rank_in_archive_order":9,"of":11,"metrics":{"CIDEr":"104.6","Pre-train (#images)":"14M","SPICE":"15"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-val-near-domain","task":"Image Captioning","dataset":"nocaps-val-near-domain","model":"OmniVL","rank_in_archive_order":8,"of":10,"metrics":{"CIDEr":"108.3","Pre-train (#images)":"14M","SPICE":"14.9"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-val-out-domain","task":"Image Captioning","dataset":"nocaps-val-out-domain","model":"OmniVL","rank_in_archive_order":8,"of":10,"metrics":{"CIDEr":"106.3","Pretrain (#images)":"14M","SPICE":"14.2"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-val-overall","task":"Image Captioning","dataset":"nocaps-val-overall","model":"OmniVL","rank_in_archive_order":8,"of":11,"metrics":{"CIDEr":"107.5","Pretrain (#images)":"14M","SPICE":"14.7"},"uses_additional_data":false},{"leaderboard":"/sota/video-captioning-on-youcook2","task":"Video Captioning","dataset":"YouCook2","model":"OmniVL","rank_in_archive_order":11,"of":14,"metrics":{"BLEU-3":"12.87","BLEU-4":"8.72","CIDEr":"1.16","METEOR":"14.83","ROUGE-L":"36.09"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-didemo","task":"Video Retrieval","dataset":"DiDeMo","model":"OmniVL","rank_in_archive_order":21,"of":40,"metrics":{"text-to-video R@1":"52.4","text-to-video R@10":"85.4","text-to-video R@5":"79.5"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msr-vtt","task":"Video Retrieval","dataset":"MSR-VTT","model":"OmniVL","rank_in_archive_order":13,"of":40,"metrics":{"text-to-video R@1":"47.8","text-to-video R@10":"83.8","text-to-video R@5":"74.2"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-msrvtt-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSRVTT-QA","model":"OmniVL","rank_in_archive_order":18,"of":34,"metrics":{"Accuracy":"0.441"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-msvd-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSVD-QA","model":"OmniVL","rank_in_archive_order":21,"of":36,"metrics":{"Accuracy":"0.510"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-didemo","task":"Zero-Shot Video Retrieval","dataset":"DiDeMo","model":"OmniVL","rank_in_archive_order":15,"of":26,"metrics":{"text-to-video R@1":"33.3","text-to-video R@10":"68.5","text-to-video R@5":"58.7"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msr-vtt","task":"Zero-Shot Video Retrieval","dataset":"MSR-VTT","model":"OmniVL","rank_in_archive_order":18,"of":41,"metrics":{"text-to-video R@1":"34.6","text-to-video R@10":"66.6","text-to-video R@5":"58.4"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2209.07526","atlas_url":"https://app.syntology.ai/?focus=2209.07526","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2209.07526"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2209.07526\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2209.07526"},"call":"get_concepts_for_paper(arxiv_id=\"2209.07526\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"0746b460a50510c159a30883782e18d4d8121330288ea10a3257a64ba2c31683","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}