{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/zero-shot-detection-via-vision-and-language","title":"Open-vocabulary Object Detection via Vision and Language Knowledge Distillation","arxiv_id":"2104.13921","date":"2021-04-28","proceeding":"ICLR 2022 4","authors":["Xiuye Gu","Tsung-Yi Lin","Weicheng Kuo","Yin Cui"],"abstract":"We aim at advancing open-vocabulary object detection, which detects objects described by arbitrary text inputs. The fundamental challenge is the availability of training data. It is costly to further scale up the number of classes contained in existing object detection datasets. To overcome this challenge, we propose ViLD, a training method via Vision and Language knowledge Distillation. Our method distills the knowledge from a pretrained open-vocabulary image classification model (teacher) into a two-stage detector (student). Specifically, we use the teacher model to encode category texts and image regions of object proposals. Then we train a student detector, whose region embeddings of detected boxes are aligned with the text and image embeddings inferred by the teacher. We benchmark on LVIS by holding out all rare categories as novel categories that are not seen during training. ViLD obtains 16.1 mask AP$_r$ with a ResNet-50 backbone, even outperforming the supervised counterpart by 3.8. When trained with a stronger teacher model ALIGN, ViLD achieves 26.3 AP$_r$. The model can directly transfer to other datasets without finetuning, achieving 72.2 AP$_{50}$ on PASCAL VOC, 36.6 AP on COCO and 11.8 AP on Objects365. On COCO, ViLD outperforms the previous state-of-the-art by 4.8 on novel AP and 11.4 on overall AP. Code and demo are open-sourced at https://github.com/tensorflow/tpu/tree/master/models/official/detection/projects/vild.","url_abs":"https://arxiv.org/abs/2104.13921v3","url_pdf":"https://arxiv.org/pdf/2104.13921v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"zero-shot-detection-via-vision-and-language","repo_url":"https://github.com/tensorflow/tpu","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"zero-shot-detection-via-vision-and-language","repo_url":"https://github.com/tensorflow/tpu/tree/master/models/official/detection/projects/vild","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"zero-shot-detection-via-vision-and-language","repo_url":"https://github.com/hanoonaR/object-centric-ovd","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"zero-shot-detection-via-vision-and-language","repo_url":"https://github.com/dyabel/detpro","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"open-vocabulary-image-classification","task_name":"Open Vocabulary Image Classification"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open Vocabulary Object Detection"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open-vocabulary object detection"},{"task_slug":"zero-shot-image-classification","task_name":"Zero-Shot Image Classification"},{"task_slug":"zero-shot-object-detection","task_name":"Zero-Shot Object Detection"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"mask-r-cnn","method_name":"Mask R-CNN"},{"method_slug":"rpn","method_name":"RPN"},{"method_slug":"roi-align","method_name":"RoIAlign"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/open-vocabulary-object-detection-on-lvis-v1-0","task":"Open Vocabulary Object Detection","dataset":"LVIS v1.0","model":"ViLD-ensemble w/ ALIGN (Eb7-FPN)","rank_in_archive_order":14,"of":28,"metrics":{"AP novel-LVIS base training":"26.3","AP novel-Unrestricted open-vocabulary training":"27.0"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-object-detection-on-lvis-v1-0","task":"Open Vocabulary Object Detection","dataset":"LVIS v1.0","model":"ViLD-ensemble (R152-FPN)","rank_in_archive_order":24,"of":28,"metrics":{"AP novel-LVIS base training":"18.7","AP novel-Unrestricted open-vocabulary training":"19.8"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-object-detection-on-lvis-v1-0","task":"Open Vocabulary Object Detection","dataset":"LVIS v1.0","model":"ViLD-ensemble (R50-FPN)","rank_in_archive_order":27,"of":28,"metrics":{"AP novel-LVIS base training":"16.6","AP novel-Unrestricted open-vocabulary training":"16.7"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-object-detection-on-lvis-v1-0","task":"Open Vocabulary Object Detection","dataset":"LVIS v1.0","model":"ViLD (R50-FPN)","rank_in_archive_order":28,"of":28,"metrics":{"AP novel-LVIS base training":"16.1","AP novel-Unrestricted open-vocabulary training":"16.3"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-object-detection-on-mscoco","task":"Open Vocabulary Object Detection","dataset":"MSCOCO","model":"ViLD","rank_in_archive_order":29,"of":32,"metrics":{"AP 0.5":"27.6"},"uses_additional_data":true},{"leaderboard":"/sota/open-vocabulary-object-detection-on-1","task":"Open Vocabulary Object Detection","dataset":"Objects365","model":"ViLD","rank_in_archive_order":2,"of":2,"metrics":{"mask AP50":"18.2"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2104.13921","atlas_url":"https://app.syntology.ai/?focus=2104.13921","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2104.13921"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2104.13921\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2104.13921"},"call":"get_concepts_for_paper(arxiv_id=\"2104.13921\")"}],"by_repo":[{"repo":"tensorflow/tpu","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2403.14715"],"harvested_for_calls":[{"arxiv_id":"2403.14715","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2403.14715"},"call":"get_harvested_code_for_paper(arxiv_id=\"2403.14715\")"}]},{"repo":"hanoonaR/object-centric-ovd","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2207.03482"],"harvested_for_calls":[{"arxiv_id":"2207.03482","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2207.03482"},"call":"get_harvested_code_for_paper(arxiv_id=\"2207.03482\")"}]},{"repo":"dyabel/detpro","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2203.14940"],"harvested_for_calls":[{"arxiv_id":"2203.14940","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2203.14940"},"call":"get_harvested_code_for_paper(arxiv_id=\"2203.14940\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"24f0d610bc8b444318a9cb88bf7f3bced29338431ff3e4f5cd2e7f93a7921cb5","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}