{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/vivo-surpassing-human-performance-in-novel","title":"VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning","arxiv_id":"2009.13682","date":"2020-09-28","proceeding":null,"authors":["Xiaowei Hu","Xi Yin","Kevin Lin","Lijuan Wang","Lei Zhang","Jianfeng Gao","Zicheng Liu"],"abstract":"It is highly desirable yet challenging to generate image captions that can describe novel objects which are unseen in caption-labeled training data, a capability that is evaluated in the novel object captioning challenge (nocaps). In this challenge, no additional image-caption training data, other thanCOCO Captions, is allowed for model training. Thus, conventional Vision-Language Pre-training (VLP) methods cannot be applied. This paper presents VIsual VOcabulary pretraining (VIVO) that performs pre-training in the absence of caption annotations. By breaking the dependency of paired image-caption training data in VLP, VIVO can leverage large amounts of paired image-tag data to learn a visual vocabulary. This is done by pre-training a multi-layer Transformer model that learns to align image-level tags with their corresponding image region features. To address the unordered nature of image tags, VIVO uses a Hungarian matching loss with masked tag prediction to conduct pre-training. We validate the effectiveness of VIVO by fine-tuning the pre-trained model for image captioning. In addition, we perform an analysis of the visual-text alignment inferred by our model. The results show that our model can not only generate fluent image captions that describe novel objects, but also identify the locations of these objects. Our single model has achieved new state-of-the-art results on nocaps and surpassed the human CIDEr score.","url_abs":"https://arxiv.org/abs/2009.13682v2","url_pdf":"https://arxiv.org/pdf/2009.13682v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"object","task_name":"Object"},{"task_slug":"tag","task_name":"TAG"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-captioning-on-nocaps-entire","task":"Image Captioning","dataset":"nocaps entire","model":"Microsoft Cognitive Services team","rank_in_archive_order":4,"of":39,"metrics":{"B1":"85.62","B2":"71.36","B3":"53.62","B4":"34.65","CIDEr":"114.25","METEOR":"31.27","ROUGE-L":"61.2","SPICE":"14.85"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-in-domain","task":"Image Captioning","dataset":"nocaps in-domain","model":"Microsoft Cognitive Services team","rank_in_archive_order":6,"of":41,"metrics":{"B1":"86.33","B2":"72.83","B3":"55.94","B4":"37.97","CIDEr":"112.82","METEOR":"32.7","ROUGE-L":"62.48","SPICE":"15.22"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-near-domain","task":"Image Captioning","dataset":"nocaps near-domain","model":"Microsoft Cognitive Services team","rank_in_archive_order":5,"of":40,"metrics":{"B1":"86.48","B2":"72.6","B3":"55.26","B4":"36.31","CIDEr":"115.54","METEOR":"31.8","ROUGE-L":"61.9","SPICE":"15.06"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-out-of-domain","task":"Image Captioning","dataset":"nocaps out-of-domain","model":"Microsoft Cognitive Services team","rank_in_archive_order":5,"of":40,"metrics":{"B1":"81.73","B2":"65.48","B3":"45.58","B4":"25.78","CIDEr":"110.14","METEOR":"28.17","ROUGE-L":"57.57","SPICE":"13.74"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-xd-entire","task":"Image Captioning","dataset":"nocaps-XD entire","model":"Microsoft Cognitive Services team","rank_in_archive_order":5,"of":12,"metrics":{"B1":"82.27","B2":"66.04","B3":"47.48","B4":"28.95","CIDEr":"100.12","METEOR":"29.47","ROUGE-L":"58.26","SPICE":"14.04"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-xd-in-domain","task":"Image Captioning","dataset":"nocaps-XD in-domain","model":"Microsoft Cognitive Services team","rank_in_archive_order":4,"of":11,"metrics":{"B1":"82.94","B2":"67.56","B3":"49.66","B4":"32.07","CIDEr":"100.62","METEOR":"30.62","ROUGE-L":"59.43","SPICE":"14.7"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-xd-near-domain","task":"Image Captioning","dataset":"nocaps-XD near-domain","model":"Microsoft Cognitive Services team","rank_in_archive_order":4,"of":11,"metrics":{"B1":"82.88","B2":"67.01","B3":"48.73","B4":"30.21","CIDEr":"101.2","METEOR":"30.0","ROUGE-L":"58.76","SPICE":"14.27"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-xd-out-of-domain","task":"Image Captioning","dataset":"nocaps-XD out-of-domain","model":"Microsoft Cognitive Services team","rank_in_archive_order":3,"of":11,"metrics":{"B1":"79.44","B2":"61.15","B3":"41.03","B4":"21.79","CIDEr":"95.5","METEOR":"26.56","ROUGE-L":"55.49","SPICE":"12.66"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2009.13682","atlas_url":"https://app.syntology.ai/?focus=2009.13682","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2009.13682"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2009.13682\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2009.13682"},"call":"get_concepts_for_paper(arxiv_id=\"2009.13682\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"e9a25707d43ace7a80239df9dedab24a131ddbee233432cc5d84ebffcc0af3d3","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}