{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/retrieval-augmented-multimodal-language","title":"Retrieval-Augmented Multimodal Language Modeling","arxiv_id":"2211.12561","date":"2022-11-22","proceeding":null,"authors":["Michihiro Yasunaga","Armen Aghajanyan","Weijia Shi","Rich James","Jure Leskovec","Percy Liang","Mike Lewis","Luke Zettlemoyer","Wen-tau Yih"],"abstract":"Recent multimodal models such as DALL-E and CM3 have achieved remarkable progress in text-to-image and image-to-text generation. However, these models store all learned knowledge (e.g., the appearance of the Eiffel Tower) in the model parameters, requiring increasingly larger models and training data to capture more knowledge. To integrate knowledge in a more scalable and modular way, we propose a retrieval-augmented multimodal model, which enables a base multimodal model (generator) to refer to relevant text and images fetched by a retriever from external memory (e.g., documents on the web). Specifically, for the retriever, we use a pretrained CLIP, and for the generator, we train a CM3 Transformer on the LAION dataset. Our resulting model, named Retrieval-Augmented CM3 (RA-CM3), is the first multimodal model that can retrieve and generate both text and images. We show that RA-CM3 significantly outperforms baseline multimodal models such as DALL-E and CM3 on both image and caption generation tasks (12 FID and 17 CIDEr improvements on MS-COCO), while requiring much less compute for training (<30% of DALL-E). Moreover, we show that RA-CM3 exhibits novel capabilities, such as faithful image generation and multimodal in-context learning (e.g., image generation from demonstrations).","url_abs":"https://arxiv.org/abs/2211.12561v2","url_pdf":"https://arxiv.org/pdf/2211.12561v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"caption-generation","task_name":"Caption Generation"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-to-text","task_name":"Image to text"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-captioning-on-coco","task":"Image Captioning","dataset":"COCO (Common Objects in Context)","model":"Flamingo (80B; 4-shot)","rank_in_archive_order":8,"of":17,"metrics":{"CIDEr":"103"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-coco","task":"Image Captioning","dataset":"COCO (Common Objects in Context)","model":"RA-CM3 (2.7B)","rank_in_archive_order":9,"of":17,"metrics":{"CIDEr":"89.1"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-coco","task":"Image Captioning","dataset":"COCO (Common Objects in Context)","model":"Flamingo (3B; 4-shot)","rank_in_archive_order":10,"of":17,"metrics":{"CIDEr":"85"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-coco","task":"Image Captioning","dataset":"COCO (Common Objects in Context)","model":"Parti","rank_in_archive_order":11,"of":17,"metrics":{"CIDEr":"83.9"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-coco","task":"Image Captioning","dataset":"COCO (Common Objects in Context)","model":"Vanilla CM3","rank_in_archive_order":13,"of":17,"metrics":{"CIDEr":"71.9"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-coco","task":"Image Captioning","dataset":"COCO (Common Objects in Context)","model":"X-LXMERT","rank_in_archive_order":14,"of":17,"metrics":{"CIDEr":"55.8"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-coco","task":"Image Captioning","dataset":"COCO (Common Objects in Context)","model":"minDALL-E","rank_in_archive_order":15,"of":17,"metrics":{"CIDEr":"48"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-coco","task":"Image Captioning","dataset":"COCO (Common Objects in Context)","model":"ruDALL-E-XL","rank_in_archive_order":16,"of":17,"metrics":{"CIDEr":"38.7"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-coco","task":"Image Captioning","dataset":"COCO (Common Objects in Context)","model":"DALL-E","rank_in_archive_order":17,"of":17,"metrics":{"CIDEr":"20.2"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-image-generation-on-coco","task":"Text-to-Image Generation","dataset":"COCO (Common Objects in Context)","model":"Stable Diffusion","rank_in_archive_order":35,"of":69,"metrics":{"FID":"12.63"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-image-generation-on-coco","task":"Text-to-Image Generation","dataset":"COCO (Common Objects in Context)","model":"RA-CM3 (2.7B)","rank_in_archive_order":41,"of":69,"metrics":{"FID":"15.7"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-image-generation-on-coco","task":"Text-to-Image Generation","dataset":"COCO (Common Objects in Context)","model":"DALL-E (12B)","rank_in_archive_order":56,"of":69,"metrics":{"FID":"28"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-image-generation-on-coco","task":"Text-to-Image Generation","dataset":"COCO (Common Objects in Context)","model":"Vanilla CM3","rank_in_archive_order":58,"of":69,"metrics":{"FID":"29.5"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2211.12561","atlas_url":"https://app.syntology.ai/?focus=2211.12561","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2211.12561"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2211.12561\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2211.12561"},"call":"get_concepts_for_paper(arxiv_id=\"2211.12561\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"85f42706b439f8a10d1d5beb78985513340282667df26adbb1b231db4a884fa1","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}