{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/ita-image-text-alignments-for-multi-modal","title":"ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition","arxiv_id":"2112.06482","date":"2021-12-13","proceeding":"NAACL 2022 7","authors":["Xinyu Wang","Min Gui","Yong Jiang","Zixia Jia","Nguyen Bach","Tao Wang","Zhongqiang Huang","Fei Huang","Kewei Tu"],"abstract":"Recently, Multi-modal Named Entity Recognition (MNER) has attracted a lot of attention. Most of the work utilizes image information through region-level visual representations obtained from a pretrained object detector and relies on an attention mechanism to model the interactions between image and text representations. However, it is difficult to model such interactions as image and text representations are trained separately on the data of their respective modality and are not aligned in the same space. As text representations take the most important role in MNER, in this paper, we propose {\\bf I}mage-{\\bf t}ext {\\bf A}lignments (ITA) to align image features into the textual space, so that the attention mechanism in transformer-based pretrained textual embeddings can be better utilized. ITA first aligns the image into regional object tags, image-level captions and optical characters as visual contexts, concatenates them with the input texts as a new cross-modal input, and then feeds it into a pretrained textual embedding model. This makes it easier for the attention module of a pretrained textual embedding model to model the interaction between the two modalities since they are both represented in the textual space. ITA further aligns the output distributions predicted from the cross-modal input and textual input views so that the MNER model can be more practical in dealing with text-only inputs and robust to noises from images. In our experiments, we show that ITA models can achieve state-of-the-art accuracy on multi-modal Named Entity Recognition datasets, even without image information.","url_abs":"https://arxiv.org/abs/2112.06482v4","url_pdf":"https://arxiv.org/pdf/2112.06482v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ita-image-text-alignments-for-multi-modal","repo_url":"https://github.com/alibaba-nlp/kb-ner","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"multi-modal-named-entity-recognition","task_name":"Multi-modal Named Entity Recognition"},{"task_slug":"named-entity-recognition-1","task_name":"Named Entity Recognition"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"},{"task_slug":"named-entity-recognition","task_name":"named-entity-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/multi-modal-named-entity-recognition-on-snap","task":"Multi-modal Named Entity Recognition","dataset":"SNAP (MNER)","model":"ITA-All","rank_in_archive_order":3,"of":6,"metrics":{"F1":"90.15"},"uses_additional_data":false},{"leaderboard":"/sota/multi-modal-named-entity-recognition-on-1","task":"Multi-modal Named Entity Recognition","dataset":"Twitter-17","model":"ITA-All","rank_in_archive_order":1,"of":2,"metrics":{"F1":"89.75"},"uses_additional_data":false},{"leaderboard":"/sota/multi-modal-named-entity-recognition-on-3","task":"Multi-modal Named Entity Recognition","dataset":"Twitter-2017","model":"ITA-All","rank_in_archive_order":3,"of":3,"metrics":{"F1":"89.75"},"uses_additional_data":false},{"leaderboard":"/sota/multi-modal-named-entity-recognition-on-2","task":"Multi-modal Named Entity Recognition","dataset":"WikiDiverse","model":"ITA-All","rank_in_archive_order":4,"of":5,"metrics":{"F1":"76.87"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2112.06482","atlas_url":"https://app.syntology.ai/?focus=2112.06482","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2112.06482"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2112.06482\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2112.06482"},"call":"get_concepts_for_paper(arxiv_id=\"2112.06482\")"}],"by_repo":[{"repo":"alibaba-nlp/kb-ner","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"1d3ffe001a721d6a1a339200d446a33d6cad0d725556633064a6117d5bc28059","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}