{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/learning-semantic-sentence-representations","title":"Learning semantic sentence representations from visually grounded language without lexical knowledge","arxiv_id":"1903.11393","date":"2019-03-27","proceeding":null,"authors":["Danny Merkx","Stefan Frank"],"abstract":"Current approaches to learning semantic representations of sentences often\nuse prior word-level knowledge. The current study aims to leverage visual\ninformation in order to capture sentence level semantics without the need for\nword embeddings. We use a multimodal sentence encoder trained on a corpus of\nimages with matching text captions to produce visually grounded sentence\nembeddings. Deep Neural Networks are trained to map the two modalities to a\ncommon embedding space such that for an image the corresponding caption can be\nretrieved and vice versa. We show that our model achieves results comparable to\nthe current state-of-the-art on two popular image-caption retrieval benchmark\ndata sets: MSCOCO and Flickr8k. We evaluate the semantic content of the\nresulting sentence embeddings using the data from the Semantic Textual\nSimilarity benchmark task and show that the multimodal embeddings correlate\nwell with human semantic similarity judgements. The system achieves\nstate-of-the-art results on several of these benchmarks, which shows that a\nsystem trained solely on multimodal data, without assuming any word\nrepresentations, is able to capture sentence level semantics. Importantly, this\nresult shows that we do not need prior knowledge of lexical level semantics in\norder to model sentence level semantics. These findings demonstrate the\nimportance of visual information in semantics.","url_abs":"http://arxiv.org/abs/1903.11393v1","url_pdf":"http://arxiv.org/pdf/1903.11393v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-semantic-sentence-representations","repo_url":"https://github.com/DannyMerkx/caption2image","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"grounded-language-learning","task_name":"Grounded language learning"},{"task_slug":"learning-semantic-representations","task_name":"Learning Semantic Representations"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"sentence-embeddings","task_name":"Sentence Embeddings"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"DannyMerkx/caption2image","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"ef5766ab21e0161df97f3bb6bcf20c59665e9eba1977985179c87001051e56bb","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}