{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/simlex-999-evaluating-semantic-models-with","title":"SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation","arxiv_id":"1408.3456","date":"2014-08-15","proceeding":"CL 2015 12","authors":["Felix Hill","Roi Reichart","Anna Korhonen"],"abstract":"We present SimLex-999, a gold standard resource for evaluating distributional\nsemantic models that improves on existing resources in several important ways.\nFirst, in contrast to gold standards such as WordSim-353 and MEN, it explicitly\nquantifies similarity rather than association or relatedness, so that pairs of\nentities that are associated but not actually similar [Freud, psychology] have\na low rating. We show that, via this focus on similarity, SimLex-999\nincentivizes the development of models with a different, and arguably wider\nrange of applications than those which reflect conceptual association. Second,\nSimLex-999 contains a range of concrete and abstract adjective, noun and verb\npairs, together with an independent rating of concreteness and (free)\nassociation strength for each pair. This diversity enables fine-grained\nanalyses of the performance of models on concepts of different types, and\nconsequently greater insight into how architectures can be improved. Further,\nunlike existing gold standard evaluations, for which automatic approaches have\nreached or surpassed the inter-annotator agreement ceiling, state-of-the-art\nmodels perform well below this ceiling on SimLex-999. There is therefore plenty\nof scope for SimLex-999 to quantify future improvements to distributional\nsemantic models, guiding the development of the next generation of\nrepresentation-learning architectures.","url_abs":"http://arxiv.org/abs/1408.3456v1","url_pdf":"http://arxiv.org/pdf/1408.3456v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"simlex-999-evaluating-semantic-models-with","repo_url":"https://github.com/obj2vec/datasets_similarity","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"simlex-999-evaluating-semantic-models-with","repo_url":"https://github.com/obj2vec/semantic_datasets","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"simlex-999-evaluating-semantic-models-with","repo_url":"https://github.com/obj2vec/similarity_datasets","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1408.3456","atlas_url":"https://app.syntology.ai/?focus=1408.3456","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1408.3456"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1408.3456\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1408.3456"},"call":"get_concepts_for_paper(arxiv_id=\"1408.3456\")"}],"by_repo":[{"repo":"obj2vec/datasets_similarity","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"obj2vec/semantic_datasets","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"obj2vec/similarity_datasets","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"204dd5e6222a5e4c2be59a5d51b91d1ba2d6fd177fc55452ac16a8268993c9e0","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}