{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/this-is-not-a-dataset-a-large-negation","title":"This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models","arxiv_id":"2310.15941","date":"2023-10-24","proceeding":null,"authors":["Iker García-Ferrero","Begoña Altuna","Javier Álvez","Itziar Gonzalez-Dios","German Rigau"],"abstract":"Although large language models (LLMs) have apparently acquired a certain level of grammatical knowledge and the ability to make generalizations, they fail to interpret negation, a crucial step in Natural Language Processing. We try to clarify the reasons for the sub-optimal performance of LLMs understanding negation. We introduce a large semi-automatically generated dataset of circa 400,000 descriptive sentences about commonsense knowledge that can be true or false in which negation is present in about 2/3 of the corpus in different forms. We have used our dataset with the largest available open LLMs in a zero-shot approach to grasp their generalization and inference capability and we have also fine-tuned some of the models to assess whether the understanding of negation can be trained. Our findings show that, while LLMs are proficient at classifying affirmative sentences, they struggle with negative sentences and lack a deep understanding of negation, often relying on superficial cues. Although fine-tuning the models on negative sentences improves their performance, the lack of generalization in handling negation is persistent, highlighting the ongoing challenges of LLMs regarding negation understanding and generalization. The dataset and code are publicly available.","url_abs":"https://arxiv.org/abs/2310.15941v1","url_pdf":"https://arxiv.org/pdf/2310.15941v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"this-is-not-a-dataset-a-large-negation","repo_url":"https://github.com/hitz-zentroa/this-is-not-a-dataset","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"negation","task_name":"Negation"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"zero-shot-text-classification","task_name":"Zero-Shot Text Classification"}],"methods":[],"datasets_introduced":[{"slug":"this-is-not-a-dataset","name":"This is not a Dataset","full_name":"This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/text-classification-on-this-is-not-a-dataset","task":"Text Classification","dataset":"This is not a Dataset","model":"Vicuna13B v1.1","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"95.7","Coherence":"81.2"},"uses_additional_data":false},{"leaderboard":"/sota/text-classification-on-this-is-not-a-dataset","task":"Text Classification","dataset":"This is not a Dataset","model":"Flan-T5-xxl","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy":"94.1","Coherence":"51.8"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.15941","atlas_url":"https://app.syntology.ai/?focus=2310.15941","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2310.15941"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2310.15941\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2310.15941"},"call":"get_concepts_for_paper(arxiv_id=\"2310.15941\")"}],"by_repo":[{"repo":"hitz-zentroa/this-is-not-a-dataset","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2106.09685"],"harvested_for_calls":[{"arxiv_id":"2106.09685","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2106.09685"},"call":"get_harvested_code_for_paper(arxiv_id=\"2106.09685\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"9e240516648b662ea20a24958cfe1e856cc6183c9e2fd595a4c25257a122321a","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}