{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/dataset/yesbut","name":"YesBut","full_name":null,"description_markdown":"# *YesBut Dataset (https://yesbut-dataset.github.io)*\r\n\r\nUnderstanding satire and humor is a challenging task for even current Vision-Language models. In this paper, we propose the challenging tasks of Satirical Image Detection (detecting whether an image is satirical), Understanding (generating the reason behind the image being satirical), and Completion (given one half of the image, selecting the other half from 2 given options, such that the complete image is satirical) and release a high-quality dataset YesBut, consisting of 2547 images, 1084 satirical and 1463 non-satirical, containing different artistic styles, to evaluate those tasks. Each satirical image in the dataset depicts a normal scenario, along with a conflicting scenario which is funny or ironic. Despite the success of current Vision-Language Models on multimodal tasks such as Visual QA and Image Captioning, our benchmarking experiments show that such models perform poorly on the proposed tasks on the YesBut Dataset in Zero-Shot Settings w.r.t both automated as well as human evaluation. Additionally, we release a dataset of 119 real, satirical photographs for further research. The dataset and code are available at https://github.com/abhi1nandy2/yesbut_dataset.\r\n\r\n## Dataset Details\r\n\r\nYesBut Dataset consists of 2547 images, 1084 satirical and 1463 non-satirical, containing different artistic styles. Each satirical image is posed in a “Yes, But” format, where the left half of the image depicts a normal scenario, while the right half depicts a conflicting scenario which is funny or ironic. \r\n\r\n**Currently on huggingface, the dataset contains the satirical images from 3 different stages of annotation, along with corresponding metadata and image descriptions.** \r\n\r\n> Download non-satirical images from the following Google Drive Links -\r\n> https://drive.google.com/file/d/1Tzs4OcEJK469myApGqOUKPQNUtVyTRDy/view?usp=sharing - Non-Satirical Images annotated in Stage 3\r\n> https://drive.google.com/file/d/1i4Fy01uBZ_2YGPzyVArZjijleNbt8xRu/view?usp=sharing - Non-Satirical Images annotated in Stage 4\r\n\r\n\r\n### Dataset Description\r\n\r\nThe YesBut dataset is a high-quality annotated dataset designed to evaluate the satire comprehension capabilities of vision-language models. It consists of 2547 multimodal images, 1084 of which are satirical, while 1463 are non-satirical. The dataset covers a variety of artistic styles, such as colorized sketches, 2D stick figures, and 3D stick figures, making it highly diverse. The satirical images follow a \"Yes, But\" structure, where the left half depicts a normal scenario, and the right half contains an ironic or humorous twist. The dataset is curated to challenge models in three main tasks: Satirical Image Detection, Understanding, and Completion. An additional dataset of 119 real, satirical photographs is also provided to further assess real-world satire comprehension.\r\n\r\n- **Curated by:** Annotators who met the qualification criteria of being undergraduate sophomore students or above, enrolled in English-medium colleges.\r\n- **Language(s) (NLP):** English\r\n- **License:** Apache license 2.0\r\n\r\n### Dataset Sources\r\n\r\n- **Repository:** https://github.com/abhi1nandy2/yesbut_dataset\r\n- **Paper:** \r\n\t- HuggingFace: https://huggingface.co/papers/2409.13592\r\n\t- ArXiv - https://arxiv.org/abs/2409.13592\r\n\t- This work has been accepted **as a long paper in EMNLP Main 2024**\r\n\r\n## Uses\r\n\r\n<!-- Address questions around how the dataset is intended to be used. -->\r\n\r\n### Direct Use\r\n\r\nThe YesBut dataset is intended for benchmarking the performance of vision-language models on satire and humor comprehension tasks. Researchers and developers can use this dataset to test their models on Satirical Image Detection (classifying an image as satirical or non-satirical), Satirical Image Understanding (generating the reason behind the satire), and Satirical Image Completion (choosing the correct half of an image that makes the completed image satirical). This dataset is particularly suitable for models developed for image-text reasoning and cross-modal understanding.\r\n\r\n### Out-of-Scope Use\r\n\r\nYesBut Dataset is not recommended for tasks unrelated to multimodal understanding, such as basic image classification without the context of humor or satire.\r\n\r\n## Dataset Structure\r\n\r\nThe YesBut dataset contains two types of images: satirical and non-satirical. Satirical images are annotated with textual descriptions for both the left and right sub-images, as well as an overall description containing the punchline. Non-satirical images are also annotated but lack the element of irony or contradiction. The dataset is divided into multiple annotation stages (Stage 2, Stage 3, Stage 4), with each stage including a mix of original and generated sub-images. The data is stored in image and metadata formats, with the annotations being key to the benchmarking tasks.\r\n\r\n## Dataset Creation\r\n\r\n### Curation Rationale\r\n\r\nThe YesBut dataset was curated to address the gap in the ability of existing vision-language models to comprehend satire and humor. Satire comprehension involves a higher level of reasoning and understanding of context, irony, and social cues, making it a challenging task for models. By including a diverse range of images and satirical scenarios, the dataset aims to push the boundaries of multimodal understanding in artificial intelligence.\r\n\r\n### Source Data\r\n\r\n<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->\r\n\r\n#### Data Collection and Processing\r\n\r\nThe images in the YesBut dataset were collected from social media platforms, particularly from the \"X\" (formerly Twitter) handle @_yesbut_. The original satirical images were manually annotated, and additional synthetic images were generated using DALL-E 3. These images were then manually labeled as satirical or non-satirical. The annotations include textual descriptions, binary features (e.g., presence of text), and difficulty ratings based on the difficulty of understanding satire/irony in the images. Data processing involved adding new variations of sub-images to expand the dataset's diversity.\r\n\r\n#### Who are the source data producers?\r\n\r\nPrior to annotation and expansion of the dataset, images were downloaded from the posts in ‘X’ (erstwhile known as Twitter) handle @_yesbut_ (with proper consent).\r\n\r\n#### Annotation process\r\n\r\nThe annotation process was carried out in four stages: (1) collecting satirical images from social media, (2) manual annotation of the images, (3) generating additional 2D stick figure images using DALL-E 3, and (4) generating 3D stick figure images.\r\n\r\n#### Who are the annotators?\r\n\r\nYesBut was curated by annotators who met the qualification criteria of being undergraduate sophomore students or above, enrolled in English-medium colleges.\r\n\r\n#### Personal and Sensitive Information\r\n\r\nThe images in YesBut Dataset do not include any personal identifiable information, and the annotations are general descriptions related to the satirical content of the images.\r\n\r\n## Bias, Risks, and Limitations\r\n\r\n- Subjectivity of annotations: The annotation task involves utilizing background knowledge that may differ among annotators. Consequently, we manually reviewed the annotations to minimize the number of incorrect annotations in the dataset. However, some subjectivity still remains. \r\n- Extension to languages other than English: This work is in the English Language. However, we plan to extend our work to languages other than English.\r\n\r\n<!-- ### Recommendations -->\r\n\r\n<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->\r\n\r\n<!-- Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations. -->\r\n\r\n## Citation\r\n\r\n<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->\r\n\r\n**BibTeX:**\r\n\r\n@article{nandy2024yesbut,\r\n  title={YesBut: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Models},\r\n  author={Nandy, Abhilash and Agarwal, Yash and Patwa, Ashish and Das, Millon Madhur and Bansal, Aman and Raj, Ankit and Goyal, Pawan and Ganguly, Niloy},\r\n  journal={arXiv preprint arXiv:2409.13592},\r\n  year={2024}\r\n}\r\n\r\n**APA:**\r\n\r\nNandy, A., Agarwal, Y., Patwa, A., Das, M. M., Bansal, A., Raj, A., ... & Ganguly, N. (2024). YesBut: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Models. arXiv preprint arXiv:2409.13592.\r\n\r\n## Dataset Card Contact\r\n\r\nGet in touch at nandyabhilash@gmail.com","description_withheld":null,"homepage":"https://yesbut-dataset.github.io","introduced_date":"2024-09-20","introduced_date_note":null,"introduced_by":{"paper":"/paper/yesbut-a-high-quality-annotated-multimodal","title":"YesBut: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Models","first_author":"Abhilash Nandy","url":null},"license":{"name":"Apache license 2.0","url":null},"modalities":[{"name":"Images","url":"/datasets/modality/images"},{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Satire Detection","url":"/task/satire-detection","datasets_with_task":"/datasets/task/satire-detection"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["YesBut"],"data_loaders":[],"num_papers_in_archive":1,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers":{"order":"date (newest first), then slug","population":"every paper with a leaderboard row on this dataset's benchmarks (the benchmark-backed subset): the archive's own papers-using-this-dataset list was never published, so this is not that list; num_papers_in_archive is the archive's own count","page":1,"pages_in_order":1,"rows_per_page":100,"shown":0,"of":0,"counts":{"papers_with_a_benchmark_row":0,"with_a_code_link":0,"where_syntology_ran_a_sample":0,"not_listed_spam_title":0,"listed":0,"listed_where_code_ran":0,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":0,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":0,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers with at least one leaderboard row on this dataset's benchmarks; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"items":[],"full_list":"papers_with_a_benchmark_row: every listed paper, in this order"},"papers_with_a_benchmark_row":[],"papers_where_code_ran":0,"papers_where_code_ran_first_page":null,"papers_where_code_ran_caption":"We ran code from the paper's repository; we did not run it on this dataset or check it against this dataset's benchmarks.","pages":[],"row_route":"each paper's syntology object carries sample_list (this site's paper page, #ran), syntology_url (syntology.ai's page for the paper, only when that route probed live) and mcp (the agent call that lists the paper's harvested samples; not a verdict on the paper)","developers":"https://syntology.ai/developers","syntology_totals":{"read_at":"2026-10-05T00:36:11+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_constructed":0,"samples_ran_checked":0,"samples_ran_instrument_failed":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"papers_with_a_sample_that_ran":0,"papers_with_a_run_with_no_instrument_failure":0,"papers_where_every_run_was_an_instrument_failure":0,"note":"the per-paper counts of every listed paper (all pages), summed; not a rate"},"key_notes":{"samples_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","samples_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"since_archive_judged":{"papers":0,"papers_placed":0,"judged_by":{"model":"a model","models_as_recorded":[],"from":"lb_rows_run_verified.jsonl: each record's `model` field (the extractor's call that wrote decisions[])"},"note":"a model's judgement from the paper's own tables that they report on this leaderboard, not a result: no value from a paper that was not placed is printed or carried here; papers counts each paper once over these leaderboards","backfill":"8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read)","backfill_complete":false,"leaderboards":[]},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks.","record_sha256":"395e1d7e781701b5cf860757e19327d3af4f13995c7a1c2bfcbc7e5ae8263ec9","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}