{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/detecting-hate-speech-in-multi-modal-memes","title":"Detecting Hate Speech in Multi-modal Memes","arxiv_id":"2012.14891","date":"2020-12-29","proceeding":null,"authors":["Abhishek Das","Japsimar Singh Wahi","SiYao Li"],"abstract":"In the past few years, there has been a surge of interest in multi-modal problems, from image captioning to visual question answering and beyond. In this paper, we focus on hate speech detection in multi-modal memes wherein memes pose an interesting multi-modal fusion problem. We aim to solve the Facebook Meme Challenge \\cite{kiela2020hateful} which aims to solve a binary classification problem of predicting whether a meme is hateful or not. A crucial characteristic of the challenge is that it includes \"benign confounders\" to counter the possibility of models exploiting unimodal priors. The challenge states that the state-of-the-art models perform poorly compared to humans. During the analysis of the dataset, we realized that majority of the data points which are originally hateful are turned into benign just be describing the image of the meme. Also, majority of the multi-modal baselines give more preference to the hate speech (language modality). To tackle these problems, we explore the visual modality using object detection and image captioning models to fetch the \"actual caption\" and then combine it with the multi-modal representation to perform binary classification. This approach tackles the benign text confounders present in the dataset to improve the performance. Another approach we experiment with is to improve the prediction with sentiment analysis. Instead of only using multi-modal representations obtained from pre-trained neural networks, we also include the unimodal sentiment to enrich the features. We perform a detailed analysis of the above two approaches, providing compelling reasons in favor of the methodologies used.","url_abs":"https://arxiv.org/abs/2012.14891v1","url_pdf":"https://arxiv.org/pdf/2012.14891v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"detecting-hate-speech-in-multi-modal-memes","repo_url":"https://github.com/Abhishek0697/Detection-of-Hate-Speech-in-Multimodal-Memes","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"binary-classification","task_name":"Binary Classification"},{"task_slug":"hate-speech-detection","task_name":"Hate Speech Detection"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2012.14891","atlas_url":"https://app.syntology.ai/?focus=2012.14891","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2012.14891"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2012.14891\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2012.14891"},"call":"get_concepts_for_paper(arxiv_id=\"2012.14891\")"}],"by_repo":[{"repo":"Abhishek0697/Detection-of-Hate-Speech-in-Multimodal-Memes","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"db693b7843e828c4a06a09c08b42fac8b3e90bf7ed9eb5487cf73165450ccea5","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}