{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/kaleidoscope-in-language-exams-for-massively","title":"Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation","arxiv_id":"2504.07072","date":"2025-04-09","proceeding":null,"authors":["Israfel Salazar","Manuel Fernández Burda","Shayekh Bin Islam","Arshia Soltani Moakhar","Shivalika Singh","Fabian Farestam","Angelika Romanou","Danylo Boiko","Dipika Khullar","Mike Zhang","Dominik Krzemiński","Jekaterina Novikova","Luísa Shimabucoro","Joseph Marvin Imperial","Rishabh Maheshwary","Sharad Duwal","Alfonso Amayuelas","Swati Rajwal","Jebish Purbey","Ahmed Ruby","Nicholas Popovič","Marek Suppa","Azmine Toushik Wasi","Ram Mohan Rao Kadiyala","Olga Tsymboi","Maksim Kostritsya","Bardia Soltani Moakhar","Gabriel da Costa Merlin","Otávio Ferracioli Coletti","Maral Jabbari Shiviari","MohammadAmin farahani fard","Silvia Fernandez","María Grandury","Dmitry Abulkhanov","Drishti Sharma","Andre Guarnier de Mitri","Leticia Bossatto Marchezi","Setayesh Heydari","Johan Obando-Ceron","Nazar Kohut","Beyza Ermis","Desmond Elliott","Enzo Ferrante","Sara Hooker","Marzieh Fadaee"],"abstract":"The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and languages, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks.","url_abs":"https://arxiv.org/abs/2504.07072v2","url_pdf":"https://arxiv.org/pdf/2504.07072v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"kaleidoscope-in-language-exams-for-massively","repo_url":"https://github.com/israfelsr/kaleidoscope","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"multiple-choice","task_name":"Multiple-choice"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[{"slug":"kaleidoscope","name":"Kaleidoscope","full_name":"Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2504.07072","atlas_url":"https://app.syntology.ai/?focus=2504.07072","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2504.07072"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2504.07072\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2504.07072"},"call":"get_concepts_for_paper(arxiv_id=\"2504.07072\")"}],"by_repo":[{"repo":"israfelsr/kaleidoscope","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"e07b8b9c613db63397e1a7ba2b402c3dab7aec8b4c9bfea8933b3c5797eb98b9","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}