{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/holistic-evaluation-of-language-models","title":"Holistic Evaluation of Language Models","arxiv_id":"2211.09110","date":"2022-11-16","proceeding":null,"authors":["Percy Liang","Rishi Bommasani","Tony Lee","Dimitris Tsipras","Dilara Soylu","Michihiro Yasunaga","Yian Zhang","Deepak Narayanan","Yuhuai Wu","Ananya Kumar","Benjamin Newman","Binhang Yuan","Bobby Yan","Ce Zhang","Christian Cosgrove","Christopher D. Manning","Christopher Ré","Diana Acosta-Navas","Drew A. Hudson","Eric Zelikman","Esin Durmus","Faisal Ladhak","Frieda Rong","Hongyu Ren","Huaxiu Yao","Jue Wang","Keshav Santhanam","Laurel Orr","Lucia Zheng","Mert Yuksekgonul","Mirac Suzgun","Nathan Kim","Neel Guha","Niladri Chatterji","Omar Khattab","Peter Henderson","Qian Huang","Ryan Chi","Sang Michael Xie","Shibani Santurkar","Surya Ganguli","Tatsunori Hashimoto","Thomas Icard","Tianyi Zhang","Vishrav Chaudhary","William Wang","Xuechen Li","Yifan Mai","Yuhui Zhang","Yuta Koreeda"],"abstract":"Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models. First, we taxonomize the vast space of potential scenarios (i.e. use cases) and metrics (i.e. desiderata) that are of interest for LMs. Then we select a broad subset based on coverage and feasibility, noting what's missing or underrepresented (e.g. question answering for neglected English dialects, metrics for trustworthiness). Second, we adopt a multi-metric approach: We measure 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency) for each of 16 core scenarios when possible (87.5% of the time). This ensures metrics beyond accuracy don't fall to the wayside, and that trade-offs are clearly exposed. We also perform 7 targeted evaluations, based on 26 targeted scenarios, to analyze specific aspects (e.g. reasoning, disinformation). Third, we conduct a large-scale evaluation of 30 prominent language models (spanning open, limited-access, and closed models) on all 42 scenarios, 21 of which were not previously used in mainstream LM evaluation. Prior to HELM, models on average were evaluated on just 17.9% of the core HELM scenarios, with some prominent models not sharing a single scenario in common. We improve this to 96.0%: now all 30 models have been densely benchmarked on the same core scenarios and metrics under standardized conditions. Our evaluation surfaces 25 top-level findings. For full transparency, we release all raw model prompts and completions publicly for further analysis, as well as a general modular toolkit. We intend for HELM to be a living benchmark for the community, continuously updated with new scenarios, metrics, and models.","url_abs":"https://arxiv.org/abs/2211.09110v2","url_pdf":"https://arxiv.org/pdf/2211.09110v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"holistic-evaluation-of-language-models","repo_url":"https://github.com/stanford-crfm/helm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"holistic-evaluation-of-language-models","repo_url":"https://github.com/IBM/helm-enterprise-benchmark","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"holistic-evaluation-of-language-models","repo_url":"https://github.com/for-ai/llm-profiling-toolkit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"fairness","task_name":"Fairness"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[{"slug":"helm","name":"HELM","full_name":"Holistic Evaluation of Language Models"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2211.09110","atlas_url":"https://app.syntology.ai/?focus=2211.09110","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2211.09110"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2211.09110\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2211.09110"},"call":"get_concepts_for_paper(arxiv_id=\"2211.09110\")"}],"by_repo":[{"repo":"stanford-crfm/helm","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2410.22456"],"harvested_for_calls":[{"arxiv_id":"2410.22456","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2410.22456"},"call":"get_harvested_code_for_paper(arxiv_id=\"2410.22456\")"}]},{"repo":"IBM/helm-enterprise-benchmark","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"for-ai/llm-profiling-toolkit","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"4090b4d4d29f2517c5cfaf5ffd78bf0a7226603d2cf9c1506555699053dfded0","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}