{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/model-card-and-evaluations-for-claude-models","title":"Model Card and Evaluations for Claude Models","arxiv_id":null,"date":"2023-07-11","proceeding":"Technical Report 2023 7","authors":["Anthropic"],"abstract":"This report includes the model card [1] for Claude models, focusing on Claude 2, along with the results of a range of safety, alignment, and capabilities evaluations. We have been iterating on the training and evaluation of Claude-type models since our first work on Reinforcement Learning from Human Feedback (RLHF) [2]; the newest Claude 2 model represents a continuous evolution from those early and less capable ‘helpful and harmless’ language assistants.\r\nThis report is not intended to be a scientific paper since most aspects of training and evaluating these models have been documented in our research papers. These include papers on preference modeling [3], reinforcement learning from human feedback for helpful and harmless models [2], red teaming language models [4], measuring representation of subjective global values in language models [5], honesty, (i.e., exploring language models’ ability to recognize what they know) [6], evaluating language models with language model-generated tests [7], moral self-correction [8], and Constitutional AI [9]. We also discussed Claude’s specific constitution in a recent blog post [10]. Our work using human evaluations to test model safety is most thoroughly documented in our paper “Red-Teaming Language Models to Reduce Harms” [4], while our recent work on automated safety evaluation is “Discovering Language Model Behaviors with Model-Written Evaluations” [7]. \r\nThis report is also not comprehensive – we expect to release new findings as we continue our research and evaluations of frontier models. However, we hope it provides useful insight into Claude 2’s capabilities and limitations.","url_abs":"https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf","url_pdf":"https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"arithmetic-reasoning","task_name":"Arithmetic Reasoning"},{"task_slug":"bug-fixing","task_name":"Bug fixing"},{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"multi-task-language-understanding","task_name":"Multi-task Language Understanding"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"red-teaming","task_name":"Red Teaming"},{"task_slug":"safety-alignment","task_name":"Safety Alignment"},{"task_slug":"model","task_name":"model"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"Claude 2 (0-shot chain-of-thought)","rank_in_archive_order":32,"of":164,"metrics":{"Accuracy":"88"},"uses_additional_data":false},{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"Claude 1.3 (0-shot chain-of-thought)","rank_in_archive_order":44,"of":164,"metrics":{"Accuracy":"85.2"},"uses_additional_data":false},{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"Claude Instant 1.1 (0-shot chain-of-thought)","rank_in_archive_order":68,"of":164,"metrics":{"Accuracy":"80.9"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-challenge","task":"Common Sense Reasoning","dataset":"ARC (Challenge)","model":"Claude 2 (few-shot, k=5)","rank_in_archive_order":5,"of":54,"metrics":{"Accuracy":"91"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-challenge","task":"Common Sense Reasoning","dataset":"ARC (Challenge)","model":"Claude 1.3 (few-shot, k=5)","rank_in_archive_order":6,"of":54,"metrics":{"Accuracy":"90"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-challenge","task":"Common Sense Reasoning","dataset":"ARC (Challenge)","model":"Claude Instant 1.1 (few-shot, k=5)","rank_in_archive_order":13,"of":54,"metrics":{"Accuracy":"85.7"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-quality","task":"Question Answering","dataset":"QuALITY","model":"Claude 1.3 (5-shot)","rank_in_archive_order":1,"of":4,"metrics":{"Accuracy":"84.1"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-quality","task":"Question Answering","dataset":"QuALITY","model":"Claude 2 (5-shot)","rank_in_archive_order":2,"of":4,"metrics":{"Accuracy":"83.2"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-quality","task":"Question Answering","dataset":"QuALITY","model":"Claude Instant 1.1 (5-shot)","rank_in_archive_order":4,"of":4,"metrics":{"Accuracy":"80.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"Claude 2 (few-shot, k=5)","rank_in_archive_order":1,"of":56,"metrics":{"EM":"87.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"Claude 1.3 (few-shot, k=5)","rank_in_archive_order":3,"of":56,"metrics":{"EM":"86.7"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"Claude Instant 1.1 (few-shot, k=5)","rank_in_archive_order":15,"of":56,"metrics":{"EM":"78.9"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"b6ceb20b12aa50ec2cc701cef1d9cbacde8a8ad4f902c5baa0128504a2549fd8","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}