{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/us-aisi-and-uk-aisi-joint-pre-deployment-test-1","title":"US AISI and UK AISI Joint Pre-Deployment Test: Anthropic’s Claude 3.5 Sonnet (October 2024 Release)","arxiv_id":null,"date":"2024-11-19","proceeding":"NIST 2024 11","authors":["US AI Safety Institute","UK AI Safety Institute"],"abstract":"This technical report details a pre-deployment evaluation of Anthropic’s upgraded version of Claude 3.5 Sonnet, released October 22, 2024 (hereafter referred to as Sonnet 3.5 (new)). This evaluation was conducted jointly by the United States Artificial Intelligence Safety Institute (US AISI) and the United Kingdom Artificial Intelligence Safety Institute (UK AISI), and this report describes in detail its technical methodology and findings. For general background and a summary of this report, see the corresponding blog post. US AISI and UK AISI’s joint pre-deployment evaluation assessed four domains: biological capabilities, cyber capabilities, software and AI development capabilities, and safeguard effectiveness. US AISI and UK AISI each ran independent tests on Sonnet 3.5 (new), working together to inform and improve methodology and interpretation of findings. US AISI and UK AISI shared their initial findings with Anthropic prior to the model’s release. The following sections introduce each evaluation domain jointly and present specific technical descriptions, methodologies, and findings in each domain as specific to either US AISI or UK AISI, as appropriate.","url_abs":"https://www.nist.gov/news-events/news/2024/11/pre-deployment-evaluation-anthropics-upgraded-claude-35-sonnet","url_pdf":"https://www.nist.gov/system/files/documents/2024/11/19/Upgraded%20Sonnet-Publication-US.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/on-cybench","task":"","dataset":"Cybench","model":"Claude 3.5 Sonnet (old, US AISI scaffold, pass@10)","rank_in_archive_order":1,"of":3,"metrics":{"Unguided Performance":"35%"},"uses_additional_data":false},{"leaderboard":"/sota/on-cybench","task":"","dataset":"Cybench","model":"o1-preview (US AISI scaffold, pass@10)","rank_in_archive_order":2,"of":3,"metrics":{"Unguided Performance":"35%"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"5aff78a47a47632a327f0696e4e629b2e300b48072400d8be7f489f60b71089e","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}