{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/transformers-as-algorithms-generalization-and","title":"Transformers as Algorithms: Generalization and Stability in In-context Learning","arxiv_id":"2301.07067","date":"2023-01-17","proceeding":null,"authors":["Yingcong Li","M. Emrullah Ildiz","Dimitris Papailiopoulos","Samet Oymak"],"abstract":"In-context learning (ICL) is a type of prompting where a transformer model operates on a sequence of (input, output) examples and performs inference on-the-fly. In this work, we formalize in-context learning as an algorithm learning problem where a transformer model implicitly constructs a hypothesis function at inference-time. We first explore the statistical aspects of this abstraction through the lens of multitask learning: We obtain generalization bounds for ICL when the input prompt is (1) a sequence of i.i.d. (input, label) pairs or (2) a trajectory arising from a dynamical system. The crux of our analysis is relating the excess risk to the stability of the algorithm implemented by the transformer. We characterize when transformer/attention architecture provably obeys the stability condition and also provide empirical verification. For generalization on unseen tasks, we identify an inductive bias phenomenon in which the transfer learning risk is governed by the task complexity and the number of MTL tasks in a highly predictable manner. Finally, we provide numerical evaluations that (1) demonstrate transformers can indeed implement near-optimal algorithms on classical regression problems with i.i.d. and dynamic data, (2) provide insights on stability, and (3) verify our theoretical predictions.","url_abs":"https://arxiv.org/abs/2301.07067v2","url_pdf":"https://arxiv.org/pdf/2301.07067v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"transformers-as-algorithms-generalization-and","repo_url":"https://github.com/yingcong-li/transformers-as-algorithms","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"transformers-as-algorithms-generalization-and","repo_url":"https://github.com/yingcong-li/dissecting-cot","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"generalization-bounds","task_name":"Generalization Bounds"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"model-selection","task_name":"Model Selection"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2301.07067","atlas_url":"https://app.syntology.ai/?focus=2301.07067","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2301.07067"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2301.07067\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2301.07067"},"call":"get_concepts_for_paper(arxiv_id=\"2301.07067\")"}],"by_repo":[{"repo":"yingcong-li/transformers-as-algorithms","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"yingcong-li/dissecting-cot","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"df16cfab00e3b7630b24cce303aef7c3ef0a7387e94675a3fe4ffef2b8c5a78b","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}