{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/critical-learning-periods-in-deep-neural","title":"Critical Learning Periods in Deep Neural Networks","arxiv_id":"1711.08856","date":"2017-11-24","proceeding":null,"authors":["Alessandro Achille","Matteo Rovere","Stefano Soatto"],"abstract":"Similar to humans and animals, deep artificial neural networks exhibit\ncritical periods during which a temporary stimulus deficit can impair the\ndevelopment of a skill. The extent of the impairment depends on the onset and\nlength of the deficit window, as in animal models, and on the size of the\nneural network. Deficits that do not affect low-level statistics, such as\nvertical flipping of the images, have no lasting effect on performance and can\nbe overcome with further training. To better understand this phenomenon, we use\nthe Fisher Information of the weights to measure the effective connectivity\nbetween layers of a network during training. Counterintuitively, information\nrises rapidly in the early phases of training, and then decreases, preventing\nredistribution of information resources in a phenomenon we refer to as a loss\nof \"Information Plasticity\". Our analysis suggests that the first few epochs\nare critical for the creation of strong connections that are optimal relative\nto the input data distribution. Once such strong connections are created, they\ndo not appear to change during additional training. These findings suggest that\nthe initial learning transient, under-scrutinized compared to asymptotic\nbehavior, plays a key role in determining the outcome of the training process.\nOur findings, combined with recent theoretical results in the literature, also\nsuggest that forgetting (decrease of information in the weights) is critical to\nachieving invariance and disentanglement in representation learning. Finally,\ncritical periods are not restricted to biological systems, but can emerge\nnaturally in learning systems, whether biological or artificial, due to\nfundamental constrains arising from learning dynamics and information\nprocessing.","url_abs":"http://arxiv.org/abs/1711.08856v3","url_pdf":"http://arxiv.org/pdf/1711.08856v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"critical-learning-periods-in-deep-neural","repo_url":"https://github.com/uw-mad-dash/Accordion","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"disentanglement","task_name":"Disentanglement"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1711.08856","atlas_url":"https://app.syntology.ai/?focus=1711.08856","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1711.08856"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1711.08856\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1711.08856"},"call":"get_concepts_for_paper(arxiv_id=\"1711.08856\")"}],"by_repo":[{"repo":"uw-mad-dash/Accordion","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"df01d41dfb6d7748a99dae9134ba98e7f7b7df85a44433bbaadad40956210a2f","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}