{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/re-neural-networks-fail-to-learn-periodic","title":"[Re] Neural Networks Fail to Learn Periodic Functions and How to Fix It","arxiv_id":null,"date":"2021-01-31","proceeding":"RC 2020","authors":["Mayur Arvind","Mustansir Mama"],"abstract":"Scope of Reproducibility\n\nNeural Networks Fail to Learn Periodic Functions and How to Fix It [1] demonstrates experimentally that standard activations such as ReLU, tanh, sigmoid and their variants all fail to learn how to extrapolate simple periodic functions. The original paper goes on to propose a new activation, which the authors name the snake function. The central claims of the paper are two-fold: (1) The properties of the activation functions are carried over to the neural networks. A tanh network will be smooth and extrapolates to a constant function, while ReLU extrapolates in a linear way. Standard neural networks with conventional activation functions are insufficient for extrapolating periodic functions. (2) The proposed activation function manages to learn periodic functions while being able to optimize as well as conventional activation functions. While both experimental proof and theoretical justifications are provided for the claims, we shall only be concerned with testing the claims via experimental means.\n\nMethodology\n\nWhile one of the authors was contacted to clarify certain difficulties, the reproduction of all experiments was completed using only the information provided in the paper. With one exception, the links to all datasets used were also provided in the original paper. This allowed us to implement most experiments from scratch.\n\nResults\n\nWe were able to successfully replicate experiments supporting the central claim of the paper, that the proposed snake non-linearity can learn periodic functions. We also analyze the suitability of the snake activation for other tasks like generative modeling and sentiment analysis.\n\nWhat was easy\n\nMany experiments included descriptions of the neural network architectures and graphs showcasing performance, giving us a clear benchmark to compare our results against.\n\nLinks to datasets for all experiments, barring one, were also included in the paper itself.\n\nWhat was difficult\n\nData for the human body temperature experiment was not available. Proper implementation details were not given for initializing the weights in neural networks with snake and using snake with RNNs.\n\nCommunication with original authors\n\nLiu Ziyin, one of the authors, was contacted to provide the dataset used for the human body temperature experiment, elaborate upon the implementation of variance correction and provide the implementation of RNNs using snake. Liu provided the GitHub link to the authorsʼ original code for the human body temperature, market index, and extrapolation experiments. Liu also provided an explanation on how to implement variance correction. While the code for the RNN implementation using the snake activation was not made public, a screenshot of the same was provided. We thank the authors for their assistance.","url_abs":"http://rescience.github.io/bibliography/Arvind_2021.html","url_pdf":"https://zenodo.org/record/4833389/files/article.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"re-neural-networks-fail-to-learn-periodic","repo_url":"https://github.com/mayurak47/Reproducibility_Challenge","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"re-neural-networks-fail-to-learn-periodic","repo_url":"https://github.com/yingkaisha/keras-unet-collection","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[{"repo":"mayurak47/Reproducibility_Challenge","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2006.08195"],"harvested_for_calls":[{"arxiv_id":"2006.08195","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2006.08195"},"call":"get_harvested_code_for_paper(arxiv_id=\"2006.08195\")"}]},{"repo":"yingkaisha/keras-unet-collection","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":5,"harvested_for":["1802.06955","1804.03999","1904.00592","2005.09007","2006.08195"],"harvested_for_calls":[{"arxiv_id":"1802.06955","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1802.06955"},"call":"get_harvested_code_for_paper(arxiv_id=\"1802.06955\")"},{"arxiv_id":"1804.03999","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1804.03999"},"call":"get_harvested_code_for_paper(arxiv_id=\"1804.03999\")"},{"arxiv_id":"1904.00592","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1904.00592"},"call":"get_harvested_code_for_paper(arxiv_id=\"1904.00592\")"},{"arxiv_id":"2005.09007","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2005.09007"},"call":"get_harvested_code_for_paper(arxiv_id=\"2005.09007\")"},{"arxiv_id":"2006.08195","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2006.08195"},"call":"get_harvested_code_for_paper(arxiv_id=\"2006.08195\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"03a80e4dfee3d9c4a4aaac3d78a509194074cef4b533e3f73c2f39d8396029ff","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}