{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/effectiveness-of-scaled-exponentially","title":"Effectiveness of Scaled Exponentially-Regularized Linear Units (SERLUs)","arxiv_id":"1807.10117","date":"2018-07-26","proceeding":null,"authors":["G. Zhang","H. Li"],"abstract":"Recently, self-normalizing neural networks (SNNs) have been proposed with the\nintention to avoid batch or weight normalization. The key step in SNNs is to\nproperly scale the exponential linear unit (referred to as SELU) to inherently\nincorporate normalization based on central limit theory. SELU is a\nmonotonically increasing function, where it has an approximately constant\nnegative output for large negative input. In this work, we propose a new\nactivation function to break the monotonicity property of SELU while still\npreserving the self-normalizing property. Differently from SELU, the new\nfunction introduces a bump-shaped function in the region of negative input by\nregularizing a linear function with a scaled exponential function, which is\nreferred to as a scaled exponentially-regularized linear unit (SERLU). The\nbump-shaped function has approximately zero response to large negative input\nwhile being able to push the output of SERLU towards zero mean statistically.\nTo effectively combat over-fitting, we develop a so-called shift-dropout for\nSERLU, which includes standard dropout as a special case. Experimental results\non MNIST, CIFAR10 and CIFAR100 show that SERLU-based neural networks provide\nconsistently promising results in comparison to other 5 activation functions\nincluding ELU, SELU, Swish, Leakly ReLU and ReLU.","url_abs":"http://arxiv.org/abs/1807.10117v2","url_pdf":"http://arxiv.org/pdf/1807.10117v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[],"methods":[{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"elu","method_name":"ELU"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"serlu","method_name":"SERLU"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"}],"datasets_introduced":[],"methods_introduced":[{"slug":"serlu","name":"SERLU","full_name":"SERLU"}],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"137c0ead4c824c040e0efaefee9dabc2c0f9e879ed323b3ce4f48e6a866554a0","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}