{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/orthogonal-weight-normalization-solution-to","title":"Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks","arxiv_id":"1709.06079","date":"2017-09-16","proceeding":null,"authors":["Lei Huang","Xianglong Liu","Bo Lang","Adams Wei Yu","Yongliang Wang","Bo Li"],"abstract":"Orthogonal matrix has shown advantages in training Recurrent Neural Networks\n(RNNs), but such matrix is limited to be square for the hidden-to-hidden\ntransformation in RNNs. In this paper, we generalize such square orthogonal\nmatrix to orthogonal rectangular matrix and formulating this problem in\nfeed-forward Neural Networks (FNNs) as Optimization over Multiple Dependent\nStiefel Manifolds (OMDSM). We show that the rectangular orthogonal matrix can\nstabilize the distribution of network activations and regularize FNNs. We also\npropose a novel orthogonal weight normalization method to solve OMDSM.\nParticularly, it constructs orthogonal transformation over proxy parameters to\nensure the weight matrix is orthogonal and back-propagates gradient information\nthrough the transformation during training. To guarantee stability, we minimize\nthe distortions between proxy parameters and canonical weights over all\ntractable orthogonal transformations. In addition, we design an orthogonal\nlinear module (OLM) to learn orthogonal filter banks in practice, which can be\nused as an alternative to standard linear module. Extensive experiments\ndemonstrate that by simply substituting OLM for standard linear module without\nrevising any experimental protocols, our method largely improves the\nperformance of the state-of-the-art networks, including Inception and residual\nnetworks on CIFAR and ImageNet datasets. In particular, we have reduced the\ntest error of wide residual network on CIFAR-100 from 20.04% to 18.61% with\nsuch simple substitution. Our code is available online for result reproduction.","url_abs":"http://arxiv.org/abs/1709.06079v2","url_pdf":"http://arxiv.org/pdf/1709.06079v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"orthogonal-weight-normalization-solution-to","repo_url":"https://github.com/huangleiBuaa/OthogonalWN","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"}],"methods":[{"method_slug":"weight-normalization","method_name":"Weight Normalization"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1709.06079","atlas_url":"https://app.syntology.ai/?focus=1709.06079","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1709.06079"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1709.06079\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1709.06079"},"call":"get_concepts_for_paper(arxiv_id=\"1709.06079\")"}],"by_repo":[{"repo":"huangleiBuaa/OthogonalWN","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"381d81c43a9f24586e5e15d286d595cc772a6be5c053b22790cb44e24284061c","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}