{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/understanding-the-disharmony-between-dropout","title":"Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift","arxiv_id":"1801.05134","date":"2018-01-16","proceeding":"CVPR 2019 6","authors":["Xiang Li","Shuo Chen","Xiaolin Hu","Jian Yang"],"abstract":"This paper first answers the question \"why do the two most powerful\ntechniques Dropout and Batch Normalization (BN) often lead to a worse\nperformance when they are combined together?\" in both theoretical and\nstatistical aspects. Theoretically, we find that Dropout would shift the\nvariance of a specific neural unit when we transfer the state of that network\nfrom train to test. However, BN would maintain its statistical variance, which\nis accumulated from the entire learning procedure, in the test phase. The\ninconsistency of that variance (we name this scheme as \"variance shift\") causes\nthe unstable numerical behavior in inference that leads to more erroneous\npredictions finally, when applying Dropout before BN. Thorough experiments on\nDenseNet, ResNet, ResNeXt and Wide ResNet confirm our findings. According to\nthe uncovered mechanism, we next explore several strategies that modifies\nDropout and try to overcome the limitations of their combination by avoiding\nthe variance shift risks.","url_abs":"http://arxiv.org/abs/1801.05134v1","url_pdf":"http://arxiv.org/pdf/1801.05134v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understanding-the-disharmony-between-dropout","repo_url":"https://github.com/m090009/Behavioral_Cloning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"understanding-the-disharmony-between-dropout","repo_url":"https://github.com/simo-bat/Crack_detection","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"understanding-the-disharmony-between-dropout","repo_url":"https://github.com/statsu1990/regression_dropout","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}},{"paper_slug":"understanding-the-disharmony-between-dropout","repo_url":"https://github.com/tcheung99/ResNet_MiniProject","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"understanding-the-disharmony-between-dropout","repo_url":"https://github.com/yang-su2000/ML-practice","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"grouped-convolution","method_name":"Grouped Convolution"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"resnext","method_name":"ResNeXt"},{"method_slug":"resnext-block","method_name":"ResNeXt Block"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1801.05134","atlas_url":"https://app.syntology.ai/?focus=1801.05134","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1801.05134"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1801.05134\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1801.05134"},"call":"get_concepts_for_paper(arxiv_id=\"1801.05134\")"}],"by_repo":[{"repo":"m090009/Behavioral_Cloning","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"simo-bat/Crack_detection","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"statsu1990/regression_dropout","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"tcheung99/ResNet_MiniProject","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"yang-su2000/ML-practice","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"818cb28c25c636ad5a733a9ba13d326a039cf8f41c4e55e1f244bfd1ae48103c","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}