{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/adaptive-dropout","slug":"adaptive-dropout","name":"Adaptive Dropout","full_name":"Adaptive Dropout","full_name_withheld":false,"description_markdown":"**Adaptive Dropout** is a regularization technique that extends dropout by allowing the dropout probability to be different for different units. The intuition is that there may be hidden units that can individually make confident predictions for the presence or absence of an important feature or combination of features. [Dropout](https://paperswithcode.com/method/dropout) will ignore this confidence and drop the unit out 50% of the time. \r\n\r\nDenote the activity of unit $j$ in a deep neural network by $a\\_{j}$ and assume that its inputs are {$a\\_{i}: i < j$}. In dropout, $a\\_{j}$ is randomly set to zero with probability 0.5. Let $m\\_{j}$ be a binary variable that is used to mask, the activity $a\\_{j}$, so that its value is:\r\n\r\n$$ a\\_{j} = m\\_{j}g \\left( \\sum\\_{i: i<j}w\\_{j, i}a\\_{i} \\right)$$\r\n\r\nwhere $w\\_{j,i}$ is the weight from unit $i$ to unit $j$ and $g\\left(·\\right)$ is the activation function and $a\\_{0} = 1$ accounts for biases. Whereas in standard dropout, $m\\_{j}$ is Bernoulli with probability $0.5$, adaptive dropout uses adaptive dropout probabilities that depends on input activities:\r\n\r\n$$ P\\left(m\\_{j} = 1\\mid{\\{a\\_{i}: i < j\\}}\\right) = f \\left( \\sum\\_{i: i<j}\\pi{\\_{j, i}a\\_{i}} \\right) $$\r\n\r\nwhere $\\pi\\_{j, i}$ is the weight from unit $i$ to unit $j$ in the standout network or the adaptive dropout network; $f(·)$ is a sigmoidal function. Here 'standout' refers to a binary belief network is that is overlaid on a neural network as part of the overall regularization technique.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Adaptive dropout for training deep neural networks","paper":"/paper/adaptive-dropout-for-training-deep-neural","first_author":"Jimmy Ba","n_authors":2,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/adaptive-dropout-for-training-deep-neural"},"source":{"url":"http://papers.nips.cc/paper/5032-adaptive-dropout-for-training-deep-neural-networks","title":"Adaptive dropout for training deep neural networks","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/mabirck/adaptative-dropout-pytorch/blob/9c81ba607c1d9c4c9845ccaf4c7b2413d89c50c1/layers.py#L5","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Regularization","url":"/methods/category/regularization","pwc_aliases":[]}],"n_papers_tagged":15,"archive_num_papers":15,"papers_newest_first":[{"paper":null,"slug":"efficient-federated-learning-with","title":"Efficient Federated Learning with Heterogeneous Data and Adaptive Dropout","date":"2025-07-14","arxiv_id":"2507.10430","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-statistical-physics-framework-for-optimal","title":"A statistical physics framework for optimal learning","date":"2025-07-10","arxiv_id":"2507.07907","n_code_links":0,"syntology":null},{"paper":null,"slug":"dota-deformable-optimized-transformer","title":"DOTA: Deformable Optimized Transformer Architecture for End-to-End Text Recognition with Retrieval-Augmented Generation","date":"2025-05-07","arxiv_id":"2505.04175","n_code_links":0,"syntology":null},{"paper":null,"slug":"beyond-overfitting-doubly-adaptive-dropout","title":"Beyond Overfitting: Doubly Adaptive Dropout for Generalizable AU Detection","date":"2025-03-12","arxiv_id":"2503.08974","n_code_links":0,"syntology":null},{"paper":"/paper/dynamic-dropconnect-enhancing-neural-network","slug":"dynamic-dropconnect-enhancing-neural-network","title":"Dynamic DropConnect: Enhancing Neural Network Robustness through Adaptive Edge Dropping Strategies","date":"2025-02-27","arxiv_id":"2502.19948","n_code_links":1,"syntology":null},{"paper":null,"slug":"2-tier-simcse-elevating-bert-for-robust","title":"2-Tier SimCSE: Elevating BERT for Robust Sentence Embeddings","date":"2025-01-23","arxiv_id":"2501.13758","n_code_links":0,"syntology":null},{"paper":null,"slug":"adaptive-dropout-unleashing-dropout-across","title":"Adaptive Dropout: Unleashing Dropout across Layers for Generalizable Image Super-Resolution","date":"2025-01-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"adaptive-dropout-for-pruning-conformers","title":"Adaptive Dropout for Pruning Conformers","date":"2024-12-06","arxiv_id":"2412.04836","n_code_links":0,"syntology":null},{"paper":null,"slug":"communication-efficient-split-learning-via","title":"Communication-Efficient Split Learning via Adaptive Feature-Wise Compression","date":"2023-07-20","arxiv_id":"2307.10805","n_code_links":0,"syntology":null},{"paper":null,"slug":"fedbiad-communication-efficient-and-accuracy","title":"FedBIAD: Communication-Efficient and Accuracy-Guaranteed Federated Learning with Bayesian Inference-Based Adaptive Dropout","date":"2023-07-14","arxiv_id":"2307.07172","n_code_links":0,"syntology":null},{"paper":"/paper/the-flip-side-of-the-reweighted-coin-duality","slug":"the-flip-side-of-the-reweighted-coin-duality","title":"The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization","date":"2021-06-14","arxiv_id":"2106.07769","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":1,"n_instrument":0,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","official":{"repos":["dlej/adaptive-dropout"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":"/paper/advanced-dropout-a-model-free-methodology-for","slug":"advanced-dropout-a-model-free-methodology-for","title":"Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization","date":"2020-10-11","arxiv_id":"2010.05244","n_code_links":1,"syntology":null},{"paper":null,"slug":"adaptive-low-rank-factorization-to-regularize","title":"Adaptive Low-Rank Factorization to regularize shallow and deep neural networks","date":"2020-05-05","arxiv_id":"2005.01995","n_code_links":0,"syntology":null},{"paper":null,"slug":"improved-dropout-for-shallow-and-deep","title":"Improved Dropout for Shallow and Deep Learning","date":"2016-02-06","arxiv_id":"1602.02220","n_code_links":0,"syntology":null},{"paper":"/paper/adaptive-dropout-for-training-deep-neural","slug":"adaptive-dropout-for-training-deep-neural","title":"Adaptive dropout for training deep neural networks","date":"2013-12-01","arxiv_id":null,"n_code_links":0,"syntology":null}],"papers_shown":15,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":15,"with_a_code_link":3,"where_syntology_ran_a_sample":1,"not_listed_spam_title":0,"listed":15,"listed_where_code_ran":1,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":1,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":1,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":1,"papers_where_code_ran_first_page":"/method/adaptive-dropout/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/adaptive-dropout/papers/ran/1","json":"/method/adaptive-dropout/papers/ran/1.json","rows":1}],"tasks":[{"task":"/task/denoising","name":"Denoising","papers":2},{"task":"/task/federated-learning","name":"Federated Learning","papers":2},{"task":"/task/attribute","name":"Attribute","papers":1},{"task":"/task/bayesian-inference","name":"Bayesian Inference","papers":1},{"task":"/task/blind-super-resolution","name":"Blind Super-Resolution","papers":1},{"task":"/task/contrastive-learning","name":"Contrastive Learning","papers":1},{"task":"/task/deep-learning","name":"Deep Learning","papers":1},{"task":"/task/image-classification","name":"Image Classification","papers":1},{"task":"/task/image-restoration","name":"Image Restoration","papers":1},{"task":"/task/image-super-resolution","name":"Image Super-Resolution","papers":1},{"task":"/task/meta-learning","name":"Meta-Learning","papers":1},{"task":"/task/network-pruning","name":"Network Pruning","papers":1},{"task":"/task/optical-character-recognition","name":"Optical Character Recognition","papers":1},{"task":"/task/optical-character-recognition","name":"Optical Character Recognition (OCR)","papers":1},{"task":"/task/quantization","name":"Quantization","papers":1},{"task":"/task/retrieval-augmented-generation","name":"Retrieval-augmented Generation","papers":1},{"task":"/task/sts","name":"STS","papers":1},{"task":"/task/semantic-textual-similarity","name":"Semantic Textual Similarity","papers":1},{"task":"/task/sentence","name":"Sentence","papers":1},{"task":"/task/sentence-embeddings","name":"Sentence Embeddings","papers":1}],"tasks_shown":20,"n_tasks":30,"usage_by_year":[{"year":"2013","papers":1},{"year":"2016","papers":1},{"year":"2020","papers":2},{"year":"2021","papers":1},{"year":"2023","papers":2},{"year":"2024","papers":1},{"year":"2025","papers":7}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/adaptive-dropout"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=Adaptive%20Dropout","read_at":"2026-10-05T00:51:31+00:00","query":"Adaptive Dropout","kind":"method","have":"declared_code","tier":"repo_known","verified_implementation":false,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":{"method":"Adaptive Dropout","origin_arxiv_id":"2506.12738"},"resolution":"exact","resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":"unknown","archive_paper_arxiv_id":null,"repositories":{"listed":1,"in_graph":1,"list_truncated":false,"strongest_link":"listed_only"},"retrieve_with":null,"name_competition":{"verdict":"sole_bearer","may_be_wrong_sense":false,"bearers_found":1,"served":{"arxiv_id":"2506.12738","citations":6},"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing else in this corpus is named 'Adaptive Dropout'. That is a statement about this corpus and not about the field: short names are reused, and a sense we never collected cannot show up here. Confirm `served.arxiv_id` if you had a specific paper in mind."},"name_competition_text":null,"near_matches":null,"code_nearby":[{"name":"Gradient Sign Dropout","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":3,"retrieve_with":"get_reference_implementation(name='Gradient Sign Dropout')","origin_arxiv_id":"2010.06808"}],"code_nearby_caption":"Code Syntology holds under other names, as the lookup lists them (code_nearby); not a complete list, and none of them is this method.","origin_paper":{"arxiv_id":"2506.12738","page":null,"syntology_url":"https://syntology.ai/paper/2506.12738","in_graph_layer":false,"samples":0},"line":"repository known, no verified implementation (declared_code; repositories: 1 listed of 1 in the graph, strongest link: listed only). The archive credits no paper with an arXiv id for this method, so whether this answer (about arXiv:2506.12738) is about this method is not checked.","level_legend":{"state":"quoted","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","card_sha256":"479785a84658dd682a11b10237b3620d651d7f87799f5dc7add970aaa83f6d39","note":"Syntology's served verification levels, quoted verbatim from its MCP server card; a row is split where the card separates clauses with a double hyphen, and the separator is not reproduced","levels":[{"level":"V0","quoted":["imports"]},{"level":"V1","quoted":["executes against a synthesized fixture and returns a value of the recorded type"]},{"level":"V2","quoted":["two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs","neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)"]},{"level":"V3","quoted":["paper-derived property tests pass on both of those same two generations."]}],"no_rung":"No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results."}},"syntology_have_state":"loaded","record_sha256":"e3cf00c8e28e49a9a3076bc6276523b90cd8f69ff7ea55ecc96d7c1a5ad594d3","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}