{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/adaptive-masking","slug":"adaptive-masking","name":"Adaptive Masking","full_name":"Adaptive Masking","full_name_withheld":false,"description_markdown":"**Adaptive Masking** is a type of attention mechanism that allows a model to learn its own context size to attend over. For each head in [Multi-Head Attention](https://paperswithcode.com/method/multi-head-attention), a masking function is added to control for the span of the attention. A masking function is a non-increasing function that maps a\r\ndistance to a value in $\\left[0, 1\\right]$. Adaptive masking takes the following soft masking function $m\\_{z}$ parametrized by a real value $z$ in $\\left[0, S\\right]$:\r\n\r\n$$ m\\_{z}\\left(x\\right) = \\min\\left[\\max\\left[\\frac{1}{R}\\left(R+z-x\\right), 0\\right], 1\\right] $$\r\n\r\nwhere $R$ is a hyper-parameter that controls its softness. The shape of this piecewise function as a function of the distance. This soft masking function is inspired by [Jernite et al. (2017)](https://arxiv.org/abs/1611.06188). The attention weights from are then computed on the masked span:\r\n\r\n$$ a\\_{tr} = \\frac{m\\_{z}\\left(t-r\\right)\\exp\\left(s\\_{tr}\\right)}{\\sum^{t-1}\\_{q=t-S}m\\_{z}\\left(t-q\\right)\\exp\\left(s\\_{tq}\\right)}$$\r\n\r\nA $\\mathcal{l}\\_{1}$ penalization is added on the parameters $z\\_{i}$ for each attention head $i$ of the model to the loss function:\r\n\r\n$$ L = - \\log{P}\\left(w\\_{1}, \\dots, w\\_{T}\\right) + \\frac{\\lambda}{M}\\sum\\_{i}z\\_{i} $$\r\n\r\nwhere $\\lambda > 0$ is the regularization hyperparameter, and $M$ is the number of heads in each\r\nlayer. This formulation is differentiable in the parameters $z\\_{i}$, and learnt jointly with the rest of the model.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/1905.07799v2","title":"Adaptive Attention Span in Transformers","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/facebookresearch/adaptive-span/blob/349f7a27a822803239b7a6ad9df06e1175275a64/adaptive_span.py#L17","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Attention Mechanisms","url":"/methods/category/attention-mechanisms","pwc_aliases":["attention-mechanisms-1"]}],"n_papers_tagged":31,"archive_num_papers":null,"papers_newest_first":[{"paper":null,"slug":"preti-patient-aware-retinal-foundation-model","title":"PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learning","date":"2025-05-18","arxiv_id":"2505.12233","n_code_links":0,"syntology":null},{"paper":null,"slug":"spat-sensitivity-based-multihead-attention","title":"SPAT: Sensitivity-based Multihead-attention Pruning on Time Series Forecasting Models","date":"2025-05-13","arxiv_id":"2505.08768","n_code_links":0,"syntology":null},{"paper":"/paper/high-frequency-prior-driven-adaptive-masking","slug":"high-frequency-prior-driven-adaptive-masking","title":"High-Frequency Prior-Driven Adaptive Masking for Accelerating Image Super-Resolution","date":"2025-05-11","arxiv_id":"2505.06975","n_code_links":1,"syntology":null},{"paper":"/paper/bqsched-a-non-intrusive-scheduler-for-batch","slug":"bqsched-a-non-intrusive-scheduler-for-batch","title":"BQSched: A Non-intrusive Scheduler for Batch Concurrent Queries via Reinforcement Learning","date":"2025-04-27","arxiv_id":"2504.19142","n_code_links":1,"syntology":null},{"paper":null,"slug":"from-gaze-to-insight-bridging-human-visual","title":"From Gaze to Insight: Bridging Human Visual Attention and Vision Language Model Explanation for Weakly-Supervised Medical Image Segmentation","date":"2025-04-15","arxiv_id":"2504.11368","n_code_links":0,"syntology":null},{"paper":"/paper/adaptive-rank-allocation-speeding-up-modern","slug":"adaptive-rank-allocation-speeding-up-modern","title":"Adaptive Rank Allocation: Speeding Up Modern Transformers with RaNA Adapters","date":"2025-03-23","arxiv_id":"2503.18216","n_code_links":1,"syntology":{"ran":3,"of":4,"n_ran_checked":2,"n_instrument":1,"unverified":1,"pointer_only":0,"phrase":"3 ran (of which 1 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified","official":{"repos":["roberto09/rana"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":1,"n_ran_no_instrument_failure":2,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":"/paper/hac-towards-100x-compression-of-3d-gaussian","slug":"hac-towards-100x-compression-of-3d-gaussian","title":"HAC++: Towards 100X Compression of 3D Gaussian Splatting","date":"2025-01-21","arxiv_id":"2501.12255","n_code_links":2,"syntology":null},{"paper":null,"slug":"efficient-masked-autoencoder-for-video-object","title":"Efficient Masked AutoEncoder for Video Object Counting and A Large-Scale Benchmark","date":"2024-11-20","arxiv_id":"2411.13056","n_code_links":0,"syntology":null},{"paper":"/paper/eh-mam-easy-to-hard-masked-acoustic-modeling","slug":"eh-mam-easy-to-hard-masked-acoustic-modeling","title":"EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning","date":"2024-10-17","arxiv_id":"2410.13179","n_code_links":1,"syntology":{"ran":6,"of":10,"n_ran_checked":6,"n_instrument":0,"unverified":4,"pointer_only":2,"phrase":"6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified","official":{"repos":["cs20s030/ehmam"],"state":"official (archive's flag): 6 ran","n_ran":6,"n_constructed":0,"n_ran_no_instrument_failure":6,"n_unverified":4,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"adaptive-masking-enhances-visual-grounding","title":"Adaptive Masking Enhances Visual Grounding","date":"2024-10-04","arxiv_id":"2410.03161","n_code_links":0,"syntology":null},{"paper":"/paper/pre-training-on-high-definition-x-ray-images","slug":"pre-training-on-high-definition-x-ray-images","title":"Pre-training on High Definition X-ray Images: An Experimental Study","date":"2024-04-27","arxiv_id":"2404.17926","n_code_links":1,"syntology":null},{"paper":null,"slug":"salience-based-adaptive-masking-revisiting","title":"Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training","date":"2024-04-12","arxiv_id":"2404.08327","n_code_links":0,"syntology":null},{"paper":null,"slug":"retentive-decision-transformer-with-adaptive","title":"Retentive Decision Transformer with Adaptive Masking for Reinforcement Learning based Recommendation Systems","date":"2024-03-26","arxiv_id":"2403.17634","n_code_links":0,"syntology":null},{"paper":"/paper/hac-hash-grid-assisted-context-for-3d","slug":"hac-hash-grid-assisted-context-for-3d","title":"HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compression","date":"2024-03-21","arxiv_id":"2403.14530","n_code_links":2,"syntology":null},{"paper":null,"slug":"exploiting-adaptive-contextual-masking-for","title":"Exploiting Adaptive Contextual Masking for Aspect-Based Sentiment Analysis","date":"2024-02-21","arxiv_id":"2402.13722","n_code_links":0,"syntology":null},{"paper":"/paper/a-novel-hybrid-time-varying-graph-neural","slug":"a-novel-hybrid-time-varying-graph-neural","title":"A novel hybrid time-varying graph neural network for traffic flow forecasting","date":"2024-01-17","arxiv_id":"2401.10155","n_code_links":0,"syntology":null},{"paper":null,"slug":"dynamic-asr-pathways-an-adaptive-masking","title":"Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Model","date":"2023-09-22","arxiv_id":"2309.13018","n_code_links":0,"syntology":null},{"paper":null,"slug":"amlp-adaptive-masking-lesion-patches-for-self","title":"AMLP:Adaptive Masking Lesion Patches for Self-supervised Medical Image Segmentation","date":"2023-09-08","arxiv_id":"2309.04312","n_code_links":0,"syntology":null},{"paper":"/paper/amom-adaptive-masking-over-masking-for","slug":"amom-adaptive-masking-over-masking-for","title":"AMOM: Adaptive Masking over Masking for Conditional Masked Language Model","date":"2023-03-13","arxiv_id":"2303.07457","n_code_links":1,"syntology":null},{"paper":null,"slug":"mps-ams-masked-patches-selection-and-adaptive","title":"MPS-AMS: Masked Patches Selection and Adaptive Masking Strategy Based Self-Supervised Medical Image Segmentation","date":"2023-02-27","arxiv_id":"2302.13699","n_code_links":0,"syntology":null},{"paper":"/paper/revisiting-pre-training-in-audio-visual","slug":"revisiting-pre-training-in-audio-visual","title":"Revisiting Pre-training in Audio-Visual Learning","date":"2023-02-07","arxiv_id":"2302.03533","n_code_links":1,"syntology":null},{"paper":null,"slug":"hat-gae-self-supervised-graph-auto-encoders","title":"HAT-GAE: Self-Supervised Graph Auto-encoders with Hierarchical Adaptive Masking and Trainable Corruption","date":"2023-01-28","arxiv_id":"2301.12063","n_code_links":0,"syntology":null},{"paper":"/paper/adamae-adaptive-masking-for-efficient","slug":"adamae-adaptive-masking-for-efficient","title":"AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders","date":"2022-11-16","arxiv_id":"2211.09120","n_code_links":2,"syntology":{"ran":18,"of":25,"n_ran_checked":13,"n_instrument":5,"unverified":7,"pointer_only":6,"phrase":"18 ran (of which 0 constructed an object rather than computing a result; 13 with no instrument failure: 1 honoured, 0 violated, 12 with no contract checked; 5 where Syntology's instrument failed) · 7 unverified","official":{"repos":["wgcban/adamae"],"state":"official (archive's flag): 14 ran","n_ran":14,"n_constructed":0,"n_ran_no_instrument_failure":10,"n_unverified":1,"ran_from_kinds":["listed","official"]}}},{"paper":"/paper/student-collaboration-improves-self","slug":"student-collaboration-improves-self","title":"Multiplexed Immunofluorescence Brain Image Analysis Using Self-Supervised Dual-Loss Adaptive Masked Autoencoder","date":"2022-05-10","arxiv_id":"2205.05194","n_code_links":1,"syntology":null},{"paper":"/paper/unsupervised-representation-learning-of-2","slug":"unsupervised-representation-learning-of-2","title":"Unsupervised Representation Learning of Player Behavioral Data with Confidence Guided Masking","date":"2022-04-25","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/generative-pre-training-from-molecules","slug":"generative-pre-training-from-molecules","title":"Generative Pre-Training from Molecules","date":"2021-09-16","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/self-attention-between-datapoints-going","slug":"self-attention-between-datapoints-going","title":"Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning","date":"2021-06-04","arxiv_id":"2106.02584","n_code_links":3,"syntology":{"ran":22,"of":30,"n_ran_checked":20,"n_instrument":2,"unverified":8,"pointer_only":0,"phrase":"22 ran (of which 4 constructed an object rather than computing a result; 20 with no instrument failure: 2 honoured, 1 violated, 17 with no contract checked; 2 where Syntology's instrument failed) · 8 unverified","official":{"repos":["OATML/Non-Parametric-Transformers"],"state":"official (archive's flag): 6 ran","n_ran":6,"n_constructed":4,"n_ran_no_instrument_failure":6,"n_unverified":5,"ran_from_kinds":["listed","official"]}}},{"paper":null,"slug":"uncertainty-maximization-in-partially","title":"Uncertainty Maximization in Partially Observable Domains: A Cognitive Perspective","date":"2021-02-22","arxiv_id":"2102.11232","n_code_links":0,"syntology":null},{"paper":"/paper/improving-transformer-models-by-reordering","slug":"improving-transformer-models-by-reordering","title":"Improving Transformer Models by Reordering their Sublayers","date":"2019-11-10","arxiv_id":"1911.03864","n_code_links":2,"syntology":null},{"paper":"/paper/augmenting-self-attention-with-persistent","slug":"augmenting-self-attention-with-persistent","title":"Augmenting Self-attention with Persistent Memory","date":"2019-07-02","arxiv_id":"1907.01470","n_code_links":2,"syntology":{"ran":5,"of":5,"n_ran_checked":5,"n_instrument":0,"unverified":0,"pointer_only":4,"phrase":"5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 3 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","official":null}},{"paper":"/paper/adaptive-attention-span-in-transformers","slug":"adaptive-attention-span-in-transformers","title":"Adaptive Attention Span in Transformers","date":"2019-05-19","arxiv_id":"1905.07799","n_code_links":8,"syntology":null}],"papers_shown":31,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":31,"with_a_code_link":17,"where_syntology_ran_a_sample":5,"not_listed_spam_title":0,"listed":31,"listed_where_code_ran":5,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":5,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":5,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":5,"papers_where_code_ran_first_page":"/method/adaptive-masking/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/adaptive-masking/papers/ran/1","json":"/method/adaptive-masking/papers/ran/1.json","rows":5}],"tasks":[{"task":"/task/language-modelling","name":"Language Modelling","papers":7},{"task":"/task/representation-learning","name":"Representation Learning","papers":7},{"task":"/task/language-modeling","name":"Language Modeling","papers":6},{"task":"/task/self-supervised-learning","name":"Self-Supervised Learning","papers":4},{"task":"/task/image-segmentation","name":"Image Segmentation","papers":3},{"task":"/task/medical-image-segmentation","name":"Medical Image Segmentation","papers":3},{"task":"/task/segmentation","name":"Segmentation","papers":3},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":3},{"task":"/task/translation","name":"Translation","papers":3},{"task":"/task/3dgs","name":"3DGS","papers":2},{"task":"/task/attribute","name":"Attribute","papers":2},{"task":"/task/contrastive-learning","name":"Contrastive Learning","papers":2},{"task":"/task/decoder","name":"Decoder","papers":2},{"task":"/task/diagnostic","name":"Diagnostic","papers":2},{"task":"/task/feature-engineering","name":"Feature Engineering","papers":2},{"task":"/task/machine-translation","name":"Machine Translation","papers":2},{"task":"/task/novel-view-synthesis","name":"Novel View Synthesis","papers":2},{"task":"/task/quantization","name":"Quantization","papers":2},{"task":"/task/speech-recognition","name":"Speech Recognition","papers":2},{"task":"/task/speech-recognition-1","name":"speech-recognition","papers":2}],"tasks_shown":20,"n_tasks":64,"usage_by_year":[{"year":"2019","papers":3},{"year":"2021","papers":3},{"year":"2022","papers":3},{"year":"2023","papers":6},{"year":"2024","papers":9},{"year":"2025","papers":7}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/adaptive-masking"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=Adaptive%20Masking","read_at":"2026-10-05T00:51:34+00:00","query":"Adaptive Masking","kind":"method","have":"no_code","tier":"task_routable","verified_implementation":false,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":{"method":"masking","origin_arxiv_id":"2004.12406"},"resolution":"token_match","resolved_name_differs":true,"absence_asserted":"for_this_exact_name_only","absence_unchecked":null,"no_code_is_about_this_method":false,"subject":"unknown","archive_paper_arxiv_id":null,"repositories":{"listed":null,"in_graph":0,"list_truncated":null,"strongest_link":null},"retrieve_with":null,"name_competition":{"verdict":"sole_bearer","may_be_wrong_sense":false,"bearers_found":1,"served":{"arxiv_id":"2004.12406","citations":138},"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing else in this corpus is named 'Adaptive Masking'. That is a statement about this corpus and not about the field: short names are reused, and a sense we never collected cannot show up here. Confirm `served.arxiv_id` if you had a specific paper in mind."},"name_competition_text":null,"near_matches":null,"code_nearby":[],"code_nearby_caption":null,"origin_paper":{"arxiv_id":"2004.12406","page":null,"syntology_url":"https://syntology.ai/paper/2004.12406","in_graph_layer":false,"samples":0},"line":"code not checked for this name. The lookup matched a different name in Syntology's graph, 'masking' (resolution: token_match), and its answer is about that name only, so it does not say whether this method has code. The archive credits no paper with an arXiv id for this method, so whether this answer (about arXiv:2004.12406) is about this method is not checked.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"2f164d0b9303a3021d4323391946bced5192db557a1e28b9a6078776cf590692","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}