Browse State-of-the-Art › Masked Language Modeling
Masked Language Modeling
254 papers with code · 0 benchmarks · 7 datasets archive 2025-07-28
Benchmarks archive 2025-07-28
No benchmark for this task in the archive.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
7 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 475 · with a code link: 254 · where Syntology ran a sample: 67 (48 with a run with no instrument failure, 19 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (67 of 475 tagged: 48 with a run with no instrument failure, 19 where every run was a failure of Syntology's instrument)
Page 1 of 5: papers 1 to 100 of 475, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
23 Mar 2020 19 repositories listed Syntology official: no sample here; runs from other or unrecorded repositories · 31 ran (of which 7 constructed an object rather than computing a result; 18 with no instrument failure: 2 honoured, 2 violated, 14 with no contract checked; 13 where Syntology's instrument failed) · 9 unverified (of 40 harvested samples) · 10 pointer-only (licence)Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample…
-
20 Aug 2019 9 repositories listed Syntology community repositories only · 4 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 11 unverified; every one of the 4 samples that ran constructed an object rather than computing a result (of 15 harvested samples) · 3 pointer-only (licence)In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder.
-
20 Apr 2020 7 repositories listed Syntology official (archive's flag): 2 ran · 8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 8 harvested samples) · 6 pointer-only (licence)Since BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for pre-training to address this problem.
-
25 Oct 2019 7 repositories listed Syntology community repositories only · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples)This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions.
-
25 Sep 2019 7 repositories listed Syntology community repositories only · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 2 pointer-only (licence)Different from previous work that applies joint random masking to both modalities, we use conditional masking on pre-training tasks (i. e., masked language/region modeling is conditioned on full observation of…
-
3 Jul 2020 6 repositories listedWhile BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning (Reimers and Gurevych, 2019), BERT based cross-lingual sentence embeddings…
-
10 Feb 2020 6 repositories listed Syntology community repositories only · 4 ran (of which 1 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 2 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples)Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering.
-
21 Dec 2020 5 repositories listedTransformer is the backbone of modern NLP models.
-
18 Apr 2022 4 repositories listedIn this paper, we propose LayoutLMv3 to pre-train multimodal Transformers for Document AI with unified text and image masking.
-
7 Aug 2021 4 repositories listedIn particular, when compared to published models such as conformer-based wav2vec 2.
-
5 Mar 2020 4 repositories listed Syntology official: no sample here; runs from other or unrecorded repositories · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 2 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 2 pointer-only (licence)We introduce "talking-heads attention" - a variation on multi-head attention which includes linearprojections across the attention-heads dimension, immediately before and after the softmax operation.
-
16 Jun 2022 3 repositories listed Syntology official: harvested, nothing ran · 14 ran (of which 9 constructed an object rather than computing a result; 14 with no instrument failure: 1 honoured, 1 violated, 12 with no contract checked; 0 where Syntology's instrument failed) · 20 unverified (of 34 harvested samples) · 1 pointer-only (licence)Manual annotation of question and answers for videos, however, is tedious and prohibits scalability.
-
1 May 2020 3 repositories listed Syntology official (archive's flag): 8 ran · 10 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 2 honoured, 0 violated, 7 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified (of 13 harvested samples) · 8 pointer-only (licence)We present HERO, a novel framework for large-scale video+language omni-representation learning.
-
17 Mar 2020 3 repositories listedThis allows the coupling of the MLM as pre-training with a downstream anomaly detection task.
-
11 Jun 2024 2 repositories listed Syntology community repositories only · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling.
-
19 May 2023 2 repositories listed Syntology official (archive's flag): 14 ran · 14 ran (of which 0 constructed an object rather than computing a result; 12 with no instrument failure: 3 honoured, 1 violated, 8 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 16 harvested samples) · 1 pointer-only (licence)In this paper, we show that representations capturing syllabic units emerge when training a self-supervised speech model with a visually-grounded training objective.
-
24 Nov 2022 2 repositories listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 4 unverified (of 10 harvested samples) · 3 pointer-only (licence)Medical image visual question answering (VQA) is a task to answer clinical questions, given a radiographic image, which is a challenging problem that requires a model to integrate both vision and language information.
-
17 Oct 2022 2 repositories listed Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 1 honoured, 0 violated, 7 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified (of 18 harvested samples) · 1 pointer-only (licence)Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks.
-
11 Oct 2022 2 repositories listed Syntology official (archive's flag): 5 ran · 13 ran (of which 2 constructed an object rather than computing a result; 8 with no instrument failure: 1 honoured, 0 violated, 7 with no contract checked; 5 where Syntology's instrument failed) · 4 unverified (of 17 harvested samples) · 4 pointer-only (licence)This paper proposes the Mixture of Attention Heads (MoA), a new architecture that combines multi-head attention with the MoE mechanism.
-
22 Aug 2022 2 repositories listedA big convergence of language, vision, and multimodal pretraining is emerging.
-
21 Jul 2022 2 repositories listedWe find that our proposed pre-training methods help in modeling the data at a patient and population level and improve performance in different fine-tuning tasks on all datasets.
-
20 Jun 2022 2 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)Accurate prediction of Drug-Target Affinity (DTA) is of vital importance in early-stage drug discovery, facilitating the identification of drugs that can effectively interact with specific targets and regulate their…
-
11 May 2022 2 repositories listedHowever, the CNN feature maps still maintain the spatial relationship and we utilize this property to design self-supervised learning approaches to train the encoder of object detection transformers in pretraining and…
-
29 Apr 2022 2 repositories listed Syntology official: no sample here; runs from other or unrecorded repositories · 6 ran (of which 4 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples)In this paper, we specifically adapt vision-language joint learning for scene text detection, a task that intrinsically involves cross-modal interaction between the two modalities: vision and language, since text is the…
-
21 Feb 2022 2 repositories listedWe revisit the design choices in Transformers, and propose methods to address their weaknesses in handling long sequences.
-
29 Dec 2021 2 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)Then it diversifies the experts and continues to train the MoE with a novel Dense-to-Sparse gate (DTS-Gate).
-
15 Nov 2021 2 repositories listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)We present a self-supervised framework iBOT that can perform masked prediction with an online tokenizer.
-
14 Oct 2021 2 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples) · 2 pointer-only (licence)Both these masks can then be composed with the pretrained model.
-
4 Aug 2021 2 repositories listedTuning pre-trained language models (PLMs) with task-specific prompts has been a promising approach for text classification.
-
3 Jun 2021 2 repositories listed Syntology official (archive's flag): 4 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 5 unverified (of 10 harvested samples) · 7 pointer-only (licence)Specifically, with the first attention function, Luna packs the input sequence into a sequence of fixed length.
-
15 May 2021 2 repositories listed
-
2 Jan 2021 2 repositories listed
-
1 Dec 2020 2 repositories listed
-
23 Oct 2020 2 repositories listed
-
5 Oct 2020 2 repositories listed
-
31 Aug 2020 2 repositories listed
-
23 Jul 2020 2 repositories listed Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 3 honoured, 0 violated, 1 with no contract checked; 5 where Syntology's instrument failed) · 2 unverified (of 11 harvested samples) · 3 pointer-only (licence)
-
26 Jun 2020 2 repositories listed Syntology 15 ran (of which 0 constructed an object rather than computing a result; 12 with no instrument failure: 3 honoured, 2 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified (of 18 harvested samples) · 3 pointer-only (licence)
-
29 May 2020 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
10 Nov 2019 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
19 Apr 2019 2 repositories listed Syntology community repositories only · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
17 Jun 2025 1 repository listed
-
9 Jun 2025 1 repository listed
-
6 Jun 2025 1 repository listed
-
1 Jun 2025 1 repository listed
-
5 May 2025 1 repository listed
-
14 Mar 2025 1 repository listed
-
18 Feb 2025 1 repository listed
-
11 Feb 2025 1 repository listed
-
7 Feb 2025 1 repository listed
-
31 Oct 2024 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)
-
30 Oct 2024 1 repository listed
-
29 Oct 2024 1 repository listed
-
11 Oct 2024 1 repository listed
-
8 Oct 2024 1 repository listed
-
2 Oct 2024 1 repository listed
-
15 Sep 2024 1 repository listed
-
13 Sep 2024 1 repository listed
-
27 Aug 2024 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
20 Aug 2024 1 repository listed
-
13 Aug 2024 1 repository listed
-
29 Jul 2024 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
28 Jul 2024 1 repository listed
-
22 Jul 2024 1 repository listed
-
8 Jul 2024 1 repository listed
-
1 Jul 2024 1 repository listed
-
25 Jun 2024 1 repository listed
-
18 Jun 2024 1 repository listed
-
10 Jun 2024 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 6 where Syntology's instrument failed) · 0 unverified (of 10 harvested samples) · 10 pointer-only (licence)
-
9 Jun 2024 1 repository listed
-
18 May 2024 1 repository listed
-
14 May 2024 1 repository listed
-
27 Apr 2024 1 repository listed
-
26 Mar 2024 1 repository listed
-
1 Mar 2024 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
27 Feb 2024 1 repository listed
-
23 Feb 2024 1 repository listed
-
21 Feb 2024 1 repository listed
-
19 Feb 2024 1 repository listed
-
13 Feb 2024 1 repository listed
-
5 Feb 2024 1 repository listed
-
2 Feb 2024 1 repository listed
-
26 Jan 2024 1 repository listed
-
15 Jan 2024 1 repository listed
-
29 Dec 2023 1 repository listed
-
20 Dec 2023 1 repository listed
-
9 Dec 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
8 Dec 2023 1 repository listed
-
29 Oct 2023 1 repository listed
-
26 Oct 2023 1 repository listed
-
25 Oct 2023 1 repository listed
-
23 Oct 2023 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 11 harvested samples) · 11 pointer-only (licence)
-
22 Oct 2023 1 repository listed
-
20 Oct 2023 1 repository listed
-
15 Oct 2023 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples)
-
5 Oct 2023 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 3 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
15 Sep 2023 1 repository listed
-
28 Aug 2023 1 repository listed
-
23 Aug 2023 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples) · 7 pointer-only (licence)
-
22 Aug 2023 1 repository listed
Syntology lines on 35 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections