{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/albert","slug":"albert","name":"ALBERT","full_name":"ALBERT","full_name_withheld":false,"description_markdown":"**ALBERT** is a [Transformer](https://paperswithcode.com/method/transformer) architecture based on [BERT](https://paperswithcode.com/method/bert) but with much fewer parameters. It achieves this through two parameter reduction techniques. The first is a factorized embeddings parameterization. By decomposing the large vocabulary embedding matrix into two small matrices, the size of the hidden layers is separated from the size of vocabulary embedding. This makes it easier to grow the hidden size without significantly increasing the parameter size of the vocabulary embeddings. The second technique is cross-layer parameter sharing. This technique prevents the parameter from growing with the depth of the network. \r\n\r\nAdditionally, ALBERT utilises a self-supervised loss for sentence-order prediction (SOP). SOP primary focuses on inter-sentence coherence and is designed to address the ineffectiveness of the next sentence prediction (NSP) loss proposed in the original BERT.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/1909.11942v6","title":"ALBERT: A Lite BERT for Self-supervised Learning of Language Representations","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Transformers","url":"/methods/category/transformers","pwc_aliases":[]}],"n_papers_tagged":172,"archive_num_papers":null,"papers_newest_first":[{"paper":null,"slug":"albert-advanced-localization-and","title":"ALBERT: Advanced Localization and Bidirectional Encoder Representations from Transformers for Automotive Damage Evaluation","date":"2025-06-12","arxiv_id":"2506.10524","n_code_links":0,"syntology":null},{"paper":null,"slug":"rapid-yet-accurate-tile-circuit-and-device","title":"Rapid yet accurate Tile-circuit and device modeling for Analog In-Memory Computing","date":"2025-05-05","arxiv_id":"2506.00004","n_code_links":0,"syntology":null},{"paper":"/paper/don-t-fight-hallucinations-use-them","slug":"don-t-fight-hallucinations-use-them","title":"Don't Fight Hallucinations, Use Them: Estimating Image Realism using NLI over Atomic Facts","date":"2025-03-20","arxiv_id":"2503.15948","n_code_links":1,"syntology":null},{"paper":null,"slug":"efficient-or-powerful-trade-offs-between","title":"Efficient or Powerful? Trade-offs Between Machine Learning and Deep Learning for Mental Illness Detection on Social Media","date":"2025-03-03","arxiv_id":"2503.01082","n_code_links":0,"syntology":null},{"paper":"/paper/robust-bias-detection-in-mlms-and-its","slug":"robust-bias-detection-in-mlms-and-its","title":"Robust Bias Detection in MLMs and its Application to Human Trait Ratings","date":"2025-02-21","arxiv_id":"2502.15600","n_code_links":1,"syntology":null},{"paper":null,"slug":"meursault-as-a-data-point","title":"Meursault as a Data Point","date":"2025-02-03","arxiv_id":"2502.01364","n_code_links":0,"syntology":null},{"paper":"/paper/punctuation-s-semantic-role-between-brain-and","slug":"punctuation-s-semantic-role-between-brain-and","title":"Aligning Brain Activity with Advanced Transformer Models: Exploring the Role of Punctuation in Semantic Processing","date":"2025-01-10","arxiv_id":"2501.06278","n_code_links":1,"syntology":null},{"paper":"/paper/a-comparative-analysis-of-transformer-and","slug":"a-comparative-analysis-of-transformer-and","title":"A Comparative Analysis of Transformer and LSTM Models for Detecting Suicidal Ideation on Reddit","date":"2024-11-23","arxiv_id":"2411.15404","n_code_links":1,"syntology":null},{"paper":"/paper/bert-based-approach-for-automating-course","slug":"bert-based-approach-for-automating-course","title":"BERT-Based Approach for Automating Course Articulation Matrix Construction with Explainable AI","date":"2024-11-21","arxiv_id":"2411.14254","n_code_links":1,"syntology":null},{"paper":"/paper/protransformer-robustify-transformers-via","slug":"protransformer-robustify-transformers-via","title":"ProTransformer: Robustify Transformers via Plug-and-Play Paradigm","date":"2024-10-30","arxiv_id":"2410.23182","n_code_links":1,"syntology":null},{"paper":null,"slug":"a-bayesian-perspective-on-the-maximum-score","title":"A Bayesian Perspective on the Maximum Score Problem","date":"2024-10-22","arxiv_id":"2410.17153","n_code_links":0,"syntology":null},{"paper":null,"slug":"meta-rtl-reinforcement-based-meta-transfer","title":"Meta-RTL: Reinforcement-Based Meta-Transfer Learning for Low-Resource Commonsense Reasoning","date":"2024-09-27","arxiv_id":"2409.19075","n_code_links":0,"syntology":null},{"paper":"/paper/profiling-patient-transcript-using-large","slug":"profiling-patient-transcript-using-large","title":"Profiling Patient Transcript Using Large Language Model Reasoning Augmentation for Alzheimer's Disease Detection","date":"2024-09-19","arxiv_id":"2409.12541","n_code_links":1,"syntology":null},{"paper":null,"slug":"biomner-a-dataset-for-biomedical-method","title":"BioMNER: A Dataset for Biomedical Method Entity Recognition","date":"2024-06-28","arxiv_id":"2406.20038","n_code_links":0,"syntology":null},{"paper":null,"slug":"concept-formation-and-alignment-in-language","title":"Concept Formation and Alignment in Language Models: Bridging Statistical Patterns in Latent Space to Concept Taxonomy","date":"2024-06-08","arxiv_id":"2406.05315","n_code_links":0,"syntology":null},{"paper":null,"slug":"effect-of-antibody-levels-on-the-spread-of","title":"Effect of antibody levels on the spread of disease in multiple infections","date":"2024-05-31","arxiv_id":"2405.20702","n_code_links":0,"syntology":null},{"paper":"/paper/ceebert-cross-domain-inference-in-early-exit","slug":"ceebert-cross-domain-inference-in-early-exit","title":"CEEBERT: Cross-Domain Inference in Early Exit BERT","date":"2024-05-23","arxiv_id":"2405.15039","n_code_links":1,"syntology":{"ran":2,"of":3,"n_ran_checked":0,"n_instrument":2,"unverified":1,"pointer_only":3,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified","official":{"repos":["Div290/CeeBERT"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"a-named-entity-recognition-and-topic-modeling","title":"A Named Entity Recognition and Topic Modeling-based Solution for Locating and Better Assessment of Natural Disasters in Social Media","date":"2024-05-01","arxiv_id":"2405.00903","n_code_links":0,"syntology":null},{"paper":null,"slug":"exploring-internal-numeracy-in-language","title":"Exploring Internal Numeracy in Language Models: A Case Study on ALBERT","date":"2024-04-25","arxiv_id":"2404.16574","n_code_links":0,"syntology":null},{"paper":"/paper/evaluating-subword-tokenization-alien-subword","slug":"evaluating-subword-tokenization-alien-subword","title":"Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge","date":"2024-04-20","arxiv_id":"2404.13292","n_code_links":1,"syntology":null},{"paper":null,"slug":"malbert-is-a-compact-multilingual-bert-model","title":"mALBERT: Is a Compact Multilingual BERT Model Still Worth It?","date":"2024-03-27","arxiv_id":"2403.18338","n_code_links":0,"syntology":null},{"paper":"/paper/an-exploratory-study-on-automatic","slug":"an-exploratory-study-on-automatic","title":"An Exploratory Study on Automatic Identification of Assumptions in the Development of Deep Learning Frameworks","date":"2024-01-08","arxiv_id":"2401.03653","n_code_links":1,"syntology":null},{"paper":"/paper/tensor-aware-energy-accounting","slug":"tensor-aware-energy-accounting","title":"Tensor-Aware Energy Accounting","date":"2023-11-19","arxiv_id":"2311.11424","n_code_links":1,"syntology":null},{"paper":null,"slug":"generative-ai-for-hate-speech-detection","title":"Generative AI for Hate Speech Detection: Evaluation and Findings","date":"2023-11-16","arxiv_id":"2311.09993","n_code_links":0,"syntology":null},{"paper":"/paper/amplify-attention-based-mixup-for-performance","slug":"amplify-attention-based-mixup-for-performance","title":"AMPLIFY:Attention-based Mixup for Performance Improvement and Label Smoothing in Transformer","date":"2023-09-22","arxiv_id":"2309.12689","n_code_links":1,"syntology":null},{"paper":"/paper/identification-of-the-relevance-of-comments","slug":"identification-of-the-relevance-of-comments","title":"Identification of the Relevance of Comments in Codes Using Bag of Words and Transformer Based Models","date":"2023-08-11","arxiv_id":"2308.06144","n_code_links":1,"syntology":null},{"paper":"/paper/performance-analysis-of-transformer-based","slug":"performance-analysis-of-transformer-based","title":"Performance Analysis of Transformer Based Models (BERT, ALBERT and RoBERTa) in Fake News Detection","date":"2023-08-09","arxiv_id":"2308.04950","n_code_links":1,"syntology":null},{"paper":null,"slug":"investigating-the-existence-of-secret","title":"Gradient-Based Word Substitution for Obstinate Adversarial Examples Generation in Language Models","date":"2023-07-24","arxiv_id":"2307.12507","n_code_links":0,"syntology":null},{"paper":null,"slug":"sparseoptimizer-sparsify-language-models","title":"SparseOptimizer: Sparsify Language Models through Moreau-Yosida Regularization and Accelerate via Compiler Co-design","date":"2023-06-27","arxiv_id":"2306.15656","n_code_links":0,"syntology":null},{"paper":null,"slug":"f-pabee-flexible-patience-based-early-exiting","title":"F-PABEE: Flexible-patience-based Early Exiting for Single-label and Multi-label text Classification Tasks","date":"2023-05-21","arxiv_id":"2305.11916","n_code_links":0,"syntology":null},{"paper":"/paper/larger-probes-tell-a-different-story","slug":"larger-probes-tell-a-different-story","title":"Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learning","date":"2023-03-29","arxiv_id":"2303.16445","n_code_links":1,"syntology":null},{"paper":null,"slug":"energy-efficient-task-adaptation-for-nlp-edge","title":"Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures","date":"2023-03-25","arxiv_id":"2303.16100","n_code_links":0,"syntology":null},{"paper":null,"slug":"elementwise-language-representation","title":"Elementwise Language Representation","date":"2023-02-27","arxiv_id":"2302.13475","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-stability-analysis-of-fine-tuning-a-pre","title":"A Stability Analysis of Fine-Tuning a Pre-Trained Model","date":"2023-01-24","arxiv_id":"2301.09820","n_code_links":0,"syntology":null},{"paper":null,"slug":"floods-relevancy-and-identification-of","title":"Floods Relevancy and Identification of Location from Twitter Posts using NLP Techniques","date":"2023-01-01","arxiv_id":"2301.00321","n_code_links":0,"syntology":null},{"paper":null,"slug":"learning-to-embed-adopting-transformer-based","title":"Learning-To-Embed: Adopting Transformer based models for E-commerce Products Representation Learning","date":"2022-12-07","arxiv_id":"2212.03725","n_code_links":0,"syntology":null},{"paper":null,"slug":"entity-assisted-language-models-for","title":"Entity-Assisted Language Models for Identifying Check-worthy Sentences","date":"2022-11-19","arxiv_id":"2211.10678","n_code_links":0,"syntology":null},{"paper":"/paper/dark-patterns-in-e-commerce-a-dataset-and-its","slug":"dark-patterns-in-e-commerce-a-dataset-and-its","title":"Dark patterns in e-commerce: a dataset and its baseline evaluations","date":"2022-11-12","arxiv_id":"2211.06543","n_code_links":1,"syntology":null},{"paper":null,"slug":"the-architectural-bottleneck-principle","title":"The Architectural Bottleneck Principle","date":"2022-11-11","arxiv_id":"2211.06420","n_code_links":0,"syntology":null},{"paper":"/paper/collateral-facilitation-in-humans-and","slug":"collateral-facilitation-in-humans-and","title":"Collateral facilitation in humans and language models","date":"2022-11-09","arxiv_id":"2211.05198","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":0,"n_instrument":1,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["jmichaelov/collateral-facilitation"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":"/paper/vote-n-rank-revision-of-benchmarking-with","slug":"vote-n-rank-revision-of-benchmarking-with","title":"Vote'n'Rank: Revision of Benchmarking with Social Choice Theory","date":"2022-10-11","arxiv_id":"2210.05769","n_code_links":1,"syntology":{"ran":6,"of":17,"n_ran_checked":6,"n_instrument":0,"unverified":11,"pointer_only":0,"phrase":"6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 11 unverified","official":{"repos":["pragmaticslab/vote_and_rank"],"state":"official (archive's flag): 6 ran","n_ran":6,"n_constructed":0,"n_ran_no_instrument_failure":6,"n_unverified":11,"ran_from_kinds":["official"]}}},{"paper":"/paper/bytetransformer-a-high-performance","slug":"bytetransformer-a-high-performance","title":"ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs","date":"2022-10-06","arxiv_id":"2210.03052","n_code_links":1,"syntology":null},{"paper":null,"slug":"deck-behavioral-tests-to-improve-1","title":"DECK: Behavioral Tests to Improve Interpretability and Generalizability of BERT Models Detecting Depression from Text","date":"2022-09-12","arxiv_id":"2209.05286","n_code_links":0,"syntology":null},{"paper":"/paper/addressing-token-uniformity-in-transformers","slug":"addressing-token-uniformity-in-transformers","title":"Addressing Token Uniformity in Transformers via Singular Value Transformation","date":"2022-08-24","arxiv_id":"2208.11790","n_code_links":1,"syntology":{"ran":2,"of":2,"n_ran_checked":0,"n_instrument":2,"unverified":0,"pointer_only":0,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","official":{"repos":["hanqi-qi/tokenuni"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"a-comparative-study-on-covid-19-fake-news","title":"A Comparative Study on COVID-19 Fake News Detection Using Different Transformer Based Models","date":"2022-08-02","arxiv_id":"2208.01355","n_code_links":0,"syntology":null},{"paper":null,"slug":"knowda-all-in-one-knowledge-mixture-model-for","title":"KnowDA: All-in-One Knowledge Mixture Model for Data Augmentation in Low-Resource NLP","date":"2022-06-21","arxiv_id":"2206.10265","n_code_links":0,"syntology":null},{"paper":null,"slug":"emoji-based-fine-grained-attention-network","title":"Emoji-based Fine-grained Attention Network for Sentiment Analysis in the Microblog Comments","date":"2022-06-10","arxiv_id":"2206.12262","n_code_links":0,"syntology":null},{"paper":"/paper/life-after-bert-what-do-other-muppets-2","slug":"life-after-bert-what-do-other-muppets-2","title":"Life after BERT: What do Other Muppets Understand about Language?","date":"2022-05-21","arxiv_id":"2205.10696","n_code_links":1,"syntology":null},{"paper":"/paper/albeto-and-distilbeto-lightweight-spanish","slug":"albeto-and-distilbeto-lightweight-spanish","title":"ALBETO and DistilBETO: Lightweight Spanish Language Models","date":"2022-04-19","arxiv_id":"2204.09145","n_code_links":2,"syntology":null},{"paper":null,"slug":"accelerating-attention-through-gradient-based","title":"Accelerating Attention through Gradient-Based Learned Runtime Pruning","date":"2022-04-07","arxiv_id":"2204.03227","n_code_links":0,"syntology":null},{"paper":"/paper/palbert-teaching-albert-to-ponder-1","slug":"palbert-teaching-albert-to-ponder-1","title":"PALBERT: Teaching ALBERT to Ponder","date":"2022-04-07","arxiv_id":"2204.03276","n_code_links":1,"syntology":{"ran":1,"of":2,"n_ran_checked":1,"n_instrument":0,"unverified":1,"pointer_only":0,"phrase":"1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; the one sample that ran constructed an object rather than computing a result","official":{"repos":["tinkoff-ai/palbert"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":1,"n_ran_no_instrument_failure":1,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"paying-more-attention-to-self-attention","title":"Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guiding","date":"2022-04-06","arxiv_id":"2204.02922","n_code_links":0,"syntology":null},{"paper":null,"slug":"anna-enhanced-language-representation-for-1","title":"ANNA: Enhanced Language Representation for Question Answering","date":"2022-03-28","arxiv_id":"2203.14507","n_code_links":0,"syntology":null},{"paper":null,"slug":"mono-vs-multilingual-bert-a-case-study-in","title":"Mono vs Multilingual BERT: A Case Study in Hindi and Marathi Named Entity Recognition","date":"2022-03-24","arxiv_id":"2203.12907","n_code_links":0,"syntology":null},{"paper":null,"slug":"detecting-and-diagnosing-terrestrial","title":"Detecting and Diagnosing Terrestrial Gravitational-Wave Mimics Through Feature Learning","date":"2022-03-09","arxiv_id":"2203.05086","n_code_links":0,"syntology":null},{"paper":"/paper/does-transliteration-help-multilingual","slug":"does-transliteration-help-multilingual","title":"Does Transliteration Help Multilingual Language Modeling?","date":"2022-01-29","arxiv_id":"2201.12501","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":0,"n_instrument":1,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["ibraheem-moosa/xlm-indic"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"an-application-of-pseudo-log-likelihoods-to","title":"An Application of Pseudo-Log-Likelihoods to Natural Language Scoring","date":"2022-01-23","arxiv_id":"2201.09377","n_code_links":0,"syntology":null},{"paper":null,"slug":"bridge-the-gap-between-cv-and-nlp-a-gradient-1","title":"Bridge the Gap Between CV and NLP! A Gradient-based Textual Adversarial Attack Framework","date":"2022-01-16","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"deck-behavioral-tests-to-improve","title":"DECK: Behavioral Tests to Improve Interpretability and Generalizability of BERT Models Detecting Depression from Text","date":"2022-01-16","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"multi-stage-pre-training-for-math","title":"Multi-Stage Pre-Training for Math-Understanding: $\\mu^2$(AL)BERT","date":"2022-01-16","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"pcee-bert-accelerating-bert-inference-via","title":"PCEE-BERT: Accelerating BERT Inference via Patient and Confident Early Exiting","date":"2022-01-16","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"a-comparative-study-of-transformers-on-word","title":"A Comparative Study of Transformers on Word Sense Disambiguation","date":"2021-11-30","arxiv_id":"2111.15417","n_code_links":0,"syntology":null},{"paper":"/paper/how-emotionally-stable-is-albert-testing-1","slug":"how-emotionally-stable-is-albert-testing-1","title":"How Emotionally Stable is ALBERT? Testing Robustness with Stochastic Weight Averaging on a Sentiment Analysis Task","date":"2021-11-18","arxiv_id":"2111.09612","n_code_links":1,"syntology":null},{"paper":null,"slug":"anna-enhanced-language-representation-for","title":"ANNA: Enhanced Language Representation for Question Answering","date":"2021-11-16","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"life-after-bert-what-do-other-muppets","title":"Life after BERT: What do Other Muppets Understand about Language?","date":"2021-11-16","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"palbert-teaching-albert-to-ponder","title":"PALBERT: Teaching ALBERT to Ponder","date":"2021-11-16","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"understanding-attention-in-machine-reading-1","title":"Understanding Attention in Machine Reading Comprehension","date":"2021-11-16","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/character-level-hypernetworks-for-hate-speech","slug":"character-level-hypernetworks-for-hate-speech","title":"Character-level HyperNetworks for Hate Speech Detection","date":"2021-11-11","arxiv_id":"2111.06336","n_code_links":1,"syntology":null},{"paper":null,"slug":"the-golden-rule-as-a-heuristic-to-measure-the","title":"The Golden Rule as a Heuristic to Measure the Fairness of Texts Using Machine Learning","date":"2021-10-29","arxiv_id":"2111.00107","n_code_links":0,"syntology":null},{"paper":"/paper/bridge-the-gap-between-cv-and-nlp-a-gradient","slug":"bridge-the-gap-between-cv-and-nlp-a-gradient","title":"Bridge the Gap Between CV and NLP! A Gradient-based Textual Adversarial Attack Framework","date":"2021-10-28","arxiv_id":"2110.15317","n_code_links":1,"syntology":null},{"paper":"/paper/ensemble-albert-on-squad-2-0","slug":"ensemble-albert-on-squad-2-0","title":"Ensemble ALBERT on SQuAD 2.0","date":"2021-10-19","arxiv_id":"2110.09665","n_code_links":1,"syntology":null},{"paper":"/paper/local-and-global-context-based-pairwise","slug":"local-and-global-context-based-pairwise","title":"Local and Global Context-Based Pairwise Models for Sentence Ordering","date":"2021-10-08","arxiv_id":"2110.04291","n_code_links":1,"syntology":null},{"paper":null,"slug":"transliteration-a-simple-technique-for","title":"Transliteration: A Simple Technique For Improving Multilingual Language Modeling","date":"2021-09-29","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"fine-tuned-transformers-show-clusters-of","title":"Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers","date":"2021-09-17","arxiv_id":"2109.08406","n_code_links":0,"syntology":null},{"paper":"/paper/transformer-based-language-models-for-factoid","slug":"transformer-based-language-models-for-factoid","title":"Transformer-based Language Models for Factoid Question Answering at BioASQ9b","date":"2021-09-15","arxiv_id":"2109.07185","n_code_links":1,"syntology":null},{"paper":null,"slug":"so-cloze-yet-so-far-n400-amplitude-is-better","title":"So Cloze yet so Far: N400 Amplitude is Better Predicted by Distributional Information than Human Predictability Judgements","date":"2021-09-02","arxiv_id":"2109.01226","n_code_links":0,"syntology":null},{"paper":null,"slug":"fight-fire-with-fire-fine-tuning-hate","title":"Fight Fire with Fire: Fine-tuning Hate Detectors using Large Samples of Generated Hate Speech","date":"2021-09-01","arxiv_id":"2109.00591","n_code_links":0,"syntology":null},{"paper":"/paper/understanding-attention-in-machine-reading","slug":"understanding-attention-in-machine-reading","title":"Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models","date":"2021-08-26","arxiv_id":"2108.11574","n_code_links":1,"syntology":null},{"paper":"/paper/bertac-enhancing-transformer-based-language","slug":"bertac-enhancing-transformer-based-language","title":"BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks","date":"2021-08-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"slug":"deepblueai-at-semeval-2021-task-7-detecting","title":"DeepBlueAI at SemEval-2021 Task 7: Detecting and Rating Humor and Offense with Stacking Diverse Language Model-Based Methods","date":"2021-08-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"gulu-at-semeval-2021-task-7-detecting-and","title":"Gulu at SemEval-2021 Task 7: Detecting and Rating Humor and Offense","date":"2021-08-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"how-many-layers-and-why-an-analysis-of-the","title":"How Many Layers and Why? An Analysis of the Model Depth in Transformers","date":"2021-08-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"hub-at-semeval-2021-task-7-fusion-of-albert","title":"hub at SemEval-2021 Task 7: Fusion of ALBERT and Word Frequency Information Detecting and Rating Humor and Offense","date":"2021-08-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/ynu-hpcc-at-semeval-2021-task-5-using-a","slug":"ynu-hpcc-at-semeval-2021-task-5-using-a","title":"YNU-HPCC at SemEval-2021 Task 5: Using a Transformer-based Model with Auxiliary Information for Toxic Span Detection","date":"2021-08-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"slug":"ynu-hpcc-at-semeval-2021-task-6-combining","title":"YNU-HPCC at SemEval-2021 Task 6: Combining ALBERT and Text-CNN for Persuasion Detection in Texts and Images","date":"2021-08-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"zyj-at-semeval-2021-task-7-hahackathon","title":"ZYJ at SemEval-2021 Task 7: HaHackathon: Detecting and Rating Humor and Offense with ALBERT-Based Model","date":"2021-08-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/go-wider-instead-of-deeper","slug":"go-wider-instead-of-deeper","title":"Go Wider Instead of Deeper","date":"2021-07-25","arxiv_id":"2107.11817","n_code_links":1,"syntology":null},{"paper":null,"slug":"digital-einstein-experience-fast-text-to","title":"Digital Einstein Experience: Fast Text-to-Speech for Conversational AI","date":"2021-07-21","arxiv_id":"2107.10658","n_code_links":0,"syntology":null},{"paper":null,"slug":"elbert-fast-albert-with-confidence-window","title":"Elbert: Fast Albert with Confidence-Window Based Early Exit","date":"2021-07-01","arxiv_id":"2107.00175","n_code_links":0,"syntology":null},{"paper":"/paper/benchmarking-differential-privacy-and","slug":"benchmarking-differential-privacy-and","title":"Benchmarking Differential Privacy and Federated Learning for BERT Models","date":"2021-06-26","arxiv_id":"2106.13973","n_code_links":1,"syntology":null},{"paper":"/paper/distributed-deep-learning-in-open","slug":"distributed-deep-learning-in-open","title":"Distributed Deep Learning in Open Collaborations","date":"2021-06-18","arxiv_id":"2106.10207","n_code_links":2,"syntology":{"ran":0,"of":5,"n_ran_checked":0,"n_instrument":0,"unverified":5,"pointer_only":0,"phrase":"0 ran · 5 unverified","official":{"repos":["learning-at-home/hivemind","yandex-research/DeDLOC"],"state":"official: harvested, nothing ran","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":5,"ran_from_kinds":[]}}},{"paper":"/paper/cblue-a-chinese-biomedical-language","slug":"cblue-a-chinese-biomedical-language","title":"CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark","date":"2021-06-15","arxiv_id":"2106.08087","n_code_links":2,"syntology":{"ran":11,"of":16,"n_ran_checked":8,"n_instrument":3,"unverified":5,"pointer_only":2,"phrase":"11 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 1 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 5 unverified","official":{"repos":["cbluebenchmark/cblue"],"state":"official (archive's flag): 11 ran","n_ran":11,"n_constructed":0,"n_ran_no_instrument_failure":8,"n_unverified":5,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"why-can-you-lay-off-heads-investigating-how","title":"Why Can You Lay Off Heads? Investigating How BERT Heads Transfer","date":"2021-06-14","arxiv_id":"2106.07137","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-semi-supervised-multi-task-learning","title":"A Semi-supervised Multi-task Learning Approach to Classify Customer Contact Intents","date":"2021-06-10","arxiv_id":"2106.07381","n_code_links":0,"syntology":null},{"paper":null,"slug":"transient-chaos-in-bert","title":"Transient Chaos in BERT","date":"2021-06-06","arxiv_id":"2106.03181","n_code_links":0,"syntology":null},{"paper":"/paper/bert-based-sentiment-analysis-a-software","slug":"bert-based-sentiment-analysis-a-software","title":"BERT-Based Sentiment Analysis: A Software Engineering Perspective","date":"2021-06-04","arxiv_id":"2106.02581","n_code_links":2,"syntology":null},{"paper":"/paper/the-case-for-translation-invariant-self","slug":"the-case-for-translation-invariant-self","title":"The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models","date":"2021-06-03","arxiv_id":"2106.01950","n_code_links":3,"syntology":{"ran":1,"of":1,"n_ran_checked":1,"n_instrument":0,"unverified":0,"pointer_only":1,"phrase":"1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result","official":{"repos":["ulmewennberg/tisa"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":1,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":"/paper/icompass-at-nlp4if-2021-fighting-the-covid-19","slug":"icompass-at-nlp4if-2021-fighting-the-covid-19","title":"iCompass at NLP4IF-2021–Fighting the COVID-19 Infodemic","date":"2021-06-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"slug":"towards-a-comprehensive-understanding-and","title":"Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers","date":"2021-06-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"neural-models-for-offensive-language","title":"Neural Models for Offensive Language Detection","date":"2021-05-30","arxiv_id":"2106.14609","n_code_links":0,"syntology":null}],"papers_shown":100,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":172,"with_a_code_link":68,"where_syntology_ran_a_sample":14,"not_listed_spam_title":0,"listed":172,"listed_where_code_ran":14,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":8,"every_run_a_failure_of_syntologys_instrument":6,"listed_with_a_run_with_no_instrument_failure":8,"listed_every_run_a_failure_of_syntologys_instrument":6,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":14,"papers_where_code_ran_first_page":"/method/albert/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"archive","page":2,"of":2,"url":"/method/albert/papers/2","json":"/method/albert/papers/2.json","rows":72},{"order":"ran","page":1,"of":1,"url":"/method/albert/papers/ran/1","json":"/method/albert/papers/ran/1.json","rows":14}],"tasks":[{"task":"/task/language-modelling","name":"Language Modelling","papers":29},{"task":"/task/language-modeling","name":"Language Modeling","papers":24},{"task":"/task/sentence","name":"Sentence","papers":23},{"task":"/task/text-classification","name":"Text Classification","papers":15},{"task":"/task/sentiment-analysis","name":"Sentiment Analysis","papers":14},{"task":"/task/question-answering","name":"Question Answering","papers":13},{"task":"/task/text-classification-1","name":"text-classification","papers":13},{"task":"/task/named-entity-recognition-ner","name":"Named Entity Recognition (NER)","papers":10},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":10},{"task":"/task/named-entity-recognition-1","name":"Named Entity Recognition","papers":9},{"task":"/task/cg","name":"NER","papers":8},{"task":"/task/named-entity-recognition","name":"named-entity-recognition","papers":8},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":7},{"task":"/task/natural-language-inference","name":"Natural Language Inference","papers":7},{"task":"/task/reading-comprehension","name":"Reading Comprehension","papers":7},{"task":"/task/classification","name":"General Classification","papers":6},{"task":"/task/machine-reading-comprehension","name":"Machine Reading Comprehension","papers":6},{"task":"/task/natural-language-understanding","name":"Natural Language Understanding","papers":6},{"task":"/task/word-embeddings","name":"Word Embeddings","papers":6},{"task":"/task/classification-1","name":"Classification","papers":5}],"tasks_shown":20,"n_tasks":167,"usage_by_year":[{"year":"2019","papers":2},{"year":"2020","papers":46},{"year":"2021","papers":63},{"year":"2022","papers":26},{"year":"2023","papers":13},{"year":"2024","papers":15},{"year":"2025","papers":7}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/albert"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=ALBERT","read_at":"2026-10-05T00:51:48+00:00","query":"ALBERT","kind":"method","have":"declared_code","tier":"repo_known","verified_implementation":false,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":{"method":"ALBERT","origin_arxiv_id":"1909.11942"},"resolution":"exact","resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":"unknown","archive_paper_arxiv_id":null,"repositories":{"listed":5,"in_graph":48,"list_truncated":true,"strongest_link":"official"},"retrieve_with":null,"name_competition":{"verdict":"contested","may_be_wrong_sense":false,"bearers_found":2,"served":{"arxiv_id":"1909.11942","citations":7671},"leading_other_bearer":{"arxiv_id":"2506.10524","title":"ALBERT: Advanced Localization and Bidirectional Encoder Representations from Transformers for Automotive Damage Evaluation","citations":null,"named_by":["title_prefix"],"bearer_name":"ALBERT: Advanced Localization and Bidirectional Encoder Representations from Transformers for Automotive Damage Evaluation","ask_with":"have('2506.10524')"},"n_other_bearers_listed":1,"note":"1 other thing(s) here are also called 'ALBERT', and none of them is cited more than the paper this answer is about. They are listed in `also_named` -- check `served.arxiv_id` is the paper you meant."},"name_competition_text":null,"near_matches":null,"code_nearby":[],"code_nearby_caption":null,"origin_paper":{"arxiv_id":"1909.11942","page":"/paper/albert-a-lite-bert-for-self-supervised","syntology_url":"https://syntology.ai/paper/1909.11942","in_graph_layer":true,"samples":126,"ran":81,"n_ran_checked":56,"n_instrument":25,"unverified":45,"phrase":"81 ran (of which 17 constructed an object rather than computing a result; 56 with no instrument failure: 4 honoured, 0 violated, 52 with no contract checked; 25 where Syntology's instrument failed) · 45 unverified","instrument_means":"failures of Syntology's instrument, not of the code"},"line":"repository known, no verified implementation (declared_code; repositories: 5 listed of 48 in the graph, strongest link: official). The archive credits no paper with an arXiv id for this method, so whether this answer (about arXiv:1909.11942) is about this method is not checked.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"8f8c1dc05d19c5bd746af9909630459b5267017f4038da954ec5455fddbdea9d","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}