{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/ulmfit","slug":"ulmfit","name":"ULMFiT","full_name":"Universal Language Model Fine-tuning","full_name_withheld":false,"description_markdown":"**Universal Language Model Fine-tuning**, or **ULMFiT**, is an architecture and transfer learning method that can be applied to NLP tasks. It involves a 3-layer [AWD-LSTM](https://paperswithcode.com/method/awd-lstm) architecture for its representations. The training consists of three steps: 1) general language model pre-training on a Wikipedia-based text, 2) fine-tuning the language model on a target task, and 3) fine-tuning the classifier on the target task.\r\n\r\nAs different layers capture different types of information, they are fine-tuned to different extents using [discriminative fine-tuning](https://paperswithcode.com/method/discriminative-fine-tuning). Training is performed using [Slanted triangular learning rates](https://paperswithcode.com/method/slanted-triangular-learning-rates) (STLR), a learning rate scheduling strategy that first linearly increases the learning rate and then linearly decays it.\r\n\r\nFine-tuning the target classifier is achieved in ULMFiT using gradual unfreezing. Rather than fine-tuning all layers at once, which risks catastrophic forgetting, ULMFiT gradually unfreezes the model starting from the last layer (i.e., closest to the output) as this contains the least general knowledge. First the last layer is unfrozen and all unfrozen layers are fine-tuned for one epoch. Then the next group of frozen layers is unfrozen and fine-tuned and repeat, until all layers are fine-tuned until convergence at the last iteration.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Universal Language Model Fine-tuning for Text Classification","paper":"/paper/universal-language-model-fine-tuning-for-text","first_author":"Jeremy Howard","n_authors":2,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/universal-language-model-fine-tuning-for-text"},"source":{"url":"http://arxiv.org/abs/1801.06146v5","title":"Universal Language Model Fine-tuning for Text Classification","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Language Models","url":"/methods/category/language-models","pwc_aliases":[]}],"n_papers_tagged":40,"archive_num_papers":40,"papers_newest_first":[{"paper":null,"slug":"advanced-deep-learning-techniques-for","title":"Advanced Deep Learning Techniques for Analyzing Earnings Call Transcripts: Methodologies and Applications","date":"2025-02-27","arxiv_id":"2503.01886","n_code_links":0,"syntology":null},{"paper":null,"slug":"no-argument-left-behind-overlapping-chunks","title":"No Argument Left Behind: Overlapping Chunks for Faster Processing of Arbitrarily Long Legal Texts","date":"2024-10-24","arxiv_id":"2410.19184","n_code_links":0,"syntology":null},{"paper":"/paper/rico-reddit-ideological-communities","slug":"rico-reddit-ideological-communities","title":"RICo: Reddit ideological communities","date":"2024-06-05","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/exploring-multi-level-threats-in-telegram","slug":"exploring-multi-level-threats-in-telegram","title":"Exploring Multi-Level Threats in Telegram Data with AI-Human Annotation: A Preliminary Study","date":"2023-12-15","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"illicit-darkweb-classification-via-natural","title":"Illicit Darkweb Classification via Natural-language Processing: Classifying Illicit Content of Webpages based on Textual Information","date":"2023-12-08","arxiv_id":"2312.04944","n_code_links":0,"syntology":null},{"paper":null,"slug":"explainable-and-high-performance-hate-and","title":"Explainable and High-Performance Hate and Offensive Speech Detection","date":"2022-06-26","arxiv_id":"2206.12983","n_code_links":0,"syntology":null},{"paper":"/paper/iiitt-dravidian-codemix-fire2021","slug":"iiitt-dravidian-codemix-fire2021","title":"IIITT@Dravidian-CodeMix-FIRE2021: Transliterate or translate? Sentiment analysis of code-mixed text in Dravidian languages","date":"2021-11-15","arxiv_id":"2111.07906","n_code_links":1,"syntology":null},{"paper":"/paper/offensive-language-identification-in-low","slug":"offensive-language-identification-in-low","title":"Offensive Language Identification in Low-resourced Code-mixed Dravidian languages using Pseudo-labeling","date":"2021-08-27","arxiv_id":"2108.12177","n_code_links":1,"syntology":null},{"paper":"/paper/sn-computer-science-towards-offensive","slug":"sn-computer-science-towards-offensive","title":"Towards Offensive Language Identification for Tamil Code-Mixed YouTube Comments and Posts","date":"2021-08-24","arxiv_id":"2108.10939","n_code_links":1,"syntology":null},{"paper":null,"slug":"learning-ulmfit-and-self-distillation-with","title":"Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System","date":"2021-07-20","arxiv_id":"2107.09625","n_code_links":0,"syntology":null},{"paper":"/paper/whose-heritage-classification-of-unesco-world","slug":"whose-heritage-classification-of-unesco-world","title":"WHOSe Heritage: Classification of UNESCO World Heritage \"Outstanding Universal Value\" Documents with Soft Labels","date":"2021-04-12","arxiv_id":"2104.05547","n_code_links":1,"syntology":null},{"paper":"/paper/l3cubemahasent-a-marathi-tweet-based","slug":"l3cubemahasent-a-marathi-tweet-based","title":"L3CubeMahaSent: A Marathi Tweet-based Sentiment Analysis Dataset","date":"2021-03-21","arxiv_id":"2103.11408","n_code_links":1,"syntology":null},{"paper":null,"slug":"experimental-evaluation-of-deep-learning","title":"Experimental Evaluation of Deep Learning models for Marathi Text Classification","date":"2021-01-13","arxiv_id":"2101.04899","n_code_links":0,"syntology":null},{"paper":"/paper/ladiff-ulmfit-a-layer-differentiated-training","slug":"ladiff-ulmfit-a-layer-differentiated-training","title":"LaDiff ULMFiT: A Layer Differentiated training approach for ULMFiT","date":"2021-01-13","arxiv_id":"2101.04965","n_code_links":1,"syntology":null},{"paper":"/paper/hinglishnlp-at-semeval-2020-task-9-fine-tuned","slug":"hinglishnlp-at-semeval-2020-task-9-fine-tuned","title":"HinglishNLP at SemEval-2020 Task 9: Fine-tuned Language Models for Hinglish Sentiment Detection","date":"2020-12-01","arxiv_id":null,"n_code_links":2,"syntology":null},{"paper":null,"slug":"smash-at-semeval-2020-task-7-optimizing-the","title":"Smash at SemEval-2020 Task 7: Optimizing the Hyperparameters of ERNIE 2.0 for Humor Ranking and Rating","date":"2020-12-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"palomino-ochoa-at-semeval-2020-task-9-robust","title":"Palomino-Ochoa at SemEval-2020 Task 9: Robust System based on Transformer for Code-Mixed Sentiment Classification","date":"2020-11-18","arxiv_id":"2011.09448","n_code_links":0,"syntology":null},{"paper":"/paper/pagsusuri-ng-rnn-based-transfer-learning","slug":"pagsusuri-ng-rnn-based-transfer-learning","title":"Pagsusuri ng RNN-based Transfer Learning Technique sa Low-Resource Language","date":"2020-10-13","arxiv_id":"2010.06447","n_code_links":2,"syntology":null},{"paper":null,"slug":"gauravarora-hasoc-dravidian-codemix-fire2020","title":"Gauravarora@HASOC-Dravidian-CodeMix-FIRE2020: Pre-training ULMFiT on Synthetically Generated Code-Mixed Data for Hate Speech Detection","date":"2020-10-05","arxiv_id":"2010.02094","n_code_links":0,"syntology":null},{"paper":"/paper/farstail-a-persian-natural-language-inference","slug":"farstail-a-persian-natural-language-inference","title":"FarsTail: A Persian Natural Language Inference Dataset","date":"2020-09-18","arxiv_id":"2009.08820","n_code_links":1,"syntology":null},{"paper":null,"slug":"fine-tuning-pre-trained-contextual-embeddings","title":"Fine-tuning Pre-trained Contextual Embeddings for Citation Content Analysis in Scholarly Publication","date":"2020-09-12","arxiv_id":"2009.05836","n_code_links":0,"syntology":null},{"paper":"/paper/hinglishnlp-fine-tuned-language-models-for","slug":"hinglishnlp-fine-tuned-language-models-for","title":"HinglishNLP: Fine-tuned Language Models for Hinglish Sentiment Detection","date":"2020-08-22","arxiv_id":"2008.09820","n_code_links":2,"syntology":null},{"paper":null,"slug":"evaluation-metrics-for-headline-generation","title":"Evaluation Metrics for Headline Generation Using Deep Pre-Trained Embeddings","date":"2020-05-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/offensive-language-detection-in-arabic-using","slug":"offensive-language-detection-in-arabic-using","title":"Offensive language detection in Arabic using ULMFiT","date":"2020-05-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"slug":"text-categorization-for-conflict-event","title":"Text Categorization for Conflict Event Annotation","date":"2020-05-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/inferring-the-source-of-official-texts-can","slug":"inferring-the-source-of-official-texts-can","title":"Inferring the source of official texts: can SVM beat ULMFiT?","date":"2020-03-02","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"slug":"localized-flood-detectionwith-minimal-labeled","title":"Localized Flood DetectionWith Minimal Labeled Social Media Data Using Transfer Learning","date":"2020-02-10","arxiv_id":"2003.04973","n_code_links":0,"syntology":null},{"paper":null,"slug":"natural-language-processing-of-mimic-iii","title":"Natural language processing of MIMIC-III clinical notes for identifying diagnosis and procedures with neural networks","date":"2019-12-28","arxiv_id":"1912.12397","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-comparative-study-of-pretrained-language","title":"A Comparative Study of Pretrained Language Models on Thai Social Text Categorization","date":"2019-12-03","arxiv_id":"1912.01580","n_code_links":0,"syntology":null},{"paper":null,"slug":"evolution-of-transfer-learning-in-natural","title":"Evolution of transfer learning in natural language processing","date":"2019-10-16","arxiv_id":"1910.07370","n_code_links":0,"syntology":null},{"paper":"/paper/the-merits-of-universal-language-model-fine","slug":"the-merits-of-universal-language-model-fine","title":"The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews","date":"2019-10-02","arxiv_id":"1910.00896","n_code_links":2,"syntology":null},{"paper":null,"slug":"analyzing-customer-feedback-for-product-fit","title":"Analyzing Customer Feedback for Product Fit Prediction","date":"2019-08-28","arxiv_id":"1908.10896","n_code_links":0,"syntology":null},{"paper":null,"slug":"low-shot-classification-a-comparison-of","title":"Low-Shot Classification: A Comparison of Classical and Deep Transfer Machine Learning Approaches","date":"2019-07-17","arxiv_id":"1907.07543","n_code_links":0,"syntology":null},{"paper":"/paper/evaluating-language-model-finetuning","slug":"evaluating-language-model-finetuning","title":"Evaluating Language Model Finetuning Techniques for Low-resource Languages","date":"2019-06-30","arxiv_id":"1907.00409","n_code_links":2,"syntology":null},{"paper":null,"slug":"exploiting-unsupervised-pre-training-and","title":"Exploiting Unsupervised Pre-training and Automated Feature Engineering for Low-resource Hate Speech Detection in Polish","date":"2019-06-17","arxiv_id":"1906.09325","n_code_links":0,"syntology":null},{"paper":null,"slug":"figure-eight-at-semeval-2019-task-3-ensemble","title":"Figure Eight at SemEval-2019 Task 3: Ensemble of Transfer Learning Methods for Contextual Emotion Detection","date":"2019-06-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"speak-up-fight-back-detection-of-social-media","title":"Speak up, Fight Back! Detection of Social Media Disclosures of Sexual Harassment","date":"2019-06-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"an-empirical-evaluation-of-text","title":"An Empirical Evaluation of Text Representation Schemes on Multilingual Social Web to Filter the Textual Aggression","date":"2019-04-16","arxiv_id":"1904.08770","n_code_links":0,"syntology":null},{"paper":"/paper/low-resource-text-classification-with-ulmfit","slug":"low-resource-text-classification-with-ulmfit","title":"Low Resource Text Classification with ULMFit and Backtranslation","date":"2019-03-21","arxiv_id":"1903.09244","n_code_links":1,"syntology":null},{"paper":"/paper/universal-language-model-fine-tuning-for-text","slug":"universal-language-model-fine-tuning-for-text","title":"Universal Language Model Fine-tuning for Text Classification","date":"2018-01-18","arxiv_id":"1801.06146","n_code_links":66,"syntology":{"ran":2,"of":5,"n_ran_checked":0,"n_instrument":2,"unverified":3,"pointer_only":3,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified","official":{"repos":["fastai/fastai"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]}}}],"papers_shown":40,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":40,"with_a_code_link":17,"where_syntology_ran_a_sample":1,"not_listed_spam_title":0,"listed":40,"listed_where_code_ran":1,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":0,"every_run_a_failure_of_syntologys_instrument":1,"listed_with_a_run_with_no_instrument_failure":0,"listed_every_run_a_failure_of_syntologys_instrument":1,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":1,"papers_where_code_ran_first_page":"/method/ulmfit/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/ulmfit/papers/ran/1","json":"/method/ulmfit/papers/ran/1.json","rows":1}],"tasks":[{"task":"/task/transfer-learning","name":"Transfer Learning","papers":14},{"task":"/task/classification","name":"General Classification","papers":12},{"task":"/task/language-modelling","name":"Language Modelling","papers":12},{"task":"/task/text-classification","name":"Text Classification","papers":12},{"task":"/task/language-modeling","name":"Language Modeling","papers":11},{"task":"/task/sentiment-analysis","name":"Sentiment Analysis","papers":9},{"task":"/task/text-classification-1","name":"text-classification","papers":8},{"task":"/task/classification-1","name":"Classification","papers":7},{"task":"/task/decision-making","name":"Decision Making","papers":3},{"task":"/task/hate-speech-detection","name":"Hate Speech Detection","papers":3},{"task":"/task/sentiment-classification","name":"Sentiment Classification","papers":3},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":2},{"task":"/task/language-identification","name":"Language Identification","papers":2},{"task":"/task/management","name":"Management","papers":2},{"task":"/task/marketing","name":"Marketing","papers":2},{"task":"/task/sentence","name":"Sentence","papers":2},{"task":"/task/text-categorization","name":"Text Categorization","papers":2},{"task":"/task/transliteration","name":"Transliteration","papers":2},{"task":"/task/word-embeddings","name":"Word Embeddings","papers":2},{"task":"/task/regression-1","name":"regression","papers":2}],"tasks_shown":20,"n_tasks":51,"usage_by_year":[{"year":"2018","papers":1},{"year":"2019","papers":12},{"year":"2020","papers":13},{"year":"2021","papers":8},{"year":"2022","papers":1},{"year":"2023","papers":2},{"year":"2024","papers":2},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/ulmfit"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=ULMFiT","read_at":"2026-10-05T01:03:25+00:00","query":"ULMFiT","kind":"method","have":"unknown_name","tier":null,"verified_implementation":null,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":null,"resolution":null,"resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":null,"archive_paper_arxiv_id":"1801.06146","repositories":null,"retrieve_with":null,"name_competition":{"verdict":"none","may_be_wrong_sense":false,"bearers_found":0,"served":null,"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing in this corpus is named 'ULMFiT' -- no method node, no paper announcing it in its title, no expanded form declaring it as an acronym."},"name_competition_text":null,"near_matches":[],"code_nearby":[],"code_nearby_caption":null,"origin_paper":null,"line":"name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"e9d2de2bdc899c0d0f80a23a2cae01f4cb052f8270b142ef57cfb5e692ca56f4","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}