Methods › General › Fine-Tuning › Discriminative Fine-Tuning
Discriminative Fine-Tuning
Introduced by Jeremy Howard et al. in Universal Language Model Fine-tuning for Text Classification
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Discriminative Fine-Tuning is a fine-tuning strategy that is used for ULMFiT type models. Instead of using the same learning rate for all layers of the model, discriminative fine-tuning allows us to tune each layer with different learning rates. For context, the regular stochastic gradient descent (SGD) update of a model’s parameters θ at time step t looks like the following (Ruder, 2016):
θₜ = θₜ₋₁ − η·∇_θJ(θ)
where η is the learning rate and ∇_θJ(θ) is the gradient with regard to the model’s objective function. For discriminative fine-tuning, we split the parameters θ into {θ₁, …, θ_L} where θₗ contains the parameters of the model at the l-th layer and L is the number of layers of the model. Similarly, we obtain {η₁, …, η_L} where θₗ where ηₗ is the learning rate of the l-th layer. The SGD update with discriminative finetuning is then:
θₜˡ = θₜ₋₁ˡ - ηˡ·∇_(θˡ)J(θ)
The authors find that empirically it worked well to first choose the learning rate ηᴸ of the last layer by fine-tuning only the last layer and using ηˡ⁻¹=ηˡ/2.6 as the learning rate for lower layers.
In Syntology for a different name, 'Discriminative Finetuning', about another paper, arXiv:2304.01662: repository known, no verified implementation (declared_code; repositories: 1 listed of 1 in the graph, strongest link: official). The lookup matched a different name in Syntology's graph, 'Discriminative Finetuning' (resolution: token_match); this answer is about that name. About another paper: this answer is about arXiv:2304.01662, while the archive credits arXiv:1801.06146.
Code Syntology holds under other names, as the lookup lists them (code_nearby); not a complete list, and none of them is this method.
- FiNE · shares_name_tokens (shares name tokens with this name; a different method) · level V2 · retrieve with
get_reference_implementation(name='FiNE') - FINE · shares_name_tokens (shares name tokens with this name; a different method) · level V3 · retrieve with
get_reference_implementation(name='FINE') - FINE-GRAINED RLHF · shares_name_tokens (shares name tokens with this name; a different method) · level V3 · retrieve with
get_reference_implementation(name='FINE-GRAINED RLHF') - GFlowNet fine-tuning · shares_name_tokens (shares name tokens with this name; a different method) · level V3 · retrieve with
get_reference_implementation(name='GFlowNet fine-tuning') - unsupervised fine-tuning · shares_name_tokens (shares name tokens with this name; a different method) · level V1 · retrieve with
get_reference_implementation(name='unsupervised fine-tuning') - Feature Shift Tuning (FST) · shares_name_tokens (shares name tokens with this name; a different method) · level V3 · retrieve with
get_reference_implementation(name='Feature Shift Tuning (FST)')
Origin paper of this answer, arXiv:2304.01662 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
What each level means, quoted from Syntology's server card: V0 “imports”; V1 “executes against a synthesized fixture and returns a value of the recorded type”; V2 “two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs” “neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)”; V3 “paper-derived property tests pass on both of those same two generations.” The card adds: “No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results.” A level quoted in two parts is one definition the card writes as two clauses joined by a double hyphen, each clause quoted on its own (how these levels are quoted).
Papers archive 2025-07-28
archive papers tagged: 1,990 · with a code link: 794 · where Syntology ran a sample: 271 (216 with a run with no instrument failure, 55 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (271 of 1,990 tagged: 216 with a run with no instrument failure, 55 where every run was a failure of Syntology's instrument)
Page 1 of 20: papers 1 to 100 of 1,990, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Making Language Model a Hierarchical Classifier and Generator 17 Jul 2025 · 1 repository · arXiv:2507.12930
-
Generative Click-through Rate Prediction with Applications to Search Advertising 15 Jul 2025 · 0 repositories · arXiv:2507.11246
-
Behaviour Space Analysis of LLM-driven Meta-heuristic Discovery 4 Jul 2025 · 0 repositories · arXiv:2507.03605
-
Agent-to-Agent Theory of Mind: Testing Interlocutor Awareness among Large Language Models 28 Jun 2025 · 0 repositories · arXiv:2506.22957
-
Cat and Mouse -- Can Fake Text Generation Outpace Detector Systems? 26 Jun 2025 · 0 repositories · arXiv:2506.21274
-
Large Language Models Acing Chartered Accountancy 26 Jun 2025 · 0 repositories · arXiv:2506.21031
-
Large Language Model-Driven Code Compliance Checking in Building Information Modeling 25 Jun 2025 · 0 repositories · arXiv:2506.20551
-
InsertRank: LLMs can reason over BM25 scores to Improve Listwise Reranking 17 Jun 2025 · 0 repositories · arXiv:2506.14086
-
M2BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Models 17 Jun 2025 · 0 repositories · arXiv:2506.14532
-
Toward a Graph Foundation Model: Pre-Training Transformers With Random Walks 17 Jun 2025 · 0 repositories · arXiv:2506.14098
-
Decomposing MLP Activations into Interpretable Features via Semi-Nonnegative Matrix Factorization 12 Jun 2025 · 1 repository · arXiv:2506.10920
-
NeuralNexus at BEA 2025 Shared Task: Retrieval-Augmented Prompting for Mistake Identification in AI Tutors 12 Jun 2025 · 1 repository · arXiv:2506.10627
-
A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning 11 Jun 2025 · 0 repositories · arXiv:2506.09429
-
Latent Multi-Head Attention for Small Language Models 11 Jun 2025 · 0 repositories · arXiv:2506.09342
-
AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP 10 Jun 2025 · 0 repositories · arXiv:2506.08768
-
Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving 10 Jun 2025 · 1 repository · arXiv:2506.08349Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
Generative Voice Bursts during Phone Call 9 Jun 2025 · 0 repositories · arXiv:2506.07526
-
LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization 9 Jun 2025 · 0 repositories · arXiv:2506.07570
-
Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models 8 Jun 2025 · 0 repositories · arXiv:2506.07121
-
RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation 7 Jun 2025 · 0 repositories · arXiv:2506.06677
-
Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey 6 Jun 2025 · 0 repositories · arXiv:2506.11102
-
The Lock-in Hypothesis: Stagnation by Algorithm 6 Jun 2025 · 0 repositories · arXiv:2506.06166
-
Dissecting Bias in LLMs: A Mechanistic Interpretability Perspective 5 Jun 2025 · 0 repositories · arXiv:2506.05166
-
Mathematical Reasoning for Unmanned Aerial Vehicles: A RAG-Based Approach for Complex Arithmetic Reasoning 5 Jun 2025 · 1 repository · arXiv:2506.04998
-
Privacy and Security Threat for OpenAI GPTs 4 Jun 2025 · 0 repositories · arXiv:2506.04036
-
An Exploratory Framework for Future SETI Applications: Detecting Generative Reactivity via Language Models 3 Jun 2025 · 0 repositories · arXiv:2506.02730
-
Rethinking the effects of data contamination in Code Intelligence 3 Jun 2025 · 0 repositories · arXiv:2506.02791
-
How Neural Networks Organize Concepts: Introducing Concept Trajectory Analysis for Deep Learning Interpretability 1 Jun 2025 · 1 repository
-
Power-of-Two (PoT) Weights in Large Language Models (LLMs) 31 May 2025 · 0 repositories · arXiv:2506.00315
-
Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks 30 May 2025 · 1 repository · arXiv:2505.24876
-
MOFGPT: Generative Design of Metal-Organic Frameworks using Language Models 30 May 2025 · 1 repository · arXiv:2506.00198
-
When GPT Spills the Tea: Comprehensive Assessment of Knowledge File Leakage in GPTs 30 May 2025 · 0 repositories · arXiv:2506.00197
-
Daunce: Data Attribution through Uncertainty Estimation 29 May 2025 · 0 repositories · arXiv:2505.23223
-
Matryoshka Model Learning for Improved Elastic Student Models 29 May 2025 · 0 repositories · arXiv:2505.23337
-
Reducing Latency in LLM-Based Natural Language Commands Processing for Robot Navigation 29 May 2025 · 0 repositories · arXiv:2506.00075
-
Multi-MLLM Knowledge Distillation for Out-of-Context News Detection 28 May 2025 · 0 repositories · arXiv:2505.22517
-
Explainability of Large Language Models using SMILE: Statistical Model-agnostic Interpretability with Local Explanations 27 May 2025 · 1 repository · arXiv:2505.21657
-
From prosthetic memory to prosthetic denial: Auditing whether large language models are prone to mass atrocity denialism 27 May 2025 · 0 repositories · arXiv:2505.21753
-
Privacy-Preserving Chest X-ray Report Generation via Multimodal Federated Learning with ViT and GPT-2 27 May 2025 · 0 repositories · arXiv:2505.21715
-
Anveshana: A New Benchmark Dataset for Cross-Lingual Information Retrieval On English Queries and Sanskrit Documents 26 May 2025 · 0 repositories · arXiv:2505.19494
-
Automated evaluation of children's speech fluency for low-resource languages 26 May 2025 · 0 repositories · arXiv:2505.19671
-
Beyond Specialization: Benchmarking LLMs for Transliteration of Indian Languages 26 May 2025 · 0 repositories · arXiv:2505.19851
-
Conversational Lexicography: Querying Lexicographic Data on Knowledge Graphs with SPARQL through Natural Language 26 May 2025 · 0 repositories · arXiv:2505.19971
-
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining 26 May 2025 · 0 repositories · arXiv:2505.19893
-
AI4Math: A Native Spanish Benchmark for University-Level Mathematical Reasoning in Large Language Models 25 May 2025 · 0 repositories · arXiv:2505.18978
-
From Reddit to Generative AI: Evaluating Large Language Models for Anxiety Support Fine-tuned on Social Media Data 24 May 2025 · 0 repositories · arXiv:2505.18464
-
Strong Membership Inference Attacks on Massive Datasets and (Moderately) Large Language Models 24 May 2025 · 0 repositories · arXiv:2505.18773
-
LLM assisted web application functional requirements generation: A case study of four popular LLMs over a Mess Management System 23 May 2025 · 0 repositories · arXiv:2505.18019
-
MMMG: a Comprehensive and Reliable Evaluation Suite for Multitask Multimodal Generation 23 May 2025 · 0 repositories · arXiv:2505.17613
-
ShIOEnv: A CLI Behavior-Capturing Environment Enabling Grammar-Guided Command Synthesis for Dataset Curation 23 May 2025 · 1 repository · arXiv:2505.18374
-
AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training 22 May 2025 · 1 repository · arXiv:2505.16363Syntology official: no sample here; runs from other or unrecorded repositories · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Data-Driven Breakthroughs and Future Directions in AI Infrastructure: A Comprehensive Review 22 May 2025 · 0 repositories · arXiv:2505.16771
-
The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm 22 May 2025 · 0 repositories · arXiv:2505.16932
-
Three Minds, One Legend: Jailbreak Large Reasoning Model with Adaptive Stacked Ciphers 22 May 2025 · 0 repositories · arXiv:2505.16241
-
An Efficient Private GPT Never Autoregressively Decodes 21 May 2025 · 0 repositories · arXiv:2505.15252
-
Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders 20 May 2025 · 0 repositories · arXiv:2505.14536Syntology 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
Choosing a Model, Shaping a Future: Comparing LLM Perspectives on Sustainability and its Relationship with AI 20 May 2025 · 0 repositories · arXiv:2505.14435
-
EEG-to-Text Translation: A Model for Deciphering Human Brain Activity 20 May 2025 · 1 repository · arXiv:2505.13936
-
Every Pixel Tells a Story: End-to-End Urdu Newspaper OCR 20 May 2025 · 0 repositories · arXiv:2505.13943
-
Generative AI at the Crossroads: Light Bulb, Dynamo, or Microscope? 20 May 2025 · 0 repositories · arXiv:2505.14588
-
Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspective 20 May 2025 · 0 repositories · arXiv:2505.14808
-
Scaling Laws for State Dynamics in Large Language Models 20 May 2025 · 0 repositories · arXiv:2505.14892
-
VesselGPT: Autoregressive Modeling of Vascular Geometry 19 May 2025 · 1 repository · arXiv:2505.13318
-
Video-GPT via Next Clip Diffusion 18 May 2025 · 1 repository · arXiv:2505.12489
-
Towards Cultural Bridge by Bahnaric-Vietnamese Translation Using Transfer Learning of Sequence-To-Sequence Pre-training Language Model 16 May 2025 · 0 repositories · arXiv:2505.11421
-
ACSE-Eval: Can LLMs threat model real-world cloud infrastructure? 16 May 2025 · 1 repository · arXiv:2505.11565
-
Can an Easy-to-Hard Curriculum Make Reasoning Emerge in Small Language Models? Evidence from a Four-Stage Curriculum on GPT-2 16 May 2025 · 0 repositories · arXiv:2505.11643
-
Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency 16 May 2025 · 0 repositories · arXiv:2505.13499
-
Are Sparse Autoencoders Useful for Java Function Bug Detection? 15 May 2025 · 1 repository · arXiv:2505.10375
-
Atomic Consistency Preference Optimization for Long-Form Question Answering 14 May 2025 · 1 repository · arXiv:2505.09039
-
Multilingual Machine Translation with Quantum Encoder Decoder Attention-based Convolutional Variational Circuits 14 May 2025 · 0 repositories · arXiv:2505.09407
-
Evaluating LLM Metrics Through Real-World Capabilities 13 May 2025 · 0 repositories · arXiv:2505.08253
-
Memorization-Compression Cycles Improve Generalization 13 May 2025 · 0 repositories · arXiv:2505.08727
-
Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies 13 May 2025 · 1 repository · arXiv:2505.08739
-
Small but Significant: On the Promise of Small Language Models for Accessible AIED 13 May 2025 · 0 repositories · arXiv:2505.08588
-
No Query, No Access 12 May 2025 · 0 repositories · arXiv:2505.07258
-
Evaluating Reasoning LLMs for Suicide Screening with the Columbia-Suicide Severity Rating Scale 11 May 2025 · 1 repository · arXiv:2505.13480
-
CellVerse: Do Large Language Models Really Understand Cell Biology? 9 May 2025 · 0 repositories · arXiv:2505.07865
-
Multi-Agent Systems for Robotic Autonomy with LLMs 9 May 2025 · 0 repositories · arXiv:2505.05762
-
AI Approaches to Qualitative and Quantitative News Analytics on NATO Unity 8 May 2025 · 0 repositories · arXiv:2505.06313
-
An Empirical Study of OpenAI API Discussions on Stack Overflow 7 May 2025 · 0 repositories · arXiv:2505.04084
-
LLM-e Guess: Can LLMs Capabilities Advance Without Hardware Progress? 7 May 2025 · 1 repository · arXiv:2505.04075
-
Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs 7 May 2025 · 0 repositories · arXiv:2505.04441
-
A Comparative Analysis of Ethical and Safety Gaps in LLMs using Relative Danger Coefficient 6 May 2025 · 0 repositories · arXiv:2505.04654
-
Evaluation of LLMs on Long-tail Entity Linking in Historical Documents 6 May 2025 · 0 repositories · arXiv:2505.03473
-
Transformers for Learning on Noisy and Task-Level Manifolds: Approximation and Generalization Insights 6 May 2025 · 0 repositories · arXiv:2505.03205
-
Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis 5 May 2025 · 0 repositories · arXiv:2505.03019
-
A Domain Adaptation of Large Language Models for Classifying Mechanical Assembly Components 2 May 2025 · 0 repositories · arXiv:2505.01627
-
UniDetox: Universal Detoxification of Large Language Models via Dataset Distillation 29 Apr 2025 · 1 repository · arXiv:2504.20500Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections 25 Apr 2025 · 0 repositories · arXiv:2504.18333
-
Optimism, Expectation, or Sarcasm? Multi-Class Hope Speech Detection in Spanish and English 24 Apr 2025 · 0 repositories · arXiv:2504.17974
-
A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms 23 Apr 2025 · 0 repositories · arXiv:2504.16420
-
Benchmarking LLM for Code Smells Detection: OpenAI GPT-4.0 vs DeepSeek-V3 22 Apr 2025 · 0 repositories · arXiv:2504.16027
-
The 1st EReL@MIR Workshop on Efficient Representation Learning for Multimodal Information Retrieval 21 Apr 2025 · 0 repositories · arXiv:2504.14788
-
AI with Emotions: Exploring Emotional Expressions in Large Language Models 20 Apr 2025 · 0 repositories · arXiv:2504.14706
-
Less is More: Adaptive Coverage for Synthetic Training Data 20 Apr 2025 · 0 repositories · arXiv:2504.14508
-
ZeroSumEval: Scaling LLM Evaluation with Inter-Model Competition 17 Apr 2025 · 1 repository · arXiv:2504.12562
-
Using customized GPT to develop prompting proficiency in architectural AI-generated images 16 Apr 2025 · 0 repositories · arXiv:2504.13948
-
Hallucination-Aware Generative Pretrained Transformer for Cooperative Aerial Mobility Control 15 Apr 2025 · 0 repositories · arXiv:2504.10831
-
VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers 15 Apr 2025 · 0 repositories · arXiv:2504.11227
Tasks archive 2025-07-28
20 shown of 725 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections