Methods › General › Knowledge Distillation › Knowledge Distillation
Knowledge Distillation
Introduced by Geoffrey Hinton et al. in Distilling the Knowledge in a Neural Network
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel. Source: Distilling the Knowledge in a Neural Network
Code snippet in the archive: a link on research.google (archive link, not checked and not linked: not a code host this site links to).
In Syntology about another paper, arXiv:2004.09813: repository known, no verified implementation (declared_code; repositories: 5 listed of 12 in the graph, strongest link: official). About another paper: this answer is about arXiv:2004.09813, while the archive credits arXiv:1503.02531.
From the lookup's note on this name: The same name also belongs to the paper “Knowledge Distillation: A Survey” (arXiv:2006.05525, 4,554 citations), which the lookup finds cited far more than the paper this answer is about (1,403 citations), so this answer may be about a different sense of the name. For arXiv:2006.05525: ask the lookup about it.
Code Syntology holds under other names, as the lookup lists them (code_nearby); not a complete list, and none of them is this method.
- knowledge fusion · shares_name_tokens (shares name tokens with this name; a different method) · level V3 · retrieve with
get_reference_implementation(name='knowledge fusion') - online distillation · shares_name_tokens (shares name tokens with this name; a different method) · level V2 · retrieve with
get_reference_implementation(name='online distillation') - EM Distillation (EMD) · shares_name_tokens (shares name tokens with this name; a different method) · level V3 · retrieve with
get_reference_implementation(name='EM Distillation (EMD)') - Weighted Distillation · shares_name_tokens (shares name tokens with this name; a different method) · level V2 · retrieve with
get_reference_implementation(name='Weighted Distillation') - Behaviour Distillation · shares_name_tokens (shares name tokens with this name; a different method) · level V2 · retrieve with
get_reference_implementation(name='Behaviour Distillation') - RFA Distillation (RFAD) · shares_name_tokens (shares name tokens with this name; a different method) · level V3 · retrieve with
get_reference_implementation(name='RFA Distillation (RFAD)')
Origin paper of this answer, arXiv:2004.09813 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 0 ran · 1 unverified (of 1 harvested sample). Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
What each level means, quoted from Syntology's server card: V0 “imports”; V1 “executes against a synthesized fixture and returns a value of the recorded type”; V2 “two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs” “neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)”; V3 “paper-derived property tests pass on both of those same two generations.” The card adds: “No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results.” A level quoted in two parts is one definition the card writes as two clauses joined by a double hyphen, each clause quoted on its own (how these levels are quoted).
Papers archive 2025-07-28
archive papers tagged: 3,071 · with a code link: 1,258 · where Syntology ran a sample: 320 (263 with a run with no instrument failure, 57 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (320 of 3,071 tagged: 263 with a run with no instrument failure, 57 where every run was a failure of Syntology's instrument)
Page 1 of 31: papers 1 to 100 of 3,071, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
DVFL-Net: A Lightweight Distilled Video Focal Modulation Network for Spatio-Temporal Action Recognition 16 Jul 2025 · 1 repository · arXiv:2507.12426
-
HanjaBridge: Resolving Semantic Ambiguity in Korean LLMs via Hanja-Augmented Pre-Training 15 Jul 2025 · 0 repositories · arXiv:2507.10920
-
Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning 14 Jul 2025 · 0 repositories · arXiv:2507.10348
-
SFedKD: Sequential Federated Learning with Discrepancy-Aware Multi-Teacher Knowledge Distillation 11 Jul 2025 · 0 repositories · arXiv:2507.08508
-
Towards Collaborative Fairness in Federated Learning Under Imbalanced Covariate Shift 11 Jul 2025 · 0 repositories · arXiv:2507.08617
-
Continual Self-Supervised Learning with Masked Autoencoders in Remote Sensing 26 Jun 2025 · 0 repositories · arXiv:2506.21312
-
Distilling Normalizing Flows 26 Jun 2025 · 0 repositories · arXiv:2506.21003
-
G²D: Boosting Multimodal Learning with Gradient-Guided Distillation 26 Jun 2025 · 1 repository · arXiv:2506.21514
-
Building Lightweight Semantic Segmentation Models for Aerial Images Using Dual Relation Distillation 25 Jun 2025 · 0 repositories · arXiv:2506.20688
-
Client Clustering Meets Knowledge Sharing: Enhancing Privacy and Robustness in Personalized Peer-to-Peer Learning 25 Jun 2025 · 0 repositories · arXiv:2506.20413
-
FedBKD: Distilled Federated Learning to Embrace Gerneralization and Personalization on Non-IID Data 25 Jun 2025 · 1 repository · arXiv:2506.20245
-
Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregation 25 Jun 2025 · 1 repository · arXiv:2506.20431
-
Towards Scalable and Generalizable Earth Observation Data Mining via Foundation Model Composition 25 Jun 2025 · 0 repositories · arXiv:2506.20174
-
Distillation-Enabled Knowledge Alignment for Generative Semantic Communications in AIGC Provisioning Tasks 24 Jun 2025 · 0 repositories · arXiv:2506.19893
-
Recalling The Forgotten Class Memberships: Unlearned Models Can Be Noisy Labelers to Leak Privacy 24 Jun 2025 · 0 repositories · arXiv:2506.19486
-
PicoSAM2: Low-Latency Segmentation In-Sensor for Edge Vision Applications 23 Jun 2025 · 0 repositories · arXiv:2506.18807
-
Multimodal Fusion SLAM with Fourier Attention 22 Jun 2025 · 1 repository · arXiv:2506.18204
-
Fine-grained Image Retrieval via Dual-Vision Adaptation 19 Jun 2025 · 0 repositories · arXiv:2506.16273
-
Factorized RVQ-GAN For Disentangled Speech Tokenization 18 Jun 2025 · 0 repositories · arXiv:2506.15456
-
Knowledge Distillation Framework for Accelerating High-Accuracy Neural Network-Based Molecular Dynamics Simulations 18 Jun 2025 · 0 repositories · arXiv:2506.15337
-
AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes 17 Jun 2025 · 0 repositories · arXiv:2506.14728
-
KDMOS:Knowledge Distillation for Motion Segmentation 17 Jun 2025 · 1 repository · arXiv:2506.14130
-
Model compression using knowledge distillation with integrated gradients 17 Jun 2025 · 0 repositories · arXiv:2506.14440
-
A Technical Study into Small Reasoning Language Models 16 Jun 2025 · 0 repositories · arXiv:2506.13404
-
HKD4VLM: A Progressive Hybrid Knowledge Distillation Framework for Robust Multimodal Hallucination and Factuality Detection in VLMs 16 Jun 2025 · 0 repositories · arXiv:2506.13038
-
Lightweight Task-Oriented Semantic Communication Empowered by Large-Scale AI Models 16 Jun 2025 · 0 repositories · arXiv:2506.13243
-
SeqPE: Transformer with Sequential Position Encoding 16 Jun 2025 · 1 repository · arXiv:2506.13277Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 2 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 3 unverified (of 12 harvested samples) · 12 pointer-only (licence)
-
Ground Reaction Force Estimation via Time-aware Knowledge Distillation 12 Jun 2025 · 0 repositories · arXiv:2506.10265
-
A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning 11 Jun 2025 · 0 repositories · arXiv:2506.09429
-
Multi-Teacher Language-Aware Knowledge Distillation for Multilingual Speech Emotion Recognition 10 Jun 2025 · 1 repository · arXiv:2506.08717
-
Towards Class-wise Fair Adversarial Training via Anti-Bias Soft Label Distillation 10 Jun 2025 · 1 repository · arXiv:2506.08611
-
Label-Context-Dependent Internal Language Model Estimation for CTC 6 Jun 2025 · 0 repositories · arXiv:2506.06096
-
hdl2v: A Code Translation Dataset for Enhanced LLM Verilog Generation 5 Jun 2025 · 0 repositories · arXiv:2506.04544
-
Static Word Embeddings for Sentence Semantic Representation 5 Jun 2025 · 0 repositories · arXiv:2506.04624Syntology 18 ran (of which 0 constructed an object rather than computing a result; 18 with no instrument failure: 1 honoured, 0 violated, 17 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified (of 22 harvested samples)
-
Building a Few-Shot Cross-Domain Multilingual NLU Model for Customer Care 4 Jun 2025 · 0 repositories · arXiv:2506.04389
-
TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models 3 Jun 2025 · 0 repositories · arXiv:2506.03099
-
Analyzing the Importance of Blank for CTC-Based Knowledge Distillation 2 Jun 2025 · 0 repositories · arXiv:2506.01503
-
KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning 2 Jun 2025 · 0 repositories · arXiv:2506.02208
-
Feature Fusion and Knowledge-Distilled Multi-Modal Multi-Target Detection 31 May 2025 · 0 repositories · arXiv:2506.00365
-
CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning 30 May 2025 · 1 repository · arXiv:2505.24816
-
Progressive Class-level Distillation 30 May 2025 · 0 repositories · arXiv:2505.24310
-
Revisiting Cross-Modal Knowledge Distillation: A Disentanglement Approach for RGBD Semantic Segmentation 30 May 2025 · 1 repository · arXiv:2505.24361
-
Knowledge Distillation for Reservoir-based Classifier: Human Activity Recognition 29 May 2025 · 0 repositories · arXiv:2505.22985
-
Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch 29 May 2025 · 0 repositories · arXiv:2505.23763
-
CAST: Contrastive Adaptation and Distillation for Semi-Supervised Instance Segmentation 28 May 2025 · 0 repositories · arXiv:2505.21904
-
Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles 28 May 2025 · 1 repository · arXiv:2505.22027
-
Multi-MLLM Knowledge Distillation for Out-of-Context News Detection 28 May 2025 · 0 repositories · arXiv:2505.22517
-
EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models 27 May 2025 · 0 repositories · arXiv:2505.20888
-
DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillation 26 May 2025 · 1 repository · arXiv:2505.19504
-
From Data to Modeling: Fully Open-vocabulary Scene Graph Generation 26 May 2025 · 0 repositories · arXiv:2505.20106
-
Light distillation for Incremental Graph Convolution Collaborative Filtering 26 May 2025 · 0 repositories · arXiv:2505.19810
-
Model Stitching by Functional Latent Alignment 26 May 2025 · 0 repositories · arXiv:2505.20142
-
Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments 26 May 2025 · 0 repositories · arXiv:2505.19699
-
Holistic White-light Polyp Classification via Alignment-free Dense Distillation of Auxiliary Optical Chromoendoscopy 25 May 2025 · 1 repository · arXiv:2505.19319
-
Online Knowledge Distillation with Reward Guidance 25 May 2025 · 0 repositories · arXiv:2505.18952
-
Remote Sensing Image Classification with Decoupled Knowledge Distillation 25 May 2025 · 0 repositories · arXiv:2505.19111
-
Tokenizing Electron Cloud in Protein-Ligand Interaction Learning 25 May 2025 · 0 repositories · arXiv:2505.19014
-
C3R: Channel Conditioned Cell Representations for unified evaluation in microscopy imaging 24 May 2025 · 0 repositories · arXiv:2505.18745
-
Knowledge Grafting of Large Language Models 24 May 2025 · 1 repository · arXiv:2505.18502
-
Single Snapshot Distillation for Phase Coded Mask Design in Phase Retrieval 23 May 2025 · 0 repositories · arXiv:2505.18352
-
SEDD-PCC: A Single Encoder-Dual Decoder Framework For End-To-End Learned Point Cloud Compression 22 May 2025 · 0 repositories · arXiv:2505.16709
-
ToDi: Token-wise Distillation via Fine-Grained Divergence Control 22 May 2025 · 0 repositories · arXiv:2505.16297
-
DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer 21 May 2025 · 1 repository · arXiv:2505.15133
-
Deliberation on Priors: Trustworthy Reasoning of Large Language Models on Knowledge Graphs 21 May 2025 · 1 repository · arXiv:2505.15210Syntology official (archive's flag): 1 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 9 unverified (of 14 harvested samples)
-
MentalMAC: Enhancing Large Language Models for Detecting Mental Manipulation via Multi-Task Anti-Curriculum Distillation 21 May 2025 · 0 repositories · arXiv:2505.15255
-
UWSAM: Segment Anything Model Guided Underwater Instance Segmentation and A Large-scale Benchmark Dataset 21 May 2025 · 1 repository · arXiv:2505.15581
-
Bridge the Gap between Past and Future: Siamese Model Optimization for Context-Aware Document Ranking 20 May 2025 · 0 repositories · arXiv:2505.14180
-
Ground-V: Teaching VLMs to Ground Complex Instructions in Pixels 20 May 2025 · 0 repositories · arXiv:2505.13788
-
Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillation 20 May 2025 · 0 repositories · arXiv:2505.13792
-
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone 19 May 2025 · 1 repository · arXiv:2505.12781
-
Bridging the Modality Gap: Enhancing Channel Prediction with Semantically Aligned LLMs and Knowledge Distillation 19 May 2025 · 0 repositories · arXiv:2505.12729
-
ORQA: A Benchmark and Foundation Model for Holistic Operating Room Modeling 19 May 2025 · 0 repositories · arXiv:2505.12890
-
Robust Multimodal Segmentation with Representation Regularization and Hybrid Prototype Distillation 19 May 2025 · 1 repository · arXiv:2505.12861
-
SMOTExT: SMOTE meets Large Language Models 19 May 2025 · 1 repository · arXiv:2505.13434
-
Towards Low-Latency Event Stream-based Visual Object Tracking: A Slow-Fast Approach 19 May 2025 · 1 repository · arXiv:2505.12903
-
Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruption 19 May 2025 · 1 repository · arXiv:2505.12912
-
Always Clear Depth: Robust Monocular Depth Estimation under Adverse Weather 18 May 2025 · 1 repository · arXiv:2505.12199
-
LAMeTA: Intent-Aware Agentic Network Optimization via a Large AI Model-Empowered Two-Stage Approach 18 May 2025 · 0 repositories · arXiv:2505.12247
-
SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning 18 May 2025 · 0 repositories · arXiv:2505.12448
-
FiGKD: Fine-Grained Knowledge Distillation via High-Frequency Detail Transfer 17 May 2025 · 0 repositories · arXiv:2505.11897
-
On Membership Inference Attacks in Knowledge Distillation 17 May 2025 · 1 repository · arXiv:2505.11837
-
Distilled Circuits: A Mechanistic Study of Internal Restructuring in Knowledge Distillation 16 May 2025 · 1 repository · arXiv:2505.10822
-
Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors 16 May 2025 · 0 repositories · arXiv:2505.11100
-
Semantically-Aware Game Image Quality Assessment 16 May 2025 · 0 repositories · arXiv:2505.11724
-
Advancing Multiple Instance Learning with Continual Learning for Whole Slide Imaging 15 May 2025 · 0 repositories · arXiv:2505.10649
-
DCSNet: A Lightweight Knowledge Distillation-Based Model with Explainable AI for Lung Cancer Diagnosis from Histopathological Images 14 May 2025 · 0 repositories · arXiv:2505.09334
-
Foundation Models Knowledge Distillation For Battery Capacity Degradation Forecast 13 May 2025 · 1 repository · arXiv:2505.08151
-
Low-Complexity Inference in Continual Learning via Compressed Knowledge Transfer 13 May 2025 · 0 repositories · arXiv:2505.08327
-
MoKD: Multi-Task Optimization for Knowledge Distillation 13 May 2025 · 0 repositories · arXiv:2505.08170
-
Channel Fingerprint Construction for Massive MIMO: A Deep Conditional Generative Approach 12 May 2025 · 0 repositories · arXiv:2505.07893
-
KDH-MLTC: Knowledge Distillation for Healthcare Multi-Label Text Classification 12 May 2025 · 0 repositories · arXiv:2505.07162
-
Ranking-aware Continual Learning for LiDAR Place Recognition 12 May 2025 · 0 repositories · arXiv:2505.07198
-
Simple Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization 12 May 2025 · 1 repository · arXiv:2505.07675
-
Topology-Guided Knowledge Distillation for Efficient Point Cloud Processing 12 May 2025 · 1 repository · arXiv:2505.08101
-
Robust & Precise Knowledge Distillation-based Novel Context-Aware Predictor for Disease Detection in Brain and Gastrointestinal 9 May 2025 · 0 repositories · arXiv:2505.06381
-
Biomed-DPT: Dual Modality Prompt Tuning for Biomedical Vision-Language Models 8 May 2025 · 1 repository · arXiv:2505.05189
-
Federated Deconfounding and Debiasing Learning for Out-of-Distribution Generalization 8 May 2025 · 0 repositories · arXiv:2505.04979
-
Theoretical Guarantees for LT-TTD: A Unified Transformer-based Architecture for Two-Level Ranking Systems 7 May 2025 · 0 repositories · arXiv:2505.04434
-
Image Recognition with Online Lightweight Vision Transformer: A Survey 6 May 2025 · 0 repositories · arXiv:2505.03113
-
Knowledge Distillation for Speech Denoising by Latent Representation Alignment with Cosine Distance 6 May 2025 · 0 repositories · arXiv:2505.03442
Tasks archive 2025-07-28
20 shown of 1,024 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Knowledge Distillation | 3,039 |
| Transfer Learning | 311 |
| Model Compression | 253 |
| Object Detection | 208 |
| object-detection | 205 |
| Image Classification | 202 |
| Semantic Segmentation | 199 |
| image-classification | 179 |
| Language Modelling | 168 |
| Language Modeling | 133 |
| Federated Learning | 123 |
| Representation Learning | 112 |
| Data Augmentation | 111 |
| Translation | 108 |
| Segmentation | 105 |
| Object | 99 |
| Incremental Learning | 98 |
| Quantization | 97 |
| Contrastive Learning | 96 |
| Continual Learning | 93 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections