Papers › Understanding the Role of the Projector in Knowledge Distillation

Understanding the Role of the Projector in Knowledge Distillation

20 Mar 2023arXiv:2303.11098archive 2025-07-28

Roy Miles, Krystian Mikolajczyk

In this paper we revisit the efficacy of knowledge distillation as a function matching and metric learning problem. In doing so we verify three important design decisions, namely the normalisation, soft maximum function, and projection layers as key ingredients. We theoretically show that the projector implicitly encodes information on past examples, enabling relational gradients for the student. We then show that the normalisation of representations is tightly coupled with the training dynamics of this projector, which can have a large impact on the students performance. Finally, we show that a simple soft maximum function can be used to address any significant capacity gap problems. Experimental results on various benchmark datasets demonstrate that using these insights can lead to superior or comparable performance to state-of-the-art knowledge distillation techniques, despite being much more computationally efficient. In particular, we obtain these results across image classification (CIFAR100 and ImageNet), object detection (COCO2017), and on more difficult distillation objectives, such as training data efficient transformers, whereby we attain a 77.2% top-1 accuracy with DeiT-Ti on ImageNet. Code and models are publicly available.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

roymiles/simple-recipe-distillation officialmentioned in papermentioned on GitHubpytorchSyntology: not harvested report
roymiles/vkd mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to VₖD: Improving Knowledge Distillation using Orthogonal Projections (arXiv:2403.06213; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report
yoshitomo-matsubara/torchdistill mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Distilling the Knowledge in a Neural Network (arXiv:1503.02531; that paper's own run record: 23 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 6 honoured, 0 violated, 8 with no contract checked; 9 where Syntology's instrument failed) · 14 unverified), Deep Residual Learning for Image Recognition (arXiv:1512.03385; that paper's own run record: 254 ran (of which 108 constructed an object rather than computing a result; 163 with no instrument failure: 3 honoured, 1 violated, 159 with no contract checked; 91 where Syntology's instrument failed) · 123 unverified), Wide Residual Networks (arXiv:1605.07146; that paper's own run record: 66 ran (of which 31 constructed an object rather than computing a result; 47 with no instrument failure: 0 honoured, 0 violated, 47 with no contract checked; 19 where Syntology's instrument failed) · 30 unverified) and 3 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
Hazqeel09/ellzaf_ml pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to SimMIM: A Simple Framework for Masked Image Modeling (arXiv:2111.09886; that paper's own run record: 9 ran (of which 7 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 1 where Syntology's instrument failed) · 5 unverified) and SpectFormer: Frequency and Attention is what you need in a Vision Transformer (arXiv:2304.06446; that paper's own run record: 5 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Image ClassificationKnowledge DistillationMetric LearningObject Detectionimage-classificationobject-detection

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Knowledge Distillation CIFAR-100 SRD (T:resnet-32x4, S:shufflenet-v2) Top-1 Accuracy (%) 79.86 #1 of 27 Archive leaderboard report
Knowledge Distillation ImageNet SRD (T:RegNety 160 S:DeiT-S) CRD training setting ✘ #12 of 52 Archive leaderboard report
Knowledge Distillation ImageNet SRD (T:RegNety 160 S:DeiT-S) Top-1 accuracy % 82.1 #12 of 52 Archive leaderboard report
Knowledge Distillation ImageNet SRD (T:RegNety 160 S:DeiT-S) model size 22M #12 of 52 Archive leaderboard report
Knowledge Distillation ImageNet SRD (T:RegNety 160 S:DeIT-Ti) CRD training setting ✘ #21 of 52 Archive leaderboard report
Knowledge Distillation ImageNet SRD (T:RegNety 160 S:DeIT-Ti) Top-1 accuracy % 77.2 #21 of 52 Archive leaderboard report
Knowledge Distillation ImageNet SRD (T:RegNety 160 S:DeIT-Ti) model size 6M #21 of 52 Archive leaderboard report
Knowledge Distillation ImageNet SRD (T: ResNet-34 S:ResNet-18) CRD training setting ✓ #38 of 52 Archive leaderboard report
Knowledge Distillation ImageNet SRD (T: ResNet-34 S:ResNet-18) Top-1 accuracy % 71.87 #38 of 52 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Knowledge Distillation

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections