Papers › Head Network Distillation: Splitting Distilled Deep Neural Networks for...

Head Network Distillation: Splitting Distilled Deep Neural Networks for Resource-Constrained Edge Computing Systems

20 Nov 2020archive 2025-07-28

Yoshitomo Matsubara, Davide Callegaro, Sabur Baidya, Marco Levorato, Sameer Singh

As the complexity of Deep Neural Network (DNN) models increases, their deployment on mobile devices becomes increasingly challenging, especially in complex vision tasks such as image classification. Many of recent contributions aim either to produce compact models matching the limited computing capabilities of mobile devices or to offload the execution of such burdensome models to a compute-capable device at the network edge - the edge servers. In this paper, we propose to modify the structure and training process of DNN models for complex image classification tasks to achieve in-network compression in the early network layers. Our training process stems from knowledge distillation, a technique that has been traditionally used to build small - student - models mimicking the output of larger - teacher - models. Here, we adopt this idea to obtain aggressive compression while preserving accuracy. Our results demonstrate that our approach is effective for state-of-the-art models trained over complex datasets, and can extend the parameter region in which edge computing is a viable and advantageous option. Additionally, we demonstrate that in many settings of practical interest we reduce the inference time with respect to specialized models such as MobileNet v2 executed at the mobile device, while improving accuracy.

PaperPDFCode

Syntology This paper has no arXiv id in the archive, and this site joins papers to Syntology's graph by arXiv id, so this page does not say whether the graph holds it.

Code

yoshitomo-matsubara/head-network-distillation officialpytorchSyntology: not harvested report
yoshitomo-matsubara/torchdistill pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Distilling the Knowledge in a Neural Network (arXiv:1503.02531; that paper's own run record: 23 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 6 honoured, 0 violated, 8 with no contract checked; 9 where Syntology's instrument failed) · 14 unverified), Deep Residual Learning for Image Recognition (arXiv:1512.03385; that paper's own run record: 254 ran (of which 108 constructed an object rather than computing a result; 163 with no instrument failure: 3 honoured, 1 violated, 159 with no contract checked; 91 where Syntology's instrument failed) · 123 unverified), Wide Residual Networks (arXiv:1605.07146; that paper's own run record: 66 ran (of which 31 constructed an object rather than computing a result; 47 with no instrument failure: 0 honoured, 0 violated, 47 with no contract checked; 19 where Syntology's instrument failed) · 30 unverified) and 3 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Edge-computingImage ClassificationKnowledge DistillationModel CompressionNeural Network Compressionimage-classification

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

Knowledge Distillation

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections