{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/knowledge-distillation","slug":"knowledge-distillation","name":"Knowledge Distillation","full_name":"Knowledge Distillation","full_name_withheld":false,"description_markdown":"A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.\r\nSource: [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531)","description_state":"present","introduced_year":null,"introduced_by":{"title":"Distilling the Knowledge in a Neural Network","paper":"/paper/distilling-the-knowledge-in-a-neural-network","first_author":"Geoffrey Hinton","n_authors":3,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/distilling-the-knowledge-in-a-neural-network"},"source":{"url":"http://arxiv.org/abs/1503.02531v1","title":"Distilling the Knowledge in a Neural Network","url_on_a_paper_host":true},"code_snippet_url":"https://research.google/blog/auto-generated-summaries-in-google-docs/","code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Knowledge Distillation","url":"/methods/category/knowledge-distillation","pwc_aliases":[]}],"n_papers_tagged":3071,"archive_num_papers":3071,"papers_newest_first":[{"paper":"/paper/dvfl-net-a-lightweight-distilled-video-focal","slug":"dvfl-net-a-lightweight-distilled-video-focal","title":"DVFL-Net: A Lightweight Distilled Video Focal Modulation Network for Spatio-Temporal Action Recognition","date":"2025-07-16","arxiv_id":"2507.12426","n_code_links":1,"syntology":null},{"paper":null,"slug":"hanjabridge-resolving-semantic-ambiguity-in","title":"HanjaBridge: Resolving Semantic Ambiguity in Korean LLMs via Hanja-Augmented Pre-Training","date":"2025-07-15","arxiv_id":"2507.10920","n_code_links":0,"syntology":null},{"paper":null,"slug":"feature-distillation-is-the-better-choice-for","title":"Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning","date":"2025-07-14","arxiv_id":"2507.10348","n_code_links":0,"syntology":null},{"paper":null,"slug":"sfedkd-sequential-federated-learning-with","title":"SFedKD: Sequential Federated Learning with Discrepancy-Aware Multi-Teacher Knowledge Distillation","date":"2025-07-11","arxiv_id":"2507.08508","n_code_links":0,"syntology":null},{"paper":null,"slug":"towards-collaborative-fairness-in-federated","title":"Towards Collaborative Fairness in Federated Learning Under Imbalanced Covariate Shift","date":"2025-07-11","arxiv_id":"2507.08617","n_code_links":0,"syntology":null},{"paper":null,"slug":"continual-self-supervised-learning-with","title":"Continual Self-Supervised Learning with Masked Autoencoders in Remote Sensing","date":"2025-06-26","arxiv_id":"2506.21312","n_code_links":0,"syntology":null},{"paper":null,"slug":"distilling-normalizing-flows","title":"Distilling Normalizing Flows","date":"2025-06-26","arxiv_id":"2506.21003","n_code_links":0,"syntology":null},{"paper":"/paper/g-2-d-boosting-multimodal-learning-with","slug":"g-2-d-boosting-multimodal-learning-with","title":"G$^{2}$D: Boosting Multimodal Learning with Gradient-Guided Distillation","date":"2025-06-26","arxiv_id":"2506.21514","n_code_links":1,"syntology":null},{"paper":null,"slug":"building-lightweight-semantic-segmentation","title":"Building Lightweight Semantic Segmentation Models for Aerial Images Using Dual Relation Distillation","date":"2025-06-25","arxiv_id":"2506.20688","n_code_links":0,"syntology":null},{"paper":null,"slug":"client-clustering-meets-knowledge-sharing","title":"Client Clustering Meets Knowledge Sharing: Enhancing Privacy and Robustness in Personalized Peer-to-Peer Learning","date":"2025-06-25","arxiv_id":"2506.20413","n_code_links":0,"syntology":null},{"paper":"/paper/fedbkd-distilled-federated-learning-to","slug":"fedbkd-distilled-federated-learning-to","title":"FedBKD: Distilled Federated Learning to Embrace Gerneralization and Personalization on Non-IID Data","date":"2025-06-25","arxiv_id":"2506.20245","n_code_links":1,"syntology":null},{"paper":"/paper/tackling-data-heterogeneity-in-federated-2","slug":"tackling-data-heterogeneity-in-federated-2","title":"Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregation","date":"2025-06-25","arxiv_id":"2506.20431","n_code_links":1,"syntology":null},{"paper":null,"slug":"towards-scalable-and-generalizable-earth","title":"Towards Scalable and Generalizable Earth Observation Data Mining via Foundation Model Composition","date":"2025-06-25","arxiv_id":"2506.20174","n_code_links":0,"syntology":null},{"paper":null,"slug":"distillation-enabled-knowledge-alignment-for","title":"Distillation-Enabled Knowledge Alignment for Generative Semantic Communications in AIGC Provisioning Tasks","date":"2025-06-24","arxiv_id":"2506.19893","n_code_links":0,"syntology":null},{"paper":null,"slug":"recalling-the-forgotten-class-memberships","title":"Recalling The Forgotten Class Memberships: Unlearned Models Can Be Noisy Labelers to Leak Privacy","date":"2025-06-24","arxiv_id":"2506.19486","n_code_links":0,"syntology":null},{"paper":null,"slug":"picosam2-low-latency-segmentation-in-sensor","title":"PicoSAM2: Low-Latency Segmentation In-Sensor for Edge Vision Applications","date":"2025-06-23","arxiv_id":"2506.18807","n_code_links":0,"syntology":null},{"paper":"/paper/multimodal-fusion-slam-with-fourier-attention","slug":"multimodal-fusion-slam-with-fourier-attention","title":"Multimodal Fusion SLAM with Fourier Attention","date":"2025-06-22","arxiv_id":"2506.18204","n_code_links":1,"syntology":null},{"paper":null,"slug":"fine-grained-image-retrieval-via-dual-vision","title":"Fine-grained Image Retrieval via Dual-Vision Adaptation","date":"2025-06-19","arxiv_id":"2506.16273","n_code_links":0,"syntology":null},{"paper":null,"slug":"factorized-rvq-gan-for-disentangled-speech","title":"Factorized RVQ-GAN For Disentangled Speech Tokenization","date":"2025-06-18","arxiv_id":"2506.15456","n_code_links":0,"syntology":null},{"paper":null,"slug":"knowledge-distillation-framework-for","title":"Knowledge Distillation Framework for Accelerating High-Accuracy Neural Network-Based Molecular Dynamics Simulations","date":"2025-06-18","arxiv_id":"2506.15337","n_code_links":0,"syntology":null},{"paper":null,"slug":"agentdistill-training-free-agent-distillation","title":"AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes","date":"2025-06-17","arxiv_id":"2506.14728","n_code_links":0,"syntology":null},{"paper":"/paper/kdmos-knowledge-distillation-for-motion","slug":"kdmos-knowledge-distillation-for-motion","title":"KDMOS:Knowledge Distillation for Motion Segmentation","date":"2025-06-17","arxiv_id":"2506.14130","n_code_links":1,"syntology":null},{"paper":null,"slug":"model-compression-using-knowledge","title":"Model compression using knowledge distillation with integrated gradients","date":"2025-06-17","arxiv_id":"2506.14440","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-technical-study-into-small-reasoning","title":"A Technical Study into Small Reasoning Language Models","date":"2025-06-16","arxiv_id":"2506.13404","n_code_links":0,"syntology":null},{"paper":null,"slug":"hkd4vlm-a-progressive-hybrid-knowledge","title":"HKD4VLM: A Progressive Hybrid Knowledge Distillation Framework for Robust Multimodal Hallucination and Factuality Detection in VLMs","date":"2025-06-16","arxiv_id":"2506.13038","n_code_links":0,"syntology":null},{"paper":null,"slug":"lightweight-task-oriented-semantic","title":"Lightweight Task-Oriented Semantic Communication Empowered by Large-Scale AI Models","date":"2025-06-16","arxiv_id":"2506.13243","n_code_links":0,"syntology":null},{"paper":"/paper/seqpe-transformer-with-sequential-position","slug":"seqpe-transformer-with-sequential-position","title":"SeqPE: Transformer with Sequential Position Encoding","date":"2025-06-16","arxiv_id":"2506.13277","n_code_links":1,"syntology":{"ran":9,"of":12,"n_ran_checked":4,"n_instrument":5,"unverified":3,"pointer_only":12,"phrase":"9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 2 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 3 unverified","official":{"repos":["ghrua/seqpe"],"state":"official (archive's flag): 9 ran","n_ran":9,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":3,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"2506-10265","title":"Ground Reaction Force Estimation via Time-aware Knowledge Distillation","date":"2025-06-12","arxiv_id":"2506.10265","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-novel-lightweight-transformer-with-edge","title":"A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning","date":"2025-06-11","arxiv_id":"2506.09429","n_code_links":0,"syntology":null},{"paper":"/paper/2506-08717","slug":"2506-08717","title":"Multi-Teacher Language-Aware Knowledge Distillation for Multilingual Speech Emotion Recognition","date":"2025-06-10","arxiv_id":"2506.08717","n_code_links":1,"syntology":null},{"paper":"/paper/towards-class-wise-fair-adversarial-training","slug":"towards-class-wise-fair-adversarial-training","title":"Towards Class-wise Fair Adversarial Training via Anti-Bias Soft Label Distillation","date":"2025-06-10","arxiv_id":"2506.08611","n_code_links":1,"syntology":null},{"paper":null,"slug":"label-context-dependent-internal-language","title":"Label-Context-Dependent Internal Language Model Estimation for CTC","date":"2025-06-06","arxiv_id":"2506.06096","n_code_links":0,"syntology":null},{"paper":null,"slug":"hdl2v-a-code-translation-dataset-for-enhanced","title":"hdl2v: A Code Translation Dataset for Enhanced LLM Verilog Generation","date":"2025-06-05","arxiv_id":"2506.04544","n_code_links":0,"syntology":null},{"paper":"/paper/static-word-embeddings-for-sentence-semantic","slug":"static-word-embeddings-for-sentence-semantic","title":"Static Word Embeddings for Sentence Semantic Representation","date":"2025-06-05","arxiv_id":"2506.04624","n_code_links":0,"syntology":{"ran":18,"of":22,"n_ran_checked":18,"n_instrument":0,"unverified":4,"pointer_only":0,"phrase":"18 ran (of which 0 constructed an object rather than computing a result; 18 with no instrument failure: 1 honoured, 0 violated, 17 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified","official":null}},{"paper":null,"slug":"building-a-few-shot-cross-domain-multilingual","title":"Building a Few-Shot Cross-Domain Multilingual NLU Model for Customer Care","date":"2025-06-04","arxiv_id":"2506.04389","n_code_links":0,"syntology":null},{"paper":null,"slug":"talkingmachines-real-time-audio-driven","title":"TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models","date":"2025-06-03","arxiv_id":"2506.03099","n_code_links":0,"syntology":null},{"paper":null,"slug":"analyzing-the-importance-of-blank-for-ctc","title":"Analyzing the Importance of Blank for CTC-Based Knowledge Distillation","date":"2025-06-02","arxiv_id":"2506.01503","n_code_links":0,"syntology":null},{"paper":null,"slug":"kdrl-post-training-reasoning-llms-via-unified","title":"KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning","date":"2025-06-02","arxiv_id":"2506.02208","n_code_links":0,"syntology":null},{"paper":null,"slug":"feature-fusion-and-knowledge-distilled-multi","title":"Feature Fusion and Knowledge-Distilled Multi-Modal Multi-Target Detection","date":"2025-05-31","arxiv_id":"2506.00365","n_code_links":0,"syntology":null},{"paper":"/paper/cl-lora-continual-low-rank-adaptation-for","slug":"cl-lora-continual-low-rank-adaptation-for","title":"CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learning","date":"2025-05-30","arxiv_id":"2505.24816","n_code_links":1,"syntology":null},{"paper":null,"slug":"progressive-class-level-distillation","title":"Progressive Class-level Distillation","date":"2025-05-30","arxiv_id":"2505.24310","n_code_links":0,"syntology":null},{"paper":"/paper/revisiting-cross-modal-knowledge-distillation","slug":"revisiting-cross-modal-knowledge-distillation","title":"Revisiting Cross-Modal Knowledge Distillation: A Disentanglement Approach for RGBD Semantic Segmentation","date":"2025-05-30","arxiv_id":"2505.24361","n_code_links":1,"syntology":null},{"paper":null,"slug":"knowledge-distillation-for-reservoir-based","title":"Knowledge Distillation for Reservoir-based Classifier: Human Activity Recognition","date":"2025-05-29","arxiv_id":"2505.22985","n_code_links":0,"syntology":null},{"paper":null,"slug":"sketch-down-the-flops-towards-efficient","title":"Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch","date":"2025-05-29","arxiv_id":"2505.23763","n_code_links":0,"syntology":null},{"paper":"/paper/cast-contrastive-adaptation-and-distillation","slug":"cast-contrastive-adaptation-and-distillation","title":"CAST: Contrastive Adaptation and Distillation for Semi-Supervised Instance Segmentation","date":"2025-05-28","arxiv_id":"2505.21904","n_code_links":0,"syntology":null},{"paper":"/paper/improving-respiratory-sound-classification","slug":"improving-respiratory-sound-classification","title":"Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles","date":"2025-05-28","arxiv_id":"2505.22027","n_code_links":1,"syntology":null},{"paper":null,"slug":"multi-mllm-knowledge-distillation-for-out-of","title":"Multi-MLLM Knowledge Distillation for Out-of-Context News Detection","date":"2025-05-28","arxiv_id":"2505.22517","n_code_links":0,"syntology":null},{"paper":null,"slug":"easydistill-a-comprehensive-toolkit-for","title":"EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models","date":"2025-05-27","arxiv_id":"2505.20888","n_code_links":0,"syntology":null},{"paper":"/paper/doge-defensive-output-generation-for-llm","slug":"doge-defensive-output-generation-for-llm","title":"DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillation","date":"2025-05-26","arxiv_id":"2505.19504","n_code_links":1,"syntology":null},{"paper":null,"slug":"from-data-to-modeling-fully-open-vocabulary","title":"From Data to Modeling: Fully Open-vocabulary Scene Graph Generation","date":"2025-05-26","arxiv_id":"2505.20106","n_code_links":0,"syntology":null},{"paper":null,"slug":"light-distillation-for-incremental-graph","title":"Light distillation for Incremental Graph Convolution Collaborative Filtering","date":"2025-05-26","arxiv_id":"2505.19810","n_code_links":0,"syntology":null},{"paper":null,"slug":"model-stitching-by-functional-latent","title":"Model Stitching by Functional Latent Alignment","date":"2025-05-26","arxiv_id":"2505.20142","n_code_links":0,"syntology":null},{"paper":null,"slug":"mosaic-data-free-knowledge-distillation-via","title":"Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments","date":"2025-05-26","arxiv_id":"2505.19699","n_code_links":0,"syntology":null},{"paper":"/paper/holistic-white-light-polyp-classification-via","slug":"holistic-white-light-polyp-classification-via","title":"Holistic White-light Polyp Classification via Alignment-free Dense Distillation of Auxiliary Optical Chromoendoscopy","date":"2025-05-25","arxiv_id":"2505.19319","n_code_links":1,"syntology":null},{"paper":null,"slug":"online-knowledge-distillation-with-reward","title":"Online Knowledge Distillation with Reward Guidance","date":"2025-05-25","arxiv_id":"2505.18952","n_code_links":0,"syntology":null},{"paper":null,"slug":"remote-sensing-image-classification-with-1","title":"Remote Sensing Image Classification with Decoupled Knowledge Distillation","date":"2025-05-25","arxiv_id":"2505.19111","n_code_links":0,"syntology":null},{"paper":null,"slug":"tokenizing-electron-cloud-in-protein-ligand","title":"Tokenizing Electron Cloud in Protein-Ligand Interaction Learning","date":"2025-05-25","arxiv_id":"2505.19014","n_code_links":0,"syntology":null},{"paper":null,"slug":"c3r-channel-conditioned-cell-representations","title":"C3R: Channel Conditioned Cell Representations for unified evaluation in microscopy imaging","date":"2025-05-24","arxiv_id":"2505.18745","n_code_links":0,"syntology":null},{"paper":"/paper/knowledge-grafting-of-large-language-models","slug":"knowledge-grafting-of-large-language-models","title":"Knowledge Grafting of Large Language Models","date":"2025-05-24","arxiv_id":"2505.18502","n_code_links":1,"syntology":null},{"paper":null,"slug":"single-snapshot-distillation-for-phase-coded","title":"Single Snapshot Distillation for Phase Coded Mask Design in Phase Retrieval","date":"2025-05-23","arxiv_id":"2505.18352","n_code_links":0,"syntology":null},{"paper":null,"slug":"sedd-pcc-a-single-encoder-dual-decoder","title":"SEDD-PCC: A Single Encoder-Dual Decoder Framework For End-To-End Learned Point Cloud Compression","date":"2025-05-22","arxiv_id":"2505.16709","n_code_links":0,"syntology":null},{"paper":null,"slug":"todi-token-wise-distillation-via-fine-grained","title":"ToDi: Token-wise Distillation via Fine-Grained Divergence Control","date":"2025-05-22","arxiv_id":"2505.16297","n_code_links":0,"syntology":null},{"paper":"/paper/deepkd-a-deeply-decoupled-and-denoised","slug":"deepkd-a-deeply-decoupled-and-denoised","title":"DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer","date":"2025-05-21","arxiv_id":"2505.15133","n_code_links":1,"syntology":null},{"paper":"/paper/deliberation-on-priors-trustworthy-reasoning","slug":"deliberation-on-priors-trustworthy-reasoning","title":"Deliberation on Priors: Trustworthy Reasoning of Large Language Models on Knowledge Graphs","date":"2025-05-21","arxiv_id":"2505.15210","n_code_links":1,"syntology":{"ran":5,"of":14,"n_ran_checked":2,"n_instrument":3,"unverified":9,"pointer_only":0,"phrase":"5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 9 unverified","official":{"repos":["reml-group/deliberation-on-priors"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":9,"ran_from_kinds":["found_in_text","official"]}}},{"paper":null,"slug":"mentalmac-enhancing-large-language-models-for","title":"MentalMAC: Enhancing Large Language Models for Detecting Mental Manipulation via Multi-Task Anti-Curriculum Distillation","date":"2025-05-21","arxiv_id":"2505.15255","n_code_links":0,"syntology":null},{"paper":"/paper/uwsam-segment-anything-model-guided","slug":"uwsam-segment-anything-model-guided","title":"UWSAM: Segment Anything Model Guided Underwater Instance Segmentation and A Large-scale Benchmark Dataset","date":"2025-05-21","arxiv_id":"2505.15581","n_code_links":1,"syntology":null},{"paper":null,"slug":"bridge-the-gap-between-past-and-future","title":"Bridge the Gap between Past and Future: Siamese Model Optimization for Context-Aware Document Ranking","date":"2025-05-20","arxiv_id":"2505.14180","n_code_links":0,"syntology":null},{"paper":null,"slug":"ground-v-teaching-vlms-to-ground-complex","title":"Ground-V: Teaching VLMs to Ground Complex Instructions in Pixels","date":"2025-05-20","arxiv_id":"2505.13788","n_code_links":0,"syntology":null},{"paper":null,"slug":"interpretable-traces-unexpected-outcomes","title":"Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillation","date":"2025-05-20","arxiv_id":"2505.13792","n_code_links":0,"syntology":null},{"paper":"/paper/a-token-is-worth-over-1000-tokens-efficient","slug":"a-token-is-worth-over-1000-tokens-efficient","title":"A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone","date":"2025-05-19","arxiv_id":"2505.12781","n_code_links":1,"syntology":null},{"paper":null,"slug":"bridging-the-modality-gap-enhancing-channel","title":"Bridging the Modality Gap: Enhancing Channel Prediction with Semantically Aligned LLMs and Knowledge Distillation","date":"2025-05-19","arxiv_id":"2505.12729","n_code_links":0,"syntology":null},{"paper":null,"slug":"orqa-a-benchmark-and-foundation-model-for","title":"ORQA: A Benchmark and Foundation Model for Holistic Operating Room Modeling","date":"2025-05-19","arxiv_id":"2505.12890","n_code_links":0,"syntology":null},{"paper":"/paper/robust-multimodal-segmentation-with","slug":"robust-multimodal-segmentation-with","title":"Robust Multimodal Segmentation with Representation Regularization and Hybrid Prototype Distillation","date":"2025-05-19","arxiv_id":"2505.12861","n_code_links":1,"syntology":null},{"paper":"/paper/smotext-smote-meets-large-language-models","slug":"smotext-smote-meets-large-language-models","title":"SMOTExT: SMOTE meets Large Language Models","date":"2025-05-19","arxiv_id":"2505.13434","n_code_links":1,"syntology":null},{"paper":"/paper/towards-low-latency-event-stream-based-visual","slug":"towards-low-latency-event-stream-based-visual","title":"Towards Low-Latency Event Stream-based Visual Object Tracking: A Slow-Fast Approach","date":"2025-05-19","arxiv_id":"2505.12903","n_code_links":1,"syntology":null},{"paper":"/paper/uniformity-first-uniformity-aware-test-time","slug":"uniformity-first-uniformity-aware-test-time","title":"Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruption","date":"2025-05-19","arxiv_id":"2505.12912","n_code_links":1,"syntology":null},{"paper":"/paper/always-clear-depth-robust-monocular-depth","slug":"always-clear-depth-robust-monocular-depth","title":"Always Clear Depth: Robust Monocular Depth Estimation under Adverse Weather","date":"2025-05-18","arxiv_id":"2505.12199","n_code_links":1,"syntology":null},{"paper":null,"slug":"lameta-intent-aware-agentic-network","title":"LAMeTA: Intent-Aware Agentic Network Optimization via a Large AI Model-Empowered Two-Stage Approach","date":"2025-05-18","arxiv_id":"2505.12247","n_code_links":0,"syntology":null},{"paper":null,"slug":"ssr-enhancing-depth-perception-in-vision","title":"SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning","date":"2025-05-18","arxiv_id":"2505.12448","n_code_links":0,"syntology":null},{"paper":null,"slug":"figkd-fine-grained-knowledge-distillation-via","title":"FiGKD: Fine-Grained Knowledge Distillation via High-Frequency Detail Transfer","date":"2025-05-17","arxiv_id":"2505.11897","n_code_links":0,"syntology":null},{"paper":"/paper/on-membership-inference-attacks-in-knowledge","slug":"on-membership-inference-attacks-in-knowledge","title":"On Membership Inference Attacks in Knowledge Distillation","date":"2025-05-17","arxiv_id":"2505.11837","n_code_links":1,"syntology":null},{"paper":"/paper/2505-10822","slug":"2505-10822","title":"Distilled Circuits: A Mechanistic Study of Internal Restructuring in Knowledge Distillation","date":"2025-05-16","arxiv_id":"2505.10822","n_code_links":1,"syntology":null},{"paper":null,"slug":"2505-11100","title":"Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors","date":"2025-05-16","arxiv_id":"2505.11100","n_code_links":0,"syntology":null},{"paper":null,"slug":"semantically-aware-game-image-quality","title":"Semantically-Aware Game Image Quality Assessment","date":"2025-05-16","arxiv_id":"2505.11724","n_code_links":0,"syntology":null},{"paper":null,"slug":"2505-10649","title":"Advancing Multiple Instance Learning with Continual Learning for Whole Slide Imaging","date":"2025-05-15","arxiv_id":"2505.10649","n_code_links":0,"syntology":null},{"paper":null,"slug":"dcsnet-a-lightweight-knowledge-distillation","title":"DCSNet: A Lightweight Knowledge Distillation-Based Model with Explainable AI for Lung Cancer Diagnosis from Histopathological Images","date":"2025-05-14","arxiv_id":"2505.09334","n_code_links":0,"syntology":null},{"paper":"/paper/foundation-models-knowledge-distillation-for","slug":"foundation-models-knowledge-distillation-for","title":"Foundation Models Knowledge Distillation For Battery Capacity Degradation Forecast","date":"2025-05-13","arxiv_id":"2505.08151","n_code_links":1,"syntology":null},{"paper":null,"slug":"low-complexity-inference-in-continual","title":"Low-Complexity Inference in Continual Learning via Compressed Knowledge Transfer","date":"2025-05-13","arxiv_id":"2505.08327","n_code_links":0,"syntology":null},{"paper":null,"slug":"mokd-multi-task-optimization-for-knowledge","title":"MoKD: Multi-Task Optimization for Knowledge Distillation","date":"2025-05-13","arxiv_id":"2505.08170","n_code_links":0,"syntology":null},{"paper":null,"slug":"channel-fingerprint-construction-for-massive","title":"Channel Fingerprint Construction for Massive MIMO: A Deep Conditional Generative Approach","date":"2025-05-12","arxiv_id":"2505.07893","n_code_links":0,"syntology":null},{"paper":null,"slug":"kdh-mltc-knowledge-distillation-for","title":"KDH-MLTC: Knowledge Distillation for Healthcare Multi-Label Text Classification","date":"2025-05-12","arxiv_id":"2505.07162","n_code_links":0,"syntology":null},{"paper":null,"slug":"ranking-aware-continual-learning-for-lidar","title":"Ranking-aware Continual Learning for LiDAR Place Recognition","date":"2025-05-12","arxiv_id":"2505.07198","n_code_links":0,"syntology":null},{"paper":"/paper/simple-semi-supervised-knowledge-distillation","slug":"simple-semi-supervised-knowledge-distillation","title":"Simple Semi-supervised Knowledge Distillation from Vision-Language Models via $\\mathbf{\\texttt{D}}$ual-$\\mathbf{\\texttt{H}}$ead $\\mathbf{\\texttt{O}}$ptimization","date":"2025-05-12","arxiv_id":"2505.07675","n_code_links":1,"syntology":null},{"paper":"/paper/topology-guided-knowledge-distillation-for","slug":"topology-guided-knowledge-distillation-for","title":"Topology-Guided Knowledge Distillation for Efficient Point Cloud Processing","date":"2025-05-12","arxiv_id":"2505.08101","n_code_links":1,"syntology":null},{"paper":null,"slug":"robust-precise-knowledge-distillation-based","title":"Robust & Precise Knowledge Distillation-based Novel Context-Aware Predictor for Disease Detection in Brain and Gastrointestinal","date":"2025-05-09","arxiv_id":"2505.06381","n_code_links":0,"syntology":null},{"paper":"/paper/biomed-dpt-dual-modality-prompt-tuning-for","slug":"biomed-dpt-dual-modality-prompt-tuning-for","title":"Biomed-DPT: Dual Modality Prompt Tuning for Biomedical Vision-Language Models","date":"2025-05-08","arxiv_id":"2505.05189","n_code_links":1,"syntology":null},{"paper":null,"slug":"federated-deconfounding-and-debiasing","title":"Federated Deconfounding and Debiasing Learning for Out-of-Distribution Generalization","date":"2025-05-08","arxiv_id":"2505.04979","n_code_links":0,"syntology":null},{"paper":null,"slug":"theoretical-guarantees-for-lt-ttd-a-unified","title":"Theoretical Guarantees for LT-TTD: A Unified Transformer-based Architecture for Two-Level Ranking Systems","date":"2025-05-07","arxiv_id":"2505.04434","n_code_links":0,"syntology":null},{"paper":null,"slug":"image-recognition-with-online-lightweight","title":"Image Recognition with Online Lightweight Vision Transformer: A Survey","date":"2025-05-06","arxiv_id":"2505.03113","n_code_links":0,"syntology":null},{"paper":null,"slug":"knowledge-distillation-for-speech-denoising","title":"Knowledge Distillation for Speech Denoising by Latent Representation Alignment with Cosine Distance","date":"2025-05-06","arxiv_id":"2505.03442","n_code_links":0,"syntology":null}],"papers_shown":100,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":3071,"with_a_code_link":1258,"where_syntology_ran_a_sample":320,"not_listed_spam_title":0,"listed":3071,"listed_where_code_ran":320,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":263,"every_run_a_failure_of_syntologys_instrument":57,"listed_with_a_run_with_no_instrument_failure":263,"listed_every_run_a_failure_of_syntologys_instrument":57,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":320,"papers_where_code_ran_first_page":"/method/knowledge-distillation/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"archive","page":2,"of":31,"url":"/method/knowledge-distillation/papers/2","json":"/method/knowledge-distillation/papers/2.json","rows":100},{"order":"archive","page":3,"of":31,"url":"/method/knowledge-distillation/papers/3","json":"/method/knowledge-distillation/papers/3.json","rows":100},{"order":"archive","page":4,"of":31,"url":"/method/knowledge-distillation/papers/4","json":"/method/knowledge-distillation/papers/4.json","rows":100},{"order":"archive","page":5,"of":31,"url":"/method/knowledge-distillation/papers/5","json":"/method/knowledge-distillation/papers/5.json","rows":100},{"order":"archive","page":6,"of":31,"url":"/method/knowledge-distillation/papers/6","json":"/method/knowledge-distillation/papers/6.json","rows":100},{"order":"archive","page":7,"of":31,"url":"/method/knowledge-distillation/papers/7","json":"/method/knowledge-distillation/papers/7.json","rows":100},{"order":"archive","page":8,"of":31,"url":"/method/knowledge-distillation/papers/8","json":"/method/knowledge-distillation/papers/8.json","rows":100},{"order":"archive","page":9,"of":31,"url":"/method/knowledge-distillation/papers/9","json":"/method/knowledge-distillation/papers/9.json","rows":100},{"order":"archive","page":10,"of":31,"url":"/method/knowledge-distillation/papers/10","json":"/method/knowledge-distillation/papers/10.json","rows":100},{"order":"archive","page":11,"of":31,"url":"/method/knowledge-distillation/papers/11","json":"/method/knowledge-distillation/papers/11.json","rows":100},{"order":"archive","page":12,"of":31,"url":"/method/knowledge-distillation/papers/12","json":"/method/knowledge-distillation/papers/12.json","rows":100},{"order":"archive","page":13,"of":31,"url":"/method/knowledge-distillation/papers/13","json":"/method/knowledge-distillation/papers/13.json","rows":100},{"order":"archive","page":14,"of":31,"url":"/method/knowledge-distillation/papers/14","json":"/method/knowledge-distillation/papers/14.json","rows":100},{"order":"archive","page":15,"of":31,"url":"/method/knowledge-distillation/papers/15","json":"/method/knowledge-distillation/papers/15.json","rows":100},{"order":"archive","page":16,"of":31,"url":"/method/knowledge-distillation/papers/16","json":"/method/knowledge-distillation/papers/16.json","rows":100},{"order":"archive","page":17,"of":31,"url":"/method/knowledge-distillation/papers/17","json":"/method/knowledge-distillation/papers/17.json","rows":100},{"order":"archive","page":18,"of":31,"url":"/method/knowledge-distillation/papers/18","json":"/method/knowledge-distillation/papers/18.json","rows":100},{"order":"archive","page":19,"of":31,"url":"/method/knowledge-distillation/papers/19","json":"/method/knowledge-distillation/papers/19.json","rows":100},{"order":"archive","page":20,"of":31,"url":"/method/knowledge-distillation/papers/20","json":"/method/knowledge-distillation/papers/20.json","rows":100},{"order":"archive","page":21,"of":31,"url":"/method/knowledge-distillation/papers/21","json":"/method/knowledge-distillation/papers/21.json","rows":100},{"order":"archive","page":22,"of":31,"url":"/method/knowledge-distillation/papers/22","json":"/method/knowledge-distillation/papers/22.json","rows":100},{"order":"archive","page":23,"of":31,"url":"/method/knowledge-distillation/papers/23","json":"/method/knowledge-distillation/papers/23.json","rows":100},{"order":"archive","page":24,"of":31,"url":"/method/knowledge-distillation/papers/24","json":"/method/knowledge-distillation/papers/24.json","rows":100},{"order":"archive","page":25,"of":31,"url":"/method/knowledge-distillation/papers/25","json":"/method/knowledge-distillation/papers/25.json","rows":100},{"order":"archive","page":26,"of":31,"url":"/method/knowledge-distillation/papers/26","json":"/method/knowledge-distillation/papers/26.json","rows":100},{"order":"archive","page":27,"of":31,"url":"/method/knowledge-distillation/papers/27","json":"/method/knowledge-distillation/papers/27.json","rows":100},{"order":"archive","page":28,"of":31,"url":"/method/knowledge-distillation/papers/28","json":"/method/knowledge-distillation/papers/28.json","rows":100},{"order":"archive","page":29,"of":31,"url":"/method/knowledge-distillation/papers/29","json":"/method/knowledge-distillation/papers/29.json","rows":100},{"order":"archive","page":30,"of":31,"url":"/method/knowledge-distillation/papers/30","json":"/method/knowledge-distillation/papers/30.json","rows":100},{"order":"archive","page":31,"of":31,"url":"/method/knowledge-distillation/papers/31","json":"/method/knowledge-distillation/papers/31.json","rows":71},{"order":"ran","page":1,"of":4,"url":"/method/knowledge-distillation/papers/ran/1","json":"/method/knowledge-distillation/papers/ran/1.json","rows":100},{"order":"ran","page":2,"of":4,"url":"/method/knowledge-distillation/papers/ran/2","json":"/method/knowledge-distillation/papers/ran/2.json","rows":100},{"order":"ran","page":3,"of":4,"url":"/method/knowledge-distillation/papers/ran/3","json":"/method/knowledge-distillation/papers/ran/3.json","rows":100},{"order":"ran","page":4,"of":4,"url":"/method/knowledge-distillation/papers/ran/4","json":"/method/knowledge-distillation/papers/ran/4.json","rows":20}],"tasks":[{"task":"/task/knowledge-distillation","name":"Knowledge Distillation","papers":3039},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":311},{"task":"/task/model-compression","name":"Model Compression","papers":253},{"task":"/task/object-detection","name":"Object Detection","papers":208},{"task":"/task/object-detection-1","name":"object-detection","papers":205},{"task":"/task/image-classification","name":"Image Classification","papers":202},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":199},{"task":"/task/image-classification","name":"image-classification","papers":179},{"task":"/task/language-modelling","name":"Language Modelling","papers":168},{"task":"/task/language-modeling","name":"Language Modeling","papers":133},{"task":"/task/federated-learning","name":"Federated Learning","papers":123},{"task":"/task/representation-learning","name":"Representation Learning","papers":112},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":111},{"task":"/task/translation","name":"Translation","papers":108},{"task":"/task/segmentation","name":"Segmentation","papers":105},{"task":"/task/object","name":"Object","papers":99},{"task":"/task/incremental-learning","name":"Incremental Learning","papers":98},{"task":"/task/quantization","name":"Quantization","papers":97},{"task":"/task/contrastive-learning","name":"Contrastive Learning","papers":96},{"task":"/task/continual-learning","name":"Continual Learning","papers":93}],"tasks_shown":20,"n_tasks":1024,"usage_by_year":[{"year":"2015","papers":1},{"year":"2016","papers":4},{"year":"2017","papers":8},{"year":"2018","papers":33},{"year":"2019","papers":130},{"year":"2020","papers":271},{"year":"2021","papers":423},{"year":"2022","papers":506},{"year":"2023","papers":637},{"year":"2024","papers":739},{"year":"2025","papers":319}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/knowledge-distillation"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=Knowledge%20Distillation","read_at":"2026-10-05T00:58:18+00:00","query":"Knowledge Distillation","kind":"method","have":"declared_code","tier":"repo_known","verified_implementation":false,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":{"method":"Knowledge Distillation","origin_arxiv_id":"2004.09813"},"resolution":"exact","resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":"different_paper","archive_paper_arxiv_id":"1503.02531","repositories":{"listed":5,"in_graph":12,"list_truncated":true,"strongest_link":"official"},"retrieve_with":null,"name_competition":{"verdict":"outranked","may_be_wrong_sense":true,"bearers_found":7,"served":{"arxiv_id":"2004.09813","citations":1403},"leading_other_bearer":{"arxiv_id":"2006.05525","title":"Knowledge Distillation: A Survey","citations":4554,"named_by":["title_prefix"],"bearer_name":"Knowledge Distillation: A Survey","ask_with":"have('2006.05525')"},"n_other_bearers_listed":5,"note":"'Knowledge Distillation' is ALSO the name of the paper 'Knowledge Distillation: A Survey' (arXiv:2006.05525, 4,554 citations), which is cited FAR more than the paper this answer is about (1,403). This answer may be about the wrong Knowledge Distillation. If you meant the better-known one, ask again with that entry of `also_named`."},"name_competition_text":"The same name also belongs to the paper “Knowledge Distillation: A Survey” (arXiv:2006.05525, 4,554 citations), which the lookup finds cited far more than the paper this answer is about (1,403 citations), so this answer may be about a different sense of the name.","near_matches":null,"code_nearby":[{"name":"knowledge fusion","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":3,"retrieve_with":"get_reference_implementation(name='knowledge fusion')","origin_arxiv_id":"2401.10491"},{"name":"online distillation","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":2,"retrieve_with":"get_reference_implementation(name='online distillation')","origin_arxiv_id":"2301.12245"},{"name":"EM Distillation (EMD)","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":3,"retrieve_with":"get_reference_implementation(name='EM Distillation (EMD)')","origin_arxiv_id":"2405.16852"},{"name":"Weighted Distillation","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":2,"retrieve_with":"get_reference_implementation(name='Weighted Distillation')","origin_arxiv_id":"2210.06711"},{"name":"Behaviour Distillation","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":2,"retrieve_with":"get_reference_implementation(name='Behaviour Distillation')","origin_arxiv_id":"2406.15042"},{"name":"RFA Distillation (RFAD)","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":3,"retrieve_with":"get_reference_implementation(name='RFA Distillation (RFAD)')","origin_arxiv_id":"2210.12067"}],"code_nearby_caption":"Code Syntology holds under other names, as the lookup lists them (code_nearby); not a complete list, and none of them is this method.","origin_paper":{"arxiv_id":"2004.09813","page":"/paper/making-monolingual-sentence-embeddings","syntology_url":"https://syntology.ai/paper/2004.09813","in_graph_layer":true,"samples":1,"ran":0,"n_ran_checked":0,"n_instrument":0,"unverified":1,"phrase":"0 ran · 1 unverified","instrument_means":"failures of Syntology's instrument, not of the code"},"line":"about another paper, arXiv:2004.09813: repository known, no verified implementation (declared_code; repositories: 5 listed of 12 in the graph, strongest link: official). About another paper: this answer is about arXiv:2004.09813, while the archive credits arXiv:1503.02531.","level_legend":{"state":"quoted","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","card_sha256":"479785a84658dd682a11b10237b3620d651d7f87799f5dc7add970aaa83f6d39","note":"Syntology's served verification levels, quoted verbatim from its MCP server card; a row is split where the card separates clauses with a double hyphen, and the separator is not reproduced","levels":[{"level":"V0","quoted":["imports"]},{"level":"V1","quoted":["executes against a synthesized fixture and returns a value of the recorded type"]},{"level":"V2","quoted":["two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs","neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)"]},{"level":"V3","quoted":["paper-derived property tests pass on both of those same two generations."]}],"no_rung":"No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results."}},"syntology_have_state":"loaded","record_sha256":"2057dd9b43f33e5a34249560df9e37154eb1f1d126890d190a1c3f84d16e018b","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}