{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/knowledge-distillation-via-route-constrained","title":"Knowledge Distillation via Route Constrained Optimization","arxiv_id":"1904.09149","date":"2019-04-19","proceeding":"ICCV 2019 10","authors":["Xiao Jin","Baoyun Peng","Yi-Chao Wu","Yu Liu","Jiaheng Liu","Ding Liang","Xiaolin Hu"],"abstract":"Distillation-based learning boosts the performance of the miniaturized neural\nnetwork based on the hypothesis that the representation of a teacher model can\nbe used as structured and relatively weak supervision, and thus would be easily\nlearned by a miniaturized model. However, we find that the representation of a\nconverged heavy model is still a strong constraint for training a small student\nmodel, which leads to a high lower bound of congruence loss. In this work,\ninspired by curriculum learning we consider the knowledge distillation from the\nperspective of curriculum learning by routing. Instead of supervising the\nstudent model with a converged teacher model, we supervised it with some anchor\npoints selected from the route in parameter space that the teacher model passed\nby, as we called route constrained optimization (RCO). We experimentally\ndemonstrate this simple operation greatly reduces the lower bound of congruence\nloss for knowledge distillation, hint and mimicking learning. On close-set\nclassification tasks like CIFAR100 and ImageNet, RCO improves knowledge\ndistillation by 2.14% and 1.5% respectively. For the sake of evaluating the\ngeneralization, we also test RCO on the open-set face recognition task\nMegaFace.","url_abs":"http://arxiv.org/abs/1904.09149v1","url_pdf":"http://arxiv.org/pdf/1904.09149v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"knowledge-distillation-via-route-constrained","repo_url":"https://github.com/SforAiDl/KD_Lib","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"face-recognition","task_name":"Face Recognition"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1904.09149","atlas_url":"https://app.syntology.ai/?focus=1904.09149","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1904.09149"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1904.09149\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1904.09149"},"call":"get_concepts_for_paper(arxiv_id=\"1904.09149\")"}],"by_repo":[{"repo":"SforAiDl/KD_Lib","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["1911.07471"],"harvested_for_calls":[{"arxiv_id":"1911.07471","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1911.07471"},"call":"get_harvested_code_for_paper(arxiv_id=\"1911.07471\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"1b672c6e2ab59a986a7d58893b7c59111e274529562a47834aa8fc35136bb01e","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}