{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/efficient-contextualized-representation","title":"Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling","arxiv_id":"1804.07827","date":"2018-04-20","proceeding":"EMNLP 2018 10","authors":["Liyuan Liu","Xiang Ren","Jingbo Shang","Jian Peng","Jiawei Han"],"abstract":"Many efforts have been made to facilitate natural language processing tasks\nwith pre-trained language models (LMs), and brought significant improvements to\nvarious applications. To fully leverage the nearly unlimited corpora and\ncapture linguistic information of multifarious levels, large-size LMs are\nrequired; but for a specific task, only parts of these information are useful.\nSuch large-sized LMs, even in the inference stage, may cause heavy computation\nworkloads, making them too time-consuming for large-scale applications. Here we\npropose to compress bulky LMs while preserving useful information with regard\nto a specific task. As different layers of the model keep different\ninformation, we develop a layer selection method for model pruning using\nsparsity-inducing regularization. By introducing the dense connectivity, we can\ndetach any layer without affecting others, and stretch shallow and wide LMs to\nbe deep and narrow. In model training, LMs are learned with layer-wise dropouts\nfor better robustness. Experiments on two benchmark datasets demonstrate the\neffectiveness of our method.","url_abs":"http://arxiv.org/abs/1804.07827v2","url_pdf":"http://arxiv.org/pdf/1804.07827v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-contextualized-representation","repo_url":"https://github.com/LiyuanLucasLiu/LD-Net","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/named-entity-recognition-ner-on-conll-2003","task":"Named Entity Recognition (NER)","dataset":"CoNLL 2003 (English)","model":"LD-Net","rank_in_archive_order":49,"of":73,"metrics":{"F1":"92.03"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1804.07827","atlas_url":"https://app.syntology.ai/?focus=1804.07827","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1804.07827"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1804.07827\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1804.07827"},"call":"get_concepts_for_paper(arxiv_id=\"1804.07827\")"}],"by_repo":[{"repo":"LiyuanLucasLiu/LD-Net","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"571ef3e3137d04a45db95c789ae991b47e1cf54e7f07d5fd5df7728339216cc2","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}