{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/prune-once-for-all-sparse-pre-trained","title":"Prune Once for All: Sparse Pre-Trained Language Models","arxiv_id":"2111.05754","date":"2021-11-10","proceeding":null,"authors":["Ofir Zafrir","Ariel Larey","Guy Boudoukh","Haihao Shen","Moshe Wasserblat"],"abstract":"Transformer-based language models are applied to a wide range of applications in natural language processing. However, they are inefficient and difficult to deploy. In recent years, many compression algorithms have been proposed to increase the implementation efficiency of large Transformer-based models on target hardware. In this work we present a new method for training sparse pre-trained Transformer language models by integrating weight pruning and model distillation. These sparse pre-trained models can be used to transfer learning for a wide range of tasks while maintaining their sparsity pattern. We demonstrate our method with three known architectures to create sparse pre-trained BERT-Base, BERT-Large and DistilBERT. We show how the compressed sparse pre-trained models we trained transfer their knowledge to five different downstream natural language tasks with minimal accuracy loss. Moreover, we show how to further compress the sparse models' weights to 8bit precision using quantization-aware training. For example, with our sparse pre-trained BERT-Large fine-tuned on SQuADv1.1 and quantized to 8bit we achieve a compression ratio of $40$X for the encoder with less than $1\\%$ accuracy loss. To the best of our knowledge, our results show the best compression-to-accuracy ratio for BERT-Base, BERT-Large, and DistilBERT.","url_abs":"https://arxiv.org/abs/2111.05754v1","url_pdf":"https://arxiv.org/pdf/2111.05754v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prune-once-for-all-sparse-pre-trained","repo_url":"https://github.com/intellabs/model-compression-research-package","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"paper_slug":"prune-once-for-all-sparse-pre-trained","repo_url":"https://github.com/intel/intel-extension-for-transformers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"distilbert","method_name":"DistilBERT"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/natural-language-inference-on-multinli-dev","task":"Natural Language Inference","dataset":"MultiNLI Dev","model":"BERT-Large-uncased-PruneOFA (90% unstruct sparse)","rank_in_archive_order":2,"of":10,"metrics":{"Matched":"83.74","Mismatched":"84.2"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-multinli-dev","task":"Natural Language Inference","dataset":"MultiNLI Dev","model":"BERT-Large-uncased-PruneOFA (90% unstruct sparse, QAT Int8)","rank_in_archive_order":3,"of":10,"metrics":{"Matched":"83.47","Mismatched":"84.08"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-multinli-dev","task":"Natural Language Inference","dataset":"MultiNLI Dev","model":"BERT-Base-uncased-PruneOFA (85% unstruct sparse)","rank_in_archive_order":4,"of":10,"metrics":{"Matched":"82.71","Mismatched":"83.67"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-multinli-dev","task":"Natural Language Inference","dataset":"MultiNLI Dev","model":"BERT-Base-uncased-PruneOFA (90% unstruct sparse)","rank_in_archive_order":5,"of":10,"metrics":{"Matched":"81.45","Mismatched":"82.43"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-multinli-dev","task":"Natural Language Inference","dataset":"MultiNLI Dev","model":"BERT-Base-uncased-PruneOFA (85% unstruct sparse, QAT Int8)","rank_in_archive_order":6,"of":10,"metrics":{"Matched":"81.4","Mismatched":"82.51"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-multinli-dev","task":"Natural Language Inference","dataset":"MultiNLI Dev","model":"DistilBERT-uncased-PruneOFA (85% unstruct sparse)","rank_in_archive_order":7,"of":10,"metrics":{"Matched":"81.35","Mismatched":"82.03"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-multinli-dev","task":"Natural Language Inference","dataset":"MultiNLI Dev","model":"DistilBERT-uncased-PruneOFA (90% unstruct sparse)","rank_in_archive_order":8,"of":10,"metrics":{"Matched":"80.68","Mismatched":"81.47"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-multinli-dev","task":"Natural Language Inference","dataset":"MultiNLI Dev","model":"DistilBERT-uncased-PruneOFA (85% unstruct sparse, QAT Int8)","rank_in_archive_order":9,"of":10,"metrics":{"Matched":"80.66","Mismatched":"81.14"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-multinli-dev","task":"Natural Language Inference","dataset":"MultiNLI Dev","model":"DistilBERT-uncased-PruneOFA (90% unstruct sparse, QAT Int8)","rank_in_archive_order":10,"of":10,"metrics":{"Matched":"78.8","Mismatched":"80.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-squad11-dev","task":"Question Answering","dataset":"SQuAD1.1 dev","model":"BERT-Large-uncased-PruneOFA (90% unstruct sparse)","rank_in_archive_order":10,"of":55,"metrics":{"EM":"83.35","F1":"90.2"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-squad11-dev","task":"Question Answering","dataset":"SQuAD1.1 dev","model":"BERT-Large-uncased-PruneOFA (90% unstruct sparse, QAT Int8)","rank_in_archive_order":11,"of":55,"metrics":{"EM":"83.22","F1":"90.02"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-squad11-dev","task":"Question Answering","dataset":"SQuAD1.1 dev","model":"BERT-Base-uncased-PruneOFA (85% unstruct sparse)","rank_in_archive_order":12,"of":55,"metrics":{"EM":"81.1","F1":"88.42"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-squad11-dev","task":"Question Answering","dataset":"SQuAD1.1 dev","model":"BERT-Base-uncased-PruneOFA (85% unstruct sparse, QAT Int8)","rank_in_archive_order":13,"of":55,"metrics":{"EM":"80.84","F1":"88.24"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-squad11-dev","task":"Question Answering","dataset":"SQuAD1.1 dev","model":"BERT-Base-uncased-PruneOFA (90% unstruct sparse)","rank_in_archive_order":14,"of":55,"metrics":{"EM":"79.83","F1":"87.25"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-squad11-dev","task":"Question Answering","dataset":"SQuAD1.1 dev","model":"DistilBERT-uncased-PruneOFA (85% unstruct sparse)","rank_in_archive_order":19,"of":55,"metrics":{"EM":"78.1","F1":"85.82"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-squad11-dev","task":"Question Answering","dataset":"SQuAD1.1 dev","model":"DistilBERT-uncased-PruneOFA (85% unstruct sparse, QAT Int8)","rank_in_archive_order":21,"of":55,"metrics":{"EM":"77.03","F1":"85.13"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-squad11-dev","task":"Question Answering","dataset":"SQuAD1.1 dev","model":"DistilBERT-uncased-PruneOFA (90% unstruct sparse)","rank_in_archive_order":22,"of":55,"metrics":{"EM":"76.91","F1":"84.82"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-squad11-dev","task":"Question Answering","dataset":"SQuAD1.1 dev","model":"DistilBERT-uncased-PruneOFA (90% unstruct sparse, QAT Int8)","rank_in_archive_order":25,"of":55,"metrics":{"EM":"75.62","F1":"83.87"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2111.05754","atlas_url":"https://app.syntology.ai/?focus=2111.05754","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2111.05754"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2111.05754\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2111.05754"},"call":"get_concepts_for_paper(arxiv_id=\"2111.05754\")"}],"by_repo":[{"repo":"intellabs/model-compression-research-package","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"intel/intel-extension-for-transformers","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":2,"harvested_for":["2309.17453","2311.16133"],"harvested_for_calls":[{"arxiv_id":"2309.17453","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2309.17453"},"call":"get_harvested_code_for_paper(arxiv_id=\"2309.17453\")"},{"arxiv_id":"2311.16133","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2311.16133"},"call":"get_harvested_code_for_paper(arxiv_id=\"2311.16133\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"013f777f2226b2d7f303eac807e342618d42c0861d5b4cb0e0d3427d1a9711ef","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}