{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/r-2-range-regularization-for-model","title":"R2 Loss: Range Restriction Loss for Model Compression and Quantization","arxiv_id":"2303.08253","date":"2023-03-14","proceeding":null,"authors":["Arnav Kundu","Chungkuk Yoo","Srijan Mishra","Minsik Cho","Saurabh Adya"],"abstract":"Model quantization and compression is widely used techniques to reduce usage of computing resource at inference time. While state-of-the-art works have been achieved reasonable accuracy with higher bit such as 4bit or 8bit, but still it is challenging to quantize/compress a model further, e.g., 1bit or 2bit. To overcome the challenge, we focus on outliers in weights of a pre-trained model which disrupt effective lower bit quantization and compression. In this work, we propose Range Restriction Loss (R2-Loss) for building lower bit quantization and compression friendly models by removing outliers from weights during pre-training. By effectively restricting range of weights, we mold the overall distribution into a tight shape to ensure high quantization bit resolution, therefore allowing model compression and quantization techniques can to utilize their limited numeric representation powers better. We introduce three different, L-inf R2-Loss, its extension Margin R2-Loss and a new Soft-Min-MaxR2-Loss to be used as an auxiliary loss during full-precision model training. These R2-Loss can be used in different cases such as L-inf and Margin R2-Loss would be effective for symmetric quantization, while Soft-Min-Max R2-Loss shows better performance for model compression. In our experiment, R2-Loss improves lower bit quantization accuracy with state-of-the-art post-training quantization (PTQ), quantization-aware training (QAT), and model compression techniques. With R2-Loss, MobileNet-V2 2bit weight and 8bit activation PTQ, MobileNet-V1 2bit weight and activation QAT, ResNet18 1bit weight compression are improved to 59.49% from 50.66%, 59.05% from 55.96%, and 52.58% from 45.54%, respectively.","url_abs":"https://arxiv.org/abs/2303.08253v2","url_pdf":"https://arxiv.org/pdf/2303.08253v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"model-optimization","task_name":"Model Optimization"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"depthwise-convolution","method_name":"Depthwise Convolution"},{"method_slug":"depthwise-separable-convolution","method_name":"Depthwise Separable Convolution"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"mobilenetv1","method_name":"MobileNetV1"},{"method_slug":"pointwise-convolution","method_name":"Pointwise Convolution"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"ResNet-18 + 4bit-1dim model compression using DKM","rank_in_archive_order":3,"of":12,"metrics":{"Top-1":"70.52"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"MobileNet-v1 + 4bit-1dim model compression using DKM","rank_in_archive_order":4,"of":12,"metrics":{"Top-1":"69.63"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"ResNet-18 + 2bit-1dim model compression using DKM","rank_in_archive_order":5,"of":12,"metrics":{"Top-1":"68.63"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"MobileNet-v1 + 2bit-1dim model compression using DKM","rank_in_archive_order":6,"of":12,"metrics":{"Top-1":"67.62"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"ResNet-18 + 4bit-4dim model compression using DKM","rank_in_archive_order":7,"of":12,"metrics":{"Top-1":"66.1"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"ResNet-18 + 2bit-2dim model compression using DKM","rank_in_archive_order":8,"of":12,"metrics":{"Top-1":"64.7"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"MobileNet-v1 + 4bit-4dim model compression using DKM","rank_in_archive_order":9,"of":12,"metrics":{"Top-1":"61.4"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"ResNet-18 + 1bit-1dim model compression using DKM","rank_in_archive_order":10,"of":12,"metrics":{"Top-1":"59.7"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"MobileNet-v1 + 2bit-2dim model compression using DKM","rank_in_archive_order":11,"of":12,"metrics":{"Top-1":"53.99"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"MobileNet-v1 + 1bit-1dim model compression using DKM","rank_in_archive_order":12,"of":12,"metrics":{"Top-1":"52.58"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-qnli","task":"Model Compression","dataset":"QNLI","model":"MobileBERT + 2bit-1dim model compression using DKM","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"82.13"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-qnli","task":"Model Compression","dataset":"QNLI","model":"MobileBERT + 1bit-1dim model compression using DKM","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy":"63.17"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"MobileNet-v1 + EWGS + R2Loss","rank_in_archive_order":25,"of":27,"metrics":{"Top-1 Accuracy (%)":"69.79","Weight bits":"4"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"MobileNet-v1 + LSQ + R2Loss","rank_in_archive_order":26,"of":27,"metrics":{"Top-1 Accuracy (%)":"69.64"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"ResNet-18 + PACT + R2Loss","rank_in_archive_order":27,"of":27,"metrics":{"Activation bits":"4","Top-1 Accuracy (%)":"68.45","Weight bits":"2"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2303.08253","atlas_url":"https://app.syntology.ai/?focus=2303.08253","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2303.08253"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2303.08253\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2303.08253"},"call":"get_concepts_for_paper(arxiv_id=\"2303.08253\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"9aec296d78dd8d017188a253d2eea1194daf99d176a901f2bb5161aece6d6f33","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}