Browse State-of-the-Art › Text Compression
Text Compression
18 papers with code · 0 benchmarks · 0 datasets archive 2025-07-28
Benchmarks archive 2025-07-28
No benchmark for this task in the archive.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
No dataset record in the archive lists this task.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 43 · with a code link: 18 · where Syntology ran a sample: 2 (2 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (2 of 43 tagged: 2 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 43 of 43, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
31 Oct 2023 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 3 pointer-only (licence)Surprisingly, our findings indicate that neural retrieval models tend to rank LLM-generated documents higher.
-
6 Jun 2023 2 repositories listedWe provide new estimates of an asymptotic upper bound on the entropy of English using the large language model LLaMA-7B as a predictor for the next token given a window of past tokens.
-
4 Jun 2025 1 repository listedThe mismatch in vocabulary also hinders deep knowledge transfer between LLMs like token-level distillation.
-
10 Feb 2025 1 repository listedAutomatically summarizing large text collections is a valuable tool for document research, with applications in journalism, academic research, legal work, and many other fields.
-
21 Dec 2024 1 repository listedLearning-based probabilistic models can be combined with an entropy coder for data compression.
-
10 Dec 2024 1 repository listedWe introduce IntellectSeeker, an innovative and personalized intelligent academic literature management platform to address these challenges.
-
25 Sep 2024 1 repository listedWe compare traditional text compression systems with neural network and LLM-based text compression methods.
-
23 Sep 2024 1 repository listedData compression continues to evolve, with traditional information theory methods being widely used for compressing text, images, and videos.
-
6 Sep 2024 1 repository listedLanguage models can largely benefit from efficient tokenization.
-
12 Aug 2024 1 repository listedXCompress offers manual, brute force, and Large Language Model (LLM) methods to determine the most suitable algorithm based on the type of text data.
-
10 Jun 2024 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)To extend the context length of Transformer-based large language models (LLMs) and improve comprehension capabilities, we often face limitations due to computational resources and bounded memory storage capacity.
-
19 Mar 2024 1 repository listedAdditionally, our model is 3x-6x faster than existing prompt compression methods, while accelerating the end-to-end latency by 1.
-
27 Jul 2023 1 repository listedThe effectiveness of compression in text classification ('gzip') has recently garnered lots of attention.
-
4 Oct 2021 1 repository listedOverlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation.
-
1 Jun 2019 1 repository listedNeural sequence-to-sequence models have been successfully applied to text compression.
-
20 Aug 2017 1 repository listedTraining large vocabulary Neural Network Language Models (NNLMs) is a difficult task due to the explicit requirement of the output layer normalization, which typically involves the evaluation of the full softmax…
-
1 Jun 2017 1 repository listedInstead, the only three key components of our method are a compressing algorithm, a dissimilarity measure and a threshold, needed to accept or reject the authorship of the questioned document.
-
8 Aug 2016 1 repository listedWe present a self-contained system for constructing natural language models for use in text compression.
-
Beyond Text Compression: Evaluating Tokenizers Across Scales3 Jun 2025 0 repositories listedThe choice of tokenizer can profoundly impact language model performance, yet accessible and reliable evaluations of tokenizer quality remain an open challenge.
-
Measuring Information Distortion in Hierarchical Ultra long Novel Generation:The Optimal Expansion Ratio18 May 2025 0 repositories listedWriting novels with Large Language Models (LLMs) raises a critical question: how much human-authored outline is necessary to generate high-quality million-word novels?
-
Hypernym Mercury: Token Optimization Through Semantic Field Constriction And Reconstruction From Hypernyms. A New Text Compression Method12 May 2025 0 repositories listedCompute optimization using token reduction of LLM prompts is an emerging task in the fields of NLP and next generation, agentic AI.
-
Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction7 May 2025 0 repositories listedAs large language models (LLMs) continue to be deployed and utilized across domains, the volume of LLM-generated data is growing rapidly.
-
Text Compression for Efficient Language Generation14 Mar 2025 0 repositories listedWe challenge the prevailing assumption that LLMs must rely fully on sub-word tokens for high-quality text generation.
-
Assessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance23 Dec 2024 0 repositories listedAssessing the extent of human edits on texts generated by Large Language Models (LLMs) is crucial to understanding the human-AI interactions and improving the quality of automated text generation systems.
-
An Enhanced Text Compression Approach Using Transformer-based Language Models15 Dec 2024 0 repositories listedText compression shrinks textual data while keeping crucial information, eradicating constraints on storage, bandwidth, and computational efficacy.
-
Theoretical Analysis of Byte-Pair Encoding13 Nov 2024 0 repositories listedByte-Pair Encoding (BPE) is a widely used method for subword tokenization, with origins in grammar-based text compression.
-
Variational Bayesian Methods for a Tree-Structured Stick-Breaking Process Mixture of Gaussians by Application of the Bayes Codes for Context Tree Models1 May 2024 0 repositories listedThe tree-structured stick-breaking process (TS-SBP) mixture model is a non-parametric Bayesian model that can represent tree-like hierarchical structures among the mixture components.
-
Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance10 Mar 2024 0 repositories listedDespite it being the cornerstone of BPE, the most common tokenization algorithm, the importance of compression in the tokenization process is still unclear.
-
Semantic Text Compression for Classification19 Sep 2023 0 repositories listedWe study semantic compression for text where meanings contained in the text are conveyed to a source decoder, e.g., for classification.
-
EntropyRank: Unsupervised Keyphrase Extraction via Side-Information Optimization for Language Model-based Text Compression25 Aug 2023 0 repositories listedWe propose an unsupervised method to extract keywords and keyphrases from texts based on a pre-trained language model (LM) and Shannon's information maximization.
-
Approximating Human-Like Few-shot Learning with GPT-based Compression14 Aug 2023 0 repositories listed
-
Optimal alphabet for single text compression13 Jan 2022 0 repositories listed
-
Text Compression-aided Transformer Encoding11 Feb 2021 0 repositories listed
-
Machine Translation with Unsupervised Length-Constraints7 Apr 2020 0 repositories listed
-
Deleter: Leveraging BERT to Perform Unsupervised Successive Text Compression7 Sep 2019 0 repositories listed
-
A study for Image compression using Re-Pair algorithm30 Jan 2019 0 repositories listed
-
Text Compression for Sentiment Analysis via Evolutionary Algorithms20 Sep 2017 0 repositories listed
-
A Neural Network Approach for Mixing Language Models23 Aug 2017 0 repositories listed
-
Long-Short Range Context Neural Networks for Language Modeling22 Aug 2017 0 repositories listed
-
Sequential Recurrent Neural Networks for Language Modeling23 Mar 2017 0 repositories listed
-
Tuple-oriented Compression for Large-scale Mini-batch Stochastic Gradient Descent22 Feb 2017 0 repositories listed
-
Language and Dialect Discrimination Using Compression-Inspired Language Models1 Dec 2016 0 repositories listed
-
Lexis: An Optimization Framework for Discovering the Hierarchical Structure of Sequential Data17 Feb 2016 0 repositories listed
Syntology lines on 2 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections