Browse State-of-the-Art › Text Compression

Text Compression

18 papers with code · 0 benchmarks · 0 datasets archive 2025-07-28

Natural Language Processing

Benchmarks archive 2025-07-28

No benchmark for this task in the archive.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

No dataset record in the archive lists this task.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Papers archive 2025-07-28

archive papers tagged: 43 · with a code link: 18 · where Syntology ran a sample: 2 (2 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology

Show: all tagged papersonly where code ran (2 of 43 tagged: 2 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 43 of 43, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.

Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.

  • Surprisingly, our findings indicate that neural retrieval models tend to rank LLM-generated documents higher.
  • 6 Jun 2023 2 repositories listed
    We provide new estimates of an asymptotic upper bound on the entropy of English using the large language model LLaMA-7B as a predictor for the next token given a window of past tokens.
  • 4 Jun 2025 1 repository listed
    The mismatch in vocabulary also hinders deep knowledge transfer between LLMs like token-level distillation.
  • 10 Feb 2025 1 repository listed
    Automatically summarizing large text collections is a valuable tool for document research, with applications in journalism, academic research, legal work, and many other fields.
  • 21 Dec 2024 1 repository listed
    Learning-based probabilistic models can be combined with an entropy coder for data compression.
  • 10 Dec 2024 1 repository listed
    We introduce IntellectSeeker, an innovative and personalized intelligent academic literature management platform to address these challenges.
  • 25 Sep 2024 1 repository listed
    We compare traditional text compression systems with neural network and LLM-based text compression methods.
  • 23 Sep 2024 1 repository listed
    Data compression continues to evolve, with traditional information theory methods being widely used for compressing text, images, and videos.
  • 6 Sep 2024 1 repository listed
    Language models can largely benefit from efficient tokenization.
  • 12 Aug 2024 1 repository listed
    XCompress offers manual, brute force, and Large Language Model (LLM) methods to determine the most suitable algorithm based on the type of text data.
  • To extend the context length of Transformer-based large language models (LLMs) and improve comprehension capabilities, we often face limitations due to computational resources and bounded memory storage capacity.
  • 19 Mar 2024 1 repository listed
    Additionally, our model is 3x-6x faster than existing prompt compression methods, while accelerating the end-to-end latency by 1.
  • 27 Jul 2023 1 repository listed
    The effectiveness of compression in text classification ('gzip') has recently garnered lots of attention.
  • 4 Oct 2021 1 repository listed
    Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation.
  • 1 Jun 2019 1 repository listed
    Neural sequence-to-sequence models have been successfully applied to text compression.
  • 20 Aug 2017 1 repository listed
    Training large vocabulary Neural Network Language Models (NNLMs) is a difficult task due to the explicit requirement of the output layer normalization, which typically involves the evaluation of the full softmax…
  • 1 Jun 2017 1 repository listed
    Instead, the only three key components of our method are a compressing algorithm, a dissimilarity measure and a threshold, needed to accept or reject the authorship of the questioned document.
  • 8 Aug 2016 1 repository listed
    We present a self-contained system for constructing natural language models for use in text compression.
  • Beyond Text Compression: Evaluating Tokenizers Across Scales
    3 Jun 2025 0 repositories listed
    The choice of tokenizer can profoundly impact language model performance, yet accessible and reliable evaluations of tokenizer quality remain an open challenge.
  • Measuring Information Distortion in Hierarchical Ultra long Novel Generation:The Optimal Expansion Ratio
    18 May 2025 0 repositories listed
    Writing novels with Large Language Models (LLMs) raises a critical question: how much human-authored outline is necessary to generate high-quality million-word novels?
  • Hypernym Mercury: Token Optimization Through Semantic Field Constriction And Reconstruction From Hypernyms. A New Text Compression Method
    12 May 2025 0 repositories listed
    Compute optimization using token reduction of LLM prompts is an emerging task in the fields of NLP and next generation, agentic AI.
  • Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction
    7 May 2025 0 repositories listed
    As large language models (LLMs) continue to be deployed and utilized across domains, the volume of LLM-generated data is growing rapidly.
  • Text Compression for Efficient Language Generation
    14 Mar 2025 0 repositories listed
    We challenge the prevailing assumption that LLMs must rely fully on sub-word tokens for high-quality text generation.
  • Assessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance
    23 Dec 2024 0 repositories listed
    Assessing the extent of human edits on texts generated by Large Language Models (LLMs) is crucial to understanding the human-AI interactions and improving the quality of automated text generation systems.
  • An Enhanced Text Compression Approach Using Transformer-based Language Models
    15 Dec 2024 0 repositories listed
    Text compression shrinks textual data while keeping crucial information, eradicating constraints on storage, bandwidth, and computational efficacy.
  • Theoretical Analysis of Byte-Pair Encoding
    13 Nov 2024 0 repositories listed
    Byte-Pair Encoding (BPE) is a widely used method for subword tokenization, with origins in grammar-based text compression.
  • Variational Bayesian Methods for a Tree-Structured Stick-Breaking Process Mixture of Gaussians by Application of the Bayes Codes for Context Tree Models
    1 May 2024 0 repositories listed
    The tree-structured stick-breaking process (TS-SBP) mixture model is a non-parametric Bayesian model that can represent tree-like hierarchical structures among the mixture components.
  • Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance
    10 Mar 2024 0 repositories listed
    Despite it being the cornerstone of BPE, the most common tokenization algorithm, the importance of compression in the tokenization process is still unclear.
  • Semantic Text Compression for Classification
    19 Sep 2023 0 repositories listed
    We study semantic compression for text where meanings contained in the text are conveyed to a source decoder, e.g., for classification.
  • EntropyRank: Unsupervised Keyphrase Extraction via Side-Information Optimization for Language Model-based Text Compression
    25 Aug 2023 0 repositories listed
    We propose an unsupervised method to extract keywords and keyphrases from texts based on a pre-trained language model (LM) and Shannon's information maximization.
  • Approximating Human-Like Few-shot Learning with GPT-based Compression
    14 Aug 2023 0 repositories listed
  • Optimal alphabet for single text compression
    13 Jan 2022 0 repositories listed
  • Text Compression-aided Transformer Encoding
    11 Feb 2021 0 repositories listed
  • Machine Translation with Unsupervised Length-Constraints
    7 Apr 2020 0 repositories listed
  • Deleter: Leveraging BERT to Perform Unsupervised Successive Text Compression
    7 Sep 2019 0 repositories listed
  • A study for Image compression using Re-Pair algorithm
    30 Jan 2019 0 repositories listed
  • Text Compression for Sentiment Analysis via Evolutionary Algorithms
    20 Sep 2017 0 repositories listed
  • A Neural Network Approach for Mixing Language Models
    23 Aug 2017 0 repositories listed
  • Long-Short Range Context Neural Networks for Language Modeling
    22 Aug 2017 0 repositories listed
  • Sequential Recurrent Neural Networks for Language Modeling
    23 Mar 2017 0 repositories listed
  • Tuple-oriented Compression for Large-scale Mini-batch Stochastic Gradient Descent
    22 Feb 2017 0 repositories listed
  • Language and Dialect Discrimination Using Compression-Inspired Language Models
    1 Dec 2016 0 repositories listed
  • Lexis: An Optimization Framework for Discovering the Hierarchical Structure of Sequential Data
    17 Feb 2016 0 repositories listed

Syntology lines on 2 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections