Methods › Natural Language Processing › Subword Segmentation › WordPiece
WordPiece
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
WordPiece is a subword segmentation algorithm used in natural language processing. The vocabulary is initialized with individual characters in the language, then the most frequent combinations of symbols in the vocabulary are iteratively added to the vocabulary. The process is:
- Initialize the word unit inventory with all the characters in the text.
- Build a language model on the training data using the inventory from 1.
- Generate a new word unit by combining two units out of the current word inventory to increment the word unit inventory by one. Choose the new word unit out of all the possible ones that increases the likelihood on the training data the most when added to the model.
- Goto 2 until a predefined limit of word units is reached or the likelihood increase falls below a certain threshold.
Text: Source
Image: WordPiece as used in BERT
In Syntology name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 7,063 · with a code link: 2,910 · where Syntology ran a sample: 650 (513 with a run with no instrument failure, 137 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (650 of 7,063 tagged: 513 with a run with no instrument failure, 137 where every run was a failure of Syntology's instrument)
Page 1 of 71: papers 1 to 100 of 7,063, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements 12 Jun 2025 · 0 repositories · arXiv:2506.10330
-
ALBERT: Advanced Localization and Bidirectional Encoder Representations from Transformers for Automotive Damage Evaluation 12 Jun 2025 · 0 repositories · arXiv:2506.10524
-
CIIR@LiveRAG 2025: Optimizing Multi-Agent Retrieval Augmented Generation through Self-Training 12 Jun 2025 · 1 repository · arXiv:2506.10844
-
Constructing and Evaluating Declarative RAG Pipelines in PyTerrier 12 Jun 2025 · 1 repository · arXiv:2506.10802
-
Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challenges 12 Jun 2025 · 0 repositories · arXiv:2506.10408
-
TableRAG: A Retrieval Augmented Generation Framework for Heterogeneous Document Reasoning 12 Jun 2025 · 1 repository · arXiv:2506.10380
-
Auto-Compressing Networks 11 Jun 2025 · 0 repositories · arXiv:2506.09714
-
Learning Efficient and Generalizable Graph Retriever for Knowledge-Graph Question Answering 11 Jun 2025 · 1 repository · arXiv:2506.09645
-
LLM-Powered CPI Prediction Inference with Online Text Time Series 11 Jun 2025 · 1 repository · arXiv:2506.09516
-
Measuring Corporate Human Capital Disclosures: Lexicon, Data, Code, and Research Opportunities 11 Jun 2025 · 0 repositories · arXiv:2506.10155
-
You Are What You Say: Exploiting Linguistic Content for VoicePrivacy Attacks 11 Jun 2025 · 0 repositories · arXiv:2506.09521
-
Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environment 10 Jun 2025 · 1 repository · arXiv:2506.10030
-
CC-RAG: Structured Multi-Hop Reasoning via Theme-Based Causal Graphs 10 Jun 2025 · 0 repositories · arXiv:2506.08364
-
DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs 10 Jun 2025 · 1 repository · arXiv:2506.08500
-
FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation 10 Jun 2025 · 1 repository · arXiv:2506.08938
-
FedRAG: A Framework for Fine-Tuning Retrieval-Augmented Generation Systems 10 Jun 2025 · 1 repository · arXiv:2506.09200
-
The Predictive Brain: Neural Correlates of Word Expectancy Align with Large Language Model Prediction Probabilities 10 Jun 2025 · 0 repositories · arXiv:2506.08511
-
Hierarchical Lexical Graph for Enhanced Multi-Hop Retrieval 9 Jun 2025 · 1 repository · arXiv:2506.08074
-
LlamaRec-LKG-RAG: A Single-Pass, Learnable Knowledge Graph-RAG Framework for LLM-Based Ranking 9 Jun 2025 · 1 repository · arXiv:2506.07449
-
Multilingual Hate Speech Detection in Social Media Using Translation-Based Approaches with Large Language Models 9 Jun 2025 · 0 repositories · arXiv:2506.08147
-
SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understanding 9 Jun 2025 · 0 repositories · arXiv:2506.07600
-
Joint-GCG: Unified Gradient-Based Poisoning Attacks on Retrieval-Augmented Generation Systems 6 Jun 2025 · 1 repository · arXiv:2506.06151
-
When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation 6 Jun 2025 · 1 repository · arXiv:2506.05690Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples)
-
Design of intelligent proofreading system for English translation based on CNN and BERT 5 Jun 2025 · 0 repositories · arXiv:2506.04811
-
Dynamic Context Tuning for Retrieval-Augmented Generation: Enhancing Multi-Turn Planning and Tool Adaptation 5 Jun 2025 · 0 repositories · arXiv:2506.11092
-
ECoRAG: Evidentiality-guided Compression for Long Context RAG 5 Jun 2025 · 1 repository · arXiv:2506.05167
-
Grapheme-Coherent Phonemic and Prosodic Annotation of Speech by Implicit and Explicit Grapheme Conditioning 5 Jun 2025 · 0 repositories · arXiv:2506.04527
-
Knowledgeable-r1: Policy Optimization for Knowledge Exploration in Retrieval-Augmented Generation 5 Jun 2025 · 1 repository · arXiv:2506.05154Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 5 harvested samples)
-
Mathematical Reasoning for Unmanned Aerial Vehicles: A RAG-Based Approach for Complex Arithmetic Reasoning 5 Jun 2025 · 1 repository · arXiv:2506.04998
-
Micro-Act: Mitigate Knowledge Conflict in Question Answering via Actionable Self-Reasoning 5 Jun 2025 · 1 repository · arXiv:2506.05278
-
On Automating Security Policies with Contemporary LLMs 5 Jun 2025 · 0 repositories · arXiv:2506.04838
-
Predicting ICU In-Hospital Mortality Using Adaptive Transformer Layer Fusion 5 Jun 2025 · 1 repository · arXiv:2506.04924
-
The NTNU System at the S&I Challenge 2025 SLA Open Track 5 Jun 2025 · 0 repositories · arXiv:2506.05121
-
Magic Mushroom: A Customizable Benchmark for Fine-grained Analysis of Retrieval Noise Erosion in RAG Systems 4 Jun 2025 · 0 repositories · arXiv:2506.03901
-
R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning 4 Jun 2025 · 1 repository · arXiv:2506.04185
-
TracLLM: A Generic Framework for Attributing Long Context LLMs 4 Jun 2025 · 1 repository · arXiv:2506.04202
-
Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds 3 Jun 2025 · 0 repositories · arXiv:2506.03100
-
CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAG 3 Jun 2025 · 0 repositories · arXiv:2506.02544
-
Enhancing Automatic PT Tagging for MEDLINE Citations Using Transformer-Based Models 3 Jun 2025 · 0 repositories · arXiv:2506.03321
-
Rethinking the effects of data contamination in Code Intelligence 3 Jun 2025 · 0 repositories · arXiv:2506.02791
-
LLMs as World Models: Data-Driven and Human-Centered Pre-Event Simulation for Disaster Impact Assessment 2 Jun 2025 · 0 repositories · arXiv:2506.06355
-
Retrieval-Augmented Generation of Ontologies from Relational Databases 2 Jun 2025 · 0 repositories · arXiv:2506.01232
-
A Graph-Retrieval-Augmented Generation Framework Enhances Decision-Making in the Circular Economy 1 Jun 2025 · 0 repositories · arXiv:2506.04252
-
RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented Generation Systems 1 Jun 2025 · 1 repository · arXiv:2506.00789
-
FinBERT2: A Specialized Bidirectional Encoder for Bridging the Gap in Finance-Specific Deployment of Large Language Models 31 May 2025 · 0 repositories · arXiv:2506.06335
-
Adversarial Threat Vectors and Risk Mitigation for Retrieval-Augmented Generation Systems 30 May 2025 · 0 repositories · arXiv:2506.00281
-
ClueAnchor: Clue-Anchored Knowledge Reasoning Exploration and Optimization for Retrieval-Augmented Generation 30 May 2025 · 1 repository · arXiv:2505.24388
-
Interpretable phenotyping of Heart Failure patients with Dutch discharge letters 30 May 2025 · 0 repositories · arXiv:2505.24619
-
LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMs 30 May 2025 · 0 repositories · arXiv:2505.24451
-
RealDrive: Retrieval-Augmented Driving with Diffusion Models 30 May 2025 · 0 repositories · arXiv:2505.24808
-
Bridging the Gap Between Semantic and User Preference Spaces for Multi-modal Music Representation Learning 29 May 2025 · 0 repositories · arXiv:2505.23298
-
Data-efficient Meta-models for Evaluation of Context-based Questions and Answers in LLMs 29 May 2025 · 0 repositories · arXiv:2505.23299
-
Evaluating AI capabilities in detecting conspiracy theories on YouTube 29 May 2025 · 1 repository · arXiv:2505.23570
-
MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignment 29 May 2025 · 0 repositories · arXiv:2505.23634
-
Query Routing for Retrieval-Augmented Language Models 29 May 2025 · 0 repositories · arXiv:2505.23052
-
Agent-UniRAG: A Trainable Open-Source LLM Agent Framework for Unified Retrieval-Augmented Generation Systems 28 May 2025 · 0 repositories · arXiv:2505.22571
-
Breaking the Cloak! Unveiling Chinese Cloaked Toxicity with Homophone Graph and Toxic Lexicon 28 May 2025 · 0 repositories · arXiv:2505.22184
-
Climate Finance Bench 28 May 2025 · 1 repository · arXiv:2505.22752
-
Contextual Memory Intelligence -- A Foundational Paradigm for Human-AI Collaboration and Reflective Generative AI Systems 28 May 2025 · 0 repositories · arXiv:2506.05370
-
Cross-modal RAG: Sub-dimensional Retrieval-Augmented Text-to-Image Generation 28 May 2025 · 1 repository · arXiv:2505.21956
-
Improving QA Efficiency with DistilBERT: Fine-Tuning and Inference on mobile Intel CPUs 28 May 2025 · 0 repositories · arXiv:2505.22937
-
RAGPPI: RAG Benchmark for Protein-Protein Interactions in Drug Discovery 28 May 2025 · 1 repository · arXiv:2505.23823
-
SkewRoute: Training-Free LLM Routing for Knowledge Graph Retrieval-Augmented Generation via Score Skewness of Retrieved Context 28 May 2025 · 0 repositories · arXiv:2505.23841
-
VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning 28 May 2025 · 1 repository · arXiv:2505.22019Syntology official: no sample here; runs from other or unrecorded repositories · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Diagnosing and Resolving Cloud Platform Instability with Multi-modal RAG LLMs 27 May 2025 · 0 repositories · arXiv:2505.21419
-
Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaboration 27 May 2025 · 0 repositories · arXiv:2505.20625
-
Anveshana: A New Benchmark Dataset for Cross-Lingual Information Retrieval On English Queries and Sanskrit Documents 26 May 2025 · 0 repositories · arXiv:2505.19494
-
Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models 26 May 2025 · 1 repository · arXiv:2505.19509
-
Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement 26 May 2025 · 1 repository · arXiv:2505.19675
-
Detection of Suicidal Risk on Social Media: A Hybrid Model 26 May 2025 · 0 repositories · arXiv:2505.23797
-
DGRAG: Distributed Graph-based Retrieval-Augmented Generation in Edge-Cloud Systems 26 May 2025 · 0 repositories · arXiv:2505.19847
-
DoctorRAG: Medical RAG Fusing Knowledge with Patient Analogy through Textual Gradients 26 May 2025 · 0 repositories · arXiv:2505.19538
-
Emotion Classification In-Context in Spanish 26 May 2025 · 0 repositories · arXiv:2505.20571
-
KnowTrace: Bootstrapping Iterative Retrieval-Augmented Generation with Structured Knowledge Tracing 26 May 2025 · 1 repository · arXiv:2505.20245
-
MA-RAG: Multi-Agent Retrieval-Augmented Generation via Collaborative Chain-of-Thought Reasoning 26 May 2025 · 0 repositories · arXiv:2505.20096
-
NeuSym-RAG: Hybrid Neural Symbolic Retrieval with Multiview Structuring for PDF Question Answering 26 May 2025 · 1 repository · arXiv:2505.19754
-
syftr: Pareto-Optimal Generative AI 26 May 2025 · 1 repository · arXiv:2505.20266
-
Conventional Contrastive Learning Often Falls Short: Improving Dense Retrieval with Cross-Encoder Listwise Distillation and Synthetic Data 25 May 2025 · 1 repository · arXiv:2505.19274
-
Hermes@DravidianLangTech 2025: Sentiment Analysis of Dravidian Languages using XLM-RoBERTa 25 May 2025 · 1 repository
-
Hypercube-RAG: Hypercube-Based Retrieval-Augmented Generation for In-domain Scientific Question-Answering 25 May 2025 · 1 repository · arXiv:2505.19288
-
Investigating Pedagogical Teacher and Student LLM Agents: Genetic Adaptation Meets Retrieval Augmented Generation Across Learning Style 25 May 2025 · 0 repositories · arXiv:2505.19173
-
Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval 25 May 2025 · 1 repository · arXiv:2505.19356
-
POQD: Performance-Oriented Query Decomposer for Multi-vector retrieval 25 May 2025 · 1 repository · arXiv:2505.19189Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 3 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 5 pointer-only (licence)
-
Retrieval-Augmented Generation for Service Discovery: Chunking Strategies and Benchmarking 25 May 2025 · 0 repositories · arXiv:2505.19310
-
A Survey of LLM × DATA 24 May 2025 · 2 repositories · arXiv:2505.18458
-
Benchmarking Poisoning Attacks against Retrieval-Augmented Generation 24 May 2025 · 0 repositories · arXiv:2505.18543
-
BRIT: Bidirectional Retrieval over Unified Image-Text Graph 24 May 2025 · 0 repositories · arXiv:2505.18450
-
Federated Retrieval-Augmented Generation: A Systematic Mapping Study 24 May 2025 · 0 repositories · arXiv:2505.18906
-
GainRAG: Preference Alignment in Retrieval-Augmented Generation through Gain Signal Synthesis 24 May 2025 · 1 repository · arXiv:2505.18710
-
LLMs for Supply Chain Management 24 May 2025 · 0 repositories · arXiv:2505.18597
-
Pruning for Performance: Efficient Idiom and Metaphor Classification in Low-Resource Konkani Using mBERT 24 May 2025 · 0 repositories · arXiv:2506.02005
-
Removal of Hallucination on Hallucination: Debate-Augmented RAG 24 May 2025 · 1 repository · arXiv:2505.18581Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 4 harvested samples)
-
The Silent Saboteur: Imperceptible Adversarial Attacks against Black-Box Retrieval-Augmented Generation Systems 24 May 2025 · 0 repositories · arXiv:2505.18583
-
FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domain 23 May 2025 · 0 repositories · arXiv:2505.17471
-
Multi-Scale Probabilistic Generation Theory: A Hierarchical Framework for Interpreting Large Language Models 23 May 2025 · 0 repositories · arXiv:2505.18244
-
QwenLong-CPRS: Towards ∞-LLMs with Dynamic Context Optimization 23 May 2025 · 0 repositories · arXiv:2505.18092
-
ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generation 23 May 2025 · 0 repositories · arXiv:2505.17632
-
Resolving Conflicting Evidence in Automated Fact-Checking: A Study on Retrieval-Augmented LLMs 23 May 2025 · 1 repository · arXiv:2505.17762
-
Align-GRAG: Reasoning-Guided Dual Alignment for Graph Retrieval-Augmented Generation 22 May 2025 · 0 repositories · arXiv:2505.16237
-
Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation 22 May 2025 · 0 repositories · arXiv:2505.16415Syntology 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
Tasks archive 2025-07-28
20 shown of 1,195 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Retrieval | 1,242 |
| RAG | 1,241 |
| Language Modelling | 1,157 |
| Retrieval-augmented Generation | 1,057 |
| Language Modeling | 935 |
| Question Answering | 758 |
| Sentence | 747 |
| Sentiment Analysis | 411 |
| Text Classification | 393 |
| text-classification | 347 |
| Transfer Learning | 314 |
| Classification | 263 |
| Word Embeddings | 260 |
| Named Entity Recognition | 248 |
| Information Retrieval | 244 |
| Named Entity Recognition (NER) | 237 |
| named-entity-recognition | 237 |
| Natural Language Understanding | 233 |
| Large Language Model | 213 |
| Articles | 193 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections