Methods › Reinforcement Learning › Offline Reinforcement Learning Methods › DPO
Direct Preference Optimization
DPO
Introduced by Rafael Rafailov et al. in Direct Preference Optimization: Your Language Model is Secretly a Reward Model
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
The archive carries no description for this method.
In Syntology about another paper, arXiv:2501.01336: repository known, no verified implementation (declared_code; repositories: 1 listed of 1 in the graph, strongest link: official). About another paper: this answer is about arXiv:2501.01336, while the archive credits arXiv:2305.18290.
From the lookup's note on this name: The same name also belongs to “Direct Preference Optimization (DPO)”, a method in the paper “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” (arXiv:2305.18290, 10,292 citations), which the lookup finds cited far more than the paper this answer is about (8 citations), so this answer may be about a different sense of the name. For arXiv:2305.18290: its page on this site · ask the lookup about it.
Code Syntology holds under other names, as the lookup lists them (code_nearby); not a complete list, and none of them is this method.
- Cal-DPO · shares_name_tokens (shares name tokens with this name; a different method) · level V3 · retrieve with
get_reference_implementation(name='Cal-DPO') - Kullback-Leibler DPO (KLDPO) · shares_name_tokens (shares name tokens with this name; a different method) · level V3 · retrieve with
get_reference_implementation(name='Kullback-Leibler DPO (KLDPO)')
Origin paper of this answer, arXiv:2501.01336 (its page on this site · its page on Syntology).
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
What each level means, quoted from Syntology's server card: V0 “imports”; V1 “executes against a synthesized fixture and returns a value of the recorded type”; V2 “two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs” “neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)”; V3 “paper-derived property tests pass on both of those same two generations.” The card adds: “No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results.” A level quoted in two parts is one definition the card writes as two clauses joined by a double hyphen, each clause quoted on its own (how these levels are quoted).
Papers archive 2025-07-28
archive papers tagged: 409 · with a code link: 184 · where Syntology ran a sample: 112 (93 with a run with no instrument failure, 19 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (112 of 409 tagged: 93 with a run with no instrument failure, 19 where every run was a failure of Syntology's instrument)
Page 1 of 5: papers 1 to 100 of 409, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Multi-Preference Lambda-weighted Listwise DPO for Dynamic Preference Alignment 24 Jun 2025 · 1 repository · arXiv:2506.19780
-
Smart-LLaMA-DPO: Reinforced Large Language Model for Explainable Smart Contract Vulnerability Detection 23 Jun 2025 · 0 repositories · arXiv:2506.18245
-
video-SALMONN 2: Captioning-Enhanced Audio-Visual Large Language Models 18 Jun 2025 · 1 repository · arXiv:2506.15220Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 8 harvested samples) · 1 pointer-only (licence)
-
TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization 17 Jun 2025 · 1 repository · arXiv:2506.14574Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
Alignment Quality Index (AQI) : Beyond Refusals: AQI as an Intrinsic Alignment Diagnostic via Latent Geometry, Cluster Divergence, and Layer wise Pooled Representations 16 Jun 2025 · 0 repositories · arXiv:2506.13901
-
CAMS: A CityGPT-Powered Agentic Framework for Urban Human Mobility Simulation 16 Jun 2025 · 0 repositories · arXiv:2506.13599
-
From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring 11 Jun 2025 · 0 repositories · arXiv:2506.09996
-
Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms 11 Jun 2025 · 0 repositories · arXiv:2506.09457
-
Vision Matters: Simple Visual Perturbations Can Boost Multimodal Math Reasoning 11 Jun 2025 · 1 repository · arXiv:2506.09736
-
Explicit Preference Optimization: No Need for an Implicit Reward Model 9 Jun 2025 · 1 repository · arXiv:2506.07492
-
LeVo: High-Quality Song Generation with Multi-Preference Alignment 9 Jun 2025 · 1 repository · arXiv:2506.07520Syntology community repositories only · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA 9 Jun 2025 · 0 repositories · arXiv:2506.08123
-
LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs 5 Jun 2025 · 1 repository · arXiv:2506.05260
-
Aligning Large Language Models with Implicit Preferences from User-Generated Content 4 Jun 2025 · 0 repositories · arXiv:2506.04463
-
SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models 4 Jun 2025 · 1 repository · arXiv:2506.04180Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples) · 6 pointer-only (licence)
-
Protein Inverse Folding From Structure Feedback 3 Jun 2025 · 0 repositories · arXiv:2506.03028
-
Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences 3 Jun 2025 · 0 repositories · arXiv:2506.02698
-
Understanding the Impact of Sampling Quality in Direct Preference Optimization 3 Jun 2025 · 0 repositories · arXiv:2506.04272
-
IF-GUIDE: Influence Function-Guided Detoxification of LLMs 2 Jun 2025 · 1 repository · arXiv:2506.01790Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)
-
Generalizable LLM Learning of Graph Synthetic Data with Reinforcement Learning 1 Jun 2025 · 0 repositories · arXiv:2506.00845
-
Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning 30 May 2025 · 1 repository · arXiv:2505.24850
-
Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization 29 May 2025 · 0 repositories · arXiv:2505.23387
-
Differential Information: An Information-Theoretic Perspective on Preference Optimization 29 May 2025 · 0 repositories · arXiv:2505.23761
-
Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? 29 May 2025 · 0 repositories · arXiv:2505.23749
-
Improving Multilingual Social Media Insights: Aspect-based Comment Analysis 29 May 2025 · 0 repositories · arXiv:2505.23037
-
MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignment 29 May 2025 · 0 repositories · arXiv:2505.23634
-
Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO 29 May 2025 · 0 repositories · arXiv:2505.23316
-
Reinforcement Learning for Better Verbalized Confidence in Long-Form Generation 29 May 2025 · 0 repositories · arXiv:2505.23912
-
D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples 28 May 2025 · 0 repositories · arXiv:2505.22002
-
Enhancing Paraphrase Type Generation: The Impact of DPO and RLHF Evaluated with Human-Ranked Data 28 May 2025 · 1 repository · arXiv:2506.02018
-
Multi-MLLM Knowledge Distillation for Out-of-Context News Detection 28 May 2025 · 0 repositories · arXiv:2505.22517
-
LPOI: Listwise Preference Optimization for Vision Language Models 27 May 2025 · 1 repository · arXiv:2505.21061
-
Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation 27 May 2025 · 0 repositories · arXiv:2505.21784
-
Energy-based Preference Optimization for Test-time Adaptation 26 May 2025 · 0 repositories · arXiv:2505.19607
-
Frictional Agent Alignment Framework: Slow Down and Don't Break Things 26 May 2025 · 1 repository · arXiv:2505.19428Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 7 harvested samples)
-
Preference Optimization by Estimating the Ratio of the Data Distribution 26 May 2025 · 0 repositories · arXiv:2505.19601
-
SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety 26 May 2025 · 0 repositories · arXiv:2505.20065
-
Token-Importance Guided Direct Preference Optimization 26 May 2025 · 0 repositories · arXiv:2505.19653Syntology 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models 26 May 2025 · 1 repository · arXiv:2505.19743
-
Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO 26 May 2025 · 1 repository · arXiv:2505.19770Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified (of 4 harvested samples)
-
What Can RL Bring to VLA Generalization? An Empirical Study 26 May 2025 · 0 repositories · arXiv:2505.19789
-
Aligning Web Query Generation with Ranking Objectives via Direct Preference Optimization 25 May 2025 · 0 repositories · arXiv:2505.19307
-
ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning 25 May 2025 · 0 repositories · arXiv:2505.19100
-
VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimization 25 May 2025 · 1 repository · arXiv:2505.19000
-
Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization 24 May 2025 · 1 repository · arXiv:2505.18720
-
Teaching with Lies: Curriculum DPO on Synthetic Negatives for Hallucination Detection 23 May 2025 · 0 repositories · arXiv:2505.17558
-
DecoupledESC: Enhancing Emotional Support Generation via Strategy-Response Decoupled Preference Optimization 22 May 2025 · 0 repositories · arXiv:2505.16995
-
Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO 22 May 2025 · 3 repositories · arXiv:2505.17017
-
Mesh-RFT: Enhancing Mesh Generation via Fine-grained Reinforcement Fine-Tuning 22 May 2025 · 0 repositories · arXiv:2505.16761
-
MPO: Multilingual Safety Alignment via Reward Gap Optimization 22 May 2025 · 1 repository · arXiv:2505.16869Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection 22 May 2025 · 0 repositories · arXiv:2505.16475
-
SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment 18 May 2025 · 0 repositories · arXiv:2505.12435
-
Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models 16 May 2025 · 0 repositories · arXiv:2505.10892
-
Towards Self-Improvement of Diffusion Models via Group Preference Optimization 16 May 2025 · 0 repositories · arXiv:2505.11070
-
Improved Algorithms for Differentially Private Language Model Alignment 13 May 2025 · 0 repositories · arXiv:2505.08849
-
Learning to Reason and Navigate: Parameter Efficient Action Planning with Large Language Models 12 May 2025 · 0 repositories · arXiv:2505.07500
-
Multimodal Integrated Knowledge Transfer to Large Language Models through Preference Optimization with Biomedical Applications 9 May 2025 · 1 repository · arXiv:2505.05736
-
Sailing AI by the Stars: A Survey of Learning from Rewards in Post-Training and Test-Time Scaling of Large Language Models 5 May 2025 · 1 repository · arXiv:2505.02686
-
SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning 5 May 2025 · 0 repositories · arXiv:2505.02363
-
Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm 3 May 2025 · 0 repositories · arXiv:2505.01706
-
Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math 30 Apr 2025 · 0 repositories · arXiv:2504.21233
-
DSPO: Direct Semantic Preference Optimization for Real-World Image Super-Resolution 21 Apr 2025 · 0 repositories · arXiv:2504.15176
-
Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-training 17 Apr 2025 · 0 repositories · arXiv:2504.13123
-
Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization 16 Apr 2025 · 0 repositories · arXiv:2504.12083
-
Discriminator-Free Direct Preference Optimization for Video Diffusion 11 Apr 2025 · 0 repositories · arXiv:2504.08542
-
Playpen: An Environment for Exploring Learning Through Conversational Interaction 11 Apr 2025 · 1 repository · arXiv:2504.08590
-
2D-Curri-DPO: Two-Dimensional Curriculum Learning for Direct Preference Optimization 10 Apr 2025 · 0 repositories · arXiv:2504.07856
-
MM-IFEngine: Towards Multimodal Instruction Following 10 Apr 2025 · 1 repository · arXiv:2504.07957Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 10 harvested samples)
-
FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion 9 Apr 2025 · 0 repositories · arXiv:2504.06562
-
LearNAT: Learning NL2SQL with AST-guided Task Decomposition for Large Language Models 3 Apr 2025 · 0 repositories · arXiv:2504.02327
-
More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment 3 Apr 2025 · 0 repositories · arXiv:2504.02193
-
De Novo Molecular Design Enabled by Direct Preference Optimization and Curriculum Learning 2 Apr 2025 · 0 repositories · arXiv:2504.01389
-
SUV: Scalable Large Language Model Copyright Compliance with Regularized Selective Unlearning 29 Mar 2025 · 0 repositories · arXiv:2503.22948
-
DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundness 28 Mar 2025 · 0 repositories · arXiv:2503.22677
-
Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHF 28 Mar 2025 · 0 repositories · arXiv:2503.22137
-
GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization 26 Mar 2025 · 1 repository · arXiv:2503.20194
-
Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach 26 Mar 2025 · 0 repositories · arXiv:2503.21819
-
ExCoT: Optimizing Reasoning for Text-to-SQL with Execution Feedback 25 Mar 2025 · 1 repository · arXiv:2503.19988Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified (of 5 harvested samples)
-
TEMPLE:Temporal Preference Learning of Video LLMs via Difficulty Scheduling and Pre-SFT Alignment 21 Mar 2025 · 1 repository · arXiv:2503.16929Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization 20 Mar 2025 · 0 repositories · arXiv:2503.15880
-
DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learning 19 Mar 2025 · 0 repositories · arXiv:2503.15265
-
Towards Harmless Multimodal Assistants with Blind Preference Optimization 18 Mar 2025 · 0 repositories · arXiv:2503.14189
-
Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation 17 Mar 2025 · 1 repository · arXiv:2503.12854
-
Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond 13 Mar 2025 · 1 repository · arXiv:2503.10460Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 1 violated, 4 with no contract checked; 4 where Syntology's instrument failed) · 3 unverified (of 13 harvested samples)
-
Aligning to What? Limits to RLHF Based Alignment 12 Mar 2025 · 1 repository · arXiv:2503.09025
-
DAST: Difficulty-Aware Self-Training on Large Language Models 12 Mar 2025 · 0 repositories · arXiv:2503.09029
-
D3PO: Preference-Based Alignment of Discrete Diffusion Models 11 Mar 2025 · 0 repositories · arXiv:2503.08295
-
LightGen: Efficient Image Generation through Knowledge Distillation and Direct Preference Optimization 11 Mar 2025 · 1 repository · arXiv:2503.08619Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 4 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified (of 10 harvested samples) · 3 pointer-only (licence)
-
SQLCritic: Correcting Text-to-SQL Generation via Clause-wise Critic 11 Mar 2025 · 0 repositories · arXiv:2503.07996
-
RePO: ReLU-based Preference Optimization 10 Mar 2025 · 1 repository · arXiv:2503.07426
-
Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment 6 Mar 2025 · 1 repository · arXiv:2503.04647
-
Improving LLM Safety Alignment with Dual-Objective Optimization 5 Mar 2025 · 1 repository · arXiv:2503.03710Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 5 pointer-only (licence)
-
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation 4 Mar 2025 · 0 repositories · arXiv:2503.02832
-
IterPref: Focal Preference Learning for Code Generation via Iterative Debugging 4 Mar 2025 · 0 repositories · arXiv:2503.02783
-
Active Learning for Direct Preference Optimization 3 Mar 2025 · 0 repositories · arXiv:2503.01076
-
CE-U: Cross Entropy Unlearning 3 Mar 2025 · 0 repositories · arXiv:2503.01224
-
In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models 3 Mar 2025 · 0 repositories · arXiv:2503.01611
-
PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation 3 Mar 2025 · 0 repositories · arXiv:2503.01233
-
Robust Multi-Objective Preference Alignment with Online DPO 1 Mar 2025 · 0 repositories · arXiv:2503.00295
-
LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spaces 27 Feb 2025 · 0 repositories · arXiv:2503.01894
Tasks archive 2025-07-28
20 shown of 181 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections