{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/dpo","slug":"dpo","name":"DPO","full_name":"Direct Preference Optimization","full_name_withheld":false,"description_markdown":null,"description_state":"absent","introduced_year":null,"introduced_by":{"title":"Direct Preference Optimization: Your Language Model is Secretly a Reward Model","paper":"/paper/direct-preference-optimization-your-language","first_author":"Rafael Rafailov","n_authors":6,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/direct-preference-optimization-your-language"},"source":{"url":"https://arxiv.org/abs/2305.18290v3","title":"Direct Preference Optimization: Your Language Model is Secretly a Reward Model","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Offline Reinforcement Learning Methods","url":"/methods/category/offline-reinforcement-learning-methods","pwc_aliases":[]}],"n_papers_tagged":409,"archive_num_papers":409,"papers_newest_first":[{"paper":"/paper/multi-preference-lambda-weighted-listwise-dpo","slug":"multi-preference-lambda-weighted-listwise-dpo","title":"Multi-Preference Lambda-weighted Listwise DPO for Dynamic Preference Alignment","date":"2025-06-24","arxiv_id":"2506.19780","n_code_links":1,"syntology":null},{"paper":null,"slug":"smart-llama-dpo-reinforced-large-language","title":"Smart-LLaMA-DPO: Reinforced Large Language Model for Explainable Smart Contract Vulnerability Detection","date":"2025-06-23","arxiv_id":"2506.18245","n_code_links":0,"syntology":null},{"paper":"/paper/video-salmonn-2-captioning-enhanced-audio","slug":"video-salmonn-2-captioning-enhanced-audio","title":"video-SALMONN 2: Captioning-Enhanced Audio-Visual Large Language Models","date":"2025-06-18","arxiv_id":"2506.15220","n_code_links":1,"syntology":{"ran":6,"of":8,"n_ran_checked":3,"n_instrument":3,"unverified":2,"pointer_only":1,"phrase":"6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified","official":{"repos":["bytedance/video-salmonn-2"],"state":"official (archive's flag): 6 ran","n_ran":6,"n_constructed":0,"n_ran_no_instrument_failure":3,"n_unverified":2,"ran_from_kinds":["official"]}}},{"paper":"/paper/tgdpo-harnessing-token-level-reward-guidance","slug":"tgdpo-harnessing-token-level-reward-guidance","title":"TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization","date":"2025-06-17","arxiv_id":"2506.14574","n_code_links":1,"syntology":{"ran":3,"of":3,"n_ran_checked":1,"n_instrument":2,"unverified":0,"pointer_only":3,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","official":{"repos":["dvlab-research/tgdpo"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"alignment-quality-index-aqi-beyond-refusals","title":"Alignment Quality Index (AQI) : Beyond Refusals: AQI as an Intrinsic Alignment Diagnostic via Latent Geometry, Cluster Divergence, and Layer wise Pooled Representations","date":"2025-06-16","arxiv_id":"2506.13901","n_code_links":0,"syntology":null},{"paper":null,"slug":"cams-a-citygpt-powered-agentic-framework-for","title":"CAMS: A CityGPT-Powered Agentic Framework for Urban Human Mobility Simulation","date":"2025-06-16","arxiv_id":"2506.13599","n_code_links":0,"syntology":null},{"paper":null,"slug":"from-judgment-to-interference-early-stopping","title":"From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring","date":"2025-06-11","arxiv_id":"2506.09996","n_code_links":0,"syntology":null},{"paper":null,"slug":"towards-bridging-the-reward-generation-gap-in","title":"Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms","date":"2025-06-11","arxiv_id":"2506.09457","n_code_links":0,"syntology":null},{"paper":"/paper/vision-matters-simple-visual-perturbations","slug":"vision-matters-simple-visual-perturbations","title":"Vision Matters: Simple Visual Perturbations Can Boost Multimodal Math Reasoning","date":"2025-06-11","arxiv_id":"2506.09736","n_code_links":1,"syntology":null},{"paper":"/paper/explicit-preference-optimization-no-need-for","slug":"explicit-preference-optimization-no-need-for","title":"Explicit Preference Optimization: No Need for an Implicit Reward Model","date":"2025-06-09","arxiv_id":"2506.07492","n_code_links":1,"syntology":null},{"paper":"/paper/levo-high-quality-song-generation-with-multi","slug":"levo-high-quality-song-generation-with-multi","title":"LeVo: High-Quality Song Generation with Multi-Preference Alignment","date":"2025-06-09","arxiv_id":"2506.07520","n_code_links":1,"syntology":{"ran":2,"of":2,"n_ran_checked":1,"n_instrument":1,"unverified":0,"pointer_only":2,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["tencent-ailab/songgeneration"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["community"]}}},{"paper":null,"slug":"qa-lign-aligning-llms-through","title":"QA-LIGN: Aligning LLMs through Constitutionally Decomposed QA","date":"2025-06-09","arxiv_id":"2506.08123","n_code_links":0,"syntology":null},{"paper":"/paper/leanpo-lean-preference-optimization-for","slug":"leanpo-lean-preference-optimization-for","title":"LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs","date":"2025-06-05","arxiv_id":"2506.05260","n_code_links":1,"syntology":null},{"paper":null,"slug":"aligning-large-language-models-with-implicit","title":"Aligning Large Language Models with Implicit Preferences from User-Generated Content","date":"2025-06-04","arxiv_id":"2506.04463","n_code_links":0,"syntology":null},{"paper":"/paper/superwriter-reflection-driven-long-form","slug":"superwriter-reflection-driven-long-form","title":"SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models","date":"2025-06-04","arxiv_id":"2506.04180","n_code_links":1,"syntology":{"ran":6,"of":6,"n_ran_checked":2,"n_instrument":4,"unverified":0,"pointer_only":6,"phrase":"6 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified","official":{"repos":["mozhu621/superwriter"],"state":"official (archive's flag): 6 ran","n_ran":6,"n_constructed":0,"n_ran_no_instrument_failure":2,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"protein-inverse-folding-from-structure","title":"Protein Inverse Folding From Structure Feedback","date":"2025-06-03","arxiv_id":"2506.03028","n_code_links":0,"syntology":null},{"paper":null,"slug":"smoothed-preference-optimization-via-renoise","title":"Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferences","date":"2025-06-03","arxiv_id":"2506.02698","n_code_links":0,"syntology":null},{"paper":null,"slug":"understanding-the-impact-of-sampling-quality","title":"Understanding the Impact of Sampling Quality in Direct Preference Optimization","date":"2025-06-03","arxiv_id":"2506.04272","n_code_links":0,"syntology":null},{"paper":"/paper/if-guide-influence-function-guided","slug":"if-guide-influence-function-guided","title":"IF-GUIDE: Influence Function-Guided Detoxification of LLMs","date":"2025-06-02","arxiv_id":"2506.01790","n_code_links":1,"syntology":{"ran":0,"of":1,"n_ran_checked":0,"n_instrument":0,"unverified":1,"pointer_only":0,"phrase":"0 ran · 1 unverified","official":{"repos":["ztcoalson/if-guide"],"state":"official: harvested, nothing ran","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":[]}}},{"paper":null,"slug":"generalizable-llm-learning-of-graph-synthetic","title":"Generalizable LLM Learning of Graph Synthetic Data with Reinforcement Learning","date":"2025-06-01","arxiv_id":"2506.00845","n_code_links":0,"syntology":null},{"paper":"/paper/harnessing-negative-signals-reinforcement","slug":"harnessing-negative-signals-reinforcement","title":"Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning","date":"2025-05-30","arxiv_id":"2505.24850","n_code_links":1,"syntology":null},{"paper":null,"slug":"afterburner-reinforcement-learning","title":"Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization","date":"2025-05-29","arxiv_id":"2505.23387","n_code_links":0,"syntology":null},{"paper":null,"slug":"differential-information-an-information","title":"Differential Information: An Information-Theoretic Perspective on Preference Optimization","date":"2025-05-29","arxiv_id":"2505.23761","n_code_links":0,"syntology":null},{"paper":null,"slug":"distortion-of-ai-alignment-does-preference","title":"Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?","date":"2025-05-29","arxiv_id":"2505.23749","n_code_links":0,"syntology":null},{"paper":null,"slug":"improving-multilingual-social-media-insights","title":"Improving Multilingual Social Media Insights: Aspect-based Comment Analysis","date":"2025-05-29","arxiv_id":"2505.23037","n_code_links":0,"syntology":null},{"paper":null,"slug":"mcp-safety-training-learning-to-refuse","title":"MCP Safety Training: Learning to Refuse Falsely Benign MCP Exploits using Improved Preference Alignment","date":"2025-05-29","arxiv_id":"2505.23634","n_code_links":0,"syntology":null},{"paper":null,"slug":"proximalized-preference-optimization-for","title":"Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO","date":"2025-05-29","arxiv_id":"2505.23316","n_code_links":0,"syntology":null},{"paper":null,"slug":"reinforcement-learning-for-better-verbalized","title":"Reinforcement Learning for Better Verbalized Confidence in Long-Form Generation","date":"2025-05-29","arxiv_id":"2505.23912","n_code_links":0,"syntology":null},{"paper":null,"slug":"d-fusion-direct-preference-optimization-for","title":"D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples","date":"2025-05-28","arxiv_id":"2505.22002","n_code_links":0,"syntology":null},{"paper":"/paper/enhancing-paraphrase-type-generation-the","slug":"enhancing-paraphrase-type-generation-the","title":"Enhancing Paraphrase Type Generation: The Impact of DPO and RLHF Evaluated with Human-Ranked Data","date":"2025-05-28","arxiv_id":"2506.02018","n_code_links":1,"syntology":null},{"paper":null,"slug":"multi-mllm-knowledge-distillation-for-out-of","title":"Multi-MLLM Knowledge Distillation for Out-of-Context News Detection","date":"2025-05-28","arxiv_id":"2505.22517","n_code_links":0,"syntology":null},{"paper":"/paper/lpoi-listwise-preference-optimization-for","slug":"lpoi-listwise-preference-optimization-for","title":"LPOI: Listwise Preference Optimization for Vision Language Models","date":"2025-05-27","arxiv_id":"2505.21061","n_code_links":1,"syntology":null},{"paper":null,"slug":"towards-safety-reasoning-in-llms-ai-agentic","title":"Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation","date":"2025-05-27","arxiv_id":"2505.21784","n_code_links":0,"syntology":null},{"paper":null,"slug":"energy-based-preference-optimization-for-test","title":"Energy-based Preference Optimization for Test-time Adaptation","date":"2025-05-26","arxiv_id":"2505.19607","n_code_links":0,"syntology":null},{"paper":"/paper/frictional-agent-alignment-framework-slow","slug":"frictional-agent-alignment-framework-slow","title":"Frictional Agent Alignment Framework: Slow Down and Don't Break Things","date":"2025-05-26","arxiv_id":"2505.19428","n_code_links":1,"syntology":{"ran":5,"of":7,"n_ran_checked":5,"n_instrument":0,"unverified":2,"pointer_only":0,"phrase":"5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified","official":{"repos":["csu-signal/faaf_acl"],"state":"official (archive's flag): 5 ran","n_ran":5,"n_constructed":0,"n_ran_no_instrument_failure":5,"n_unverified":2,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"preference-optimization-by-estimating-the","title":"Preference Optimization by Estimating the Ratio of the Data Distribution","date":"2025-05-26","arxiv_id":"2505.19601","n_code_links":0,"syntology":null},{"paper":null,"slug":"safedpo-a-simple-approach-to-direct","title":"SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety","date":"2025-05-26","arxiv_id":"2505.20065","n_code_links":0,"syntology":null},{"paper":"/paper/token-importance-guided-direct-preference","slug":"token-importance-guided-direct-preference","title":"Token-Importance Guided Direct Preference Optimization","date":"2025-05-26","arxiv_id":"2505.19653","n_code_links":0,"syntology":{"ran":1,"of":1,"n_ran_checked":0,"n_instrument":1,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":null}},{"paper":"/paper/token-level-accept-or-reject-a-micro","slug":"token-level-accept-or-reject-a-micro","title":"Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models","date":"2025-05-26","arxiv_id":"2505.19743","n_code_links":1,"syntology":null},{"paper":"/paper/understanding-the-performance-gap-in","slug":"understanding-the-performance-gap-in","title":"Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO","date":"2025-05-26","arxiv_id":"2505.19770","n_code_links":1,"syntology":{"ran":1,"of":4,"n_ran_checked":0,"n_instrument":1,"unverified":3,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified","official":{"repos":["srzer/Gap-in-Preference-Learning"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":3,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"what-can-rl-bring-to-vla-generalization-an","title":"What Can RL Bring to VLA Generalization? An Empirical Study","date":"2025-05-26","arxiv_id":"2505.19789","n_code_links":0,"syntology":null},{"paper":null,"slug":"aligning-web-query-generation-with-ranking","title":"Aligning Web Query Generation with Ranking Objectives via Direct Preference Optimization","date":"2025-05-25","arxiv_id":"2505.19307","n_code_links":0,"syntology":null},{"paper":null,"slug":"aspo-adaptive-sentence-level-preference","title":"ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning","date":"2025-05-25","arxiv_id":"2505.19100","n_code_links":0,"syntology":null},{"paper":"/paper/veripo-cultivating-long-reasoning-in-video","slug":"veripo-cultivating-long-reasoning-in-video","title":"VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimization","date":"2025-05-25","arxiv_id":"2505.19000","n_code_links":1,"syntology":null},{"paper":"/paper/optimal-transport-based-token-weighting","slug":"optimal-transport-based-token-weighting","title":"Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization","date":"2025-05-24","arxiv_id":"2505.18720","n_code_links":1,"syntology":null},{"paper":null,"slug":"teaching-with-lies-curriculum-dpo-on","title":"Teaching with Lies: Curriculum DPO on Synthetic Negatives for Hallucination Detection","date":"2025-05-23","arxiv_id":"2505.17558","n_code_links":0,"syntology":null},{"paper":null,"slug":"decoupledesc-enhancing-emotional-support","title":"DecoupledESC: Enhancing Emotional Support Generation via Strategy-Response Decoupled Preference Optimization","date":"2025-05-22","arxiv_id":"2505.16995","n_code_links":0,"syntology":null},{"paper":"/paper/delving-into-rl-for-image-generation-with-cot","slug":"delving-into-rl-for-image-generation-with-cot","title":"Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO","date":"2025-05-22","arxiv_id":"2505.17017","n_code_links":3,"syntology":null},{"paper":null,"slug":"mesh-rft-enhancing-mesh-generation-via-fine","title":"Mesh-RFT: Enhancing Mesh Generation via Fine-grained Reinforcement Fine-Tuning","date":"2025-05-22","arxiv_id":"2505.16761","n_code_links":0,"syntology":null},{"paper":"/paper/mpo-multilingual-safety-alignment-via-reward","slug":"mpo-multilingual-safety-alignment-via-reward","title":"MPO: Multilingual Safety Alignment via Reward Gap Optimization","date":"2025-05-22","arxiv_id":"2505.16869","n_code_links":1,"syntology":{"ran":3,"of":3,"n_ran_checked":1,"n_instrument":2,"unverified":0,"pointer_only":0,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","official":{"repos":["circle-hit/mpo"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"reflectevo-improving-meta-introspection-of","title":"ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection","date":"2025-05-22","arxiv_id":"2505.16475","n_code_links":0,"syntology":null},{"paper":null,"slug":"sgdpo-self-guided-direct-preference","title":"SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment","date":"2025-05-18","arxiv_id":"2505.12435","n_code_links":0,"syntology":null},{"paper":null,"slug":"2505-10892","title":"Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models","date":"2025-05-16","arxiv_id":"2505.10892","n_code_links":0,"syntology":null},{"paper":null,"slug":"2505-11070","title":"Towards Self-Improvement of Diffusion Models via Group Preference Optimization","date":"2025-05-16","arxiv_id":"2505.11070","n_code_links":0,"syntology":null},{"paper":null,"slug":"improved-algorithms-for-differentially","title":"Improved Algorithms for Differentially Private Language Model Alignment","date":"2025-05-13","arxiv_id":"2505.08849","n_code_links":0,"syntology":null},{"paper":null,"slug":"learning-to-reason-and-navigate-parameter","title":"Learning to Reason and Navigate: Parameter Efficient Action Planning with Large Language Models","date":"2025-05-12","arxiv_id":"2505.07500","n_code_links":0,"syntology":null},{"paper":"/paper/multimodal-integrated-knowledge-transfer-to","slug":"multimodal-integrated-knowledge-transfer-to","title":"Multimodal Integrated Knowledge Transfer to Large Language Models through Preference Optimization with Biomedical Applications","date":"2025-05-09","arxiv_id":"2505.05736","n_code_links":1,"syntology":null},{"paper":"/paper/sailing-ai-by-the-stars-a-survey-of-learning","slug":"sailing-ai-by-the-stars-a-survey-of-learning","title":"Sailing AI by the Stars: A Survey of Learning from Rewards in Post-Training and Test-Time Scaling of Large Language Models","date":"2025-05-05","arxiv_id":"2505.02686","n_code_links":1,"syntology":null},{"paper":null,"slug":"simplemix-frustratingly-simple-mixing-of-off","title":"SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning","date":"2025-05-05","arxiv_id":"2505.02363","n_code_links":0,"syntology":null},{"paper":null,"slug":"inducing-robustness-in-a-2-dimensional-direct","title":"Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm","date":"2025-05-03","arxiv_id":"2505.01706","n_code_links":0,"syntology":null},{"paper":null,"slug":"phi-4-mini-reasoning-exploring-the-limits-of","title":"Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math","date":"2025-04-30","arxiv_id":"2504.21233","n_code_links":0,"syntology":null},{"paper":null,"slug":"dspo-direct-semantic-preference-optimization","title":"DSPO: Direct Semantic Preference Optimization for Real-World Image Super-Resolution","date":"2025-04-21","arxiv_id":"2504.15176","n_code_links":0,"syntology":null},{"paper":null,"slug":"low-hallucination-synthetic-captions-for","title":"Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-training","date":"2025-04-17","arxiv_id":"2504.13123","n_code_links":0,"syntology":null},{"paper":"/paper/self-alignment-of-large-video-language-models","slug":"self-alignment-of-large-video-language-models","title":"Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization","date":"2025-04-16","arxiv_id":"2504.12083","n_code_links":0,"syntology":null},{"paper":null,"slug":"discriminator-free-direct-preference","title":"Discriminator-Free Direct Preference Optimization for Video Diffusion","date":"2025-04-11","arxiv_id":"2504.08542","n_code_links":0,"syntology":null},{"paper":"/paper/playpen-an-environment-for-exploring-learning","slug":"playpen-an-environment-for-exploring-learning","title":"Playpen: An Environment for Exploring Learning Through Conversational Interaction","date":"2025-04-11","arxiv_id":"2504.08590","n_code_links":1,"syntology":null},{"paper":null,"slug":"2d-curri-dpo-two-dimensional-curriculum","title":"2D-Curri-DPO: Two-Dimensional Curriculum Learning for Direct Preference Optimization","date":"2025-04-10","arxiv_id":"2504.07856","n_code_links":0,"syntology":null},{"paper":"/paper/mm-ifengine-towards-multimodal-instruction","slug":"mm-ifengine-towards-multimodal-instruction","title":"MM-IFEngine: Towards Multimodal Instruction Following","date":"2025-04-10","arxiv_id":"2504.07957","n_code_links":1,"syntology":{"ran":7,"of":10,"n_ran_checked":7,"n_instrument":0,"unverified":3,"pointer_only":0,"phrase":"7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified","official":{"repos":["syuan03/mm-ifengine"],"state":"official (archive's flag): 7 ran","n_ran":7,"n_constructed":0,"n_ran_no_instrument_failure":7,"n_unverified":3,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"fuserl-dense-preference-optimization-for","title":"FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion","date":"2025-04-09","arxiv_id":"2504.06562","n_code_links":0,"syntology":null},{"paper":null,"slug":"learnat-learning-nl2sql-with-ast-guided-task","title":"LearNAT: Learning NL2SQL with AST-guided Task Decomposition for Large Language Models","date":"2025-04-03","arxiv_id":"2504.02327","n_code_links":0,"syntology":null},{"paper":null,"slug":"more-is-less-the-pitfalls-of-multi-model","title":"More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment","date":"2025-04-03","arxiv_id":"2504.02193","n_code_links":0,"syntology":null},{"paper":null,"slug":"de-novo-molecular-design-enabled-by-direct","title":"De Novo Molecular Design Enabled by Direct Preference Optimization and Curriculum Learning","date":"2025-04-02","arxiv_id":"2504.01389","n_code_links":0,"syntology":null},{"paper":null,"slug":"suv-scalable-large-language-model-copyright","title":"SUV: Scalable Large Language Model Copyright Compliance with Regularized Selective Unlearning","date":"2025-03-29","arxiv_id":"2503.22948","n_code_links":0,"syntology":null},{"paper":null,"slug":"dso-aligning-3d-generators-with-simulation","title":"DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundness","date":"2025-03-28","arxiv_id":"2503.22677","n_code_links":0,"syntology":null},{"paper":null,"slug":"sharpe-ratio-guided-active-learning-for","title":"Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHF","date":"2025-03-28","arxiv_id":"2503.22137","n_code_links":0,"syntology":null},{"paper":"/paper/gapo-learning-preferential-prompt-through","slug":"gapo-learning-preferential-prompt-through","title":"GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization","date":"2025-03-26","arxiv_id":"2503.20194","n_code_links":1,"syntology":null},{"paper":null,"slug":"optimizing-safe-and-aligned-language","title":"Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach","date":"2025-03-26","arxiv_id":"2503.21819","n_code_links":0,"syntology":null},{"paper":"/paper/excot-optimizing-reasoning-for-text-to-sql","slug":"excot-optimizing-reasoning-for-text-to-sql","title":"ExCoT: Optimizing Reasoning for Text-to-SQL with Execution Feedback","date":"2025-03-25","arxiv_id":"2503.19988","n_code_links":1,"syntology":{"ran":1,"of":5,"n_ran_checked":1,"n_instrument":0,"unverified":4,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified","official":{"repos":["snowflakedb/ArcticTraining"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":4,"ran_from_kinds":["official"]}}},{"paper":"/paper/temple-temporal-preference-learning-of-video","slug":"temple-temporal-preference-learning-of-video","title":"TEMPLE:Temporal Preference Learning of Video LLMs via Difficulty Scheduling and Pre-SFT Alignment","date":"2025-03-21","arxiv_id":"2503.16929","n_code_links":1,"syntology":{"ran":1,"of":2,"n_ran_checked":0,"n_instrument":1,"unverified":1,"pointer_only":2,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified","official":{"repos":["lscpku/temple"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"inco-dpo-balancing-distribution-shift-and","title":"InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization","date":"2025-03-20","arxiv_id":"2503.15880","n_code_links":0,"syntology":null},{"paper":null,"slug":"deepmesh-auto-regressive-artist-mesh-creation","title":"DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learning","date":"2025-03-19","arxiv_id":"2503.15265","n_code_links":0,"syntology":null},{"paper":null,"slug":"towards-harmless-multimodal-assistants-with","title":"Towards Harmless Multimodal Assistants with Blind Preference Optimization","date":"2025-03-18","arxiv_id":"2503.14189","n_code_links":0,"syntology":null},{"paper":"/paper/enhancing-llm-reasoning-with-iterative-dpo-a","slug":"enhancing-llm-reasoning-with-iterative-dpo-a","title":"Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation","date":"2025-03-17","arxiv_id":"2503.12854","n_code_links":1,"syntology":null},{"paper":"/paper/light-r1-curriculum-sft-dpo-and-rl-for-long","slug":"light-r1-curriculum-sft-dpo-and-rl-for-long","title":"Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond","date":"2025-03-13","arxiv_id":"2503.10460","n_code_links":1,"syntology":{"ran":10,"of":13,"n_ran_checked":6,"n_instrument":4,"unverified":3,"pointer_only":0,"phrase":"10 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 1 violated, 4 with no contract checked; 4 where Syntology's instrument failed) · 3 unverified","official":{"repos":["qihoo360/light-r1"],"state":"official (archive's flag): 10 ran","n_ran":10,"n_constructed":0,"n_ran_no_instrument_failure":6,"n_unverified":3,"ran_from_kinds":["official"]}}},{"paper":"/paper/aligning-to-what-limits-to-rlhf-based","slug":"aligning-to-what-limits-to-rlhf-based","title":"Aligning to What? Limits to RLHF Based Alignment","date":"2025-03-12","arxiv_id":"2503.09025","n_code_links":1,"syntology":null},{"paper":null,"slug":"dast-difficulty-aware-self-training-on-large","title":"DAST: Difficulty-Aware Self-Training on Large Language Models","date":"2025-03-12","arxiv_id":"2503.09029","n_code_links":0,"syntology":null},{"paper":null,"slug":"d3po-preference-based-alignment-of-discrete","title":"D3PO: Preference-Based Alignment of Discrete Diffusion Models","date":"2025-03-11","arxiv_id":"2503.08295","n_code_links":0,"syntology":null},{"paper":"/paper/lightgen-efficient-image-generation-through","slug":"lightgen-efficient-image-generation-through","title":"LightGen: Efficient Image Generation through Knowledge Distillation and Direct Preference Optimization","date":"2025-03-11","arxiv_id":"2503.08619","n_code_links":1,"syntology":{"ran":9,"of":10,"n_ran_checked":4,"n_instrument":5,"unverified":1,"pointer_only":3,"phrase":"9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 4 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified","official":{"repos":["xianfengwu01/lightgen"],"state":"official (archive's flag): 9 ran","n_ran":9,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"sqlcritic-correcting-text-to-sql-generation","title":"SQLCritic: Correcting Text-to-SQL Generation via Clause-wise Critic","date":"2025-03-11","arxiv_id":"2503.07996","n_code_links":0,"syntology":null},{"paper":"/paper/repo-relu-based-preference-optimization","slug":"repo-relu-based-preference-optimization","title":"RePO: ReLU-based Preference Optimization","date":"2025-03-10","arxiv_id":"2503.07426","n_code_links":1,"syntology":null},{"paper":"/paper/implicit-cross-lingual-rewarding-for","slug":"implicit-cross-lingual-rewarding-for","title":"Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment","date":"2025-03-06","arxiv_id":"2503.04647","n_code_links":1,"syntology":null},{"paper":"/paper/improving-llm-safety-alignment-with-dual","slug":"improving-llm-safety-alignment-with-dual","title":"Improving LLM Safety Alignment with Dual-Objective Optimization","date":"2025-03-05","arxiv_id":"2503.03710","n_code_links":1,"syntology":{"ran":5,"of":5,"n_ran_checked":2,"n_instrument":3,"unverified":0,"pointer_only":5,"phrase":"5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified","official":{"repos":["wicai24/door-alignment"],"state":"official (archive's flag): 5 ran","n_ran":5,"n_constructed":0,"n_ran_no_instrument_failure":2,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"aligndistil-token-level-language-model","title":"AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation","date":"2025-03-04","arxiv_id":"2503.02832","n_code_links":0,"syntology":null},{"paper":null,"slug":"iterpref-focal-preference-learning-for-code","title":"IterPref: Focal Preference Learning for Code Generation via Iterative Debugging","date":"2025-03-04","arxiv_id":"2503.02783","n_code_links":0,"syntology":null},{"paper":null,"slug":"active-learning-for-direct-preference","title":"Active Learning for Direct Preference Optimization","date":"2025-03-03","arxiv_id":"2503.01076","n_code_links":0,"syntology":null},{"paper":null,"slug":"ce-u-cross-entropy-unlearning","title":"CE-U: Cross Entropy Unlearning","date":"2025-03-03","arxiv_id":"2503.01224","n_code_links":0,"syntology":null},{"paper":null,"slug":"in-context-learning-vs-instruction-tuning-the","title":"In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models","date":"2025-03-03","arxiv_id":"2503.01611","n_code_links":0,"syntology":null},{"paper":null,"slug":"peo-improving-bi-factorial-preference","title":"PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation","date":"2025-03-03","arxiv_id":"2503.01233","n_code_links":0,"syntology":null},{"paper":null,"slug":"robust-multi-objective-preference-alignment","title":"Robust Multi-Objective Preference Alignment with Online DPO","date":"2025-03-01","arxiv_id":"2503.00295","n_code_links":0,"syntology":null},{"paper":null,"slug":"livs-a-pluralistic-alignment-dataset-for","title":"LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spaces","date":"2025-02-27","arxiv_id":"2503.01894","n_code_links":0,"syntology":null}],"papers_shown":100,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":409,"with_a_code_link":184,"where_syntology_ran_a_sample":112,"not_listed_spam_title":0,"listed":409,"listed_where_code_ran":112,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":93,"every_run_a_failure_of_syntologys_instrument":19,"listed_with_a_run_with_no_instrument_failure":93,"listed_every_run_a_failure_of_syntologys_instrument":19,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":112,"papers_where_code_ran_first_page":"/method/dpo/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"archive","page":2,"of":5,"url":"/method/dpo/papers/2","json":"/method/dpo/papers/2.json","rows":100},{"order":"archive","page":3,"of":5,"url":"/method/dpo/papers/3","json":"/method/dpo/papers/3.json","rows":100},{"order":"archive","page":4,"of":5,"url":"/method/dpo/papers/4","json":"/method/dpo/papers/4.json","rows":100},{"order":"archive","page":5,"of":5,"url":"/method/dpo/papers/5","json":"/method/dpo/papers/5.json","rows":9},{"order":"ran","page":1,"of":2,"url":"/method/dpo/papers/ran/1","json":"/method/dpo/papers/ran/1.json","rows":100},{"order":"ran","page":2,"of":2,"url":"/method/dpo/papers/ran/2","json":"/method/dpo/papers/ran/2.json","rows":12}],"tasks":[{"task":"/task/language-modelling","name":"Language Modelling","papers":70},{"task":"/task/language-modeling","name":"Language Modeling","papers":61},{"task":"/task/instruction-following","name":"Instruction Following","papers":29},{"task":"/task/math","name":"Math","papers":26},{"task":"/task/large-language-model","name":"Large Language Model","papers":25},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":25},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":23},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":21},{"task":"/task/question-answering","name":"Question Answering","papers":19},{"task":"/task/mathematical-reasoning","name":"Mathematical Reasoning","papers":18},{"task":"/task/hallucination","name":"Hallucination","papers":16},{"task":"/task/model","name":"model","papers":15},{"task":"/task/diversity","name":"Diversity","papers":14},{"task":"/task/image-generation","name":"Image Generation","papers":13},{"task":"/task/gsm8k","name":"GSM8K","papers":12},{"task":"/task/text-generation","name":"Text Generation","papers":9},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":8},{"task":"/task/text-to-image-generation","name":"Text-to-Image Generation","papers":8},{"task":"/task/safety-alignment","name":"Safety Alignment","papers":7},{"task":"/task/sentence","name":"Sentence","papers":7}],"tasks_shown":20,"n_tasks":181,"usage_by_year":[{"year":"2023","papers":20},{"year":"2024","papers":240},{"year":"2025","papers":149}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/dpo"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=DPO","read_at":"2026-10-05T00:55:15+00:00","query":"DPO","kind":"method","have":"declared_code","tier":"repo_known","verified_implementation":false,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":{"method":"DPO","origin_arxiv_id":"2501.01336"},"resolution":"exact","resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":"different_paper","archive_paper_arxiv_id":"2305.18290","repositories":{"listed":1,"in_graph":1,"list_truncated":false,"strongest_link":"official"},"retrieve_with":null,"name_competition":{"verdict":"outranked","may_be_wrong_sense":true,"bearers_found":8,"served":{"arxiv_id":"2501.01336","citations":8},"leading_other_bearer":{"arxiv_id":"2305.18290","title":"Direct Preference Optimization: Your Language Model is Secretly a Reward Model","citations":10292,"named_by":["acronym_expansion"],"bearer_name":"Direct Preference Optimization (DPO)","ask_with":"have('2305.18290')"},"n_other_bearers_listed":5,"note":"'DPO' is ALSO the name of 'Direct Preference Optimization (DPO)', a method in 'Direct Preference Optimization: Your Language Model is Secretly a Reward Model' (arXiv:2305.18290, 10,292 citations), which is cited FAR more than the paper this answer is about (8). This answer may be about the wrong DPO. If you meant the better-known one, ask again with that entry of `also_named`."},"name_competition_text":"The same name also belongs to “Direct Preference Optimization (DPO)”, a method in the paper “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” (arXiv:2305.18290, 10,292 citations), which the lookup finds cited far more than the paper this answer is about (8 citations), so this answer may be about a different sense of the name.","near_matches":null,"code_nearby":[{"name":"Cal-DPO","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":3,"retrieve_with":"get_reference_implementation(name='Cal-DPO')","origin_arxiv_id":"2412.14516"},{"name":"Kullback-Leibler DPO (KLDPO)","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":3,"retrieve_with":"get_reference_implementation(name='Kullback-Leibler DPO (KLDPO)')","origin_arxiv_id":"2502.01930"}],"code_nearby_caption":"Code Syntology holds under other names, as the lookup lists them (code_nearby); not a complete list, and none of them is this method.","origin_paper":{"arxiv_id":"2501.01336","page":"/paper/aligning-large-language-models-for-faithful","syntology_url":"https://syntology.ai/paper/2501.01336","in_graph_layer":false,"samples":0},"line":"about another paper, arXiv:2501.01336: repository known, no verified implementation (declared_code; repositories: 1 listed of 1 in the graph, strongest link: official). About another paper: this answer is about arXiv:2501.01336, while the archive credits arXiv:2305.18290.","level_legend":{"state":"quoted","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","card_sha256":"479785a84658dd682a11b10237b3620d651d7f87799f5dc7add970aaa83f6d39","note":"Syntology's served verification levels, quoted verbatim from its MCP server card; a row is split where the card separates clauses with a double hyphen, and the separator is not reproduced","levels":[{"level":"V0","quoted":["imports"]},{"level":"V1","quoted":["executes against a synthesized fixture and returns a value of the recorded type"]},{"level":"V2","quoted":["two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs","neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)"]},{"level":"V3","quoted":["paper-derived property tests pass on both of those same two generations."]}],"no_rung":"No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results."}},"syntology_have_state":"loaded","record_sha256":"d7298573350ccd5c2bff5f4be739516f4c0647472c81d23e283affb46fa1b1c6","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}