{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/ppo","slug":"ppo","name":"PPO","full_name":"Proximal Policy Optimization","full_name_withheld":false,"description_markdown":"**Proximal Policy Optimization**, or **PPO**, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable performance of [TRPO](https://paperswithcode.com/method/trpo), while using only first-order optimization. \r\n\r\nLet $r\\_{t}\\left(\\theta\\right)$ denote the probability ratio $r\\_{t}\\left(\\theta\\right) = \\frac{\\pi\\_{\\theta}\\left(a\\_{t}\\mid{s\\_{t}}\\right)}{\\pi\\_{\\theta\\_{old}}\\left(a\\_{t}\\mid{s\\_{t}}\\right)}$, so $r\\left(\\theta\\_{old}\\right) = 1$. TRPO maximizes a “surrogate” objective:\r\n\r\n$$ L^{\\text{CPI}}\\left({\\theta}\\right) = \\hat{\\mathbb{E}}\\_{t}\\left[\\frac{\\pi\\_{\\theta}\\left(a\\_{t}\\mid{s\\_{t}}\\right)}{\\pi\\_{\\theta\\_{old}}\\left(a\\_{t}\\mid{s\\_{t}}\\right)})\\hat{A}\\_{t}\\right] = \\hat{\\mathbb{E}}\\_{t}\\left[r\\_{t}\\left(\\theta\\right)\\hat{A}\\_{t}\\right] $$\r\n\r\nWhere $CPI$ refers to a conservative policy iteration. Without a constraint, maximization of $L^{CPI}$ would lead to an excessively large policy update; hence, we PPO modifies the objective, to penalize changes to the policy that move $r\\_{t}\\left(\\theta\\right)$ away from 1:\r\n\r\n$$ J^{\\text{CLIP}}\\left({\\theta}\\right) = \\hat{\\mathbb{E}}\\_{t}\\left[\\min\\left(r\\_{t}\\left(\\theta\\right)\\hat{A}\\_{t}, \\text{clip}\\left(r\\_{t}\\left(\\theta\\right), 1-\\epsilon, 1+\\epsilon\\right)\\hat{A}\\_{t}\\right)\\right] $$\r\n\r\nwhere $\\epsilon$ is a hyperparameter, say, $\\epsilon = 0.2$. The motivation for this objective is as follows. The first term inside the min is $L^{CPI}$. The second term, $\\text{clip}\\left(r\\_{t}\\left(\\theta\\right), 1-\\epsilon, 1+\\epsilon\\right)\\hat{A}\\_{t}$ modifies the surrogate\r\nobjective by clipping the probability ratio, which removes the incentive for moving $r\\_{t}$ outside of the interval $\\left[1 − \\epsilon, 1 + \\epsilon\\right]$. Finally, we take the minimum of the clipped and unclipped objective, so the final objective is a lower bound (i.e., a pessimistic bound) on the unclipped objective. With this scheme, we only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse. \r\n\r\nOne detail to note is that when we apply PPO for a network where we have shared parameters for actor and critic functions, we typically add to the objective function an error term on value estimation and an entropy term to encourage exploration.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Proximal Policy Optimization Algorithms","paper":"/paper/proximal-policy-optimization-algorithms","first_author":"John Schulman","n_authors":5,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/proximal-policy-optimization-algorithms"},"source":{"url":"http://arxiv.org/abs/1707.06347v2","title":"Proximal Policy Optimization Algorithms","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Policy Gradient Methods","url":"/methods/category/policy-gradient-methods","pwc_aliases":[]}],"n_papers_tagged":949,"archive_num_papers":949,"papers_newest_first":[{"paper":null,"slug":"airllm-diffusion-policy-based-adaptive-lora","title":"AirLLM: Diffusion Policy-based Adaptive LoRA for Remote Fine-Tuning of LLM over the Air","date":"2025-07-15","arxiv_id":"2507.11515","n_code_links":0,"syntology":null},{"paper":"/paper/step-wise-policy-for-rare-tool-knowledge","slug":"step-wise-policy-for-rare-tool-knowledge","title":"Step-wise Policy for Rare-tool Knowledge (SPaRK): Offline RL that Drives Diverse Tool Use in LLMs","date":"2025-07-15","arxiv_id":"2507.11371","n_code_links":1,"syntology":null},{"paper":null,"slug":"scene-aware-conversational-adas-with","title":"Scene-Aware Conversational ADAS with Generative AI for Real-Time Driver Assistance","date":"2025-07-14","arxiv_id":"2507.10500","n_code_links":0,"syntology":null},{"paper":null,"slug":"meta-reinforcement-learning-for-fast-and-data","title":"Meta-Reinforcement Learning for Fast and Data-Efficient Spectrum Allocation in Dynamic Wireless Networks","date":"2025-07-13","arxiv_id":"2507.10619","n_code_links":0,"syntology":null},{"paper":"/paper/deep-reinforcement-learning-with-gradient-1","slug":"deep-reinforcement-learning-with-gradient-1","title":"Deep Reinforcement Learning with Gradient Eligibility Traces","date":"2025-07-12","arxiv_id":"2507.09087","n_code_links":1,"syntology":null},{"paper":"/paper/geo-orbit-a-federated-digital-twin-framework","slug":"geo-orbit-a-federated-digital-twin-framework","title":"Geo-ORBIT: A Federated Digital Twin Framework for Scene-Adaptive Lane Geometry Detection","date":"2025-07-11","arxiv_id":"2507.08743","n_code_links":1,"syntology":null},{"paper":null,"slug":"lead-the-llm-enhanced-planning-system","title":"LeAD: The LLM Enhanced Planning System Converged with End-to-end Autonomous Driving","date":"2025-07-08","arxiv_id":"2507.05754","n_code_links":0,"syntology":null},{"paper":null,"slug":"model-free-optical-processors-using-in-situ","title":"Model-free Optical Processors using In Situ Reinforcement Learning with Proximal Policy Optimization","date":"2025-07-08","arxiv_id":"2507.05583","n_code_links":0,"syntology":null},{"paper":null,"slug":"2048-reinforcement-learning-in-a-delayed","title":"2048: Reinforcement Learning in a Delayed Reward Environment","date":"2025-07-07","arxiv_id":"2507.05465","n_code_links":0,"syntology":null},{"paper":null,"slug":"yolo-apd-enhancing-yolov8-for-robust","title":"YOLO-APD: Enhancing YOLOv8 for Robust Pedestrian Detection on Complex Road Geometries","date":"2025-07-07","arxiv_id":"2507.05376","n_code_links":0,"syntology":null},{"paper":null,"slug":"llm-based-realistic-safety-critical-driving","title":"LLM-based Realistic Safety-Critical Driving Video Generation","date":"2025-07-02","arxiv_id":"2507.01264","n_code_links":0,"syntology":null},{"paper":null,"slug":"bida-a-bi-level-interaction-decision-making","title":"BIDA: A Bi-level Interaction Decision-making Algorithm for Autonomous Vehicles in Dynamic Traffic Scenarios","date":"2025-06-19","arxiv_id":"2506.16546","n_code_links":0,"syntology":null},{"paper":null,"slug":"multi-agent-reinforcement-learning-for-28","title":"Multi-Agent Reinforcement Learning for Autonomous Multi-Satellite Earth Observation: A Realistic Case Study","date":"2025-06-18","arxiv_id":"2506.15207","n_code_links":0,"syntology":null},{"paper":null,"slug":"algorithmic-approaches-to-enhance-safety-in","title":"Algorithmic Approaches to Enhance Safety in Autonomous Vehicles: Minimizing Lane Changes and Merging","date":"2025-06-17","arxiv_id":"2506.15026","n_code_links":0,"syntology":null},{"paper":"/paper/light-aircraft-game-basic-implementation-and","slug":"light-aircraft-game-basic-implementation-and","title":"Light Aircraft Game : Basic Implementation and training results analysis","date":"2025-06-17","arxiv_id":"2506.14164","n_code_links":1,"syntology":null},{"paper":"/paper/tgdpo-harnessing-token-level-reward-guidance","slug":"tgdpo-harnessing-token-level-reward-guidance","title":"TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization","date":"2025-06-17","arxiv_id":"2506.14574","n_code_links":1,"syntology":{"ran":3,"of":3,"n_ran_checked":1,"n_instrument":2,"unverified":0,"pointer_only":3,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","official":{"repos":["dvlab-research/tgdpo"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"adaptive-guidance-accelerates-reinforcement","title":"Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models","date":"2025-06-16","arxiv_id":"2506.13923","n_code_links":0,"syntology":null},{"paper":"/paper/autovla-a-vision-language-action-model-for","slug":"autovla-a-vision-language-action-model-for","title":"AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning","date":"2025-06-16","arxiv_id":"2506.13757","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":0,"n_instrument":1,"unverified":0,"pointer_only":1,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["ucla-mobility/AutoVLA"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"how-real-is-carlas-dynamic-vision-sensor-a","title":"How Real is CARLAs Dynamic Vision Sensor? A Study on the Sim-to-Real Gap in Traffic Object Detection","date":"2025-06-16","arxiv_id":"2506.13722","n_code_links":0,"syntology":null},{"paper":null,"slug":"2506-08149","title":"Ego-centric Learning of Communicative World Models for Autonomous Driving","date":"2025-06-09","arxiv_id":"2506.08149","n_code_links":0,"syntology":null},{"paper":null,"slug":"autonomous-vehicle-lateral-control-using-deep","title":"Autonomous Vehicle Lateral Control Using Deep Reinforcement Learning with MPC-PID Demonstration","date":"2025-06-04","arxiv_id":"2506.04040","n_code_links":0,"syntology":null},{"paper":null,"slug":"ppo-in-the-fisher-rao-geometry","title":"PPO in the Fisher-Rao geometry","date":"2025-06-04","arxiv_id":"2506.03757","n_code_links":0,"syntology":null},{"paper":"/paper/redrft-a-light-weight-benchmark-for","slug":"redrft-a-light-weight-benchmark-for","title":"RedRFT: A Light-Weight Benchmark for Reinforcement Fine-Tuning-Based Red Teaming","date":"2025-06-04","arxiv_id":"2506.04302","n_code_links":1,"syntology":null},{"paper":null,"slug":"response-level-rewards-are-all-you-need-for","title":"Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective","date":"2025-06-03","arxiv_id":"2506.02553","n_code_links":0,"syntology":null},{"paper":null,"slug":"rewarding-the-unlikely-lifting-grpo-beyond","title":"Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening","date":"2025-06-03","arxiv_id":"2506.02355","n_code_links":0,"syntology":null},{"paper":"/paper/the-surprising-effectiveness-of-negative","slug":"the-surprising-effectiveness-of-negative","title":"The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning","date":"2025-06-02","arxiv_id":"2506.01347","n_code_links":1,"syntology":{"ran":3,"of":3,"n_ran_checked":1,"n_instrument":2,"unverified":0,"pointer_only":0,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","official":{"repos":["tianhongzxy/rlvr-decomposed"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"drivemind-a-dual-vlm-based-reinforcement","title":"DriveMind: A Dual-VLM based Reinforcement Learning Framework for Autonomous Driving","date":"2025-06-01","arxiv_id":"2506.00819","n_code_links":0,"syntology":null},{"paper":null,"slug":"language-guided-multi-agent-learning-in","title":"Language-Guided Multi-Agent Learning in Simulations: A Unified Framework and Evaluation","date":"2025-06-01","arxiv_id":"2506.04251","n_code_links":0,"syntology":null},{"paper":"/paper/using-diffusion-ensembles-to-estimate","slug":"using-diffusion-ensembles-to-estimate","title":"Using Diffusion Ensembles to Estimate Uncertainty for End-to-End Autonomous Driving","date":"2025-05-31","arxiv_id":"2506.00560","n_code_links":0,"syntology":null},{"paper":"/paper/areal-a-large-scale-asynchronous","slug":"areal-a-large-scale-asynchronous","title":"AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning","date":"2025-05-30","arxiv_id":"2505.24298","n_code_links":1,"syntology":{"ran":10,"of":13,"n_ran_checked":10,"n_instrument":0,"unverified":3,"pointer_only":0,"phrase":"10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified","official":{"repos":["inclusionai/areal"],"state":"official (archive's flag): 10 ran","n_ran":10,"n_constructed":0,"n_ran_no_instrument_failure":10,"n_unverified":3,"ran_from_kinds":["official"]}}},{"paper":"/paper/composite-reward-design-in-ppo-driven","slug":"composite-reward-design-in-ppo-driven","title":"Composite Reward Design in PPO-Driven Adaptive Filtering","date":"2025-05-29","arxiv_id":"2506.06323","n_code_links":1,"syntology":null},{"paper":"/paper/segment-policy-optimization-effective-segment","slug":"segment-policy-optimization-effective-segment","title":"Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models","date":"2025-05-29","arxiv_id":"2505.23564","n_code_links":1,"syntology":{"ran":7,"of":9,"n_ran_checked":3,"n_instrument":4,"unverified":2,"pointer_only":0,"phrase":"7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified","official":{"repos":["aiframeresearch/spo"],"state":"official (archive's flag): 7 ran","n_ran":7,"n_constructed":0,"n_ran_no_instrument_failure":3,"n_unverified":2,"ran_from_kinds":["official"]}}},{"paper":"/paper/r1-code-interpreter-training-llms-to-reason","slug":"r1-code-interpreter-training-llms-to-reason","title":"R1-Code-Interpreter: Training LLMs to Reason with Code via Supervised and Reinforcement Learning","date":"2025-05-27","arxiv_id":"2505.21668","n_code_links":1,"syntology":null},{"paper":null,"slug":"what-can-rl-bring-to-vla-generalization-an","title":"What Can RL Bring to VLA Generalization? An Empirical Study","date":"2025-05-26","arxiv_id":"2505.19789","n_code_links":0,"syntology":null},{"paper":"/paper/improving-value-estimation-critically","slug":"improving-value-estimation-critically","title":"Improving Value Estimation Critically Enhances Vanilla Policy Gradient","date":"2025-05-25","arxiv_id":"2505.19247","n_code_links":1,"syntology":null},{"paper":null,"slug":"genpo-generative-diffusion-models-meet-on","title":"GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning","date":"2025-05-24","arxiv_id":"2505.18763","n_code_links":0,"syntology":null},{"paper":"/paper/a-robust-ppo-optimized-tabular-transformer","slug":"a-robust-ppo-optimized-tabular-transformer","title":"A Robust PPO-optimized Tabular Transformer Framework for Intrusion Detection in Industrial IoT Systems","date":"2025-05-23","arxiv_id":"2505.18234","n_code_links":1,"syntology":null},{"paper":"/paper/shioenv-a-cli-behavior-capturing-environment","slug":"shioenv-a-cli-behavior-capturing-environment","title":"ShIOEnv: A CLI Behavior-Capturing Environment Enabling Grammar-Guided Command Synthesis for Dataset Curation","date":"2025-05-23","arxiv_id":"2505.18374","n_code_links":1,"syntology":null},{"paper":"/paper/the-cell-must-go-on-agar-io-for-continual","slug":"the-cell-must-go-on-agar-io-for-continual","title":"The Cell Must Go On: Agar.io for Continual Reinforcement Learning","date":"2025-05-23","arxiv_id":"2505.18347","n_code_links":1,"syntology":null},{"paper":"/paper/raw2drive-reinforcement-learning-with-aligned","slug":"raw2drive-reinforcement-learning-with-aligned","title":"Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)","date":"2025-05-22","arxiv_id":"2505.16394","n_code_links":0,"syntology":null},{"paper":null,"slug":"am-ppo-advantage-alpha-modulation-with","title":"AM-PPO: (Advantage) Alpha-Modulation with Proximal Policy Optimization","date":"2025-05-21","arxiv_id":"2505.15514","n_code_links":0,"syntology":null},{"paper":null,"slug":"hcrmp-a-llm-hinted-contextual-reinforcement","title":"HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Driving","date":"2025-05-21","arxiv_id":"2505.15793","n_code_links":0,"syntology":null},{"paper":"/paper/ipad-iterative-proposal-centric-end-to-end","slug":"ipad-iterative-proposal-centric-end-to-end","title":"iPad: Iterative Proposal-centric End-to-End Autonomous Driving","date":"2025-05-21","arxiv_id":"2505.15111","n_code_links":1,"syntology":null},{"paper":"/paper/trajectory-bellman-residual-minimization-a","slug":"trajectory-bellman-residual-minimization-a","title":"Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning","date":"2025-05-21","arxiv_id":"2505.15311","n_code_links":0,"syntology":{"ran":2,"of":2,"n_ran_checked":0,"n_instrument":2,"unverified":0,"pointer_only":0,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","official":null}},{"paper":null,"slug":"interpretable-reinforcement-learning-for-load","title":"Interpretable Reinforcement Learning for Load Balancing using Kolmogorov-Arnold Networks","date":"2025-05-20","arxiv_id":"2505.14459","n_code_links":0,"syntology":null},{"paper":null,"slug":"kippo-koopman-inspired-proximal-policy","title":"KIPPO: Koopman-Inspired Proximal Policy Optimization","date":"2025-05-20","arxiv_id":"2505.14566","n_code_links":0,"syntology":null},{"paper":null,"slug":"power-allocation-for-delay-optimization-in","title":"Power Allocation for Delay Optimization in Device-to-Device Networks: A Graph Reinforcement Learning Approach","date":"2025-05-19","arxiv_id":"2505.12902","n_code_links":0,"syntology":null},{"paper":null,"slug":"bi-level-policy-optimization-with-nystrom","title":"Bi-Level Policy Optimization with Nyström Hypergradients","date":"2025-05-16","arxiv_id":"2505.11714","n_code_links":0,"syntology":null},{"paper":"/paper/reinforcement-learning-finetunes-small","slug":"reinforcement-learning-finetunes-small","title":"Reinforcement Learning Finetunes Small Subnetworks in Large Language Models","date":"2025-05-16","arxiv_id":"2505.11711","n_code_links":0,"syntology":{"ran":2,"of":2,"n_ran_checked":0,"n_instrument":2,"unverified":0,"pointer_only":2,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","official":null}},{"paper":null,"slug":"fine-tuning-diffusion-policies-with","title":"Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps","date":"2025-05-15","arxiv_id":"2505.10482","n_code_links":0,"syntology":null},{"paper":null,"slug":"monte-carlo-beam-search-for-actor-critic","title":"Monte Carlo Beam Search for Actor-Critic Reinforcement Learning in Continuous Control","date":"2025-05-13","arxiv_id":"2505.09029","n_code_links":0,"syntology":null},{"paper":null,"slug":"object-detection-in-adverse-weather","title":"Object detection in adverse weather conditions for autonomous vehicles using Instruct Pix2Pix","date":"2025-05-13","arxiv_id":"2505.08228","n_code_links":0,"syntology":null},{"paper":null,"slug":"dialoguereason-rule-based-rl-sparks-dialogue","title":"DialogueReason: Rule-Based RL Sparks Dialogue Reasoning in LLMs","date":"2025-05-11","arxiv_id":"2505.07049","n_code_links":0,"syntology":null},{"paper":"/paper/a-critical-assessment-of-reinforcement","slug":"a-critical-assessment-of-reinforcement","title":"A critical assessment of reinforcement learning methods for microswimmer navigation in complex flows","date":"2025-05-08","arxiv_id":"2505.05525","n_code_links":1,"syntology":null},{"paper":null,"slug":"multi-objective-reinforcement-learning-for-4","title":"Multi-Objective Reinforcement Learning for Adaptive Personalized Autonomous Driving","date":"2025-05-08","arxiv_id":"2505.05223","n_code_links":0,"syntology":null},{"paper":null,"slug":"reinforcement-learning-for-game-theoretic","title":"Reinforcement Learning for Game-Theoretic Resource Allocation on Graphs","date":"2025-05-08","arxiv_id":"2505.06319","n_code_links":0,"syntology":null},{"paper":"/paper/the-city-that-never-settles-simulation-based","slug":"the-city-that-never-settles-simulation-based","title":"The City that Never Settles: Simulation-based LiDAR Dataset for Long-Term Place Recognition Under Extreme Structural Changes","date":"2025-05-08","arxiv_id":"2505.05076","n_code_links":1,"syntology":null},{"paper":"/paper/x-driver-explainable-autonomous-driving-with","slug":"x-driver-explainable-autonomous-driving-with","title":"X-Driver: Explainable Autonomous Driving with Vision-Language Models","date":"2025-05-08","arxiv_id":"2505.05098","n_code_links":0,"syntology":null},{"paper":null,"slug":"putting-the-value-back-in-rl-better-test-time","title":"Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers","date":"2025-05-07","arxiv_id":"2505.04842","n_code_links":0,"syntology":null},{"paper":null,"slug":"interpretable-learning-dynamics-in","title":"Interpretable Learning Dynamics in Unsupervised Reinforcement Learning","date":"2025-05-06","arxiv_id":"2505.06279","n_code_links":0,"syntology":null},{"paper":"/paper/hypercontroller-a-hyperparameter-controller","slug":"hypercontroller-a-hyperparameter-controller","title":"HyperController: A Hyperparameter Controller for Fast and Stable Training of Reinforcement Learning Neural Networks","date":"2025-04-27","arxiv_id":"2504.19382","n_code_links":1,"syntology":null},{"paper":null,"slug":"federated-learning-based-semantic","title":"Federated Learning-based Semantic Segmentation for Lane and Object Detection in Autonomous Driving","date":"2025-04-26","arxiv_id":"2504.18939","n_code_links":0,"syntology":null},{"paper":null,"slug":"hierarchical-reinforcement-learning-in-multi","title":"Hierarchical Reinforcement Learning in Multi-Goal Spatial Navigation with Autonomous Mobile Robots","date":"2025-04-26","arxiv_id":"2504.18794","n_code_links":0,"syntology":null},{"paper":"/paper/carl-learning-scalable-planning-policies-with","slug":"carl-learning-scalable-planning-policies-with","title":"CaRL: Learning Scalable Planning Policies with Simple Rewards","date":"2025-04-24","arxiv_id":"2504.17838","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":1,"n_instrument":0,"unverified":0,"pointer_only":1,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified","official":{"repos":["autonomousvision/CaRL"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"training-large-language-models-to-reason-via","title":"Training Large Language Models to Reason via EM Policy Gradient","date":"2025-04-24","arxiv_id":"2504.18587","n_code_links":0,"syntology":null},{"paper":null,"slug":"alphagrad-non-linear-gradient-normalization","title":"AlphaGrad: Non-Linear Gradient Normalization Optimizer","date":"2025-04-22","arxiv_id":"2504.16020","n_code_links":0,"syntology":null},{"paper":"/paper/langcoop-collaborative-driving-with-language","slug":"langcoop-collaborative-driving-with-language","title":"LangCoop: Collaborative Driving with Language","date":"2025-04-18","arxiv_id":"2504.13406","n_code_links":1,"syntology":null},{"paper":null,"slug":"evaluating-the-diversity-and-quality-of-llm","title":"Evaluating the Diversity and Quality of LLM Generated Content","date":"2025-04-16","arxiv_id":"2504.12522","n_code_links":0,"syntology":null},{"paper":"/paper/a-minimalist-approach-to-llm-reasoning-from","slug":"a-minimalist-approach-to-llm-reasoning-from","title":"A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce","date":"2025-04-15","arxiv_id":"2504.11343","n_code_links":1,"syntology":{"ran":1,"of":1,"n_ran_checked":0,"n_instrument":1,"unverified":0,"pointer_only":0,"phrase":"1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["rlhflow/minimal-rl"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"measures-of-variability-for-risk-averse","title":"Measures of Variability for Risk-averse Policy Gradient","date":"2025-04-15","arxiv_id":"2504.11412","n_code_links":0,"syntology":null},{"paper":null,"slug":"iterative-self-training-for-code-generation","title":"Iterative Self-Training for Code Generation via Reinforced Re-Ranking","date":"2025-04-13","arxiv_id":"2504.09643","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-survey-of-frontiers-in-llm-reasoning","title":"A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems","date":"2025-04-12","arxiv_id":"2504.09037","n_code_links":0,"syntology":null},{"paper":"/paper/a-reinforcement-learning-method-for","slug":"a-reinforcement-learning-method-for","title":"A Reinforcement Learning Method for Environments with Stochastic Variables: Post-Decision Proximal Policy Optimization with Dual Critic Networks","date":"2025-04-07","arxiv_id":"2504.05150","n_code_links":1,"syntology":null},{"paper":"/paper/concise-reasoning-via-reinforcement-learning","slug":"concise-reasoning-via-reinforcement-learning","title":"Concise Reasoning via Reinforcement Learning","date":"2025-04-07","arxiv_id":"2504.05185","n_code_links":1,"syntology":{"ran":12,"of":14,"n_ran_checked":6,"n_instrument":6,"unverified":2,"pointer_only":13,"phrase":"12 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 5 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 2 unverified","official":{"repos":["ai-wand/concise-reasoning"],"state":"official (archive's flag): 11 ran","n_ran":11,"n_constructed":0,"n_ran_no_instrument_failure":6,"n_unverified":2,"ran_from_kinds":["official","unlocated"]}}},{"paper":"/paper/deep-reinforcement-learning-algorithms-for-1","slug":"deep-reinforcement-learning-algorithms-for-1","title":"Deep Reinforcement Learning Algorithms for Option Hedging","date":"2025-04-07","arxiv_id":"2504.05521","n_code_links":1,"syntology":null},{"paper":null,"slug":"resource-efficient-beam-prediction-in-mmwave","title":"Resource-Efficient Beam Prediction in mmWave Communications with Multimodal Realistic Simulation Framework","date":"2025-04-07","arxiv_id":"2504.05187","n_code_links":0,"syntology":null},{"paper":null,"slug":"cortex-avd-a-framework-for-corner-case","title":"CORTEX-AVD: A Framework for CORner Case Testing and EXploration in Autonomous Vehicle Development","date":"2025-04-04","arxiv_id":"2504.03989","n_code_links":0,"syntology":null},{"paper":"/paper/end-to-end-driving-with-online-trajectory","slug":"end-to-end-driving-with-online-trajectory","title":"End-to-End Driving with Online Trajectory Evaluation via BEV World Model","date":"2025-04-02","arxiv_id":"2504.01941","n_code_links":1,"syntology":{"ran":7,"of":11,"n_ran_checked":7,"n_instrument":0,"unverified":4,"pointer_only":0,"phrase":"7 ran (of which 7 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified; every one of the 7 samples that ran constructed an object rather than computing a result","official":{"repos":["liyingyanucas/wote"],"state":"official (archive's flag): 7 ran","n_ran":7,"n_constructed":7,"n_ran_no_instrument_failure":7,"n_unverified":4,"ran_from_kinds":["official"]}}},{"paper":"/paper/open-reasoner-zero-an-open-source-approach-to","slug":"open-reasoner-zero-an-open-source-approach-to","title":"Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model","date":"2025-03-31","arxiv_id":"2503.24290","n_code_links":1,"syntology":{"ran":2,"of":4,"n_ran_checked":1,"n_instrument":1,"unverified":2,"pointer_only":0,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified","official":null}},{"paper":"/paper/uniocc-a-unified-benchmark-for-occupancy","slug":"uniocc-a-unified-benchmark-for-occupancy","title":"UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving","date":"2025-03-31","arxiv_id":"2503.24381","n_code_links":1,"syntology":{"ran":4,"of":6,"n_ran_checked":4,"n_instrument":0,"unverified":2,"pointer_only":0,"phrase":"4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified","official":{"repos":["tasl-lab/uniocc"],"state":"official (archive's flag): 4 ran","n_ran":4,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":2,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"on-the-mistaken-assumption-of-interchangeable","title":"On the Mistaken Assumption of Interchangeable Deep Reinforcement Learning Implementations","date":"2025-03-28","arxiv_id":"2503.22575","n_code_links":0,"syntology":null},{"paper":null,"slug":"knowledge-graphs-as-world-models-for-semantic","title":"Knowledge Graphs as World Models for Semantic Material-Aware Obstacle Handling in Autonomous Vehicles","date":"2025-03-27","arxiv_id":"2503.21232","n_code_links":0,"syntology":null},{"paper":"/paper/gapo-learning-preferential-prompt-through","slug":"gapo-learning-preferential-prompt-through","title":"GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization","date":"2025-03-26","arxiv_id":"2503.20194","n_code_links":1,"syntology":null},{"paper":null,"slug":"evolutionary-policy-optimization","title":"Evolutionary Policy Optimization","date":"2025-03-24","arxiv_id":"2503.19037","n_code_links":0,"syntology":null},{"paper":"/paper/curriculum-rl-meets-monte-carlo-planning","slug":"curriculum-rl-meets-monte-carlo-planning","title":"Curriculum RL meets Monte Carlo Planning: Optimization of a Real World Container Management Problem","date":"2025-03-21","arxiv_id":"2503.17194","n_code_links":1,"syntology":null},{"paper":null,"slug":"optimizing-2d-1-packing-in-constrained","title":"Optimizing 2D+1 Packing in Constrained Environments Using Deep Reinforcement Learning","date":"2025-03-21","arxiv_id":"2503.17573","n_code_links":0,"syntology":null},{"paper":null,"slug":"resfl-an-uncertainty-aware-framework-for","title":"RESFL: An Uncertainty-Aware Framework for Responsible Federated Learning by Balancing Privacy, Fairness and Utility in Autonomous Vehicles","date":"2025-03-20","arxiv_id":"2503.16251","n_code_links":0,"syntology":null},{"paper":null,"slug":"good-actions-succeed-bad-actions-generalize-a","title":"Good Actions Succeed, Bad Actions Generalize: A Case Study on Why RL Generalizes Better","date":"2025-03-19","arxiv_id":"2503.15693","n_code_links":0,"syntology":null},{"paper":"/paper/reinforcement-learning-based-motion-imitation","slug":"reinforcement-learning-based-motion-imitation","title":"Reinforcement learning-based motion imitation for physiologically plausible musculoskeletal motor control","date":"2025-03-18","arxiv_id":"2503.14637","n_code_links":1,"syntology":null},{"paper":"/paper/diffad-a-unified-diffusion-modeling-approach","slug":"diffad-a-unified-diffusion-modeling-approach","title":"DiffAD: A Unified Diffusion Modeling Approach for Autonomous Driving","date":"2025-03-15","arxiv_id":"2503.12170","n_code_links":0,"syntology":null},{"paper":null,"slug":"exploring-competitive-and-collusive-behaviors","title":"Exploring Competitive and Collusive Behaviors in Algorithmic Pricing with Deep Reinforcement Learning","date":"2025-03-14","arxiv_id":"2503.11270","n_code_links":0,"syntology":null},{"paper":"/paper/an-real-sim-real-rsr-loop-framework-for","slug":"an-real-sim-real-rsr-loop-framework-for","title":"An Real-Sim-Real (RSR) Loop Framework for Generalizable Robotic Policy Transfer with Differentiable Simulation","date":"2025-03-13","arxiv_id":"2503.10118","n_code_links":1,"syntology":null},{"paper":null,"slug":"edge-ai-powered-real-time-decision-making-for","title":"Edge AI-Powered Real-Time Decision-Making for Autonomous Vehicles in Adverse Weather Conditions","date":"2025-03-12","arxiv_id":"2503.09638","n_code_links":0,"syntology":null},{"paper":"/paper/maneuvergpt-agentic-control-for-safe","slug":"maneuvergpt-agentic-control-for-safe","title":"ManeuverGPT Agentic Control for Safe Autonomous Stunt Maneuvers","date":"2025-03-12","arxiv_id":"2503.09035","n_code_links":1,"syntology":null},{"paper":"/paper/simlingo-vision-only-closed-loop-autonomous","slug":"simlingo-vision-only-closed-loop-autonomous","title":"SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment","date":"2025-03-12","arxiv_id":"2503.09594","n_code_links":1,"syntology":{"ran":11,"of":14,"n_ran_checked":8,"n_instrument":3,"unverified":3,"pointer_only":0,"phrase":"11 ran (of which 8 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified","official":null}},{"paper":null,"slug":"jisam-alleviate-labeling-burden-and-corner","title":"JiSAM: Alleviate Labeling Burden and Corner Case Problems in Autonomous Driving via Minimal Real-World Data","date":"2025-03-11","arxiv_id":"2503.08422","n_code_links":0,"syntology":null},{"paper":null,"slug":"task-oriented-co-design-of-communication","title":"Task-Oriented Co-Design of Communication, Computing, and Control for Edge-Enabled Industrial Cyber-Physical Systems","date":"2025-03-11","arxiv_id":"2503.08661","n_code_links":0,"syntology":null},{"paper":null,"slug":"combating-partial-perception-deficit-in","title":"Combating Partial Perception Deficit in Autonomous Driving with Multimodal LLM Commonsense","date":"2025-03-10","arxiv_id":"2503.07020","n_code_links":0,"syntology":null},{"paper":null,"slug":"using-a-single-actor-to-output-personalized","title":"Using a single actor to output personalized policy for different intersections","date":"2025-03-10","arxiv_id":"2503.07678","n_code_links":0,"syntology":null},{"paper":null,"slug":"gflowvlm-enhancing-multi-step-reasoning-in","title":"GFlowVLM: Enhancing Multi-step Reasoning in Vision-Language Models with Generative Flow Networks","date":"2025-03-09","arxiv_id":"2503.06514","n_code_links":0,"syntology":null}],"papers_shown":100,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":949,"with_a_code_link":397,"where_syntology_ran_a_sample":139,"not_listed_spam_title":0,"listed":949,"listed_where_code_ran":139,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":112,"every_run_a_failure_of_syntologys_instrument":27,"listed_with_a_run_with_no_instrument_failure":112,"listed_every_run_a_failure_of_syntologys_instrument":27,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":139,"papers_where_code_ran_first_page":"/method/ppo/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"archive","page":2,"of":10,"url":"/method/ppo/papers/2","json":"/method/ppo/papers/2.json","rows":100},{"order":"archive","page":3,"of":10,"url":"/method/ppo/papers/3","json":"/method/ppo/papers/3.json","rows":100},{"order":"archive","page":4,"of":10,"url":"/method/ppo/papers/4","json":"/method/ppo/papers/4.json","rows":100},{"order":"archive","page":5,"of":10,"url":"/method/ppo/papers/5","json":"/method/ppo/papers/5.json","rows":100},{"order":"archive","page":6,"of":10,"url":"/method/ppo/papers/6","json":"/method/ppo/papers/6.json","rows":100},{"order":"archive","page":7,"of":10,"url":"/method/ppo/papers/7","json":"/method/ppo/papers/7.json","rows":100},{"order":"archive","page":8,"of":10,"url":"/method/ppo/papers/8","json":"/method/ppo/papers/8.json","rows":100},{"order":"archive","page":9,"of":10,"url":"/method/ppo/papers/9","json":"/method/ppo/papers/9.json","rows":100},{"order":"archive","page":10,"of":10,"url":"/method/ppo/papers/10","json":"/method/ppo/papers/10.json","rows":49},{"order":"ran","page":1,"of":2,"url":"/method/ppo/papers/ran/1","json":"/method/ppo/papers/ran/1.json","rows":100},{"order":"ran","page":2,"of":2,"url":"/method/ppo/papers/ran/2","json":"/method/ppo/papers/ran/2.json","rows":39}],"tasks":[{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":314},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":274},{"task":"/task/autonomous-driving","name":"Autonomous Driving","papers":249},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":242},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":157},{"task":"/task/autonomous-vehicles","name":"Autonomous Vehicles","papers":83},{"task":"/task/decision-making","name":"Decision Making","papers":65},{"task":"/task/imitation-learning","name":"Imitation Learning","papers":60},{"task":"/task/object-detection","name":"Object Detection","papers":55},{"task":"/task/object-detection-1","name":"object-detection","papers":51},{"task":"/task/continuous-control","name":"Continuous Control","papers":41},{"task":"/task/mujoco","name":"MuJoCo","papers":41},{"task":"/task/continuous-control","name":"continuous-control","papers":37},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":34},{"task":"/task/language-modelling","name":"Language Modelling","papers":33},{"task":"/task/multi-agent-reinforcement-learning","name":"Multi-agent Reinforcement Learning","papers":28},{"task":"/task/q-learning","name":"Q-Learning","papers":26},{"task":"/task/language-modeling","name":"Language Modeling","papers":24},{"task":"/task/openai-gym","name":"OpenAI Gym","papers":22},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":21}],"tasks_shown":20,"n_tasks":357,"usage_by_year":[{"year":"2017","papers":4},{"year":"2018","papers":21},{"year":"2019","papers":50},{"year":"2020","papers":95},{"year":"2021","papers":113},{"year":"2022","papers":140},{"year":"2023","papers":152},{"year":"2024","papers":232},{"year":"2025","papers":142}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/ppo"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=PPO","read_at":"2026-10-05T01:00:36+00:00","query":"PPO","kind":"method","have":"declared_code","tier":"repo_known","verified_implementation":false,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":{"method":"Proximal Policy Optimization (PPO)","origin_arxiv_id":"1707.06347"},"resolution":"alias","resolved_name_differs":true,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":"same_paper","archive_paper_arxiv_id":"1707.06347","repositories":{"listed":4,"in_graph":187,"list_truncated":true,"strongest_link":"mentioned_in_github"},"retrieve_with":null,"name_competition":{"verdict":"sole_bearer","may_be_wrong_sense":false,"bearers_found":1,"served":{"arxiv_id":"1707.06347","citations":30578},"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing else in this corpus is named 'PPO'. That is a statement about this corpus and not about the field: short names are reused, and a sense we never collected cannot show up here. Confirm `served.arxiv_id` if you had a specific paper in mind."},"name_competition_text":null,"near_matches":null,"code_nearby":[],"code_nearby_caption":null,"origin_paper":{"arxiv_id":"1707.06347","page":"/paper/proximal-policy-optimization-algorithms","syntology_url":"https://syntology.ai/paper/1707.06347","in_graph_layer":true,"samples":176,"ran":99,"n_ran_checked":71,"n_instrument":28,"unverified":77,"phrase":"99 ran (of which 53 constructed an object rather than computing a result; 71 with no instrument failure: 7 honoured, 2 violated, 62 with no contract checked; 28 where Syntology's instrument failed) · 77 unverified","instrument_means":"failures of Syntology's instrument, not of the code"},"line":"repository known, no verified implementation (declared_code; repositories: 4 listed of 187 in the graph, strongest link: mentioned on GitHub). The lookup matched a different name in Syntology's graph, 'Proximal Policy Optimization (PPO)' (resolution: alias); this answer is about that name. This answer is about arXiv:1707.06347, the paper the archive credits with this method.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"bc54fbc7eb50ce8fb1ce2bd9d6d3f30484b120501a79eb98ef38260bc70af5e5","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}