{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/trpo","slug":"trpo","name":"TRPO","full_name":"Trust Region Policy Optimization","full_name_withheld":false,"description_markdown":"**Trust Region Policy Optimization**, or **TRPO**, is a policy gradient method in reinforcement learning that avoids parameter updates that change the policy too much with a KL divergence constraint on the size of the policy update at each iteration.\r\n\r\nTake the case of off-policy reinforcement learning, where the policy $\\beta$ for collecting trajectories on rollout workers is different from the policy $\\pi$ to optimize for. The objective function in an off-policy model measures the total advantage over the state visitation distribution and actions, while the mismatch between the training data distribution and the true policy state distribution is compensated with an importance sampling estimator:\r\n\r\n$$ J\\left(\\theta\\right) = \\sum\\_{s\\in{S}}p^{\\pi\\_{\\theta\\_{old}}}\\sum\\_{a\\in\\mathcal{A}}\\left(\\pi\\_{\\theta}\\left(a\\mid{s}\\right)\\hat{A}\\_{\\theta\\_{old}}\\left(s, a\\right)\\right) $$\r\n\r\n$$ J\\left(\\theta\\right) = \\sum\\_{s\\in{S}}p^{\\pi\\_{\\theta\\_{old}}}\\sum\\_{a\\in\\mathcal{A}}\\left(\\beta\\left(a\\mid{s}\\right)\\frac{\\pi\\_{\\theta}\\left(a\\mid{s}\\right)}{\\beta\\left(a\\mid{s}\\right)}\\hat{A}\\_{\\theta\\_{old}}\\left(s, a\\right)\\right) $$\r\n\r\n$$ J\\left(\\theta\\right) = \\mathbb{E}\\_{s\\sim{p}^{\\pi\\_{\\theta\\_{old}}}, a\\sim{\\beta}} \\left(\\frac{\\pi\\_{\\theta}\\left(a\\mid{s}\\right)}{\\beta\\left(a\\mid{s}\\right)}\\hat{A}\\_{\\theta\\_{old}}\\left(s, a\\right)\\right)$$\r\n\r\nWhen training on policy, theoretically the policy for collecting data is same as the policy that we want to optimize. However, when rollout workers and optimizers are running in parallel asynchronously, the behavior policy can get stale. TRPO considers this subtle difference: It labels the behavior policy as $\\pi\\_{\\theta\\_{old}}\\left(a\\mid{s}\\right)$ and thus the objective function becomes:\r\n\r\n$$ J\\left(\\theta\\right) = \\mathbb{E}\\_{s\\sim{p}^{\\pi\\_{\\theta\\_{old}}}, a\\sim{\\pi\\_{\\theta\\_{old}}}} \\left(\\frac{\\pi\\_{\\theta}\\left(a\\mid{s}\\right)}{\\pi\\_{\\theta\\_{old}}\\left(a\\mid{s}\\right)}\\hat{A}\\_{\\theta\\_{old}}\\left(s, a\\right)\\right)$$\r\n\r\nTRPO aims to maximize the objective function $J\\left(\\theta\\right)$ subject to a trust region constraint which enforces the distance between old and new policies measured by KL-divergence to be small enough, within a parameter $\\delta$:\r\n\r\n$$ \\mathbb{E}\\_{s\\sim{p}^{\\pi\\_{\\theta\\_{old}}}} \\left[D\\_{KL}\\left(\\pi\\_{\\theta\\_{old}}\\left(.\\mid{s}\\right)\\mid\\mid\\pi\\_{\\theta}\\left(.\\mid{s}\\right)\\right)\\right] \\leq \\delta$$","description_state":"present","introduced_year":null,"introduced_by":{"title":"Trust Region Policy Optimization","paper":"/paper/trust-region-policy-optimization","first_author":"John Schulman","n_authors":5,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/trust-region-policy-optimization"},"source":{"url":"http://arxiv.org/abs/1502.05477v5","title":"Trust Region Policy Optimization","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Policy Gradient Methods","url":"/methods/category/policy-gradient-methods","pwc_aliases":[]}],"n_papers_tagged":81,"archive_num_papers":81,"papers_newest_first":[{"paper":null,"slug":"staq-it-growing-neural-networks-for-policy","title":"StaQ it! Growing neural networks for Policy Mirror Descent","date":"2025-06-16","arxiv_id":"2506.13862","n_code_links":0,"syntology":null},{"paper":null,"slug":"finite-sample-convergence-bounds-for-trust","title":"Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games","date":"2025-05-28","arxiv_id":"2505.22781","n_code_links":0,"syntology":null},{"paper":"/paper/improving-value-estimation-critically","slug":"improving-value-estimation-critically","title":"Improving Value Estimation Critically Enhances Vanilla Policy Gradient","date":"2025-05-25","arxiv_id":"2505.19247","n_code_links":1,"syntology":null},{"paper":null,"slug":"energy-efficient-rsma-based-leo-satellite","title":"Energy Efficient RSMA-Based LEO Satellite Communications Assisted by UAV-Mounted BD-Active RIS: A DRL Approach","date":"2025-05-07","arxiv_id":"2505.04148","n_code_links":0,"syntology":null},{"paper":null,"slug":"deep-reinforcement-learning-based-user-1","title":"Deep Reinforcement Learning-Based User Association in Hybrid LiFi/WiFi Indoor Networks","date":"2025-03-03","arxiv_id":"2503.01803","n_code_links":0,"syntology":null},{"paper":null,"slug":"fast-convergence-of-softmax-policy-mirror","title":"Fast Convergence of Softmax Policy Mirror Ascent","date":"2024-11-18","arxiv_id":"2411.12042","n_code_links":0,"syntology":null},{"paper":null,"slug":"dynamics-of-resource-allocation-in-o-rans-an","title":"Dynamics of Resource Allocation in O-RANs: An In-depth Exploration of On-Policy and Off-Policy Deep Reinforcement Learning for Real-Time Applications","date":"2024-11-17","arxiv_id":"2412.01839","n_code_links":0,"syntology":null},{"paper":null,"slug":"embedding-safety-into-rl-a-new-take-on-trust","title":"Embedding Safety into RL: A New Take on Trust Region Methods","date":"2024-11-05","arxiv_id":"2411.02957","n_code_links":0,"syntology":null},{"paper":null,"slug":"linear-function-approximation-as-a","title":"Linear Function Approximation as a Computationally Efficient Method to Solve Classical Reinforcement Learning Challenges","date":"2024-05-27","arxiv_id":"2405.20350","n_code_links":0,"syntology":null},{"paper":"/paper/matrix-low-rank-trust-region-policy","slug":"matrix-low-rank-trust-region-policy","title":"Matrix Low-Rank Trust Region Policy Optimization","date":"2024-05-27","arxiv_id":"2405.17625","n_code_links":1,"syntology":null},{"paper":"/paper/joint-physical-digital-facial-attack","slug":"joint-physical-digital-facial-attack","title":"Joint Physical-Digital Facial Attack Detection Via Simulating Spoofing Clues","date":"2024-04-12","arxiv_id":"2404.08450","n_code_links":3,"syntology":null},{"paper":"/paper/policy-mirror-descent-with-lookahead","slug":"policy-mirror-descent-with-lookahead","title":"Policy Mirror Descent with Lookahead","date":"2024-03-21","arxiv_id":"2403.14156","n_code_links":1,"syntology":null},{"paper":null,"slug":"convergence-for-natural-policy-gradient-on","title":"Convergence for Natural Policy Gradient on Infinite-State Queueing MDPs","date":"2024-02-07","arxiv_id":"2402.05274","n_code_links":0,"syntology":null},{"paper":"/paper/simple-policy-optimization","slug":"simple-policy-optimization","title":"Simple Policy Optimization","date":"2024-01-29","arxiv_id":"2401.16025","n_code_links":1,"syntology":null},{"paper":"/paper/clipped-objective-policy-gradients-for","slug":"clipped-objective-policy-gradients-for","title":"Clipped-Objective Policy Gradients for Pessimistic Policy Optimization","date":"2023-11-10","arxiv_id":"2311.05846","n_code_links":1,"syntology":null},{"paper":null,"slug":"dropout-strategy-in-reinforcement-learning","title":"Dropout Strategy in Reinforcement Learning: Limiting the Surrogate Objective Variance in Policy Optimization Methods","date":"2023-10-31","arxiv_id":"2310.20380","n_code_links":0,"syntology":null},{"paper":"/paper/dsac-t-distributional-soft-actor-critic-with","slug":"dsac-t-distributional-soft-actor-critic-with","title":"Distributional Soft Actor-Critic with Three Refinements","date":"2023-10-09","arxiv_id":"2310.05858","n_code_links":2,"syntology":{"ran":5,"of":5,"n_ran_checked":4,"n_instrument":1,"unverified":0,"pointer_only":5,"phrase":"5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["jingliang-duan/dsac-t","jingliang-duan/dsac-v2"],"state":"official (archive's flag): 5 ran","n_ran":5,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"general-munchausen-reinforcement-learning","title":"General Munchausen Reinforcement Learning with Tsallis Kullback-Leibler Divergence","date":"2023-09-21","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"distributional-estimation-of-data-uncertainty","title":"Distributional Estimation of Data Uncertainty for Surveillance Face Anti-spoofing","date":"2023-09-18","arxiv_id":"2309.09485","n_code_links":0,"syntology":null},{"paper":"/paper/containergym-a-real-world-reinforcement","slug":"containergym-a-real-world-reinforcement","title":"ContainerGym: A Real-World Reinforcement Learning Benchmark for Resource Allocation","date":"2023-07-06","arxiv_id":"2307.02991","n_code_links":1,"syntology":null},{"paper":null,"slug":"safe-efficient-comfort-and-energy-saving","title":"Safe, Efficient, Comfort, and Energy-saving Automated Driving through Roundabout Based on Deep Reinforcement Learning","date":"2023-06-20","arxiv_id":"2306.11465","n_code_links":0,"syntology":null},{"paper":"/paper/decision-aware-actor-critic-with-function-1","slug":"decision-aware-actor-critic-with-function-1","title":"Decision-Aware Actor-Critic with Function Approximation and Theoretical Guarantees","date":"2023-05-24","arxiv_id":"2305.15249","n_code_links":1,"syntology":{"ran":1,"of":2,"n_ran_checked":1,"n_instrument":0,"unverified":1,"pointer_only":2,"phrase":"1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; the one sample that ran constructed an object rather than computing a result","official":{"repos":["amirrezakazemi/acpg"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":1,"n_ran_no_instrument_failure":1,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":null,"slug":"trust-region-free-policy-optimization-for","title":"Trust-Region-Free Policy Optimization for Stochastic Policies","date":"2023-02-15","arxiv_id":"2302.07985","n_code_links":0,"syntology":null},{"paper":"/paper/sample-dropout-a-simple-yet-effective","slug":"sample-dropout-a-simple-yet-effective","title":"Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimization","date":"2023-02-05","arxiv_id":"2302.02299","n_code_links":1,"syntology":{"ran":6,"of":12,"n_ran_checked":4,"n_instrument":2,"unverified":6,"pointer_only":12,"phrase":"6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 6 unverified","official":{"repos":["linzichuan/sdpo"],"state":"official (archive's flag): 6 ran","n_ran":6,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":6,"ran_from_kinds":["official"]}}},{"paper":"/paper/learning-fast-and-slow-a-goal-directed-memory","slug":"learning-fast-and-slow-a-goal-directed-memory","title":"Learning, Fast and Slow: A Goal-Directed Memory-Based Approach for Dynamic Environments","date":"2023-01-31","arxiv_id":"2301.13758","n_code_links":1,"syntology":null},{"paper":"/paper/a-novel-framework-for-policy-mirror-descent","slug":"a-novel-framework-for-policy-mirror-descent","title":"A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergence","date":"2023-01-30","arxiv_id":"2301.13139","n_code_links":1,"syntology":null},{"paper":null,"slug":"generalized-munchausen-reinforcement-learning","title":"Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence","date":"2023-01-27","arxiv_id":"2301.11476","n_code_links":0,"syntology":null},{"paper":null,"slug":"pdvn-a-patch-based-dual-view-network-for-face","title":"PDVN: A Patch-based Dual-view Network for Face Liveness Detection using Light Field Focal Stack","date":"2023-01-17","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/relative-sparsity-for-medical-decision","slug":"relative-sparsity-for-medical-decision","title":"Relative Sparsity for Medical Decision Problems","date":"2022-11-29","arxiv_id":"2211.16566","n_code_links":1,"syntology":null},{"paper":null,"slug":"normality-guided-distributional-reinforcement","title":"Normality-Guided Distributional Reinforcement Learning for Continuous Control","date":"2022-08-28","arxiv_id":"2208.13125","n_code_links":0,"syntology":null},{"paper":null,"slug":"entropy-augmented-reinforcement-learning","title":"Entropy Augmented Reinforcement Learning","date":"2022-08-19","arxiv_id":"2208.09322","n_code_links":0,"syntology":null},{"paper":null,"slug":"asynchronous-curriculum-experience-replay-a","title":"Asynchronous Curriculum Experience Replay: A Deep Reinforcement Learning Approach for UAV Autonomous Motion Control in Unknown Dynamic Environments","date":"2022-07-04","arxiv_id":"2207.01251","n_code_links":0,"syntology":null},{"paper":"/paper/truly-deterministic-policy-optimization-1","slug":"truly-deterministic-policy-optimization-1","title":"Truly Deterministic Policy Optimization","date":"2022-05-30","arxiv_id":"2205.15379","n_code_links":1,"syntology":null},{"paper":null,"slug":"is-word-error-rate-a-good-evaluation-metric","title":"Is Word Error Rate a good evaluation metric for Speech Recognition in Indic Languages?","date":"2022-03-30","arxiv_id":"2203.16601","n_code_links":0,"syntology":null},{"paper":"/paper/mirror-learning-a-unifying-framework-of","slug":"mirror-learning-a-unifying-framework-of","title":"Mirror Learning: A Unifying Framework of Policy Optimisation","date":"2022-01-07","arxiv_id":"2201.02373","n_code_links":1,"syntology":null},{"paper":null,"slug":"learning-reward-machines-a-study-in-partially","title":"Learning Reward Machines: A Study in Partially Observable Reinforcement Learning","date":"2021-12-17","arxiv_id":"2112.09477","n_code_links":0,"syntology":null},{"paper":null,"slug":"entrpo-trust-region-policy-optimization","title":"EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization","date":"2021-10-26","arxiv_id":"2110.13373","n_code_links":0,"syntology":null},{"paper":"/paper/a-functional-mirror-ascent-view-of-policy","slug":"a-functional-mirror-ascent-view-of-policy","title":"A general class of surrogate functions for stable and efficient reinforcement learning","date":"2021-08-12","arxiv_id":"2108.05828","n_code_links":1,"syntology":null},{"paper":null,"slug":"refined-policy-improvement-bounds-for-mdps","title":"Refined Policy Improvement Bounds for MDPs","date":"2021-07-16","arxiv_id":"2107.08068","n_code_links":0,"syntology":null},{"paper":null,"slug":"curious-explorer-a-provable-exploration","title":"Curious Explorer: a provable exploration strategy in Policy Learning","date":"2021-06-29","arxiv_id":"2106.15503","n_code_links":0,"syntology":null},{"paper":null,"slug":"on-policy-deep-reinforcement-learning-for-the","title":"On-Policy Deep Reinforcement Learning for the Average-Reward Criterion","date":"2021-06-14","arxiv_id":"2106.07329","n_code_links":0,"syntology":null},{"paper":null,"slug":"on-the-linear-convergence-of-natural-policy","title":"On the Linear convergence of Natural Policy Gradient Algorithm","date":"2021-05-04","arxiv_id":"2105.01424","n_code_links":0,"syntology":null},{"paper":"/paper/a-deeppixbis-attentional-angular-margin-for","slug":"a-deeppixbis-attentional-angular-margin-for","title":"A-DeepPixBis: Attentional Angular Margin for Face Anti-Spoofing","date":"2021-03-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"optimization-issues-in-kl-constrained","title":"Optimization Issues in KL-Constrained Approximate Policy Iteration","date":"2021-02-11","arxiv_id":"2102.06234","n_code_links":0,"syntology":null},{"paper":null,"slug":"a-survey-of-motion-planning-algorithms-for","title":"A review of motion planning algorithms for intelligent robotics","date":"2021-02-04","arxiv_id":"2102.02376","n_code_links":0,"syntology":null},{"paper":null,"slug":"trust-region-policy-optimization-regularized","title":"Trust Region Policy Optimization Regularized via PID Control","date":"2020-12-18","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/tonic-a-deep-reinforcement-learning-library","slug":"tonic-a-deep-reinforcement-learning-library","title":"Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking","date":"2020-11-15","arxiv_id":"2011.07537","n_code_links":1,"syntology":{"ran":3,"of":4,"n_ran_checked":0,"n_instrument":3,"unverified":1,"pointer_only":0,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified","official":{"repos":["fabiopardo/tonic"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":"/paper/control-with-adaptive-q-learning","slug":"control-with-adaptive-q-learning","title":"Control with adaptive Q-learning","date":"2020-11-03","arxiv_id":"2011.02141","n_code_links":1,"syntology":null},{"paper":"/paper/multi-agent-trust-region-policy-optimization","slug":"multi-agent-trust-region-policy-optimization","title":"Multi-Agent Trust Region Policy Optimization","date":"2020-10-15","arxiv_id":"2010.07916","n_code_links":1,"syntology":{"ran":16,"of":18,"n_ran_checked":14,"n_instrument":2,"unverified":2,"pointer_only":7,"phrase":"16 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified","official":null}},{"paper":null,"slug":"faded-experience-trust-region-policy","title":"Faded-Experience Trust Region Policy Optimization for Model-Free Power Allocation in Interference Channel","date":"2020-08-04","arxiv_id":"2008.01705","n_code_links":0,"syntology":null},{"paper":null,"slug":"lagrangian-duality-in-reinforcement-learning","title":"Lagrangian Duality in Reinforcement Learning","date":"2020-07-20","arxiv_id":"2007.09998","n_code_links":0,"syntology":null},{"paper":"/paper/optimistic-distributionally-robust-policy","slug":"optimistic-distributionally-robust-policy","title":"Optimistic Distributionally Robust Policy Optimization","date":"2020-06-14","arxiv_id":"2006.07815","n_code_links":1,"syntology":{"ran":4,"of":5,"n_ran_checked":4,"n_instrument":0,"unverified":1,"pointer_only":0,"phrase":"4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified","official":{"repos":["kadysongbb/dr-trpo"],"state":"official (archive's flag): 4 ran","n_ran":4,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":1,"ran_from_kinds":["official"]}}},{"paper":"/paper/implementation-matters-in-deep-policy","slug":"implementation-matters-in-deep-policy","title":"Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO","date":"2020-05-25","arxiv_id":"2005.12729","n_code_links":3,"syntology":{"ran":2,"of":2,"n_ran_checked":1,"n_instrument":1,"unverified":0,"pointer_only":0,"phrase":"2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified","official":{"repos":["MadryLab/implementation-matters"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":"/paper/mirror-descent-policy-optimization","slug":"mirror-descent-policy-optimization","title":"Mirror Descent Policy Optimization","date":"2020-05-20","arxiv_id":"2005.09814","n_code_links":1,"syntology":null},{"paper":"/paper/implementation-matters-in-deep-rl-a-case","slug":"implementation-matters-in-deep-rl-a-case","title":"Implementation Matters in Deep RL: A Case Study on PPO and TRPO","date":"2020-05-01","arxiv_id":null,"n_code_links":2,"syntology":null},{"paper":"/paper/first-order-optimization-in-policy-space-for","slug":"first-order-optimization-in-policy-space-for","title":"First Order Constrained Optimization in Policy Space","date":"2020-02-16","arxiv_id":"2002.06506","n_code_links":2,"syntology":null},{"paper":"/paper/analyzing-policy-distillation-on-multi-task","slug":"analyzing-policy-distillation-on-multi-task","title":"Analyzing Policy Distillation on Multi-Task Learning and Meta-Reinforcement Learning in Meta-World","date":"2020-02-08","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"risk-averse-trust-region-optimization-for","title":"Risk-Averse Trust Region Optimization for Reward-Volatility Reduction","date":"2019-12-06","arxiv_id":"1912.03193","n_code_links":0,"syntology":null},{"paper":"/paper/learning-reward-machines-for-partially","slug":"learning-reward-machines-for-partially","title":"Learning Reward Machines for Partially Observable Reinforcement Learning","date":"2019-12-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"slug":"neural-trust-regionproximal-policy","title":"Neural Trust Region/Proximal Policy Optimization Attains Globally Optimal Policy","date":"2019-12-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"slug":"multi-step-greedy-policies-in-model-free-deep-1","title":"Multi-step Greedy Reinforcement Learning Algorithms","date":"2019-10-07","arxiv_id":"1910.02919","n_code_links":0,"syntology":null},{"paper":null,"slug":"revisit-policy-optimization-in-matrix-form","title":"Revisit Policy Optimization in Matrix Form","date":"2019-09-19","arxiv_id":"1909.09186","n_code_links":0,"syntology":null},{"paper":null,"slug":"adaptive-trust-region-policy-optimization","title":"Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPs","date":"2019-09-06","arxiv_id":"1909.02769","n_code_links":0,"syntology":null},{"paper":"/paper/hindsight-trust-region-policy-optimization","slug":"hindsight-trust-region-policy-optimization","title":"Hindsight Trust Region Policy Optimization","date":"2019-07-29","arxiv_id":"1907.12439","n_code_links":1,"syntology":null},{"paper":null,"slug":"neural-proximaltrust-region-policy","title":"Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy","date":"2019-06-25","arxiv_id":"1906.10306","n_code_links":0,"syntology":null},{"paper":null,"slug":"policy-optimization-via-stochastic-recursive","title":"Policy Optimization via Stochastic Recursive Gradient Algorithm","date":"2019-05-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/supervised-policy-update","slug":"supervised-policy-update","title":"SUPERVISED POLICY UPDATE","date":"2019-05-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/discretizing-continuous-action-space-for-on","slug":"discretizing-continuous-action-space-for-on","title":"Discretizing Continuous Action Space for On-Policy Optimization","date":"2019-01-29","arxiv_id":"1901.10500","n_code_links":2,"syntology":null},{"paper":"/paper/on-policy-trust-region-policy-optimisation","slug":"on-policy-trust-region-policy-optimisation","title":"On-Policy Trust Region Policy Optimisation with Replay Buffers","date":"2019-01-18","arxiv_id":"1901.06212","n_code_links":2,"syntology":null},{"paper":"/paper/multi-objective-model-based-policy-search-for","slug":"multi-objective-model-based-policy-search-for","title":"Multi-objective Model-based Policy Search for Data-efficient Learning with Sparse Rewards","date":"2018-06-25","arxiv_id":"1806.09351","n_code_links":1,"syntology":null},{"paper":"/paper/supervised-policy-update-for-deep","slug":"supervised-policy-update-for-deep","title":"Supervised Policy Update for Deep Reinforcement Learning","date":"2018-05-29","arxiv_id":"1805.11706","n_code_links":1,"syntology":null},{"paper":null,"slug":"variational-inference-for-policy-gradient","title":"Variational Inference for Policy Gradient","date":"2018-02-21","arxiv_id":"1802.07833","n_code_links":0,"syntology":null},{"paper":null,"slug":"sample-efficient-deep-reinforcement-learning","title":"Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spaces","date":"2018-02-11","arxiv_id":"1802.03753","n_code_links":0,"syntology":null},{"paper":null,"slug":"pretraining-deep-actor-critic-reinforcement","title":"Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrations","date":"2018-01-31","arxiv_id":"1801.10459","n_code_links":0,"syntology":null},{"paper":null,"slug":"multi-task-learning-with-gradient-guided","title":"Multi-task Learning with Gradient Guided Policy Specialization","date":"2017-09-23","arxiv_id":"1709.07979","n_code_links":0,"syntology":null},{"paper":"/paper/trust-pcl-an-off-policy-trust-region-method","slug":"trust-pcl-an-off-policy-trust-region-method","title":"Trust-PCL: An Off-Policy Trust Region Method for Continuous Control","date":"2017-07-06","arxiv_id":"1707.01891","n_code_links":1,"syntology":null},{"paper":"/paper/parameter-space-noise-for-exploration","slug":"parameter-space-noise-for-exploration","title":"Parameter Space Noise for Exploration","date":"2017-06-06","arxiv_id":"1706.01905","n_code_links":10,"syntology":{"ran":4,"of":5,"n_ran_checked":2,"n_instrument":2,"unverified":1,"pointer_only":5,"phrase":"4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified","official":null}},{"paper":null,"slug":"a-unified-view-of-entropy-regularized-markov","title":"A unified view of entropy-regularized Markov decision processes","date":"2017-05-22","arxiv_id":"1705.07798","n_code_links":0,"syntology":null},{"paper":"/paper/model-agnostic-meta-learning-for-fast","slug":"model-agnostic-meta-learning-for-fast","title":"Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks","date":"2017-03-09","arxiv_id":"1703.03400","n_code_links":85,"syntology":{"ran":103,"of":154,"n_ran_checked":72,"n_instrument":31,"unverified":51,"pointer_only":57,"phrase":"103 ran (of which 35 constructed an object rather than computing a result; 72 with no instrument failure: 6 honoured, 1 violated, 65 with no contract checked; 31 where Syntology's instrument failed) · 51 unverified","official":{"repos":["cbfinn/maml","cbfinn/maml_rl"],"state":"official (archive's flag): 1 ran","n_ran":1,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["listed","official","unlocated"]}}},{"paper":"/paper/sample-efficient-actor-critic-with-experience","slug":"sample-efficient-actor-critic-with-experience","title":"Sample Efficient Actor-Critic with Experience Replay","date":"2016-11-03","arxiv_id":"1611.01224","n_code_links":7,"syntology":{"ran":0,"of":1,"n_ran_checked":0,"n_instrument":0,"unverified":1,"pointer_only":1,"phrase":"0 ran · 1 unverified","official":null}},{"paper":"/paper/trust-region-policy-optimization","slug":"trust-region-policy-optimization","title":"Trust Region Policy Optimization","date":"2015-02-19","arxiv_id":"1502.05477","n_code_links":24,"syntology":{"ran":7,"of":17,"n_ran_checked":6,"n_instrument":1,"unverified":10,"pointer_only":7,"phrase":"7 ran (of which 2 constructed an object rather than computing a result; 6 with no instrument failure: 2 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 10 unverified","official":null}}],"papers_shown":81,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":81,"with_a_code_link":36,"where_syntology_ran_a_sample":10,"not_listed_spam_title":0,"listed":81,"listed_where_code_ran":10,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":9,"every_run_a_failure_of_syntologys_instrument":1,"listed_with_a_run_with_no_instrument_failure":9,"listed_every_run_a_failure_of_syntologys_instrument":1,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":10,"papers_where_code_ran_first_page":"/method/trpo/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/trpo/papers/ran/1","json":"/method/trpo/papers/ran/1.json","rows":10}],"tasks":[{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":46},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":32},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":30},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":23},{"task":"/task/continuous-control","name":"Continuous Control","papers":11},{"task":"/task/policy-gradient-methods","name":"Policy Gradient Methods","papers":9},{"task":"/task/mujoco","name":"MuJoCo","papers":8},{"task":"/task/continuous-control","name":"continuous-control","papers":8},{"task":"/task/decision-making","name":"Decision Making","papers":5},{"task":"/task/atari-games","name":"Atari Games","papers":3},{"task":"/task/face-anti-spoofing","name":"Face Anti-Spoofing","papers":3},{"task":"/task/face-recognition","name":"Face Recognition","papers":3},{"task":"/task/multi-task-learning","name":"Multi-Task Learning","papers":3},{"task":"/task/benchmarking","name":"Benchmarking","papers":2},{"task":"/task/meta-learning","name":"Meta-Learning","papers":2},{"task":"/task/model-based-reinforcement-learning","name":"Model-based Reinforcement Learning","papers":2},{"task":"/task/openai-gym","name":"OpenAI Gym","papers":2},{"task":"/task/partially-observable-reinforcement-learning","name":"Partially Observable Reinforcement Learning","papers":2},{"task":"/task/problem-decomposition","name":"Problem Decomposition","papers":2},{"task":"/task/q-learning","name":"Q-Learning","papers":2}],"tasks_shown":20,"n_tasks":55,"usage_by_year":[{"year":"2015","papers":1},{"year":"2016","papers":1},{"year":"2017","papers":5},{"year":"2018","papers":5},{"year":"2019","papers":12},{"year":"2020","papers":12},{"year":"2021","papers":10},{"year":"2022","papers":7},{"year":"2023","papers":14},{"year":"2024","papers":9},{"year":"2025","papers":5}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/trpo"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=TRPO","read_at":"2026-10-05T01:03:13+00:00","query":"TRPO","kind":"method","have":"declared_code","tier":"repo_known","verified_implementation":false,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":{"method":"TRPO","origin_arxiv_id":"2606.17664"},"resolution":"exact","resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":"different_paper","archive_paper_arxiv_id":"1502.05477","repositories":{"listed":1,"in_graph":1,"list_truncated":false,"strongest_link":"listed_only"},"retrieve_with":null,"name_competition":{"verdict":"sole_bearer","may_be_wrong_sense":false,"bearers_found":1,"served":{"arxiv_id":"2606.17664","citations":0},"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing else in this corpus is named 'TRPO'. That is a statement about this corpus and not about the field: short names are reused, and a sense we never collected cannot show up here. Confirm `served.arxiv_id` if you had a specific paper in mind."},"name_competition_text":null,"near_matches":null,"code_nearby":[{"name":"Optimal Transport Trust Region Policy Optimization (OT-TRPO)","relation":"shares_name_tokens","relation_means":"shares name tokens with this name; a different method","level":2,"retrieve_with":"get_reference_implementation(name='Optimal Transport Trust Region Policy Optimization (OT-TRPO)')","origin_arxiv_id":"2210.11137"}],"code_nearby_caption":"Code Syntology holds under other names, as the lookup lists them (code_nearby); not a complete list, and none of them is this method.","origin_paper":{"arxiv_id":"2606.17664","page":"/paper/arxiv-2606-17664","syntology_url":"https://syntology.ai/paper/2606.17664","in_graph_layer":true,"samples":5,"ran":4,"n_ran_checked":3,"n_instrument":1,"unverified":1,"phrase":"4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified","instrument_means":"failures of Syntology's instrument, not of the code"},"line":"about another paper, arXiv:2606.17664: repository known, no verified implementation (declared_code; repositories: 1 listed of 1 in the graph, strongest link: listed only). About another paper: this answer is about arXiv:2606.17664, while the archive credits arXiv:1502.05477.","level_legend":{"state":"quoted","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","card_sha256":"479785a84658dd682a11b10237b3620d651d7f87799f5dc7add970aaa83f6d39","note":"Syntology's served verification levels, quoted verbatim from its MCP server card; a row is split where the card separates clauses with a double hyphen, and the separator is not reproduced","levels":[{"level":"V0","quoted":["imports"]},{"level":"V1","quoted":["executes against a synthesized fixture and returns a value of the recorded type"]},{"level":"V2","quoted":["two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs","neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)"]},{"level":"V3","quoted":["paper-derived property tests pass on both of those same two generations."]}],"no_rung":"No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results."}},"syntology_have_state":"loaded","record_sha256":"74ce3d32b582e4270c668e81d87467a10cefcf986c1d7a0b45110e28c12f614b","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}