{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/logically-constrained-reinforcement-learning","title":"Logically-Constrained Reinforcement Learning","arxiv_id":"1801.08099","date":"2018-01-24","proceeding":null,"authors":["Mohammadhosein Hasanbeig","Alessandro Abate","Daniel Kroening"],"abstract":"We present the first model-free Reinforcement Learning (RL) algorithm to\nsynthesise policies for an unknown Markov Decision Process (MDP), such that a\nlinear time property is satisfied. The given temporal property is converted\ninto a Limit Deterministic Buchi Automaton (LDBA) and a robust reward function\nis defined over the state-action pairs of the MDP according to the resulting\nLDBA. With this reward function, the policy synthesis procedure is\n\"constrained\" by the given specification. These constraints guide the MDP\nexploration so as to minimize the solution time by only considering the portion\nof the MDP that is relevant to satisfaction of the LTL property. This improves\nperformance and scalability of the proposed method by avoiding an exhaustive\nupdate over the whole state space while the efficiency of standard methods such\nas dynamic programming is hindered by excessive memory requirements, caused by\nthe need to store a full-model in memory. Additionally, we show that the RL\nprocedure sets up a local value iteration method to efficiently calculate the\nmaximum probability of satisfying the given property, at any given state of the\nMDP. We prove that our algorithm is guaranteed to find a policy whose traces\nprobabilistically satisfy the LTL property if such a policy exists, and\nadditionally we show that our method produces reasonable control policies even\nwhen the LTL property cannot be satisfied. The performance of the algorithm is\nevaluated via a set of numerical examples. We observe an improvement of one\norder of magnitude in the number of iterations required for the synthesis\ncompared to existing approaches.","url_abs":"http://arxiv.org/abs/1801.08099v8","url_pdf":"http://arxiv.org/pdf/1801.08099v8.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"logically-constrained-reinforcement-learning","repo_url":"https://github.com/grockious/lcrl","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"decision-making-under-uncertainty","task_name":"Decision Making Under Uncertainty"},{"task_slug":"hierarchical-reinforcement-learning","task_name":"Hierarchical Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"safe-reinforcement-learning","task_name":"Safe Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1801.08099","atlas_url":"https://app.syntology.ai/?focus=1801.08099","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1801.08099"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1801.08099\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1801.08099"},"call":"get_concepts_for_paper(arxiv_id=\"1801.08099\")"}],"by_repo":[{"repo":"grockious/lcrl","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["1909.05304"],"harvested_for_calls":[{"arxiv_id":"1909.05304","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1909.05304"},"call":"get_harvested_code_for_paper(arxiv_id=\"1909.05304\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"b6b0b33e7618fa79f3219cc5e180870ea63498a5e99c126b32c85c991c821005","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}