{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/model-bellman-inconsistency-for-model-based","title":"Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning","arxiv_id":null,"date":"2023-07-01","proceeding":"PMLR 2023 7","authors":["Yihao Sun","Jiaji Zhang","Chengxing Jia","Haoxin Lin","Junyin Ye","Yang Yu"],"abstract":"For offline reinforcement learning (RL), model-based methods are expected to be data-efficient as they incorporate dynamics models to generate more data. However, due to inevitable model errors, straightforwardly learning a policy in the model typically fails in the offline setting. Previous studies have incorporated conservatism to prevent out-of-distribution exploration. For example, MOPO penalizes rewards through uncertainty measures from predicting the next states, which we have discovered are loose bounds of the ideal uncertainty, i.e., the Bellman error. In this work, we propose MOdel-Bellman Inconsistency penalized offLinE Policy Optimization (MOBILE), a novel uncertainty-driven offline RL algorithm. MOBILE conducts uncertainty quantification through the inconsistency of Bellman estimations under an ensemble of learned dynamics models, which can be a better approximator to the true Bellman error, and penalizes the Bellman estimation based on this uncertainty. Empirically we have verified that our proposed uncertainty quantification can be significantly closer to the true Bellman error than the compared methods. Consequently, MOBILE outperforms prior offline RL approaches on most tasks of D4RL and NeoRL benchmarks.","url_abs":"https://proceedings.mlr.press/v202/sun23q.html","url_pdf":"https://proceedings.mlr.press/v202/sun23q/sun23q.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"model-bellman-inconsistency-for-model-based","repo_url":"https://github.com/yihaosun1124/mobile","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"model-bellman-inconsistency-for-model-based","repo_url":"https://github.com/yihaosun1124/OfflineRL-Kit","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"d4rl","task_name":"D4RL"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"uncertainty-quantification","task_name":"Uncertainty Quantification"},{"task_slug":"model","task_name":"model"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[{"repo":"yihaosun1124/mobile","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"yihaosun1124/OfflineRL-Kit","kind":"named_in_paper","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":6,"harvested_for":["2005.13239","2102.08363","2110.01548","2110.06169","2204.12581","2206.04745"],"harvested_for_calls":[{"arxiv_id":"2005.13239","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2005.13239"},"call":"get_harvested_code_for_paper(arxiv_id=\"2005.13239\")"},{"arxiv_id":"2102.08363","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2102.08363"},"call":"get_harvested_code_for_paper(arxiv_id=\"2102.08363\")"},{"arxiv_id":"2110.01548","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2110.01548"},"call":"get_harvested_code_for_paper(arxiv_id=\"2110.01548\")"},{"arxiv_id":"2110.06169","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2110.06169"},"call":"get_harvested_code_for_paper(arxiv_id=\"2110.06169\")"},{"arxiv_id":"2204.12581","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2204.12581"},"call":"get_harvested_code_for_paper(arxiv_id=\"2204.12581\")"},{"arxiv_id":"2206.04745","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2206.04745"},"call":"get_harvested_code_for_paper(arxiv_id=\"2206.04745\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"ed224f54a98288fe15274ad8d09e2400e32d3d39e6ab13dd1f355e7efe16ebf2","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}