{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/bayesian-bandits-balancing-the-exploration","title":"Bayesian bandits: balancing the exploration-exploitation tradeoff via double sampling","arxiv_id":"1709.03162","date":"2017-09-10","proceeding":null,"authors":["Iñigo Urteaga","Chris H. Wiggins"],"abstract":"Reinforcement learning studies how to balance exploration and exploitation in\nreal-world systems, optimizing interactions with the world while simultaneously\nlearning how the world operates. One general class of algorithms for such\nlearning is the multi-armed bandit setting. Randomized probability matching,\nbased upon the Thompson sampling approach introduced in the 1930s, has recently\nbeen shown to perform well and to enjoy provable optimality properties. It\npermits generative, interpretable modeling in a Bayesian setting, where prior\nknowledge is incorporated, and the computed posteriors naturally capture the\nfull state of knowledge. In this work, we harness the information contained in\nthe Bayesian posterior and estimate its sufficient statistics via sampling. In\nseveral application domains, for example in health and medicine, each\ninteraction with the world can be expensive and invasive, whereas drawing\nsamples from the model is relatively inexpensive. Exploiting this viewpoint, we\ndevelop a double sampling technique driven by the uncertainty in the learning\nprocess: it favors exploitation when certain about the properties of each arm,\nexploring otherwise. The proposed algorithm does not make any distributional\nassumption and it is applicable to complex reward distributions, as long as\nBayesian posterior updates are computable. Utilizing the estimated posterior\nsufficient statistics, double sampling autonomously balances the\nexploration-exploitation tradeoff to make better informed decisions. We\nempirically show its reduced cumulative regret when compared to\nstate-of-the-art alternatives in representative bandit settings.","url_abs":"http://arxiv.org/abs/1709.03162v2","url_pdf":"http://arxiv.org/pdf/1709.03162v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bayesian-bandits-balancing-the-exploration","repo_url":"https://github.com/iurteaga/bandits","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"thompson-sampling","task_name":"Thompson Sampling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1709.03162","atlas_url":"https://app.syntology.ai/?focus=1709.03162","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1709.03162"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1709.03162\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1709.03162"},"call":"get_concepts_for_paper(arxiv_id=\"1709.03162\")"}],"by_repo":[{"repo":"iurteaga/bandits","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"bcb6dcac027bae4695d0ee38553ad5ae7927026eed5c02478af5728403b28a0b","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}