{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/unbiased-learning-to-rank-meets-reality","title":"Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset","arxiv_id":"2404.02543","date":"2024-04-03","proceeding":null,"authors":["Philipp Hager","Romain Deffayet","Jean-Michel Renders","Onno Zoeter","Maarten de Rijke"],"abstract":"Unbiased learning-to-rank (ULTR) is a well-established framework for learning from user clicks, which are often biased by the ranker collecting the data. While theoretically justified and extensively tested in simulation, ULTR techniques lack empirical validation, especially on modern search engines. The Baidu-ULTR dataset released for the WSDM Cup 2023, collected from Baidu's search engine, offers a rare opportunity to assess the real-world performance of prominent ULTR techniques. Despite multiple submissions during the WSDM Cup 2023 and the subsequent NTCIR ULTRE-2 task, it remains unclear whether the observed improvements stem from applying ULTR or other learning techniques. In this work, we revisit and extend the available experiments on the Baidu-ULTR dataset. We find that standard unbiased learning-to-rank techniques robustly improve click predictions but struggle to consistently improve ranking performance, especially considering the stark differences obtained by choice of ranking loss and query-document features. Our experiments reveal that gains in click prediction do not necessarily translate to enhanced ranking performance on expert relevance annotations, implying that conclusions strongly depend on how success is measured in this benchmark.","url_abs":"https://arxiv.org/abs/2404.02543v3","url_pdf":"https://arxiv.org/pdf/2404.02543v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unbiased-learning-to-rank-meets-reality","repo_url":"https://github.com/philipphager/baidu-bert-model","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"unbiased-learning-to-rank-meets-reality","repo_url":"https://github.com/philipphager/ultr-reproducibility","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[{"task_slug":"learning-to-rank","task_name":"Learning-To-Rank"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"philipphager/baidu-bert-model","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["1910.14424"],"harvested_for_calls":[{"arxiv_id":"1910.14424","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1910.14424"},"call":"get_harvested_code_for_paper(arxiv_id=\"1910.14424\")"}]},{"repo":"philipphager/ultr-reproducibility","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"f2ccba9b24c7f2679681abd7725a786b9a4e326630cdaf994cca04a9eae6c652","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}