{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/impact-of-the-query-set-on-the-evaluation-of","title":"Impact of the Query Set on the Evaluation of Expert Finding Systems","arxiv_id":"1806.10813","date":"2018-06-28","proceeding":null,"authors":["Brochier Robin  ERIC","Guille Adrien  ERIC","Rothan Benjamin  ERIC","Velcin Julien  ERIC"],"abstract":"Expertise is a loosely defined concept that is hard to formalize. Much\nresearch has focused on designing efficient algorithms for expert finding in\nlarge databases in various application domains. The evaluation of such\nrecommender systems lies most of the time on human-annotated sets of experts\nassociated with topics. The protocol of evaluation consists in using the\nnamings or short descriptions of these topics as raw queries in order to rank\nthe available set of candidates. Several measures taken from the field of\ninformation retrieval are then applied to rate the rankings of candidates\nagainst the ground truth set of experts. In this paper, we apply this\ntopic-query evaluation methodology with the AMiner data and explore a new\ndocument-query methodology to evaluate experts retrieval from a set of queries\nsampled directly from the experts documents. Specifically, we describe two\ndatasets extracted from AMiner, three baseline algorithms from the literature\nbased on several document representations and provide experiment results to\nshow that using a wide range of more realistic queries provides different\nevaluation results to the usual topic-queries.","url_abs":"http://arxiv.org/abs/1806.10813v1","url_pdf":"http://arxiv.org/pdf/1806.10813v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"impact-of-the-query-set-on-the-evaluation-of","repo_url":"https://github.com/brochier/impact_query_expert_finding","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"brochier/impact_query_expert_finding","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"f8862c5e085c13f1aba73d3e9216412c657c7a55bea32cda36a87e3964fd02d9","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}