{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/data-splits-and-metrics-for-method","title":"Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets","arxiv_id":"2204.05235","date":"2022-04-11","proceeding":null,"authors":["Chinedu Innocent Nwoye","Nicolas Padoy"],"abstract":"In addition to generating data and annotations, devising sensible data splitting strategies and evaluation metrics is essential for the creation of a benchmark dataset. This practice ensures consensus on the usage of the data, homogeneous assessment, and uniform comparison of research methods on the dataset. This study focuses on CholecT50, which is a 50 video surgical dataset that formalizes surgical activities as triplets of <instrument, verb, target>. In this paper, we introduce the standard splits for the CholecT50 and CholecT45 datasets and show how they compare with existing use of the dataset. CholecT45 is the first public release of 45 videos of CholecT50 dataset. We also develop a metrics library, ivtmetrics, for model evaluation on surgical triplets. Furthermore, we conduct a benchmark study by reproducing baseline methods in the most predominantly used deep learning frameworks (PyTorch and TensorFlow) to evaluate them using the proposed data splits and metrics and release them publicly to support future research. The proposed data splits and evaluation metrics will enable global tracking of research progress on the dataset and facilitate optimal model selection for further deployment.","url_abs":"https://arxiv.org/abs/2204.05235v2","url_pdf":"https://arxiv.org/pdf/2204.05235v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/camma-public/attention-tripnet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/camma-public/tripnet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/camma-public/ivtmetrics","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"BSD-2-Clause"}},{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/CAMMA-public/cholect45","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/camma-public/rendezvous","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/CAMMA-public/cholect50","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/camma-public/mcit-ig","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/camma-public/rendezvous-in-time","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-triplet-recognition","task_name":"Action Triplet Recognition"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"model-selection","task_name":"Model Selection"},{"task_slug":null,"task_name":"Triplet"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-triplet-recognition-on-cholect45","task":"Action Triplet Recognition","dataset":"CholecT45","model":"Rendezvous","rank_in_archive_order":1,"of":3,"metrics":{"mAP":"29.4±2.8"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect45","task":"Action Triplet Recognition","dataset":"CholecT45","model":"Attention Tripnet","rank_in_archive_order":2,"of":3,"metrics":{"mAP":"27.2±2.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect45","task":"Action Triplet Recognition","dataset":"CholecT45","model":"Tripnet","rank_in_archive_order":3,"of":3,"metrics":{"mAP":"24.4±4.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect45-cross","task":"Action Triplet Recognition","dataset":"CholecT45 (cross-val)","model":"Rendezvous","rank_in_archive_order":3,"of":5,"metrics":{"mAP":"29.4±2.8"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect45-cross","task":"Action Triplet Recognition","dataset":"CholecT45 (cross-val)","model":"Attention Tripnet","rank_in_archive_order":4,"of":5,"metrics":{"mAP":"27.2±2.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect45-cross","task":"Action Triplet Recognition","dataset":"CholecT45 (cross-val)","model":"Tripnet","rank_in_archive_order":5,"of":5,"metrics":{"mAP":"24.4±4.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50","task":"Action Triplet Recognition","dataset":"CholecT50","model":"Rendezvous (PyTorch)","rank_in_archive_order":2,"of":6,"metrics":{"Mean AP":"29.5"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50","task":"Action Triplet Recognition","dataset":"CholecT50","model":"Attention Tripnet (PyTorch)","rank_in_archive_order":4,"of":6,"metrics":{"Mean AP":"23.3"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50","task":"Action Triplet Recognition","dataset":"CholecT50","model":"Tripnet (PyTorch)","rank_in_archive_order":5,"of":6,"metrics":{"Mean AP":"21.6"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50-1","task":"Action Triplet Recognition","dataset":"CholecT50 (Challenge)","model":"Rendezvous (PyTorch)","rank_in_archive_order":6,"of":27,"metrics":{"mAP":"32.8"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50-1","task":"Action Triplet Recognition","dataset":"CholecT50 (Challenge)","model":"Attention Tripnet (PyTorch)","rank_in_archive_order":12,"of":27,"metrics":{"mAP":"27.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50-1","task":"Action Triplet Recognition","dataset":"CholecT50 (Challenge)","model":"Tripnet (PyTorch)","rank_in_archive_order":13,"of":27,"metrics":{"mAP":"27.4"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50-cross-1","task":"Action Triplet Recognition","dataset":"CholecT50 (cross-val)","model":"Rendezvous","rank_in_archive_order":2,"of":4,"metrics":{"mAP":"29.4±2.5"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50-cross-1","task":"Action Triplet Recognition","dataset":"CholecT50 (cross-val)","model":"Attention Tripnet","rank_in_archive_order":3,"of":4,"metrics":{"mAP":"27.2±2.9"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50-cross-1","task":"Action Triplet Recognition","dataset":"CholecT50 (cross-val)","model":"Tripnet","rank_in_archive_order":4,"of":4,"metrics":{"mAP":"25.3±2.4"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50-cross","task":"Action Triplet Recognition","dataset":"CholecT50(cross-val)","model":"Rendezvous","rank_in_archive_order":1,"of":1,"metrics":{"mAP":"29.4±2.5"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2204.05235","atlas_url":"https://app.syntology.ai/?focus=2204.05235","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2204.05235"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2204.05235\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2204.05235"},"call":"get_concepts_for_paper(arxiv_id=\"2204.05235\")"}],"by_repo":[{"repo":"camma-public/attention-tripnet","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"camma-public/tripnet","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"camma-public/ivtmetrics","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"CAMMA-public/cholect45","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"camma-public/rendezvous","kind":"named_in_paper","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"CAMMA-public/cholect50","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"camma-public/mcit-ig","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"camma-public/rendezvous-in-time","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"ed4393cdab601d5261215a5539197ed2401e9e8625cc5c83b651f35ace38e10d","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}