{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/comparative-validation-of-machine-learning","title":"Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark","arxiv_id":"2109.14956","date":"2021-09-30","proceeding":null,"authors":["Martin Wagner","Beat-Peter Müller-Stich","Anna Kisilenko","Duc Tran","Patrick Heger","Lars Mündermann","David M Lubotsky","Benjamin Müller","Tornike Davitashvili","Manuela Capek","Annika Reinke","Tong Yu","Armine Vardazaryan","Chinedu Innocent Nwoye","Nicolas Padoy","Xinyang Liu","Eung-Joo Lee","Constantin Disch","Hans Meine","Tong Xia","Fucang Jia","Satoshi Kondo","Wolfgang Reiter","Yueming Jin","Yonghao Long","Meirui Jiang","Qi Dou","Pheng Ann Heng","Isabell Twick","Kadir Kirtac","Enes Hosgor","Jon Lindström Bolmgren","Michael Stenzel","Björn von Siemens","Hannes G. Kenngott","Felix Nickel","Moritz von Frankenberg","Franziska Mathis-Ullrich","Lena Maier-Hein","Stefanie Speidel","Sebastian Bodenstedt"],"abstract":"PURPOSE: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91% average precision has been reported for phase recognition on an open data single-center dataset. In this work we investigated the generalizability of phase recognition algorithms in a multi-center setting including more difficult recognition tasks such as surgical action and surgical skill. METHODS: To achieve this goal, a dataset with 33 laparoscopic cholecystectomy videos from three surgical centers with a total operation time of 22 hours was created. Labels included annotation of seven surgical phases with 250 phase transitions, 5514 occurences of four surgical actions, 6980 occurences of 21 surgical instruments from seven instrument categories and 495 skill classifications in five skill dimensions. The dataset was used in the 2019 Endoscopic Vision challenge, sub-challenge for surgical workflow and skill analysis. Here, 12 teams submitted their machine learning algorithms for recognition of phase, action, instrument and/or skill assessment. RESULTS: F1-scores were achieved for phase recognition between 23.9% and 67.7% (n=9 teams), for instrument presence detection between 38.5% and 63.8% (n=8 teams), but for action recognition only between 21.8% and 23.3% (n=5 teams). The average absolute error for skill assessment was 0.78 (n=1 team). CONCLUSION: Surgical workflow and skill analysis are promising technologies to support the surgical team, but are not solved yet, as shown by our comparison of algorithms. This novel benchmark can be used for comparable evaluation and validation of future work.","url_abs":"https://arxiv.org/abs/2109.14956v1","url_pdf":"https://arxiv.org/pdf/2109.14956v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"surgical-phase-recognition","task_name":"Surgical phase recognition"}],"methods":[],"datasets_introduced":[{"slug":"heichole-benchmark","name":"HeiChole Benchmark","full_name":"Surgical Workflow and Skill Analysis Challenge (HeiChole Benchmark)"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/surgical-phase-recognition-on-heichole","task":"Surgical phase recognition","dataset":"HeiChole Benchmark","model":"CAMMA1 (challenge model)","rank_in_archive_order":2,"of":5,"metrics":{"F1":"68.8"},"uses_additional_data":false},{"leaderboard":"/sota/surgical-phase-recognition-on-heichole","task":"Surgical phase recognition","dataset":"HeiChole Benchmark","model":"HIKVision (challenge model)","rank_in_archive_order":3,"of":5,"metrics":{"F1":"65.4"},"uses_additional_data":false},{"leaderboard":"/sota/surgical-phase-recognition-on-heichole","task":"Surgical phase recognition","dataset":"HeiChole Benchmark","model":"CUHK (challenge model)","rank_in_archive_order":4,"of":5,"metrics":{"F1":"65"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2109.14956","atlas_url":"https://app.syntology.ai/?focus=2109.14956","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2109.14956"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2109.14956\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2109.14956"},"call":"get_concepts_for_paper(arxiv_id=\"2109.14956\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"013f5b9ea7a4aa73ebb179e6c2b90e206dd0f4261d1d2019f12b5e6fb1138c9a","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}