{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/mm-or-a-large-multimodal-operating-room","title":"MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments","arxiv_id":"2503.02579","date":"2025-03-04","proceeding":"CVPR 2025 1","authors":["Ege Özsoy","Chantal Pellegrini","Tobias Czempiel","Felix Tristram","Kun Yuan","David Bani-Harouni","Ulrich Eck","Benjamin Busam","Matthias Keicher","Nassir Navab"],"abstract":"Operating rooms (ORs) are complex, high-stakes environments requiring precise understanding of interactions among medical staff, tools, and equipment for enhancing surgical assistance, situational awareness, and patient safety. Current datasets fall short in scale, realism and do not capture the multimodal nature of OR scenes, limiting progress in OR modeling. To this end, we introduce MM-OR, a realistic and large-scale multimodal spatiotemporal OR dataset, and the first dataset to enable multimodal scene graph generation. MM-OR captures comprehensive OR scenes containing RGB-D data, detail views, audio, speech transcripts, robotic logs, and tracking data and is annotated with panoptic segmentations, semantic scene graphs, and downstream task labels. Further, we propose MM2SG, the first multimodal large vision-language model for scene graph generation, and through extensive experiments, demonstrate its ability to effectively leverage multimodal inputs. Together, MM-OR and MM2SG establish a new benchmark for holistic OR understanding, and open the path towards multimodal scene analysis in complex, high-stakes environments. Our code, and data is available at https://github.com/egeozsoy/MM-OR.","url_abs":"https://arxiv.org/abs/2503.02579v1","url_pdf":"https://arxiv.org/pdf/2503.02579v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mm-or-a-large-multimodal-operating-room","repo_url":"https://github.com/egeozsoy/MM-OR","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"2d-panoptic-segmentation","task_name":"2D Panoptic Segmentation"},{"task_slug":"graph-generation","task_name":"Graph Generation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"scene-graph-generation","task_name":"Scene Graph Generation"},{"task_slug":"video-panoptic-segmentation","task_name":"Video Panoptic Segmentation"}],"methods":[],"datasets_introduced":[{"slug":"mm-or","name":"MM-OR","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/2d-panoptic-segmentation-on-4d-or","task":"2D Panoptic Segmentation","dataset":"4D-OR","model":"MM-OR","rank_in_archive_order":1,"of":1,"metrics":{"VPQ":"71.8"},"uses_additional_data":true},{"leaderboard":"/sota/2d-panoptic-segmentation-on-mm-or","task":"2D Panoptic Segmentation","dataset":"MM-OR","model":"MM-OR","rank_in_archive_order":1,"of":1,"metrics":{"VPQ":"67.5"},"uses_additional_data":true},{"leaderboard":"/sota/scene-graph-generation-on-4d-or","task":"Scene Graph Generation","dataset":"4D-OR","model":"MM2SG","rank_in_archive_order":2,"of":5,"metrics":{"F1":"0.901"},"uses_additional_data":true},{"leaderboard":"/sota/scene-graph-generation-on-mm-or","task":"Scene Graph Generation","dataset":"MM-OR","model":"MM2SG","rank_in_archive_order":1,"of":1,"metrics":{"Macro F1":"0.529"},"uses_additional_data":false},{"leaderboard":"/sota/video-panoptic-segmentation-on-4d-or","task":"Video Panoptic Segmentation","dataset":"4D-OR","model":"MM-OR-VPQ4","rank_in_archive_order":1,"of":2,"metrics":{"VPQ":"69.8"},"uses_additional_data":true},{"leaderboard":"/sota/video-panoptic-segmentation-on-4d-or","task":"Video Panoptic Segmentation","dataset":"4D-OR","model":"MM-OR-VPQ8","rank_in_archive_order":2,"of":2,"metrics":{"VPQ":"69.2"},"uses_additional_data":true},{"leaderboard":"/sota/video-panoptic-segmentation-on-mm-or","task":"Video Panoptic Segmentation","dataset":"MM-OR","model":"MM-OR-VPQ4","rank_in_archive_order":1,"of":2,"metrics":{"VPQ":"67.0"},"uses_additional_data":false},{"leaderboard":"/sota/video-panoptic-segmentation-on-mm-or","task":"Video Panoptic Segmentation","dataset":"MM-OR","model":"MM-OR-VPQ8","rank_in_archive_order":2,"of":2,"metrics":{"VPQ":"66.4"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2503.02579","atlas_url":"https://app.syntology.ai/?focus=2503.02579","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2503.02579"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2503.02579\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2503.02579"},"call":"get_concepts_for_paper(arxiv_id=\"2503.02579\")"}],"by_repo":[{"repo":"egeozsoy/MM-OR","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"48ae8bedff098aa6b71e10868035b1ae913ea6ed91be0a2b779c1461b9fa47fa","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}