{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/memorysam-memorize-modalities-and-semantics","title":"MemorySAM: Memorize Modalities and Semantics with Segment Anything Model 2 for Multi-modal Semantic Segmentation","arxiv_id":"2503.06700","date":"2025-03-09","proceeding":null,"authors":["Chenfei Liao","Xu Zheng","Yuanhuiyi Lyu","Haiwei Xue","Yihong Cao","Jiawen Wang","Kailun Yang","Xuming Hu"],"abstract":"Research has focused on Multi-Modal Semantic Segmentation (MMSS), where pixel-wise predictions are derived from multiple visual modalities captured by diverse sensors. Recently, the large vision model, Segment Anything Model 2 (SAM2), has shown strong zero-shot segmentation performance on both images and videos. When extending SAM2 to MMSS, two issues arise: 1. How can SAM2 be adapted to multi-modal data? 2. How can SAM2 better understand semantics? Inspired by cross-frame correlation in videos, we propose to treat multi-modal data as a sequence of frames representing the same scene. Our key idea is to ''memorize'' the modality-agnostic information and 'memorize' the semantics related to the targeted scene. To achieve this, we apply SAM2's memory mechanisms across multi-modal data to capture modality-agnostic features. Meanwhile, to memorize the semantic knowledge, we propose a training-only Semantic Prototype Memory Module (SPMM) to store category-level prototypes across training for facilitating SAM2's transition from instance to semantic segmentation. A prototypical adaptation loss is imposed between global and local prototypes iteratively to align and refine SAM2's semantic understanding. Extensive experimental results demonstrate that our proposed MemorySAM outperforms SoTA methods by large margins on both synthetic and real-world benchmarks (65.38% on DELIVER, 52.88% on MCubeS). Source code will be made publicly available.","url_abs":"https://arxiv.org/abs/2503.06700v1","url_pdf":"https://arxiv.org/pdf/2503.06700v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"memorysam-memorize-modalities-and-semantics","repo_url":"https://github.com/Chenfei-Liao/MemorySAM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"multi-modal-image-segmentation","task_name":"Multi-modal image segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"sam","method_name":"SAM"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/semantic-segmentation-on-deliver","task":"Semantic Segmentation","dataset":"DeLiVER","model":"MemorySAM-B+(R-D-E-L)","rank_in_archive_order":8,"of":26,"metrics":{"mIoU":"65.38"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-deliver","task":"Semantic Segmentation","dataset":"DeLiVER","model":"MemorySAM-B+(R-D)","rank_in_archive_order":9,"of":26,"metrics":{"mIoU":"63.48"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-deliver","task":"Semantic Segmentation","dataset":"DeLiVER","model":"MemorySAM-B+(R-D-E)","rank_in_archive_order":11,"of":26,"metrics":{"mIoU":"62.42"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-deliver","task":"Semantic Segmentation","dataset":"DeLiVER","model":"MemorySAM-B+(RGB)","rank_in_archive_order":17,"of":26,"metrics":{"mIoU":"53.22"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-mcubes","task":"Semantic Segmentation","dataset":"MCubeS","model":"MemorySAM-B+(RGB-A-D-N)","rank_in_archive_order":5,"of":22,"metrics":{"mIoU":"52.88"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-mcubes","task":"Semantic Segmentation","dataset":"MCubeS","model":"MemorySAM-B+(RGB-A-D)","rank_in_archive_order":8,"of":22,"metrics":{"mIoU":"52.20"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-mcubes","task":"Semantic Segmentation","dataset":"MCubeS","model":"MemorySAM-B+(RGB-A)","rank_in_archive_order":12,"of":22,"metrics":{"mIoU":"51.20"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2503.06700","atlas_url":"https://app.syntology.ai/?focus=2503.06700","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2503.06700"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2503.06700\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2503.06700"},"call":"get_concepts_for_paper(arxiv_id=\"2503.06700\")"}],"by_repo":[{"repo":"Chenfei-Liao/MemorySAM","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"eb86bd177860dad04a507ead26bec29bef85067a3877e12a73f7db304f049aa3","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}