{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/efficient-multimodal-semantic-segmentation","title":"Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning","arxiv_id":"2312.00360","date":"2023-12-01","proceeding":null,"authors":["Shaohua Dong","Yunhe Feng","Qing Yang","Yan Huang","Dongfang Liu","Heng Fan"],"abstract":"Multimodal (e.g., RGB-Depth/RGB-Thermal) fusion has shown great potential for improving semantic segmentation in complex scenes (e.g., indoor/low-light conditions). Existing approaches often fully fine-tune a dual-branch encoder-decoder framework with a complicated feature fusion strategy for achieving multimodal semantic segmentation, which is training-costly due to the massive parameter updates in feature extraction and fusion. To address this issue, we propose a surprisingly simple yet effective dual-prompt learning network (dubbed DPLNet) for training-efficient multimodal (e.g., RGB-D/T) semantic segmentation. The core of DPLNet is to directly adapt a frozen pre-trained RGB model to multimodal semantic segmentation, reducing parameter updates. For this purpose, we present two prompt learning modules, comprising multimodal prompt generator (MPG) and multimodal feature adapter (MFA). MPG works to fuse the features from different modalities in a compact manner and is inserted from shadow to deep stages to generate the multi-level multimodal prompts that are injected into the frozen backbone, while MPG adapts prompted multimodal features in the frozen backbone for better multimodal semantic segmentation. Since both the MPG and MFA are lightweight, only a few trainable parameters (3.88M, 4.4% of the pre-trained backbone parameters) are introduced for multimodal feature fusion and learning. Using a simple decoder (3.27M parameters), DPLNet achieves new state-of-the-art performance or is on a par with other complex approaches on four RGB-D/T semantic segmentation datasets while satisfying parameter efficiency. Moreover, we show that DPLNet is general and applicable to other multimodal tasks such as salient object detection and video semantic segmentation. Without special design, DPLNet outperforms many complicated models. Our code will be available at github.com/ShaohuaDong2021/DPLNet.","url_abs":"https://arxiv.org/abs/2312.00360v2","url_pdf":"https://arxiv.org/pdf/2312.00360v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-multimodal-semantic-segmentation","repo_url":"https://github.com/shaohuadong2021/dplnet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"prompt-learning","task_name":"Prompt Learning"},{"task_slug":null,"task_name":"RGBD Semantic Segmentation"},{"task_slug":"salient-object-detection-1","task_name":"Salient Object Detection"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"thermal-image-segmentation","task_name":"Thermal Image Segmentation"},{"task_slug":"video-semantic-segmentation","task_name":"Video Semantic Segmentation"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/semantic-segmentation-on-nyu-depth-v2","task":"Semantic Segmentation","dataset":"NYU Depth v2","model":"DPLNet","rank_in_archive_order":7,"of":121,"metrics":{"Mean IoU":"59.3"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-sun-rgbd","task":"Semantic Segmentation","dataset":"SUN-RGBD","model":"DPLNet","rank_in_archive_order":7,"of":44,"metrics":{"Mean IoU":"52.8%"},"uses_additional_data":false},{"leaderboard":"/sota/thermal-image-segmentation-on-mfn-dataset","task":"Thermal Image Segmentation","dataset":"MFN Dataset","model":"DPLNet","rank_in_archive_order":8,"of":55,"metrics":{"mIOU":"59.3"},"uses_additional_data":false},{"leaderboard":"/sota/thermal-image-segmentation-on-pst900","task":"Thermal Image Segmentation","dataset":"PST900","model":"DPLNet","rank_in_archive_order":6,"of":22,"metrics":{"mIoU":"86.7"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.00360","atlas_url":"https://app.syntology.ai/?focus=2312.00360","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2312.00360"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2312.00360\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2312.00360"},"call":"get_concepts_for_paper(arxiv_id=\"2312.00360\")"}],"by_repo":[{"repo":"shaohuadong2021/dplnet","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2401.01578"],"harvested_for_calls":[{"arxiv_id":"2401.01578","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2401.01578"},"call":"get_harvested_code_for_paper(arxiv_id=\"2401.01578\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"bf0596baeae2a155c7acf45b3c73446b0e54b446fa524f4fb347ef684fda8c06","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}