{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/unimd-towards-unifying-moment-retrieval-and","title":"UniMD: Towards Unifying Moment Retrieval and Temporal Action Detection","arxiv_id":"2404.04933","date":"2024-04-07","proceeding":null,"authors":["Yingsen Zeng","Yujie Zhong","Chengjian Feng","Lin Ma"],"abstract":"Temporal Action Detection (TAD) focuses on detecting pre-defined actions, while Moment Retrieval (MR) aims to identify the events described by open-ended natural language within untrimmed videos. Despite that they focus on different events, we observe they have a significant connection. For instance, most descriptions in MR involve multiple actions from TAD. In this paper, we aim to investigate the potential synergy between TAD and MR. Firstly, we propose a unified architecture, termed Unified Moment Detection (UniMD), for both TAD and MR. It transforms the inputs of the two tasks, namely actions for TAD or events for MR, into a common embedding space, and utilizes two novel query-dependent decoders to generate a uniform output of classification score and temporal segments. Secondly, we explore the efficacy of two task fusion learning approaches, pre-training and co-training, in order to enhance the mutual benefits between TAD and MR. Extensive experiments demonstrate that the proposed task fusion learning scheme enables the two tasks to help each other and outperform the separately trained counterparts. Impressively, UniMD achieves state-of-the-art results on three paired datasets Ego4D, Charades-STA, and ActivityNet. Our code is available at https://github.com/yingsen1/UniMD.","url_abs":"https://arxiv.org/abs/2404.04933v2","url_pdf":"https://arxiv.org/pdf/2404.04933v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unimd-towards-unifying-moment-retrieval-and","repo_url":"https://github.com/yingsen1/unimd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"action-detection","task_name":"Action Detection"},{"task_slug":"moment-queries","task_name":"Moment Queries"},{"task_slug":"moment-retrieval","task_name":"Moment Retrieval"},{"task_slug":"natural-language-moment-retrieval","task_name":"Natural Language Moment Retrieval"},{"task_slug":"natural-language-queries","task_name":"Natural Language Queries"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-detection-on-charades","task":"Action Detection","dataset":"Charades","model":"UniMD+Sync. (RGB+Flow)","rank_in_archive_order":4,"of":16,"metrics":{"mAP":"26.53"},"uses_additional_data":false},{"leaderboard":"/sota/moment-retrieval-on-charades-sta","task":"Moment Retrieval","dataset":"Charades-STA","model":"UniMD+Sync.","rank_in_archive_order":8,"of":25,"metrics":{"R@1 IoU=0.5":"63.98","R@1 IoU=0.7":"44.46","R@5 IoU=0.5":"91.94","R@5 IoU=0.7":"67.72"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-moment-retrieval-on","task":"Natural Language Moment Retrieval","dataset":"ActivityNet Captions","model":"UniMD+Sync.","rank_in_archive_order":8,"of":8,"metrics":{"R@5,IoU=0.5":"80.54","R@5,IoU=0.7":"57.04"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-queries-on-ego4d","task":"Natural Language Queries","dataset":"Ego4D","model":"UniMD+Sync.","rank_in_archive_order":7,"of":10,"metrics":{"R@1 IoU=0.3":"14.16","R@1 IoU=0.5":"10.06","R@1 Mean(0.3 and 0.5)":"12.11","R@5 IoU=0.3":"26.95","R@5 IoU=0.5":"19.16"},"uses_additional_data":false},{"leaderboard":"/sota/temporal-action-localization-on-activitynet","task":"Temporal Action Localization","dataset":"ActivityNet-1.3","model":"UniMD+Sync.","rank_in_archive_order":7,"of":33,"metrics":{"mAP":"39.83","mAP IOU@0.5":"60.29"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.04933","atlas_url":"https://app.syntology.ai/?focus=2404.04933","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2404.04933"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2404.04933\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2404.04933"},"call":"get_concepts_for_paper(arxiv_id=\"2404.04933\")"}],"by_repo":[{"repo":"yingsen1/unimd","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"cf647172a94687340fd1ef9fd28d409cf8a85187584aab9ec0a4bb8a69db6f7c","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}