{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/tdsm-triplet-diffusion-for-skeleton-text","title":"TDSM: Triplet Diffusion for Skeleton-Text Matching in Zero-Shot Action Recognition","arxiv_id":"2411.10745","date":"2024-11-16","proceeding":null,"authors":["Jeonghyeok Do","Munchurl Kim"],"abstract":"We firstly present a diffusion-based action recognition with zero-shot learning for skeleton inputs. In zero-shot skeleton-based action recognition, aligning skeleton features with the text features of action labels is essential for accurately predicting unseen actions. Previous methods focus on direct alignment between skeleton and text latent spaces, but the modality gaps between these spaces hinder robust generalization learning. Motivated from the remarkable performance of text-to-image diffusion models, we leverage their alignment capabilities between different modalities mostly by focusing on the training process during reverse diffusion rather than using their generative power. Based on this, our framework is designed as a Triplet Diffusion for Skeleton-Text Matching (TDSM) method which aligns skeleton features with text prompts through reverse diffusion, embedding the prompts into the unified skeleton-text latent space to achieve robust matching. To enhance discriminative power, we introduce a novel triplet diffusion (TD) loss that encourages our TDSM to correct skeleton-text matches while pushing apart incorrect ones. Our TDSM significantly outperforms the very recent state-of-the-art methods with large margins of 2.36%-point to 13.05%-point, demonstrating superior accuracy and scalability in zero-shot settings through effective skeleton-text matching.","url_abs":"https://arxiv.org/abs/2411.10745v2","url_pdf":"https://arxiv.org/pdf/2411.10745v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tdsm-triplet-diffusion-for-skeleton-text","repo_url":"https://github.com/KAIST-VICLab/TDSM","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"skeleton-based-action-recognition","task_name":"Skeleton Based Action Recognition"},{"task_slug":"text-matching","task_name":"Text Matching"},{"task_slug":null,"task_name":"Triplet"},{"task_slug":"zero-shot-skeletal-action-recognition","task_name":"Zero Shot Skeletal Action Recognition"},{"task_slug":"zero-shot-action-recognition","task_name":"Zero-Shot Action Recognition"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"latent-diffusion-model","method_name":"Latent Diffusion Model"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/zero-shot-skeletal-action-recognition-on-ntu","task":"Zero Shot Skeletal Action Recognition","dataset":"NTU RGB+D","model":"TDSM","rank_in_archive_order":1,"of":9,"metrics":{"Accuracy (12 unseen classes)":"56.03","Accuracy (5 unseen classes)":"86.49","Random Split Accuracy":"88.88"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-skeletal-action-recognition-on-ntu-1","task":"Zero Shot Skeletal Action Recognition","dataset":"NTU RGB+D 120","model":"TDSM","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy (10 unseen classes)":"74.15","Accuracy (24 unseen classes)":"65.06","Random Split Accuracy":"69.47"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-skeletal-action-recognition-on-pku","task":"Zero Shot Skeletal Action Recognition","dataset":"PKU-MMD","model":"TDSM","rank_in_archive_order":2,"of":7,"metrics":{"Random Split Accuracy":"70.76"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.10745","atlas_url":"https://app.syntology.ai/?focus=2411.10745","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2411.10745"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2411.10745\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2411.10745"},"call":"get_concepts_for_paper(arxiv_id=\"2411.10745\")"}],"by_repo":[{"repo":"KAIST-VICLab/TDSM","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"dbe506193140f9a56fdc48d6d24631915ecc0cfc065dd06d3d30c945fc896849","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}