{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/multi-stream-keypoint-attention-network-for","title":"Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation","arxiv_id":"2405.05672","date":"2024-05-09","proceeding":null,"authors":["Mo Guan","Yan Wang","Guangkun Ma","Jiarui Liu","Mingzu Sun"],"abstract":"Sign language serves as a non-vocal means of communication, transmitting information and significance through gestures, facial expressions, and bodily movements. The majority of current approaches for sign language recognition (SLR) and translation rely on RGB video inputs, which are vulnerable to fluctuations in the background. Employing a keypoint-based strategy not only mitigates the effects of background alterations but also substantially diminishes the computational demands of the model. Nevertheless, contemporary keypoint-based methodologies fail to fully harness the implicit knowledge embedded in keypoint sequences. To tackle this challenge, our inspiration is derived from the human cognition mechanism, which discerns sign language by analyzing the interplay between gesture configurations and supplementary elements. We propose a multi-stream keypoint attention network to depict a sequence of keypoints produced by a readily available keypoint estimator. In order to facilitate interaction across multiple streams, we investigate diverse methodologies such as keypoint fusion strategies, head fusion, and self-distillation. The resulting framework is denoted as MSKA-SLR, which is expanded into a sign language translation (SLT) model through the straightforward addition of an extra translation network. We carry out comprehensive experiments on well-known benchmarks like Phoenix-2014, Phoenix-2014T, and CSL-Daily to showcase the efficacy of our methodology. Notably, we have attained a novel state-of-the-art performance in the sign language translation task of Phoenix-2014T. The code and models can be accessed at: https://github.com/sutwangyan/MSKA.","url_abs":"https://arxiv.org/abs/2405.05672v1","url_pdf":"https://arxiv.org/pdf/2405.05672v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multi-stream-keypoint-attention-network-for","repo_url":"https://github.com/sutwangyan/MSKA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"sign-language-recognition","task_name":"Sign Language Recognition"},{"task_slug":"sign-language-translation","task_name":"Sign Language Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/sign-language-recognition-on-csl-daily","task":"Sign Language Recognition","dataset":"CSL-Daily","model":"MSKA-SLR","rank_in_archive_order":6,"of":14,"metrics":{"Word Error Rate (WER)":"27.8"},"uses_additional_data":false},{"leaderboard":"/sota/sign-language-recognition-on-rwth-phoenix","task":"Sign Language Recognition","dataset":"RWTH-PHOENIX-Weather 2014","model":"MSKA-SLR","rank_in_archive_order":14,"of":22,"metrics":{"Word Error Rate (WER)":"22.1"},"uses_additional_data":false},{"leaderboard":"/sota/sign-language-recognition-on-rwth-phoenix-1","task":"Sign Language Recognition","dataset":"RWTH-PHOENIX-Weather 2014 T","model":"MSKA-SLR","rank_in_archive_order":8,"of":15,"metrics":{"Word Error Rate (WER)":"20.5"},"uses_additional_data":false},{"leaderboard":"/sota/sign-language-translation-on-csl-daily","task":"Sign Language Translation","dataset":"CSL-Daily","model":"MSKA-SLT","rank_in_archive_order":2,"of":8,"metrics":{"BLEU-4":"25.52"},"uses_additional_data":false},{"leaderboard":"/sota/sign-language-translation-on-rwth-phoenix","task":"Sign Language Translation","dataset":"RWTH-PHOENIX-Weather 2014 T","model":"MSKA-SLT","rank_in_archive_order":1,"of":11,"metrics":{"BLEU-4":"29.03"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"sutwangyan/MSKA","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"3066c6560a1a2bd8e63b8f5e4ea6d23ec2324b233833b45497f665fa31e6f084","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}