{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/egocentric-whole-body-motion-capture-with","title":"Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement","arxiv_id":"2311.16495","date":"2023-11-28","proceeding":"CVPR 2024 1","authors":["Jian Wang","Zhe Cao","Diogo Luvizon","Lingjie Liu","Kripasindhu Sarkar","Danhang Tang","Thabo Beeler","Christian Theobalt"],"abstract":"In this work, we explore egocentric whole-body motion capture using a single fisheye camera, which simultaneously estimates human body and hand motion. This task presents significant challenges due to three factors: the lack of high-quality datasets, fisheye camera distortion, and human body self-occlusion. To address these challenges, we propose a novel approach that leverages FisheyeViT to extract fisheye image features, which are subsequently converted into pixel-aligned 3D heatmap representations for 3D human body pose prediction. For hand tracking, we incorporate dedicated hand detection and hand pose estimation networks for regressing 3D hand poses. Finally, we develop a diffusion-based whole-body motion prior model to refine the estimated whole-body motion while accounting for joint uncertainties. To train these networks, we collect a large synthetic dataset, EgoWholeBody, comprising 840,000 high-quality egocentric images captured across a diverse range of whole-body motion sequences. Quantitative and qualitative evaluations demonstrate the effectiveness of our method in producing high-quality whole-body motion estimates from a single egocentric camera.","url_abs":"https://arxiv.org/abs/2311.16495v2","url_pdf":"https://arxiv.org/pdf/2311.16495v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"egocentric-pose-estimation","task_name":"Egocentric Pose Estimation"},{"task_slug":"hand-detection","task_name":"Hand Detection"},{"task_slug":"hand-pose-estimation","task_name":"Hand Pose Estimation"},{"task_slug":"pose-estimation","task_name":"Pose Estimation"},{"task_slug":"pose-prediction","task_name":"Pose Prediction"}],"methods":[{"method_slug":"heatmap","method_name":"Heatmap"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/egocentric-pose-estimation-on-globalegomocap","task":"Egocentric Pose Estimation","dataset":"GlobalEgoMocap Test Dataset","model":"EgoWholeMocap-Temporal","rank_in_archive_order":1,"of":7,"metrics":{"Average MPJPE (mm)":"65.83","PA-MPJPE":"53.47"},"uses_additional_data":true},{"leaderboard":"/sota/egocentric-pose-estimation-on-globalegomocap","task":"Egocentric Pose Estimation","dataset":"GlobalEgoMocap Test Dataset","model":"EgoWholeMocap-Single Frame","rank_in_archive_order":2,"of":7,"metrics":{"Average MPJPE (mm)":"68.59","PA-MPJPE":"55.92"},"uses_additional_data":true},{"leaderboard":"/sota/egocentric-pose-estimation-on-sceneego","task":"Egocentric Pose Estimation","dataset":"SceneEgo","model":"EgoWholeMocap-Temporal","rank_in_archive_order":1,"of":8,"metrics":{"Average MPJPE (mm)":"57.59","PA-MPJPE":"46.55"},"uses_additional_data":true},{"leaderboard":"/sota/egocentric-pose-estimation-on-sceneego","task":"Egocentric Pose Estimation","dataset":"SceneEgo","model":"EgoWholeMocap-Single Frame","rank_in_archive_order":2,"of":8,"metrics":{"Average MPJPE (mm)":"64.19","PA-MPJPE":"50.06"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.16495","atlas_url":"https://app.syntology.ai/?focus=2311.16495","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2311.16495"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2311.16495\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2311.16495"},"call":"get_concepts_for_paper(arxiv_id=\"2311.16495\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"15c4fc2df66f499468f833961d1ee9050adb3422214cdf6fdaec36a903ece591","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}