{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/masked-video-distillation-rethinking-masked","title":"Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning","arxiv_id":"2212.04500","date":"2022-12-08","proceeding":"CVPR 2023 1","authors":["Rui Wang","Dongdong Chen","Zuxuan Wu","Yinpeng Chen","Xiyang Dai","Mengchen Liu","Lu Yuan","Yu-Gang Jiang"],"abstract":"Benefiting from masked visual modeling, self-supervised video representation learning has achieved remarkable progress. However, existing methods focus on learning representations from scratch through reconstructing low-level features like raw pixel RGB values. In this paper, we propose masked video distillation (MVD), a simple yet effective two-stage masked feature modeling framework for video representation learning: firstly we pretrain an image (or video) model by recovering low-level features of masked patches, then we use the resulting features as targets for masked feature modeling. For the choice of teacher models, we observe that students taught by video teachers perform better on temporally-heavy video tasks, while image teachers transfer stronger spatial representations for spatially-heavy video tasks. Visualization analysis also indicates different teachers produce different learned patterns for students. Motivated by this observation, we design a spatial-temporal co-teaching method for MVD. Specifically, we distill student models from both video teachers and image teachers by masked feature modeling. Extensive experimental results demonstrate that video transformers pretrained with spatial-temporal co-teaching outperform models distilled with a single teacher on a multitude of video datasets. Our MVD with vanilla ViT achieves state-of-the-art performance compared with previous supervised or self-supervised methods on several challenging video downstream tasks. For example, with the ViT-Large model, our MVD achieves 86.4% and 76.7% Top-1 accuracy on Kinetics-400 and Something-Something-v2, outperforming VideoMAE by 1.2% and 2.4% respectively. When a larger ViT-Huge model is adopted, MVD achieves the state-of-the-art performance with 77.3% Top-1 accuracy on Something-Something-v2 and 41.1 mAP on AVA v2.2. Code will be available at \\url{https://github.com/ruiwang2021/mvd}.","url_abs":"https://arxiv.org/abs/2212.04500v2","url_pdf":"https://arxiv.org/pdf/2212.04500v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"masked-video-distillation-rethinking-masked","repo_url":"https://github.com/ruiwang2021/mvd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"masked-video-distillation-rethinking-masked","repo_url":"https://github.com/2023-MindSpore-4/Code-5/tree/main/MVD","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"masked-video-distillation-rethinking-masked","repo_url":"https://github.com/Mind23-2/MindCode-101/tree/main/MVD","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"masked-video-distillation-rethinking-masked","repo_url":"https://github.com/Mind23-2/MindCode-3/tree/main/MVD","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-action-recognition","task_name":"Self-Supervised Action Recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"MVD (K400 pretrain, ViT-H, 16x224x224)","rank_in_archive_order":40,"of":207,"metrics":{"Acc@1":"87.2","Acc@5":"97.4"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"MVD (K400 pretrain, ViT-L, 16x224x224)","rank_in_archive_order":47,"of":207,"metrics":{"Acc@1":"86.4","Acc@5":"97.0"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"MVD (K400 pretrain, ViT-B, 16x224x224)","rank_in_archive_order":67,"of":207,"metrics":{"Acc@1":"83.4","Acc@5":"95.8"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"MVD (K400 pretrain, ViT-S, 16x224x224)","rank_in_archive_order":89,"of":207,"metrics":{"Acc@1":"81.0","Acc@5":"94.8"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-on-ava-v2-2","task":"Action Recognition","dataset":"AVA v2.2","model":"MVD (Kinetics400 pretrain+finetune, ViT-H, 16x4)","rank_in_archive_order":5,"of":38,"metrics":{"mAP":"41.1"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-on-ava-v2-2","task":"Action Recognition","dataset":"AVA v2.2","model":"MVD (Kinetics400 pretrain, ViT-H, 16x4)","rank_in_archive_order":7,"of":38,"metrics":{"mAP":"40.1"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-on-ava-v2-2","task":"Action Recognition","dataset":"AVA v2.2","model":"MVD (Kinetics400 pretrain+finetune, ViT-L, 16x4)","rank_in_archive_order":12,"of":38,"metrics":{"mAP":"38.7"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-on-ava-v2-2","task":"Action Recognition","dataset":"AVA v2.2","model":"MVD (Kinetics400 pretrain, ViT-L, 16x4)","rank_in_archive_order":14,"of":38,"metrics":{"mAP":"37.7"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-on-ava-v2-2","task":"Action Recognition","dataset":"AVA v2.2","model":"MVD (Kinetics400 pretrain+finetune, ViT-B, 16x4)","rank_in_archive_order":20,"of":38,"metrics":{"mAP":"34.2"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-on-ava-v2-2","task":"Action Recognition","dataset":"AVA v2.2","model":"MVD (Kinetics400 pretrain, ViT-B, 16x4)","rank_in_archive_order":25,"of":38,"metrics":{"mAP":"31.1"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"MVD (Kinetics400 pretrain, ViT-H, 16 frame)","rank_in_archive_order":1,"of":123,"metrics":{"GFLOPs":"1192x6","Parameters":"633","Top-1 Accuracy":"77.3","Top-5 Accuracy":"95.7"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"MVD (Kinetics400 pretrain, ViT-L, 16 frame)","rank_in_archive_order":6,"of":123,"metrics":{"GFLOPs":"597x6","Parameters":"305","Top-1 Accuracy":"76.7","Top-5 Accuracy":"95.5"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"MVD (Kinetics400 pretrain, ViT-B, 16 frame)","rank_in_archive_order":17,"of":123,"metrics":{"GFLOPs":"180x6","Parameters":"87","Top-1 Accuracy":"73.7","Top-5 Accuracy":"94.0"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"MVD (Kinetics400 pretrain, ViT-S, 16 frame)","rank_in_archive_order":34,"of":123,"metrics":{"GFLOPs":"57x6","Parameters":"22","Top-1 Accuracy":"70.9","Top-5 Accuracy":"92.8"},"uses_additional_data":true},{"leaderboard":"/sota/self-supervised-action-recognition-on-hmdb51","task":"Self-Supervised Action Recognition","dataset":"HMDB51","model":"MVD (ViT-B)","rank_in_archive_order":1,"of":48,"metrics":{"Frozen":"false","Pre-Training Dataset":"Kinetics400","Top-1 Accuracy":"79.7"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-action-recognition-on-ucf101","task":"Self-Supervised Action Recognition","dataset":"UCF101","model":"MVD (ViT-B)","rank_in_archive_order":2,"of":53,"metrics":{"3-fold Accuracy":"97.5","Frozen":"false","Pre-Training Dataset":"Kinetics400"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2212.04500","atlas_url":"https://app.syntology.ai/?focus=2212.04500","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2212.04500"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2212.04500\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2212.04500"},"call":"get_concepts_for_paper(arxiv_id=\"2212.04500\")"}],"by_repo":[{"repo":"ruiwang2021/mvd","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"2023-MindSpore-4/Code-5","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"Mind23-2/MindCode-101","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"Mind23-2/MindCode-3","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"12628bdbbdb41b8b4f16aac2c952da24681b3a663d3622fd23210cfbc76b78e4","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}