{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/multiscale-multimodal-transformer-for","title":"Multiscale Multimodal Transformer for Multimodal Action Recognition","arxiv_id":null,"date":"2022-09-22","proceeding":"Submitted to ICLR 2022 9","authors":["Wentao Zhu","Jingru Yi","Xiaohang Sun","Xiang Hao","Linda Liu","Mohamed Omar"],"abstract":"While action recognition has been an active research area for several years, most existing approaches merely leverage the video modality as opposed to humans that efficiently process video and audio cues simultaneously. This limits the usage of recent models to applications where the actions are visually well-defined. On the other hand, audio and video can be perceived in a hierarchical structure, e.g., from audio signal per sampling time point to audio activities and the whole category in the audio classification. In this work, we develop a multiscale multimodal Transformer (MMT) that employs hierarchical representation learning. Particularly, MMT is composed of a novel multiscale audio Transformer (MAT) and a multiscale video Transformer. Furthermore, we propose a set of multimodal supervised contrastive objectives called audio-video contrastive loss (AVC) and intra-modal contrastive loss (IMC) that specifically align the two modalities for robust multimodal representation fusion. MMT surpasses previous state-of-the-art approaches by 7.3%, 1.6% and 2.1% on Kinetics-Sounds, Epic-Kitchens-100 and VGGSound in terms of the top-1 accuracy without external training data. Moreover, our MAT significantly outperforms AST by 22.2%, 4.4% and 4.7% on the three public benchmark datasets and is 3x more efficient based on the number of FLOPs. Through extensive ablation studies and visualizations, we demonstrate that the proposed MMT can effectively capture semantically more separable feature representations from a combination of video and audio signals.","url_abs":"https://openreview.net/pdf?id=aqP3WFwMPbe","url_pdf":"https://openreview.net/pdf?id=aqP3WFwMPbe","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"audio-classification","task_name":"Audio Classification"},{"task_slug":"multi-modal-classification","task_name":"Multi-modal Classification"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-recognition-on-epic-kitchens-100","task":"Action Recognition","dataset":"EPIC-KITCHENS-100","model":"MMT","rank_in_archive_order":13,"of":32,"metrics":{"Action@1":"47.8","Noun@1":"61.0","Verb@1":"70.1"},"uses_additional_data":false},{"leaderboard":"/sota/audio-classification-on-vggsound","task":"Audio Classification","dataset":"VGGSound","model":"MMT (Audio-Visual)","rank_in_archive_order":7,"of":23,"metrics":{"Top 1 Accuracy":"66.2","Top 5 Accuracy":"85.7"},"uses_additional_data":false},{"leaderboard":"/sota/audio-classification-on-vggsound","task":"Audio Classification","dataset":"VGGSound","model":"MMT (Video)","rank_in_archive_order":17,"of":23,"metrics":{"Top 1 Accuracy":"56.1","Top 5 Accuracy":"77.9"},"uses_additional_data":false},{"leaderboard":"/sota/multi-modal-classification-on-vgg-sound","task":"Multi-modal Classification","dataset":"VGG-Sound","model":"MMT","rank_in_archive_order":1,"of":4,"metrics":{"Top-1 Accuracy":"66.2","Top-5 Accuracy":"85.7"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"c44c0eecfc52ea9fa620e43c537171660a3195d1fad8e845f25e5dfa8c2076e0","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}