{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/m2d","slug":"m2d","name":"M2D","full_name":"Masked Modeling Duo","full_name_withheld":false,"description_markdown":"Self-supervised learning (SSL) using masked prediction has made great strides in general-purpose audio representation. This study proposes Masked Modeling Duo (M2D), an improved masked prediction SSL, which learns by predicting representations of masked input signals that serve as training signals. Unlike conventional methods, M2D obtains a training signal by encoding only the masked part, encouraging the two networks in M2D to model the input. While M2D improves general-purpose audio representations, a specialized representation is essential for real-world applications, such as in industrial and medical domains. The often confidential and proprietary data in such domains is typically limited in size and has a different distribution from that in pre-training datasets. Therefore, we propose M2D for X (M2D-X), which extends M2D to enable the pre-training of specialized representations for an application X. M2D-X learns from M2D and an additional task and inputs background noise. We make the additional task configurable to serve diverse applications, while the background noise helps learn on small data and forms a denoising task that makes representation robust. With these design choices, M2D-X should learn a representation specialized to serve various application needs. Our experiments confirmed that the representations for general-purpose audio, specialized for the highly competitive AudioSet and speech domain, and a small-data medical task achieve top-level performance, demonstrating the potential of using our models as a universal audio pre-training framework.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Masked Modeling Duo: Towards a Universal Audio Pre-training Framework","paper":"/paper/masked-modeling-duo-towards-a-universal-audio","first_author":"Daisuke Niizumi","n_authors":5,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/masked-modeling-duo-towards-a-universal-audio"},"source":{"url":"https://arxiv.org/abs/2404.06095v1","title":"Masked Modeling Duo: Towards a Universal Audio Pre-training Framework","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Self-Supervised Learning","url":"/methods/category/self-supervised-learning","pwc_aliases":[]}],"n_papers_tagged":7,"archive_num_papers":7,"papers_newest_first":[{"paper":"/paper/m2d2-exploring-general-purpose-audio-language","slug":"m2d2-exploring-general-purpose-audio-language","title":"M2D2: Exploring General-purpose Audio-Language Representations Beyond CLAP","date":"2025-03-28","arxiv_id":"2503.22104","n_code_links":2,"syntology":null},{"paper":"/paper/boxr-body-and-head-motion-optimization","slug":"boxr-body-and-head-motion-optimization","title":"BOXR: Body and head motion Optimization framework for eXtended Reality","date":"2024-10-16","arxiv_id":"2410.13084","n_code_links":1,"syntology":null},{"paper":"/paper/m2d-clap-masked-modeling-duo-meets-clap-for","slug":"m2d-clap-masked-modeling-duo-meets-clap-for","title":"M2D-CLAP: Masked Modeling Duo Meets CLAP for Learning General-purpose Audio-Language Representation","date":"2024-06-04","arxiv_id":"2406.02032","n_code_links":2,"syntology":null},{"paper":"/paper/exploring-pre-trained-general-purpose-audio","slug":"exploring-pre-trained-general-purpose-audio","title":"Exploring Pre-trained General-purpose Audio Representations for Heart Murmur Detection","date":"2024-04-26","arxiv_id":"2404.17107","n_code_links":2,"syntology":null},{"paper":"/paper/masked-modeling-duo-towards-a-universal-audio","slug":"masked-modeling-duo-towards-a-universal-audio","title":"Masked Modeling Duo: Towards a Universal Audio Pre-training Framework","date":"2024-04-09","arxiv_id":"2404.06095","n_code_links":2,"syntology":{"ran":7,"of":7,"n_ran_checked":4,"n_instrument":3,"unverified":0,"pointer_only":7,"phrase":"7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 1 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified","official":{"repos":["nttcslab/m2d","nttcslab/eval-audio-repr"],"state":"official (archive's flag): 7 ran","n_ran":7,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":0,"ran_from_kinds":["official"]}}},{"paper":"/paper/masked-modeling-duo-for-speech-specializing","slug":"masked-modeling-duo-for-speech-specializing","title":"Masked Modeling Duo for Speech: Specializing General-Purpose Audio Representation to Speech using Denoising Distillation","date":"2023-05-23","arxiv_id":"2305.14079","n_code_links":1,"syntology":null},{"paper":"/paper/masked-modeling-duo-learning-representations","slug":"masked-modeling-duo-learning-representations","title":"Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Input","date":"2022-10-26","arxiv_id":"2210.14648","n_code_links":1,"syntology":{"ran":3,"of":3,"n_ran_checked":1,"n_instrument":2,"unverified":0,"pointer_only":3,"phrase":"3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified","official":{"repos":["nttcslab/m2d"],"state":"official (archive's flag): 3 ran","n_ran":3,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]}}}],"papers_shown":7,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":7,"with_a_code_link":7,"where_syntology_ran_a_sample":2,"not_listed_spam_title":0,"listed":7,"listed_where_code_ran":2,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":2,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":2,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":2,"papers_where_code_ran_first_page":"/method/m2d/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[{"order":"ran","page":1,"of":1,"url":"/method/m2d/papers/ran/1","json":"/method/m2d/papers/ran/1.json","rows":2}],"tasks":[{"task":"/task/self-supervised-learning","name":"Self-Supervised Learning","papers":6},{"task":"/task/audio-classification","name":"Audio Classification","papers":4},{"task":"/task/audio-tagging","name":"Audio Tagging","papers":2},{"task":"/task/denoising","name":"Denoising","papers":2},{"task":"/task/linear-evaluation","name":"Linear evaluation","papers":2},{"task":"/task/music-genre-classification","name":"Music Genre Classification","papers":2},{"task":"/task/speaker-identification","name":"Speaker Identification","papers":2},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":2},{"task":"/task/audio-captioning","name":"Audio captioning","papers":1},{"task":"/task/audio-to-text-retrieval","name":"Audio to Text Retrieval","papers":1},{"task":"/task/classify-murmurs","name":"Classify murmurs","papers":1},{"task":"/task/emotion-recognition","name":"Emotion Recognition","papers":1},{"task":"/task/environment-sound-classification","name":"Environment Sound Classification","papers":1},{"task":null,"name":"GPU","papers":1},{"task":"/task/instrument-recognition","name":"Instrument Recognition","papers":1},{"task":"/task/keyword-spotting","name":"Keyword Spotting","papers":1},{"task":null,"name":"Keyword Spotting on Google Speech Commands","papers":1},{"task":"/task/knowledge-distillation","name":"Knowledge Distillation","papers":1},{"task":"/task/music-auto-tagging","name":"Music Auto-Tagging","papers":1},{"task":"/task/music-classification","name":"Music Classification","papers":1}],"tasks_shown":20,"n_tasks":28,"usage_by_year":[{"year":"2022","papers":1},{"year":"2023","papers":1},{"year":"2024","papers":4},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/m2d"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=M2D","read_at":"2026-10-05T00:58:50+00:00","query":"M2D","kind":"method","have":"unknown_name","tier":null,"verified_implementation":null,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":null,"resolution":null,"resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":null,"archive_paper_arxiv_id":"2404.06095","repositories":null,"retrieve_with":null,"name_competition":{"verdict":"none","may_be_wrong_sense":false,"bearers_found":0,"served":null,"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing in this corpus is named 'M2D' -- no method node, no paper announcing it in its title, no expanded form declaring it as an acronym."},"name_competition_text":null,"near_matches":["M2D-LIF","Model-to-Data (M2D) distillation"],"code_nearby":[],"code_nearby_caption":null,"origin_paper":null,"line":"name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"1d9b4ac4f22c8d1640a3a08ab6983ac689a58d4e36420d131796fe9dc9a797a2","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}