{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/m2d2-exploring-general-purpose-audio-language","title":"M2D2: Exploring General-purpose Audio-Language Representations Beyond CLAP","arxiv_id":"2503.22104","date":"2025-03-28","proceeding":null,"authors":["Daisuke Niizumi","Daiki Takeuchi","Masahiro Yasuda","Binh Thien Nguyen","Yasunori Ohishi","Noboru Harada"],"abstract":"Contrastive language-audio pre-training (CLAP) has addressed audio-language tasks such as audio-text retrieval by aligning audio and text in a common feature space. While CLAP addresses general audio-language tasks, its audio features do not generalize well in audio tasks. In contrast, self-supervised learning (SSL) models learn general-purpose audio features that perform well in diverse audio tasks. We pursue representation learning that can be widely used in audio applications and hypothesize that a method that learns both general audio features and CLAP features should achieve our goal, which we call a general-purpose audio-language representation. To implement our hypothesis, we propose M2D2, a second-generation masked modeling duo (M2D) that combines an SSL M2D and CLAP. M2D2 learns two types of features using two modalities (audio and text) in a two-stage training process. It also utilizes advanced LLM-based sentence embeddings in CLAP training for powerful semantic supervision. In the first stage, M2D2 learns generalizable audio features from M2D and CLAP, where CLAP aligns the features with the fine LLM-based semantic embeddings. In the second stage, it learns CLAP features using the audio features learned from the LLM-based embeddings. Through these pre-training stages, M2D2 should enhance generalizability and performance in its audio and CLAP features. Experiments validated that M2D2 achieves effective general-purpose audio-language representation, highlighted with SOTA fine-tuning mAP of 49.0 for AudioSet, SOTA performance in music tasks, and top-level performance in audio-language tasks.","url_abs":"https://arxiv.org/abs/2503.22104v1","url_pdf":"https://arxiv.org/pdf/2503.22104v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"m2d2-exploring-general-purpose-audio-language","repo_url":"https://github.com/nttcslab/eval-audio-repr","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"m2d2-exploring-general-purpose-audio-language","repo_url":"https://github.com/nttcslab/m2d","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"audio-classification","task_name":"Audio Classification"},{"task_slug":"audio-tagging","task_name":"Audio Tagging"},{"task_slug":"audio-captioning","task_name":"Audio captioning"},{"task_slug":"audio-to-text-retrieval","task_name":"Audio to Text Retrieval"},{"task_slug":"emotion-recognition","task_name":"Emotion Recognition"},{"task_slug":"instrument-recognition","task_name":"Instrument Recognition"},{"task_slug":"music-auto-tagging","task_name":"Music Auto-Tagging"},{"task_slug":"music-classification","task_name":"Music Classification"},{"task_slug":"music-genre-classification","task_name":"Music Genre Classification"},{"task_slug":"music-tagging","task_name":"Music Tagging"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"sentence-embeddings","task_name":"Sentence Embeddings"},{"task_slug":"singer-identification","task_name":"Singer Identification"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"text-to-audio-retrieval","task_name":"Text to Audio Retrieval"},{"task_slug":"vocal-technique-classification","task_name":"Vocal technique classification"}],"methods":[{"method_slug":"m2d","method_name":"M2D"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/audio-classification-on-audioset","task":"Audio Classification","dataset":"AudioSet","model":"M2D2","rank_in_archive_order":15,"of":51,"metrics":{"Test mAP":"0.490"},"uses_additional_data":false},{"leaderboard":"/sota/audio-classification-on-esc-50","task":"Audio Classification","dataset":"ESC-50","model":"M2D2 AS+","rank_in_archive_order":3,"of":29,"metrics":{"Accuracy (5-fold)":"98.5","PRE-TRAINING DATASET":"AudioSet,WavCaps","Top-1 Accuracy":"98.5"},"uses_additional_data":true},{"leaderboard":"/sota/emotion-recognition-on-emomusic","task":"Emotion Recognition","dataset":"Emomusic","model":"M2D-CLAP","rank_in_archive_order":1,"of":5,"metrics":{"EmoA":"77.4","EmoV":"61.9"},"uses_additional_data":false},{"leaderboard":"/sota/emotion-recognition-on-emomusic","task":"Emotion Recognition","dataset":"Emomusic","model":"M2D2","rank_in_archive_order":2,"of":5,"metrics":{"EmoA":"76.7","EmoV":"59.3"},"uses_additional_data":false},{"leaderboard":"/sota/emotion-recognition-on-emomusic","task":"Emotion Recognition","dataset":"Emomusic","model":"M2D","rank_in_archive_order":3,"of":5,"metrics":{"EmoA":"76.1","EmoV":"59.4"},"uses_additional_data":false},{"leaderboard":"/sota/instrument-recognition-on-nsynth","task":"Instrument Recognition","dataset":"NSynth","model":"M2D-CLAP","rank_in_archive_order":1,"of":7,"metrics":{"Accuracy":"80.6"},"uses_additional_data":false},{"leaderboard":"/sota/instrument-recognition-on-nsynth","task":"Instrument Recognition","dataset":"NSynth","model":"M2D2 AS+","rank_in_archive_order":2,"of":7,"metrics":{"Accuracy":"79.7"},"uses_additional_data":false},{"leaderboard":"/sota/instrument-recognition-on-nsynth","task":"Instrument Recognition","dataset":"NSynth","model":"M2D AS","rank_in_archive_order":3,"of":7,"metrics":{"Accuracy":"78.7"},"uses_additional_data":false},{"leaderboard":"/sota/music-auto-tagging-on-magnatagatune","task":"Music Auto-Tagging","dataset":"MagnaTagATune","model":"M2D2 AS+","rank_in_archive_order":1,"of":3,"metrics":{"PR-AUC":"41.6","ROC AUC":"91.8"},"uses_additional_data":false},{"leaderboard":"/sota/singer-identification-on-vocalset-1","task":"Singer Identification","dataset":"VocalSet","model":"M2D2 AS+","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"92.7"},"uses_additional_data":true},{"leaderboard":"/sota/singer-identification-on-vocalset-1","task":"Singer Identification","dataset":"VocalSet","model":"M2D2","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy":"91.8"},"uses_additional_data":true},{"leaderboard":"/sota/vocal-technique-classification-on-vocalset-1","task":"Vocal technique classification","dataset":"VocalSet","model":"M2D2 AS+","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy ":"78.9"},"uses_additional_data":true},{"leaderboard":"/sota/vocal-technique-classification-on-vocalset-1","task":"Vocal technique classification","dataset":"VocalSet","model":"M2D2","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy ":"77.4"},"uses_additional_data":true}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"nttcslab/eval-audio-repr","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2404.06095"],"harvested_for_calls":[{"arxiv_id":"2404.06095","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2404.06095"},"call":"get_harvested_code_for_paper(arxiv_id=\"2404.06095\")"}]},{"repo":"nttcslab/m2d","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2210.14648"],"harvested_for_calls":[{"arxiv_id":"2210.14648","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2210.14648"},"call":"get_harvested_code_for_paper(arxiv_id=\"2210.14648\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"86686419e26b5c1cc774ed173f1de8f573981b5b1ca701121abb6db11ff40064","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}