{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/dynamic-convolutional-neural-networks-as","title":"Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Models","arxiv_id":"2310.15648","date":"2023-10-24","proceeding":null,"authors":["Florian Schmid","Khaled Koutini","Gerhard Widmer"],"abstract":"The introduction of large-scale audio datasets, such as AudioSet, paved the way for Transformers to conquer the audio domain and replace CNNs as the state-of-the-art neural network architecture for many tasks. Audio Spectrogram Transformers are excellent at exploiting large datasets, creating powerful pre-trained models that surpass CNNs when fine-tuned on downstream tasks. However, current popular Audio Spectrogram Transformers are demanding in terms of computational complexity compared to CNNs. Recently, we have shown that, by employing Transformer-to-CNN Knowledge Distillation, efficient CNNs can catch up with and even outperform Transformers on large datasets. In this work, we extend this line of research and increase the capacity of efficient CNNs by introducing dynamic CNN blocks, constructed of dynamic non-linearities, dynamic convolutions and attention mechanisms. We show that these dynamic CNNs outperform traditional efficient CNNs, in terms of the performance-complexity trade-off and parameter efficiency, at the task of audio tagging on the large-scale AudioSet. Our experiments further indicate that the introduced dynamic CNNs achieve better performance on downstream tasks and scale up well, attaining Transformer performance and even outperforming them on AudioSet and several downstream tasks.","url_abs":"https://arxiv.org/abs/2310.15648v1","url_pdf":"https://arxiv.org/pdf/2310.15648v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dynamic-convolutional-neural-networks-as","repo_url":"https://github.com/fschmid56/efficientat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"audio-classification","task_name":"Audio Classification"},{"task_slug":"audio-tagging","task_name":"Audio Tagging"},{"task_slug":"instrument-recognition","task_name":"Instrument Recognition"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/audio-classification-on-audioset","task":"Audio Classification","dataset":"AudioSet","model":"DyMN-L (Audio-Only, Single)","rank_in_archive_order":14,"of":51,"metrics":{"Test mAP":"0.490"},"uses_additional_data":true},{"leaderboard":"/sota/audio-classification-on-esc-50","task":"Audio Classification","dataset":"ESC-50","model":"DyMN-L","rank_in_archive_order":7,"of":29,"metrics":{"Accuracy (5-fold)":"97.4","PRE-TRAINING DATASET":"AudioSet","Top-1 Accuracy":"97.4"},"uses_additional_data":true},{"leaderboard":"/sota/audio-classification-on-fsd50k","task":"Audio Classification","dataset":"FSD50K","model":"MN","rank_in_archive_order":2,"of":10,"metrics":{"mAP":"65.6"},"uses_additional_data":true},{"leaderboard":"/sota/audio-classification-on-fsd50k","task":"Audio Classification","dataset":"FSD50K","model":"DyMN-L","rank_in_archive_order":4,"of":10,"metrics":{"mAP":"65.5"},"uses_additional_data":true},{"leaderboard":"/sota/audio-tagging-on-audioset","task":"Audio Tagging","dataset":"AudioSet","model":"DyMN-L (Audio-Only, Single)","rank_in_archive_order":4,"of":11,"metrics":{"mean average precision":"0.490"},"uses_additional_data":true},{"leaderboard":"/sota/instrument-recognition-on-openmic-2018","task":"Instrument Recognition","dataset":"OpenMIC-2018","model":"DyMN-L","rank_in_archive_order":1,"of":5,"metrics":{"mean average precision":"0.855"},"uses_additional_data":true}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"fschmid56/efficientat","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"36d4c9ccc685b4f9d1b2a88aa12dcf426f53b8bcc91a7c76fd4a055b2272671f","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}