{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/audiobox-unified-audio-generation-with","title":"Audiobox: Unified Audio Generation with Natural Language Prompts","arxiv_id":"2312.15821","date":"2023-12-25","proceeding":null,"authors":["Apoorv Vyas","Bowen Shi","Matthew Le","Andros Tjandra","Yi-Chiao Wu","Baishan Guo","Jiemin Zhang","Xinyue Zhang","Robert Adkins","William Ngan","Jeff Wang","Ivan Cruz","Bapi Akula","Akinniyi Akinyemi","Brian Ellis","Rashel Moritz","Yael Yungster","Alice Rakotoarison","Liang Tan","Chris Summers","Carleigh Wood","Joshua Lane","Mary Williamson","Wei-Ning Hsu"],"abstract":"Audio is an essential part of our life, but creating it often requires expertise and is time-consuming. Research communities have made great progress over the past year advancing the performance of large scale audio generative models for a single modality (speech, sound, or music) through adopting more powerful generative models and scaling data. However, these models lack controllability in several aspects: speech generation models cannot synthesize novel styles based on text description and are limited on domain coverage such as outdoor environments; sound generation models only provide coarse-grained control based on descriptions like \"a person speaking\" and would only generate mumbling human voices. This paper presents Audiobox, a unified model based on flow-matching that is capable of generating various audio modalities. We design description-based and example-based prompting to enhance controllability and unify speech and sound generation paradigms. We allow transcript, vocal, and other audio styles to be controlled independently when generating speech. To improve model generalization with limited labels, we adapt a self-supervised infilling objective to pre-train on large quantities of unlabeled audio. Audiobox sets new benchmarks on speech and sound generation (0.745 similarity on Librispeech for zero-shot TTS; 0.77 FAD on AudioCaps for text-to-sound) and unlocks new methods for generating audio with novel vocal and acoustic styles. We further integrate Bespoke Solvers, which speeds up generation by over 25 times compared to the default ODE solver for flow-matching, without loss of performance on several tasks. Our demo is available at https://audiobox.metademolab.com/","url_abs":"https://arxiv.org/abs/2312.15821v1","url_pdf":"https://arxiv.org/pdf/2312.15821v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"audio-generation","task_name":"Audio Generation"},{"task_slug":null,"task_name":"AudioCaps"},{"task_slug":"fad","task_name":"FAD"}],"methods":[{"method_slug":"alibi","method_name":"ALiBi"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/audio-generation-on-audiocaps","task":"Audio Generation","dataset":"AudioCaps","model":"Audiobox Sound","rank_in_archive_order":10,"of":23,"metrics":{"CLAP_LAION":"0.71","FAD":"0.77","FD":"8.30","IS":"12.70"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.15821","atlas_url":"https://app.syntology.ai/?focus=2312.15821","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2312.15821"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2312.15821\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2312.15821"},"call":"get_concepts_for_paper(arxiv_id=\"2312.15821\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"47852b58cf099346a9a435a001d994c696fdd14e3cfd159eb0bc9348b6d6c104","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}