{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/billion-scale-pretraining-with-vision","title":"Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual Representations","arxiv_id":"2108.05887","date":"2021-08-12","proceeding":null,"authors":["Josh Beal","Hao-Yu Wu","Dong Huk Park","Andrew Zhai","Dmitry Kislyuk"],"abstract":"Large-scale pretraining of visual representations has led to state-of-the-art performance on a range of benchmark computer vision tasks, yet the benefits of these techniques at extreme scale in complex production systems has been relatively unexplored. We consider the case of a popular visual discovery product, where these representations are trained with multi-task learning, from use-case specific visual understanding (e.g. skin tone classification) to general representation learning for all visual content (e.g. embeddings for retrieval). In this work, we describe how we (1) generate a dataset with over a billion images via large weakly-supervised pretraining to improve the performance of these visual representations, and (2) leverage Transformers to replace the traditional convolutional backbone, with insights into both system and performance improvements, especially at 1B+ image scale. To support this backbone model, we detail a systematic approach to deriving weakly-supervised image annotations from heterogenous text signals, demonstrating the benefits of clustering techniques to handle the long-tail distribution of image labels. Through a comprehensive study of offline and online evaluation, we show that large-scale Transformer-based pretraining provides significant benefits to industry computer vision applications. The model is deployed in a production visual shopping system, with 36% improvement in top-1 relevance and 23% improvement in click-through volume. We conduct extensive experiments to better understand the empirical relationships between Transformer-based architectures, dataset scale, and the performance of production vision systems.","url_abs":"https://arxiv.org/abs/2108.05887v1","url_pdf":"https://arxiv.org/pdf/2108.05887v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-objectnet","task":"Image Classification","dataset":"ObjectNet","model":"ViT-B/16 (ANN-1.3B)","rank_in_archive_order":26,"of":106,"metrics":{"Top-1 Accuracy":"50.7"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-objectnet","task":"Image Classification","dataset":"ObjectNet","model":"ResNet-101 (JFT-300M)","rank_in_archive_order":28,"of":106,"metrics":{"Top-1 Accuracy":"49.1"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-objectnet","task":"Image Classification","dataset":"ObjectNet","model":"ViT-B/32","rank_in_archive_order":30,"of":106,"metrics":{"Top-1 Accuracy":"48.4"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-objectnet","task":"Image Classification","dataset":"ObjectNet","model":"ResNet-50 (JFT-300M)","rank_in_archive_order":39,"of":106,"metrics":{"Top-1 Accuracy":"42.5"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2108.05887","atlas_url":"https://app.syntology.ai/?focus=2108.05887","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2108.05887"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2108.05887\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2108.05887"},"call":"get_concepts_for_paper(arxiv_id=\"2108.05887\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"0522978f5ae4b8e31ae7433c53fd5256b6ede24c1f037068da892ba9b393e7c0","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}