{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/text-and-code-embeddings-by-contrastive-pre","title":"Text and Code Embeddings by Contrastive Pre-Training","arxiv_id":"2201.10005","date":"2022-01-24","proceeding":null,"authors":["Arvind Neelakantan","Tao Xu","Raul Puri","Alec Radford","Jesse Michael Han","Jerry Tworek","Qiming Yuan","Nikolas Tezak","Jong Wook Kim","Chris Hallacy","Johannes Heidecke","Pranav Shyam","Boris Power","Tyna Eloundou Nekoul","Girish Sastry","Gretchen Krueger","David Schnurr","Felipe Petroski Such","Kenny Hsu","Madeleine Thompson","Tabarak Khan","Toki Sherbakov","Joanne Jang","Peter Welinder","Lilian Weng"],"abstract":"Text embeddings are useful features in many applications such as semantic search and computing text similarity. Previous work typically trains models customized for different use cases, varying in dataset choice, training objective and model architecture. In this work, we show that contrastive pre-training on unsupervised data at scale leads to high quality vector representations of text and code. The same unsupervised text embeddings that achieve new state-of-the-art results in linear-probe classification also display impressive semantic search capabilities and sometimes even perform competitively with fine-tuned models. On linear-probe classification accuracy averaging over 7 tasks, our best unsupervised model achieves a relative improvement of 4% and 1.8% over previous best unsupervised and supervised text embedding models respectively. The same text embeddings when evaluated on large-scale semantic search attains a relative improvement of 23.4%, 14.7%, and 10.6% over previous best unsupervised methods on MSMARCO, Natural Questions and TriviaQA benchmarks, respectively. Similarly to text embeddings, we train code embedding models on (text, code) pairs, obtaining a 20.8% relative improvement over prior best work on code search.","url_abs":"https://arxiv.org/abs/2201.10005v1","url_pdf":"https://arxiv.org/pdf/2201.10005v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"text-and-code-embeddings-by-contrastive-pre","repo_url":"https://github.com/openmatch/coco-dr","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"code-search","task_name":"Code Search"},{"task_slug":"linear-probe-classification","task_name":"Linear-Probe Classification"},{"task_slug":"natural-questions","task_name":"Natural Questions"},{"task_slug":"passage-ranking","task_name":"Passage Ranking"},{"task_slug":"triviaqa","task_name":"TriviaQA"},{"task_slug":"zero-shot-text-search","task_name":"Zero-shot Text Search"},{"task_slug":"text-similarity","task_name":"text similarity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/code-search-on-codesearchnet","task":"Code Search","dataset":"CodeSearchNet","model":"cpt-code M","rank_in_archive_order":1,"of":6,"metrics":{"Go":"97.5","JS":"86.5","Java":"94.4","Overall":"93.5","PHP":"97.2","Python":"99.9","Ruby":"85.5"},"uses_additional_data":false},{"leaderboard":"/sota/code-search-on-codesearchnet","task":"Code Search","dataset":"CodeSearchNet","model":"cpt-code S","rank_in_archive_order":2,"of":6,"metrics":{"Go":"97.7","JS":"86.0","Java":"94.0","Overall":"93.4","PHP":"96.7","Python":"99.8","Ruby":"86.3"},"uses_additional_data":false},{"leaderboard":"/sota/passage-ranking-on-ms-marco","task":"Passage Ranking","dataset":"MS MARCO","model":"Fine-tuned SOTA","rank_in_archive_order":1,"of":4,"metrics":{"MRR@10":"44.3"},"uses_additional_data":false},{"leaderboard":"/sota/passage-ranking-on-ms-marco","task":"Passage Ranking","dataset":"MS MARCO","model":"cpt-text XL","rank_in_archive_order":2,"of":4,"metrics":{"MRR@10":"22.7"},"uses_additional_data":false},{"leaderboard":"/sota/passage-ranking-on-ms-marco","task":"Passage Ranking","dataset":"MS MARCO","model":"cpt-text L","rank_in_archive_order":3,"of":4,"metrics":{"MRR@10":"21.5"},"uses_additional_data":false},{"leaderboard":"/sota/passage-ranking-on-ms-marco","task":"Passage Ranking","dataset":"MS MARCO","model":"BM25","rank_in_archive_order":4,"of":4,"metrics":{"MRR@10":"18.4"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2201.10005","atlas_url":"https://app.syntology.ai/?focus=2201.10005","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2201.10005"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2201.10005\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2201.10005"},"call":"get_concepts_for_paper(arxiv_id=\"2201.10005\")"}],"by_repo":[{"repo":"openmatch/coco-dr","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2210.15212"],"harvested_for_calls":[{"arxiv_id":"2210.15212","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2210.15212"},"call":"get_harvested_code_for_paper(arxiv_id=\"2210.15212\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"2494cc3aef21321820f24e055510c575db6c5010b3ac03601254a65ed87cb507","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}