{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/cynical-selection-of-language-model-training","title":"Cynical Selection of Language Model Training Data","arxiv_id":"1709.02279","date":"2017-09-07","proceeding":null,"authors":["Amittai Axelrod"],"abstract":"The Moore-Lewis method of \"intelligent selection of language model training\ndata\" is very effective, cheap, efficient... and also has structural problems.\n(1) The method defines relevance by playing language models trained on the\nin-domain and the out-of-domain (or data pool) corpora against each other. This\npowerful idea-- which we set out to preserve-- treats the two corpora as the\nopposing ends of a single spectrum. This lack of nuance does not allow for the\ntwo corpora to be very similar. In the extreme case where the come from the\nsame distribution, all of the sentences have a Moore-Lewis score of zero, so\nthere is no resulting ranking. (2) The selected sentences are not guaranteed to\nbe able to model the in-domain data, nor to even cover the in-domain data. They\nare simply well-liked by the in-domain model; this is necessary, but not\nsufficient. (3) There is no way to tell what is the optimal number of sentences\nto select, short of picking various thresholds and building the systems.\n  We present a greedy, lazy, approximate, and generally efficient\ninformation-theoretic method of accomplishing the same goal using only\nvocabulary counts. The method has the following properties: (1) Is responsive\nto the extent to which two corpora differ. (2) Quickly reaches near-optimal\nvocabulary coverage. (3) Takes into account what has already been selected. (4)\nDoes not involve defining any kind of domain, nor any kind of classifier. (6)\nKnows approximately when to stop. This method can be used as an\ninherently-meaningful measure of similarity, as it measures the bits of\ninformation to be gained by adding one text to another.","url_abs":"http://arxiv.org/abs/1709.02279v1","url_pdf":"http://arxiv.org/pdf/1709.02279v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cynical-selection-of-language-model-training","repo_url":"https://github.com/amittai/cynical","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1709.02279","atlas_url":"https://app.syntology.ai/?focus=1709.02279","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1709.02279"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1709.02279\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1709.02279"},"call":"get_concepts_for_paper(arxiv_id=\"1709.02279\")"}],"by_repo":[{"repo":"amittai/cynical","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"945938374abf93bbc6584371222901bd8cb5f8d2665df598b9a0c4192a6e3588","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}