{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/method/unigram-segmentation","slug":"unigram-segmentation","name":"Unigram Segmentation","full_name":"Unigram Segmentation","full_name_withheld":false,"description_markdown":"**Unigram Segmentation** is a subword segmentation algorithm based on a unigram language model. It provides multiple segmentations with probabilities. The language model allows for emulating the noise generated during the segmentation of actual data.\r\n\r\nThe unigram language model makes an assumption that each subword occurs independently, and consequently, the probability of a subword sequence $\\mathbf{x} = (x_1,\\ldots,x_M)$ is\r\nformulated as the product of the subword occurrence probabilities\r\n$p(x_i)$:\r\n\r\n$$\r\n  P(\\mathbf{x}) = \\prod_{i=1}^{M} p(x_i), \\\\\\\\\r\n  \\forall i\\,\\, x_i \\in \\mathcal{V},\\,\\,\\,\r\n  \\sum_{x \\in \\mathcal{V}} p(x) = 1, \\nonumber\r\n$$\r\n\r\nwhere $\\mathcal{V}$ is a pre-determined vocabulary.  The most probable\r\nsegmentation $\\mathbf{x}^*$ for the input sentence $X$ is then given by:\r\n\r\n$$\r\n  \\mathbf{x}^{*} = \\text{argmax}_{\\mathbf{x} \\in \\mathcal{S}(X)} P(\\mathbf{x}),\r\n$$\r\n\r\nwhere $\\mathcal{S}(X)$ is a set of segmentation candidates built from\r\nthe input sentence $X$.  $\\mathbf{x}^*$ is obtained with the Viterbi\r\nalgorithm.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates","paper":"/paper/subword-regularization-improving-neural","first_author":"Taku Kudo","n_authors":1,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/subword-regularization-improving-neural"},"source":{"url":"http://arxiv.org/abs/1804.10959v1","title":"Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Subword Segmentation","url":"/methods/category/subword-segmentation","pwc_aliases":[]}],"n_papers_tagged":1,"archive_num_papers":1,"papers_newest_first":[{"paper":null,"slug":"benchmarking-azerbaijani-neural-machine","title":"Benchmarking Azerbaijani Neural Machine Translation","date":"2022-07-29","arxiv_id":"2207.14473","n_code_links":0,"syntology":null}],"papers_shown":1,"papers_order":"date (newest first), then slug","paper_counts":{"archive_papers_tagged":1,"with_a_code_link":0,"where_syntology_ran_a_sample":0,"not_listed_spam_title":0,"listed":1,"listed_where_code_ran":0,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":0,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":0,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers the archive tags; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"papers_where_code_ran":0,"papers_where_code_ran_first_page":null,"papers_where_code_ran_caption":"We ran code from the paper's repository; we did not isolate this method inside it.","pages":[],"tasks":[{"task":"/task/benchmarking","name":"Benchmarking","papers":1},{"task":"/task/domain-generalization","name":"Domain Generalization","papers":1},{"task":"/task/machine-translation","name":"Machine Translation","papers":1},{"task":"/task/nmt","name":"NMT","papers":1},{"task":"/task/segmentation","name":"Segmentation","papers":1},{"task":"/task/translation","name":"Translation","papers":1}],"tasks_shown":6,"n_tasks":6,"usage_by_year":[{"year":"2022","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/unigram-segmentation"},"syntology_read_at":"2026-10-05T00:36:11+00:00","syntology":{"source":"Syntology's have check for this name (the free coverage lookup), read from Syntology's graph","have_url":"https://syntology.ai/api/graph/have?x=Unigram%20Segmentation","read_at":"2026-10-05T01:03:26+00:00","query":"Unigram Segmentation","kind":"method","have":"unknown_name","tier":null,"verified_implementation":null,"level":null,"level_meaning":"the served verification level, per answer; never summed across methods","resolved_to":null,"resolution":null,"resolved_name_differs":false,"absence_asserted":false,"absence_unchecked":null,"no_code_is_about_this_method":false,"subject":null,"archive_paper_arxiv_id":"1804.10959","repositories":null,"retrieve_with":null,"name_competition":{"verdict":"none","may_be_wrong_sense":false,"bearers_found":0,"served":null,"leading_other_bearer":null,"n_other_bearers_listed":0,"note":"Nothing in this corpus is named 'Unigram Segmentation' -- no method node, no paper announcing it in its title, no expanded form declaring it as an acronym."},"name_competition_text":null,"near_matches":["Model Segmentation","Play Segmentation (PS)","Latent Noise Segmentation","zero-guidance segmentation","simple segmentation method","Quantum Motion Segmentation","Amodal Panoptic Segmentation","real-time segmentation model"],"code_nearby":[],"code_nearby_caption":null,"origin_paper":null,"line":"name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.","level_legend":null},"syntology_have_state":"loaded","record_sha256":"87348b92f7af808e37f3daf1bbb6fde816bd5b7f944468b64d4a071bba7e15da","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}