{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/latte-lattice-attentive-encoding-for","title":"LATTE: Lattice ATTentive Encoding for Character-based Word Segmentation","arxiv_id":null,"date":"2023-06-01","proceeding":"Journal of Natural Language Processing 2023 6","authors":["Thodsaporn Chay-intr","Hidetaka Kamigaito","Kotaro Funakoshi","Manabu Okumura"],"abstract":"A character sequence comprises at least one or more segmentation alternatives. This can be considered segmentation ambiguity and may weaken segmentation performance in word segmentation. Proper handling of such ambiguity lessens ambiguous decisions on word boundaries. Previous works have achieved remarkable segmentation performance and alleviated the ambiguity problem by incorporating the lattice, owing to its ability to capture segmentation alternatives, along with graph-based and pre-trained models. However, multiple granularity information, including character and word, in a lattice that encodes with such models may not be attentively exploited. To strengthen multi-granularity representations in a lattice, we propose the Lattice ATTentive Encoding (LATTE) method for character-based word segmentation. Our model employs the lattice structure to handle segmentation alternatives and utilizes graph neural networks along with an attention mechanism to attentively extract multi-granularity representation from the lattice for complementing character representations. Our experimental results demonstrated improvements in segmentation performance on the BCCWJ, CTB6, and BEST2010 datasets in three languages, particularly Japanese, Chinese, and Thai.","url_abs":"https://www.jstage.jst.go.jp/article/jnlp/30/2/30_456/_article/-char/ja","url_pdf":"https://www.jstage.jst.go.jp/article/jnlp/30/2/30_456/_pdf/-char/ja","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"latte-lattice-attentive-encoding-for","repo_url":"https://github.com/tchayintr/latte-ptm-ws","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"latte-lattice-attentive-encoding-for","repo_url":"https://github.com/tchayintr/latte-ws","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"chinese-word-segmentation","task_name":"Chinese Word Segmentation"},{"task_slug":"japanese-word-segmentation","task_name":"Japanese Word Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"thai-word-segmentation","task_name":"Thai Word Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/chinese-word-segmentation-on-ctb6","task":"Chinese Word Segmentation","dataset":"CTB6","model":"LATTE (Linguistic units, lattices, PTMs, GNNs)","rank_in_archive_order":1,"of":4,"metrics":{"F1":"98.07"},"uses_additional_data":true},{"leaderboard":"/sota/japanese-word-segmentation-on-bccwj","task":"Japanese Word Segmentation","dataset":"BCCWJ","model":"LATTE (Linguistic units, lattices, PTMs, GNNs)","rank_in_archive_order":1,"of":3,"metrics":{"F1-score (Word)":"0.9936"},"uses_additional_data":true},{"leaderboard":"/sota/thai-word-tokenization-on-best-2010","task":"Thai Word Segmentation","dataset":"BEST-2010","model":"LATTE (Linguistic units, lattices, PTMs, GNNs)","rank_in_archive_order":1,"of":5,"metrics":{"F1-Score":"0.9907"},"uses_additional_data":true}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[{"repo":"tchayintr/latte-ptm-ws","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"tchayintr/latte-ws","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"5968523eec780ea043ee1fb23bb143dbaf48a7f27eef587e7c2c71f337d42032","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}