{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/automated-evaluation-of-out-of-context-errors","title":"Automated Evaluation of Out-of-Context Errors","arxiv_id":"1803.08983","date":"2018-03-23","proceeding":"LREC 2018 5","authors":["Patrick Huber","Jan Niehues","Alex Waibel"],"abstract":"We present a new approach to evaluate computational models for the task of\ntext understanding by the means of out-of-context error detection. Through the\nnovel design of our automated modification process, existing large-scale data\nsources can be adopted for a vast number of text understanding tasks. The data\nis thereby altered on a semantic level, allowing models to be tested against a\nchallenging set of modified text passages that require to comprise a broader\nnarrative discourse. Our newly introduced task targets actual real-world\nproblems of transcription and translation systems by inserting authentic\nout-of-context errors. The automated modification process is applied to the\n2016 TEDTalk corpus. Entirely automating the process allows the adoption of\ncomplete datasets at low cost, facilitating supervised learning procedures and\ndeeper networks to be trained and tested. To evaluate the quality of the\nmodification algorithm a language model and a supervised binary classification\nmodel are trained and tested on the altered dataset. A human baseline\nevaluation is examined to compare the results with human performance. The\noutcome of the evaluation task indicates the difficulty to detect semantic\nerrors for machine-learning algorithms and humans, showing that the errors\ncannot be identified when limited to a single sentence.","url_abs":"http://arxiv.org/abs/1803.08983v1","url_pdf":"http://arxiv.org/pdf/1803.08983v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"automated-evaluation-of-out-of-context-errors","repo_url":"https://github.com/isl-mt/SemanticWordReplacement-LREC2018","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"binary-classification","task_name":"Binary Classification"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"arxiv_id_not_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"by_repo":[{"repo":"isl-mt/SemanticWordReplacement-LREC2018","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"b6723b776deb5e2d2df06567c1b141581742e9f9c636e21be3e90b473082a967","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}