{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/bringing-replication-and-reproduction","title":"Bringing replication and reproduction together with generalisability in NLP: Three reproduction studies for Target Dependent Sentiment Analysis","arxiv_id":"1806.05219","date":"2018-06-13","proceeding":"COLING 2018 8","authors":["Andrew Moore","Paul Rayson"],"abstract":"Lack of repeatability and generalisability are two significant threats to\ncontinuing scientific development in Natural Language Processing. Language\nmodels and learning methods are so complex that scientific conference papers no\nlonger contain enough space for the technical depth required for replication or\nreproduction. Taking Target Dependent Sentiment Analysis as a case study, we\nshow how recent work in the field has not consistently released code, or\ndescribed settings for learning methods in enough detail, and lacks\ncomparability and generalisability in train, test or validation data. To\ninvestigate generalisability and to enable state of the art comparative\nevaluations, we carry out the first reproduction studies of three groups of\ncomplementary methods and perform the first large-scale mass evaluation on six\ndifferent English datasets. Reflecting on our experiences, we recommend that\nfuture replication or reproduction experiments should always consider a variety\nof datasets alongside documenting and releasing their methods and published\ncode in order to minimise the barriers to both repeatability and\ngeneralisability. We have released our code with a model zoo on GitHub with\nJupyter Notebooks to aid understanding and full documentation, and we recommend\nthat others do the same with their papers at submission time through an\nanonymised GitHub account.","url_abs":"http://arxiv.org/abs/1806.05219v2","url_pdf":"http://arxiv.org/pdf/1806.05219v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bringing-replication-and-reproduction","repo_url":"https://github.com/apmoore1/Bella","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1806.05219","atlas_url":"https://app.syntology.ai/?focus=1806.05219","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1806.05219"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1806.05219\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1806.05219"},"call":"get_concepts_for_paper(arxiv_id=\"1806.05219\")"}],"by_repo":[{"repo":"apmoore1/Bella","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"4790b3bae9c6016c31daaeeceb10f9a22a84b8eec6d05692feb54869af2635ee","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}