{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/overview-of-the-2022-validity-and-novelty","title":"Overview of the 2022 Validity and Novelty Prediction Shared Task","arxiv_id":null,"date":"2022-10-01","proceeding":"ArgMining (ACL) 2022 10","authors":["Philipp Heinisch","Anette Frank","Juri Opitz","Moritz Plenz","Philipp Cimiano"],"abstract":"This paper provides an overview of the Argument Validity and Novelty Prediction Shared Task that was organized as part of the 9th Workshop on Argument Mining (ArgMining 2022). The task focused on the prediction of the validity and novelty of a conclusion given a textual premise. Validity is defined as the degree to which the conclusion is justified with respect to the given premise. Novelty defines the degree to which the conclusion contains content that is new in relation to the premise. Six groups participated in the task, submitting overall 13 system runs for the subtask of binary classification and 2 system runs for the subtask of relative classification. The results reveal that the task is challenging, with best results obtained for Validity prediction in the range of 75% F1 score, for Novelty prediction of 70% F1 score and for correctly predicting both Validity and Novelty of 45% F1 score. In this paper we summarize the task definition and dataset. We give an overview of the results obtained by the participating systems, as well as insights to be gained from the diverse contributions.","url_abs":"https://aclanthology.org/2022.argmining-1.7","url_pdf":"https://aclanthology.org/2022.argmining-1.7.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"overview-of-the-2022-validity-and-novelty","repo_url":"https://github.com/phhei/argsvalidnovel","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"argument-mining","task_name":"Argument Mining"},{"task_slug":"binary-classification","task_name":"Binary Classification"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"valnov","task_name":"ValNov"}],"methods":[],"datasets_introduced":[{"slug":"valnov-subtask-a","name":"ValNov Subtask A","full_name":""},{"slug":"valnov-subtask-b","name":"ValNov Subtask B","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/valnov-on-valnov-subtask-a","task":"ValNov","dataset":"ValNov Subtask A","model":"CLTeamL-3","rank_in_archive_order":1,"of":8,"metrics":{"JOINT-F1":"45.16","NOV-F1":"61.75","VAL-F1":"74.64"},"uses_additional_data":false},{"leaderboard":"/sota/valnov-on-valnov-subtask-a","task":"ValNov","dataset":"ValNov Subtask A","model":"ACCEPT-1","rank_in_archive_order":3,"of":8,"metrics":{"JOINT-F1":"43.13","NOV-F1":"70.00","VAL-F1":"59.20"},"uses_additional_data":false},{"leaderboard":"/sota/valnov-on-valnov-subtask-a","task":"ValNov","dataset":"ValNov Subtask A","model":"CSS","rank_in_archive_order":4,"of":8,"metrics":{"JOINT-F1":"42.40","NOV-F1":"59.86","VAL-F1":"70.76"},"uses_additional_data":false},{"leaderboard":"/sota/valnov-on-valnov-subtask-a","task":"ValNov","dataset":"ValNov Subtask A","model":"System Average","rank_in_archive_order":5,"of":8,"metrics":{"JOINT-F1":"35.94","NOV-F1":"52.97","VAL-F1":"62.74"},"uses_additional_data":false},{"leaderboard":"/sota/valnov-on-valnov-subtask-a","task":"ValNov","dataset":"ValNov Subtask A","model":"NLP@UIT","rank_in_archive_order":6,"of":8,"metrics":{"JOINT-F1":"25.89","NOV-F1":"43.36","VAL-F1":"61.72"},"uses_additional_data":false},{"leaderboard":"/sota/valnov-on-valnov-subtask-a","task":"ValNov","dataset":"ValNov Subtask A","model":"Baseline","rank_in_archive_order":7,"of":8,"metrics":{"JOINT-F1":"23.90","NOV-F1":"36.12","VAL-F1":"59.96"},"uses_additional_data":false},{"leaderboard":"/sota/valnov-on-valnov-subtask-a","task":"ValNov","dataset":"ValNov Subtask A","model":"Harshad","rank_in_archive_order":8,"of":8,"metrics":{"JOINT-F1":"17.35","NOV-F1":"39.00","VAL-F1":"56.31"},"uses_additional_data":false},{"leaderboard":"/sota/valnov-on-valnov-subtask-b","task":"ValNov","dataset":"ValNov Subtask B","model":"NLP@UIT","rank_in_archive_order":1,"of":3,"metrics":{"JOINT-F1":"41.50","NOV-F1":"38.39","VAL-F1":"44.60"},"uses_additional_data":false},{"leaderboard":"/sota/valnov-on-valnov-subtask-b","task":"ValNov","dataset":"ValNov Subtask B","model":"AXiS@EdUni","rank_in_archive_order":2,"of":3,"metrics":{"JOINT-F1":"29.16","NOV-F1":"25.86","VAL-F1":"32.47"},"uses_additional_data":false},{"leaderboard":"/sota/valnov-on-valnov-subtask-b","task":"ValNov","dataset":"ValNov Subtask B","model":"Baseline","rank_in_archive_order":3,"of":3,"metrics":{"JOINT-F1":"21.46","NOV-F1":"23.09","VAL-F1":"19.82"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[{"repo":"phhei/argsvalidnovel","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"501658cd302c51db2e69e5b3f2c73de595fb0956c2a9750f4b49f8ea738fb1ea","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}