{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/egraffbench-evaluation-of-equivariant-graph","title":"EGraFFBench: Evaluation of Equivariant Graph Neural Network Force Fields for Atomistic Simulations","arxiv_id":"2310.02428","date":"2023-10-03","proceeding":null,"authors":["Vaibhav Bihani","Utkarsh Pratiush","Sajid Mannan","Tao Du","Zhimin Chen","Santiago Miret","Matthieu Micoulaut","Morten M Smedskjaer","Sayan Ranu","N M Anoop Krishnan"],"abstract":"Equivariant graph neural networks force fields (EGraFFs) have shown great promise in modelling complex interactions in atomic systems by exploiting the graphs' inherent symmetries. Recent works have led to a surge in the development of novel architectures that incorporate equivariance-based inductive biases alongside architectural innovations like graph transformers and message passing to model atomic interactions. However, thorough evaluations of these deploying EGraFFs for the downstream task of real-world atomistic simulations, is lacking. To this end, here we perform a systematic benchmarking of 6 EGraFF algorithms (NequIP, Allegro, BOTNet, MACE, Equiformer, TorchMDNet), with the aim of understanding their capabilities and limitations for realistic atomistic simulations. In addition to our thorough evaluation and analysis on eight existing datasets based on the benchmarking literature, we release two new benchmark datasets, propose four new metrics, and three challenging tasks. The new datasets and tasks evaluate the performance of EGraFF to out-of-distribution data, in terms of different crystal structures, temperatures, and new molecules. Interestingly, evaluation of the EGraFF models based on dynamic simulations reveals that having a lower error on energy or force does not guarantee stable or reliable simulation or faithful replication of the atomic structures. Moreover, we find that no model clearly outperforms other models on all datasets and tasks. Importantly, we show that the performance of all the models on out-of-distribution datasets is unreliable, pointing to the need for the development of a foundation model for force fields that can be used in real-world simulations. In summary, this work establishes a rigorous framework for evaluating machine learning force fields in the context of atomic simulations and points to open research challenges within this domain.","url_abs":"https://arxiv.org/abs/2310.02428v2","url_pdf":"https://arxiv.org/pdf/2310.02428v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"atomic-forces","task_name":"Atomic Forces"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"formation-energy","task_name":"Formation Energy"},{"task_slug":"graph-neural-network","task_name":"Graph Neural Network"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/formation-energy-on-3bpa","task":"Formation Energy","dataset":"3BPA","model":"BOTNet","rank_in_archive_order":1,"of":4,"metrics":{"MAE":"5"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-3bpa","task":"Formation Energy","dataset":"3BPA","model":"Allegro","rank_in_archive_order":2,"of":4,"metrics":{"MAE":"4.13"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-3bpa","task":"Formation Energy","dataset":"3BPA","model":"MACE","rank_in_archive_order":3,"of":4,"metrics":{"MAE":"4"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-3bpa","task":"Formation Energy","dataset":"3BPA","model":"NequIP","rank_in_archive_order":4,"of":4,"metrics":{"MAE":"3.15"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-acetylacetone","task":"Formation Energy","dataset":"Acetylacetone","model":"BOTNet","rank_in_archive_order":1,"of":4,"metrics":{"MAE":"2"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-acetylacetone","task":"Formation Energy","dataset":"Acetylacetone","model":"MACE","rank_in_archive_order":2,"of":4,"metrics":{"MAE":"2"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-acetylacetone","task":"Formation Energy","dataset":"Acetylacetone","model":"NequIP","rank_in_archive_order":3,"of":4,"metrics":{"MAE":"1.38"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-acetylacetone","task":"Formation Energy","dataset":"Acetylacetone","model":"Allegro","rank_in_archive_order":4,"of":4,"metrics":{"MAE":"0.92"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-aspirin","task":"Formation Energy","dataset":"Aspirin","model":"Allegro","rank_in_archive_order":1,"of":4,"metrics":{"MAE":"14.36"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-aspirin","task":"Formation Energy","dataset":"Aspirin","model":"MACE","rank_in_archive_order":2,"of":4,"metrics":{"MAE":"13.79"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-aspirin","task":"Formation Energy","dataset":"Aspirin","model":"BOTNet","rank_in_archive_order":3,"of":4,"metrics":{"MAE":"12.63"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-aspirin","task":"Formation Energy","dataset":"Aspirin","model":"NequIP","rank_in_archive_order":4,"of":4,"metrics":{"MAE":"9.27"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-ethanol","task":"Formation Energy","dataset":"Ethanol","model":"MACE","rank_in_archive_order":1,"of":4,"metrics":{"MAE":"209.96"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-ethanol","task":"Formation Energy","dataset":"Ethanol","model":"BOTNet","rank_in_archive_order":2,"of":4,"metrics":{"MAE":"203.83"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-ethanol","task":"Formation Energy","dataset":"Ethanol","model":"Allegro","rank_in_archive_order":3,"of":4,"metrics":{"MAE":"6.94"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-ethanol","task":"Formation Energy","dataset":"Ethanol","model":"NequIP","rank_in_archive_order":4,"of":4,"metrics":{"MAE":"4.99"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-gete","task":"Formation Energy","dataset":"GeTe","model":"BOTNet","rank_in_archive_order":1,"of":4,"metrics":{"MAE":"3034"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-gete","task":"Formation Energy","dataset":"GeTe","model":"MACE","rank_in_archive_order":2,"of":4,"metrics":{"MAE":"2670"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-gete","task":"Formation Energy","dataset":"GeTe","model":"NequIP","rank_in_archive_order":3,"of":4,"metrics":{"MAE":"1780.951"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-gete","task":"Formation Energy","dataset":"GeTe","model":"Allegro","rank_in_archive_order":4,"of":4,"metrics":{"MAE":"1009.4"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-lips","task":"Formation Energy","dataset":"LiPS","model":"NequIP","rank_in_archive_order":1,"of":4,"metrics":{"MAE":"165.43"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-lips","task":"Formation Energy","dataset":"LiPS","model":"Allegro","rank_in_archive_order":2,"of":4,"metrics":{"MAE":"31.75"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-lips","task":"Formation Energy","dataset":"LiPS","model":"MACE","rank_in_archive_order":3,"of":4,"metrics":{"MAE":"30"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-lips","task":"Formation Energy","dataset":"LiPS","model":"BOTNet","rank_in_archive_order":4,"of":4,"metrics":{"MAE":"28"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-lips20","task":"Formation Energy","dataset":"LiPS20","model":"Allegro","rank_in_archive_order":1,"of":4,"metrics":{"MAE":"33.17"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-lips20","task":"Formation Energy","dataset":"LiPS20","model":"NequIP","rank_in_archive_order":2,"of":4,"metrics":{"MAE":"26.8"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-lips20","task":"Formation Energy","dataset":"LiPS20","model":"BOTNet","rank_in_archive_order":3,"of":4,"metrics":{"MAE":"24.59"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-lips20","task":"Formation Energy","dataset":"LiPS20","model":"MACE","rank_in_archive_order":4,"of":4,"metrics":{"MAE":"14.05"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-naphthalene","task":"Formation Energy","dataset":"Naphthalene","model":"BOTNet","rank_in_archive_order":1,"of":4,"metrics":{"MAE":"182.55"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-naphthalene","task":"Formation Energy","dataset":"Naphthalene","model":"MACE","rank_in_archive_order":2,"of":4,"metrics":{"MAE":"161.74"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-naphthalene","task":"Formation Energy","dataset":"Naphthalene","model":"Allegro","rank_in_archive_order":3,"of":4,"metrics":{"MAE":"5.82"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-naphthalene","task":"Formation Energy","dataset":"Naphthalene","model":"NequIP","rank_in_archive_order":4,"of":4,"metrics":{"MAE":"2.66"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-salicylic-acid","task":"Formation Energy","dataset":"Salicylic Acid","model":"MACE","rank_in_archive_order":1,"of":4,"metrics":{"MAE":"165.29"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-salicylic-acid","task":"Formation Energy","dataset":"Salicylic Acid","model":"BOTNet","rank_in_archive_order":2,"of":4,"metrics":{"MAE":"153.06"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-salicylic-acid","task":"Formation Energy","dataset":"Salicylic Acid","model":"Allegro","rank_in_archive_order":3,"of":4,"metrics":{"MAE":"8.59"},"uses_additional_data":false},{"leaderboard":"/sota/formation-energy-on-salicylic-acid","task":"Formation Energy","dataset":"Salicylic Acid","model":"NequIP","rank_in_archive_order":4,"of":4,"metrics":{"MAE":"6.29"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.02428","atlas_url":"https://app.syntology.ai/?focus=2310.02428","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2310.02428"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2310.02428\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2310.02428"},"call":"get_concepts_for_paper(arxiv_id=\"2310.02428\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"b539fb05dbabc4a212460ad2be3a08d7d80b9405af141bebc7e007beef365e9f","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}