{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/synthesize-step-by-step-tools-templates-and-1","title":"Synthesize Step-by-Step: Tools Templates and LLMs as Data Generators for Reasoning-Based Chart VQA","arxiv_id":null,"date":"2024-01-01","proceeding":"CVPR 2024 1","authors":["Zhuowan Li","Bhavan Jasani","Peng Tang","Shabnam Ghadar"],"abstract":"    Understanding data visualizations like charts and plots requires reasoning about both visual elements and numerics. Although strong in extractive questions current chart visual question answering (chart VQA) models suffer on complex reasoning questions. In this work we address the lack of reasoning ability by data augmentation. We leverage Large Language Models (LLMs) which have shown to have strong reasoning ability as an automatic data annotator that generates question-answer annotations for chart images. The key innovation in our method lies in the Synthesize Step-by-Step strategy: our LLM-based data generator learns to decompose the complex question into step-by-step sub-questions (rationales) which are then used to derive the final answer using external tools i.e. Python. This step-wise generation procedure is trained on synthetic data generated using a template-based QA generation pipeline. Experimental results highlight the significance of the proposed step-by-step generation. By training with the LLM-augmented data (LAMENDA) we significantly enhance the chart VQA models achieving the state-of-the-art accuracy on the ChartQA and PlotQA datasets. In particular our approach improves the accuracy of the previous state-of-the-art approach from 38% to 54% on the human-written questions in the ChartQA dataset which needs strong reasoning. We hope our work underscores the potential of synthetic data and encourages further exploration of data augmentation using LLMs for reasoning-heavy tasks.    ","url_abs":"http://openaccess.thecvf.com//content/CVPR2024/html/Li_Synthesize_Step-by-Step_Tools_Templates_and_LLMs_as_Data_Generators_for_CVPR_2024_paper.html","url_pdf":"http://openaccess.thecvf.com//content/CVPR2024/papers/Li_Synthesize_Step-by-Step_Tools_Templates_and_LLMs_as_Data_Generators_for_CVPR_2024_paper.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"chart-question-answering","task_name":"Chart Question Answering"},{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/chart-question-answering-on-chartqa","task":"Chart Question Answering","dataset":"ChartQA","model":"MatCha4096 + LaMenDa","rank_in_archive_order":9,"of":27,"metrics":{"1:1 Accuracy":"72.64"},"uses_additional_data":true},{"leaderboard":"/sota/chart-question-answering-on-plotqa","task":"Chart Question Answering","dataset":"PlotQA","model":"MatCha4096 + LaMenDa","rank_in_archive_order":1,"of":6,"metrics":{"1:1 Accuracy":"92.89"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-plotqa-d1-1","task":"Visual Question Answering","dataset":"PlotQA-D1","model":"MatCha4096 + LaMenDa","rank_in_archive_order":1,"of":2,"metrics":{"1:1 Accuracy":"93.94"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-plotqa-d2-1","task":"Visual Question Answering","dataset":"PlotQA-D2","model":"MatCha4096 + LaMenDa","rank_in_archive_order":1,"of":2,"metrics":{"1:1 Accuracy":"91.84"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-plotqa-d1","task":"Visual Question Answering (VQA)","dataset":"PlotQA-D1","model":"MatCha4096 + LaMenDa","rank_in_archive_order":1,"of":4,"metrics":{"1:1 Accuracy":"93.94"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-plotqa-d2","task":"Visual Question Answering (VQA)","dataset":"PlotQA-D2","model":"MatCha4096 + LaMenDa","rank_in_archive_order":1,"of":4,"metrics":{"1:1 Accuracy":"91.84"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"5bbf137cdbc7d64a8a92e973370f55e5f02e577c158616b59cdbc30f02151811","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}