{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/dataset/vnbench","name":"VNBench","full_name":null,"description_markdown":"VNBench is a comprehensive benchmark suite for video generative models, which evaluates video generation quality across specific, hierarchical, and disentangled dimensions, each with tailored prompts and evaluation methods. The suite includes 16 dimensions for evaluating Text-to-Video (T2V) models, such as subject consistency, motion smoothness, and overall consistency. VNBench also supports evaluating Image-to-Video (I2V) models and has recently introduced VBench-Long for evaluating longer videos¹. It's designed to align with human perceptions and provide valuable insights for future developments in video generation.\r\n\r\n# Diversity of “Needle” Types\r\nEdit: Using artificially added subtitles as the \"needle\". These subtitles are embedded in video frames to simulate the scenario of finding specific textual information in a video.\r\n\r\nInsert: Using images as the \"needle\". These images are inserted as static segments between video frames to assess the model's ability to recognize and remember static images in a video.\r\n\r\nLevel Classification: Classified into two levels based on image recognizability. The first level uses common objects (e.g., fruit images), while the second level uses more challenging landmark or object images, increasing the task's difficulty.\r\n\r\n# Diversity of Video “Haystack”\r\nTemporal Distribution: The video \"haystack\" used by VNBench comes from different data sources, with video durations ranging from 10 seconds to 180 seconds. This covers short, medium, and long video lengths to evaluate the model's adaptability to different video lengths.\r\n\r\nContent Coverage: The video content includes various scenes, ensuring the evaluation's broadness and the diversity of video sources.\r\n\r\n# Diversity of Queries\r\nRetrieval Task: Requires the model to retrieve specific \"needles\" from videos, assessing the model's fine-grained understanding and information extraction ability.\r\n\r\nOrdering Task: Requires the model to identify and order the timestamps of all inserted \"needles\" in the video, assessing the model's understanding of video temporal dynamics and event sequences.\r\n\r\nCounting Task: Requires the model to count the occurrences of specific objects in the video, including recognizing and tracking repetitive patterns within and across frames. This assesses the model's understanding of spatial and temporal dimensions.\r\n\r\n(1) Vchitect/VBench: [CVPR2024 Highlight] VBench - GitHub. https://github.com/Vchitect/VBench.\r\n(2) VBench: Comprehensive Benchmark Suite for Video Generative Models. https://arxiv.org/abs/2311.17982.\r\n(3) Linux中vdbench的安装与使用_vdbench参数详解-CSDN博客. https://blog.csdn.net/SweeNeil/article/details/95338293.\r\n(4) Cinebench 2024 Downloads - Maxon. https://www.maxon.net/zh/downloads/cinebench-2024-downloads.\r\n(5) undefined. https://doi.org/10.48550/arXiv.2311.17982.","description_withheld":null,"homepage":"https://github.com/joez17/VideoNIAH","introduced_date":"2024-06-13","introduced_date_note":null,"introduced_by":{"paper":"/paper/needle-in-a-video-haystack-a-scalable","title":"Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs","first_author":"Zijia Zhao","url":null},"license":null,"modalities":[],"tasks":[{"name":"Zero-Shot Video Question Answer","url":"/task/zeroshot-video-question-answer","datasets_with_task":"/datasets/task/zeroshot-video-question-answer"}],"languages":[],"variants":["VNBench"],"data_loaders":[],"num_papers_in_archive":11,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/zero-shot-video-question-answer-on-vnbench","task":"Zero-Shot Video Question Answer","dataset_variant":"VNBench","rows":9,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"BIMBA-LLaVA-Qwen2-7B","paper":"/paper/bimba-selective-scan-compression-for-long","metrics":{"Accuracy":"77.88"},"code_links":[{"title":"md-mohaiminul/BIMBA","url":"https://github.com/md-mohaiminul/BIMBA"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers":{"order":"date (newest first), then slug","population":"every paper with a leaderboard row on this dataset's benchmarks (the benchmark-backed subset): the archive's own papers-using-this-dataset list was never published, so this is not that list; num_papers_in_archive is the archive's own count","page":1,"pages_in_order":1,"rows_per_page":100,"shown":8,"of":8,"counts":{"papers_with_a_benchmark_row":8,"with_a_code_link":8,"where_syntology_ran_a_sample":3,"not_listed_spam_title":0,"listed":8,"listed_where_code_ran":3,"where_syntology_ran_a_sample_split":{"with_a_run_with_no_instrument_failure":3,"every_run_a_failure_of_syntologys_instrument":0,"listed_with_a_run_with_no_instrument_failure":3,"listed_every_run_a_failure_of_syntologys_instrument":0,"filter":{"states":["a run with no instrument failure","any run, instrument failures included"],"default":"a run with no instrument failure","note":"on the 'only where code ran' pages the default hides, in the browser, the rows where every run was a failure of Syntology's instrument; the second state shows them again. Rows are hidden, never re-ordered; these twins list every row"}},"definition":"distinct papers with at least one leaderboard row on this dataset's benchmarks; 'where Syntology ran a sample' counts papers with at least one harvested sample that ran, which is not a correctness claim"},"items":[{"paper":"/paper/bimba-selective-scan-compression-for-long","slug":"bimba-selective-scan-compression-for-long","title":"BIMBA: Selective-Scan Compression for Long-Range Video Question Answering","date":"2025-03-12","arxiv_id":"2503.09590","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/qwen2-vl-enhancing-vision-language-model-s","slug":"qwen2-vl-enhancing-vision-language-model-s","title":"Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution","date":"2024-09-18","arxiv_id":"2409.12191","rows_on_this_dataset":1,"code_links":8,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":12,"samples_ran":12,"samples_constructed":1,"samples_ran_checked":10,"samples_ran_instrument_failed":2,"samples_unverified":0,"pointer_only_for_licence":1,"official":{"repos":["qwenlm/qwen2-vl"],"state":"official (archive's flag): 4 ran","n_ran":4,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":0,"ran_from_kinds":["listed","official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/qwen2-vl-enhancing-vision-language-model-s#ran","syntology_url":"https://syntology.ai/paper/2409.12191","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.12191"}}}},{"paper":"/paper/llava-onevision-easy-visual-task-transfer","slug":"llava-onevision-easy-visual-task-transfer","title":"LLaVA-OneVision: Easy Visual Task Transfer","date":"2024-08-06","arxiv_id":"2408.03326","rows_on_this_dataset":2,"code_links":2,"syntology":null},{"paper":"/paper/llava-next-interleave-tackling-multi-image","slug":"llava-next-interleave-tackling-multi-image","title":"LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models","date":"2024-07-10","arxiv_id":"2407.07895","rows_on_this_dataset":1,"code_links":3,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":4,"samples_ran":2,"samples_constructed":0,"samples_ran_checked":1,"samples_ran_instrument_failed":1,"samples_unverified":2,"pointer_only_for_licence":4,"official":{"repos":["LLaVA-VL/LLaVA-NeXT"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/llava-next-interleave-tackling-multi-image#ran","syntology_url":"https://syntology.ai/paper/2407.07895","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.07895"}}}},{"paper":"/paper/videollama-2-advancing-spatial-temporal","slug":"videollama-2-advancing-spatial-temporal","title":"VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs","date":"2024-06-11","arxiv_id":"2406.07476","rows_on_this_dataset":1,"code_links":3,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":17,"samples_ran":12,"samples_constructed":0,"samples_ran_checked":7,"samples_ran_instrument_failed":5,"samples_unverified":5,"pointer_only_for_licence":8,"official":{"repos":["damo-nlp-sg/videollama2"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/videollama-2-advancing-spatial-temporal#ran","syntology_url":"https://syntology.ai/paper/2406.07476","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.07476"}}}},{"paper":"/paper/gemini-1-5-unlocking-multimodal-understanding","slug":"gemini-1-5-unlocking-multimodal-understanding","title":"Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context","date":"2024-03-08","arxiv_id":"2403.05530","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/video-chatgpt-towards-detailed-video","slug":"video-chatgpt-towards-detailed-video","title":"Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models","date":"2023-06-08","arxiv_id":"2306.05424","rows_on_this_dataset":1,"code_links":2,"syntology":null},{"paper":"/paper/videochat-chat-centric-video-understanding","slug":"videochat-chat-centric-video-understanding","title":"VideoChat: Chat-Centric Video Understanding","date":"2023-05-10","arxiv_id":"2305.06355","rows_on_this_dataset":1,"code_links":1,"syntology":null}],"full_list":"papers_with_a_benchmark_row: every listed paper, in this order"},"papers_with_a_benchmark_row":[{"paper":"/paper/bimba-selective-scan-compression-for-long","slug":"bimba-selective-scan-compression-for-long","title":"BIMBA: Selective-Scan Compression for Long-Range Video Question Answering","date":"2025-03-12","arxiv_id":"2503.09590","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/qwen2-vl-enhancing-vision-language-model-s","slug":"qwen2-vl-enhancing-vision-language-model-s","title":"Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution","date":"2024-09-18","arxiv_id":"2409.12191","rows_on_this_dataset":1,"code_links":8,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":12,"samples_ran":12,"samples_constructed":1,"samples_ran_checked":10,"samples_ran_instrument_failed":2,"samples_unverified":0,"pointer_only_for_licence":1,"official":{"repos":["qwenlm/qwen2-vl"],"state":"official (archive's flag): 4 ran","n_ran":4,"n_constructed":0,"n_ran_no_instrument_failure":4,"n_unverified":0,"ran_from_kinds":["listed","official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/qwen2-vl-enhancing-vision-language-model-s#ran","syntology_url":"https://syntology.ai/paper/2409.12191","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.12191"}}}},{"paper":"/paper/llava-onevision-easy-visual-task-transfer","slug":"llava-onevision-easy-visual-task-transfer","title":"LLaVA-OneVision: Easy Visual Task Transfer","date":"2024-08-06","arxiv_id":"2408.03326","rows_on_this_dataset":2,"code_links":2,"syntology":null},{"paper":"/paper/llava-next-interleave-tackling-multi-image","slug":"llava-next-interleave-tackling-multi-image","title":"LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models","date":"2024-07-10","arxiv_id":"2407.07895","rows_on_this_dataset":1,"code_links":3,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":4,"samples_ran":2,"samples_constructed":0,"samples_ran_checked":1,"samples_ran_instrument_failed":1,"samples_unverified":2,"pointer_only_for_licence":4,"official":{"repos":["LLaVA-VL/LLaVA-NeXT"],"state":"official (archive's flag): 2 ran","n_ran":2,"n_constructed":0,"n_ran_no_instrument_failure":1,"n_unverified":0,"ran_from_kinds":["official"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/llava-next-interleave-tackling-multi-image#ran","syntology_url":"https://syntology.ai/paper/2407.07895","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.07895"}}}},{"paper":"/paper/videollama-2-advancing-spatial-temporal","slug":"videollama-2-advancing-spatial-temporal","title":"VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs","date":"2024-06-11","arxiv_id":"2406.07476","rows_on_this_dataset":1,"code_links":3,"syntology":{"read_at":"2026-10-05T00:36:11+00:00","samples_harvested":17,"samples_ran":12,"samples_constructed":0,"samples_ran_checked":7,"samples_ran_instrument_failed":5,"samples_unverified":5,"pointer_only_for_licence":8,"official":{"repos":["damo-nlp-sg/videollama2"],"state":"community repositories only","n_ran":0,"n_constructed":0,"n_ran_no_instrument_failure":0,"n_unverified":0,"ran_from_kinds":["listed"]},"claim":"Per-sample execution on synthesized fixtures; not a correctness claim.","sample_list":"/paper/videollama-2-advancing-spatial-temporal#ran","syntology_url":"https://syntology.ai/paper/2406.07476","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.07476"}}}},{"paper":"/paper/gemini-1-5-unlocking-multimodal-understanding","slug":"gemini-1-5-unlocking-multimodal-understanding","title":"Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context","date":"2024-03-08","arxiv_id":"2403.05530","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/video-chatgpt-towards-detailed-video","slug":"video-chatgpt-towards-detailed-video","title":"Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models","date":"2023-06-08","arxiv_id":"2306.05424","rows_on_this_dataset":1,"code_links":2,"syntology":null},{"paper":"/paper/videochat-chat-centric-video-understanding","slug":"videochat-chat-centric-video-understanding","title":"VideoChat: Chat-Centric Video Understanding","date":"2023-05-10","arxiv_id":"2305.06355","rows_on_this_dataset":1,"code_links":1,"syntology":null}],"papers_where_code_ran":3,"papers_where_code_ran_first_page":"/dataset/vnbench/papers/ran/1","papers_where_code_ran_caption":"We ran code from the paper's repository; we did not run it on this dataset or check it against this dataset's benchmarks.","pages":[{"order":"ran","page":1,"of":1,"url":"/dataset/vnbench/papers/ran/1","json":"/dataset/vnbench/papers/ran/1.json","rows":3}],"row_route":"each paper's syntology object carries sample_list (this site's paper page, #ran), syntology_url (syntology.ai's page for the paper, only when that route probed live) and mcp (the agent call that lists the paper's harvested samples; not a verdict on the paper)","developers":"https://syntology.ai/developers","syntology_totals":{"read_at":"2026-10-05T00:36:11+00:00","papers_with_samples":3,"samples_harvested":33,"samples_ran":26,"samples_constructed":1,"samples_ran_checked":18,"samples_ran_instrument_failed":8,"samples_unverified":7,"pointer_only_for_licence":13,"papers_with_no_sample_that_ran":0,"papers_with_a_sample_that_ran":3,"papers_with_a_run_with_no_instrument_failure":3,"papers_where_every_run_was_an_instrument_failure":0,"note":"the per-paper counts of every listed paper (all pages), summed; not a rate"},"key_notes":{"samples_ran_checked":"legacy name, kept unchanged so existing readers do not break: it counts the samples that ran with no instrument failure (honoured, violated, and ran with no contract checked); it does not mean a contract was checked, and the pages print it as 'K with no instrument failure', not 'K checked'","samples_constructed":"a sub-count of the samples that ran, never subtracted from them and never a failure: an executed sample whose run returned an instance of its own class (fixture_out_type equals the entry name): the run built an object and did not compute a result (Syntology's RAN record, counts.constructed)"},"since_archive_judged":{"papers":1,"papers_placed":0,"judged_by":{"model":"Claude Sonnet 4.5","models_as_recorded":["global.anthropic.claude-sonnet-4-5-20250929-v1:0"],"from":"lb_rows_run_verified.jsonl: each record's `model` field (the extractor's call that wrote decisions[])"},"note":"a model's judgement from the paper's own tables that they report on this leaderboard, not a result: no value from a paper that was not placed is printed or carried here; papers counts each paper once over these leaderboards","backfill":"8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read)","backfill_complete":false,"leaderboards":[{"leaderboard":"/sota/zero-shot-video-question-answer-on-vnbench","since_archive":"/sota/zero-shot-video-question-answer-on-vnbench#since-judged","papers_judged":1,"placed":0}]},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks.","record_sha256":"36cd88e270d49b294b55ec92b8fbd2a390ddb27465d3b6af08ea6fe2117492a1","record_changed_at":"2026-10-05","record_changed_at_basis":"changed"}