{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/hitea-hierarchical-temporal-aware-video","title":"HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training","arxiv_id":"2212.14546","date":"2022-12-30","proceeding":"ICCV 2023 1","authors":["Qinghao Ye","Guohai Xu","Ming Yan","Haiyang Xu","Qi Qian","Ji Zhang","Fei Huang"],"abstract":"Video-language pre-training has advanced the performance of various downstream video-language tasks. However, most previous methods directly inherit or adapt typical image-language pre-training paradigms to video-language pre-training, thus not fully exploiting the unique characteristic of video, i.e., temporal. In this paper, we propose a Hierarchical Temporal-Aware video-language pre-training framework, HiTeA, with two novel pre-training tasks for modeling cross-modal alignment between moments and texts as well as the temporal relations of video-text pairs. Specifically, we propose a cross-modal moment exploration task to explore moments in videos, which results in detailed video moment representation. Besides, the inherent temporal relations are captured by aligning video-text pairs as a whole in different time resolutions with multi-modal temporal relation exploration task. Furthermore, we introduce the shuffling test to evaluate the temporal reliance of datasets and video-language pre-training models. We achieve state-of-the-art results on 15 well-established video-language understanding and generation tasks, especially on temporal-oriented datasets (e.g., SSv2-Template and SSv2-Label) with 8.6% and 11.1% improvement respectively. HiTeA also demonstrates strong generalization ability when directly transferred to downstream tasks in a zero-shot manner. Models and demo will be available on ModelScope.","url_abs":"https://arxiv.org/abs/2212.14546v1","url_pdf":"https://arxiv.org/pdf/2212.14546v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"tgif-action","task_name":"TGIF-Action"},{"task_slug":"tgif-frame","task_name":"TGIF-Frame"},{"task_slug":"tgif-transition","task_name":"TGIF-Transition"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":"zero-shot-video-retrieval","task_name":"Zero-Shot Video Retrieval"},{"task_slug":"cross-modal-alignment","task_name":"cross-modal alignment"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-captioning-on-msr-vtt-1","task":"Video Captioning","dataset":"MSR-VTT","model":"HiTeA","rank_in_archive_order":11,"of":24,"metrics":{"BLEU-4":"49.2","CIDEr":"65.1","METEOR":"30.7","ROUGE-L":"65.0"},"uses_additional_data":true},{"leaderboard":"/sota/video-captioning-on-msvd-1","task":"Video Captioning","dataset":"MSVD","model":"HiTeA","rank_in_archive_order":7,"of":16,"metrics":{"BLEU-4":"71.0","CIDEr":"146.9","METEOR":"45.3","ROUGE-L":"81.4"},"uses_additional_data":true},{"leaderboard":"/sota/video-question-answering-on-msrvtt-mc","task":"Video Question Answering","dataset":"MSRVTT-MC","model":"HiTeA","rank_in_archive_order":2,"of":7,"metrics":{"Accuracy":"97.4"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-next-qa","task":"Video Question Answering","dataset":"NExT-QA","model":"HiTeA","rank_in_archive_order":33,"of":47,"metrics":{"Accuracy":"63.1"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-activitynet","task":"Video Retrieval","dataset":"ActivityNet","model":"HiTeA","rank_in_archive_order":17,"of":31,"metrics":{"text-to-video R@1":"49.7","text-to-video R@10":"86.7","text-to-video R@5":"77.1"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-didemo","task":"Video Retrieval","dataset":"DiDeMo","model":"HiTeA","rank_in_archive_order":13,"of":40,"metrics":{"text-to-video R@1":"56.5","text-to-video R@10":"89.7","text-to-video R@5":"81.7"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"HiTeA","rank_in_archive_order":12,"of":38,"metrics":{"text-to-video R@1":"28.7","text-to-video R@10":"59.0","text-to-video R@5":"50.3"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"HiTeA","rank_in_archive_order":33,"of":63,"metrics":{"text-to-video R@1":"46.8","text-to-video R@10":"81.9","text-to-video R@5":"71.2"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-ssv2-label-retrieval","task":"Video Retrieval","dataset":"SSv2-label retrieval","model":"HiTeA","rank_in_archive_order":3,"of":5,"metrics":{"text-to-video R@1":"55.2","text-to-video R@10":"81.4","text-to-video R@5":"89.1"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-ssv2-template-retrieval","task":"Video Retrieval","dataset":"SSv2-template retrieval","model":"HiTeA","rank_in_archive_order":3,"of":5,"metrics":{"text-to-video R@1":"85.6","text-to-video R@10":"100","text-to-video R@5":"100"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-msrvtt-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSRVTT-QA","model":"HiTeA","rank_in_archive_order":12,"of":34,"metrics":{"Accuracy":"0.459"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-msvd-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSVD-QA","model":"HiTeA","rank_in_archive_order":11,"of":36,"metrics":{"Accuracy":"0.556"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-tgif-qa","task":"Visual Question Answering (VQA)","dataset":"TGIF-QA","model":"HiTeA","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"0.732"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-learning-on-msrvtt-qa","task":"Zero-Shot Learning","dataset":"MSRVTT-QA","model":"HiTeA","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"21.7"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-learning-on-msvd-qa","task":"Zero-Shot Learning","dataset":"MSVD-QA","model":"HiTeA","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"37.4"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-didemo","task":"Zero-Shot Video Retrieval","dataset":"DiDeMo","model":"HiTeA-17M","rank_in_archive_order":8,"of":26,"metrics":{"text-to-video R@1":"43.2","text-to-video R@10":"79.0","text-to-video R@5":"69.3"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-didemo","task":"Zero-Shot Video Retrieval","dataset":"DiDeMo","model":"HiTeA-5M","rank_in_archive_order":13,"of":26,"metrics":{"text-to-video R@1":"36.1","text-to-video R@10":"70.3","text-to-video R@5":"60.1"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-lsmdc","task":"Zero-Shot Video Retrieval","dataset":"LSMDC","model":"HiTeA-17M","rank_in_archive_order":7,"of":16,"metrics":{"text-to-video R@1":"18.3","text-to-video R@10":"44.2","text-to-video R@5":"36.7"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-lsmdc","task":"Zero-Shot Video Retrieval","dataset":"LSMDC","model":"HiTeA-5M","rank_in_archive_order":11,"of":16,"metrics":{"text-to-video R@1":"15.5","text-to-video R@10":"39.8","text-to-video R@5":"31.1"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msr-vtt","task":"Zero-Shot Video Retrieval","dataset":"MSR-VTT","model":"HiTeA-17M","rank_in_archive_order":19,"of":41,"metrics":{"text-to-video R@1":"34.4","text-to-video R@10":"69.9","text-to-video R@5":"60.0"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msr-vtt","task":"Zero-Shot Video Retrieval","dataset":"MSR-VTT","model":"HiTeA-5M","rank_in_archive_order":23,"of":41,"metrics":{"text-to-video R@1":"29.9","text-to-video R@10":"62.9","text-to-video R@5":"54.2"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2212.14546","atlas_url":"https://app.syntology.ai/?focus=2212.14546","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2212.14546"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2212.14546\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2212.14546"},"call":"get_concepts_for_paper(arxiv_id=\"2212.14546\")"}],"by_repo":[],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":"the archive holds no code link for this paper","arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"315c3b66ab4f3810de3606014a85e87dd282ad4978bb3905c62f1f978187c158","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}