{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/grounded-situation-recognition","title":"Grounded Situation Recognition","arxiv_id":"2003.12058","date":"2020-03-26","proceeding":"ECCV 2020 8","authors":["Sarah Pratt","Mark Yatskar","Luca Weihs","Ali Farhadi","Aniruddha Kembhavi"],"abstract":"We introduce Grounded Situation Recognition (GSR), a task that requires producing structured semantic summaries of images describing: the primary activity, entities engaged in the activity with their roles (e.g. agent, tool), and bounding-box groundings of entities. GSR presents important technical challenges: identifying semantic saliency, categorizing and localizing a large and diverse set of entities, overcoming semantic sparsity, and disambiguating roles. Moreover, unlike in captioning, GSR is straightforward to evaluate. To study this new task we create the Situations With Groundings (SWiG) dataset which adds 278,336 bounding-box groundings to the 11,538 entity classes in the imsitu dataset. We propose a Joint Situation Localizer and find that jointly predicting situations and groundings with end-to-end training handily outperforms independent training on the entire grounding metric suite with relative gains between 8% and 32%. Finally, we show initial findings on three exciting future directions enabled by our models: conditional querying, visual chaining, and grounded semantic aware image retrieval. Code and data available at https://prior.allenai.org/projects/gsr.","url_abs":"https://arxiv.org/abs/2003.12058v1","url_pdf":"https://arxiv.org/pdf/2003.12058v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"grounded-situation-recognition","repo_url":"https://github.com/allenai/swig","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"grounded-situation-recognition","task_name":"Grounded Situation Recognition"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"situation-recognition","task_name":"Situation Recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/grounded-situation-recognition-on-swig","task":"Grounded Situation Recognition","dataset":"SWiG","model":"JSL","rank_in_archive_order":7,"of":13,"metrics":{"Top-1 Verb":"39.94","Top-1 Verb & Grounded-Value":"24.86","Top-1 Verb & Value":"31.44","Top-5 Verbs":"67.6","Top-5 Verbs & Grounded-Value":"40.6","Top-5 Verbs & Value":"51.88"},"uses_additional_data":false},{"leaderboard":"/sota/grounded-situation-recognition-on-swig","task":"Grounded Situation Recognition","dataset":"SWiG","model":"ISL","rank_in_archive_order":8,"of":13,"metrics":{"Top-1 Verb":"39.36","Top-1 Verb & Grounded-Value":"22.73","Top-1 Verb & Value":"30.09","Top-5 Verbs":"65.51","Top-5 Verbs & Grounded-Value":"36.6","Top-5 Verbs & Value":"50.16"},"uses_additional_data":false},{"leaderboard":"/sota/situation-recognition-on-imsitu","task":"Situation Recognition","dataset":"imSitu","model":"JSL","rank_in_archive_order":7,"of":13,"metrics":{"Top-1 Verb":"39.94","Top-1 Verb & Value":"31.44","Top-5 Verbs":"67.6","Top-5 Verbs & Value":"51.88"},"uses_additional_data":false},{"leaderboard":"/sota/situation-recognition-on-imsitu","task":"Situation Recognition","dataset":"imSitu","model":"ISL","rank_in_archive_order":8,"of":13,"metrics":{"Top-1 Verb":"39.36","Top-1 Verb & Value":"30.09","Top-5 Verbs":"65.51","Top-5 Verbs & Value":"50.16"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2003.12058","atlas_url":"https://app.syntology.ai/?focus=2003.12058","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"2003.12058"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"2003.12058\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"2003.12058"},"call":"get_concepts_for_paper(arxiv_id=\"2003.12058\")"}],"by_repo":[{"repo":"allenai/swig","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"4ee50b33c5e341dea8c8fc37d697a048e8b7fa54657933cf61a2241c6f104d63","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}