{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/knowzrel-common-sense-knowledge-based-zero","title":"KnowZRel: Common Sense Knowledge-based Zero-Shot Relationship Retrieval for Generalised Scene Graph Generation","arxiv_id":null,"date":"2025-02-21","proceeding":"IEEE Transactions on Artificial Intelligence 2025 2","authors":["M. Jaleed Khan","John G. Breslin","Edward Curry"],"abstract":"A scene graph is a key image representation in visual reasoning. The generalisability of Scene Graph Generation (SGG) methods is crucial for reliable reasoning and real-world applicability. However, imbalanced training datasets limit this, underrepresenting meaningful visual relationships. Current SGG methods using external knowledge sources face limitations due to these imbalances or restricted relationship coverage, impacting their reasoning and generalisation capabilities. We propose a novel neurosymbolic approach that integrates data-driven object detection with heterogeneous knowledge graph-based object refinement and zero-shot relationship retrieval, highlighting the loosely coupled synergy between neural and symbolic components. This combination addresses the limitations of imbalanced training datasets in scene graph generation and enables effective prediction of unseen visual relationships. Objects are detected using a region-based deep neural network and refined based on their positional and structural similarity, followed by retrieval of pairwise visual relationships using a heterogeneous knowledge graph. The redundant and irrelevant visual relationships are discarded based on the similarity of relationship labels and node embeddings. Finally, the visual relationships are interlinked to generate the scene graph. The employed heterogeneous knowledge graph combines diverse knowledge sources, offering rich common sense knowledge about objects and their interactions in the world. Our method, evaluated using the benchmark Visual Genome dataset and zero-shot recall (zR@K) metric, shows a 59.96% improvement over existing state-of-the-art methods, highlighting its effectiveness in generalised SGG. The object refinement step effectively improved the object detection performance by 57.1%. Additional evaluation using the GQA dataset confirms the cross-dataset generalisability of our method. We also compared various knowledge sources and embedding models to determine an optimal combination for zero-shot SGG. The source code is available at https://github.com/jaleedkhan/zsrr-sgg.","url_abs":"https://ieeexplore.ieee.org/document/10897903","url_pdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=10897903","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"knowzrel-common-sense-knowledge-based-zero","repo_url":"https://github.com/jaleedkhan/zsrr-sgg","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"graph-generation","task_name":"Graph Generation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"scene-graph-generation","task_name":"Scene Graph Generation"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"},{"task_slug":"zero-shot-scene-graph-generation","task_name":"Zero-Shot Scene Graph Generation"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/object-detection-on-gqa","task":"Object Detection","dataset":"GQA","model":"KnowZRel","rank_in_archive_order":1,"of":1,"metrics":{"mAP":"39"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-visual-genome","task":"Object Detection","dataset":"Visual Genome","model":"KnowZRel","rank_in_archive_order":1,"of":4,"metrics":{"MAP":"44"},"uses_additional_data":false},{"leaderboard":"/sota/scene-graph-generation-on-gqa","task":"Scene Graph Generation","dataset":"GQA","model":"KnowZRel","rank_in_archive_order":1,"of":1,"metrics":{"zR@100":"29.56","zR@20":"12.47","zR@50":"22.51"},"uses_additional_data":false},{"leaderboard":"/sota/scene-graph-generation-on-visual-genome","task":"Scene Graph Generation","dataset":"Visual Genome","model":"KnowZRel","rank_in_archive_order":19,"of":19,"metrics":{"zR@100":"35.65","zR@20":"14.22","zR@50":"25.43"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers","held":null,"reason":"no_arxiv_id","checked_against":null,"by_repo":[{"repo":"jaleedkhan/zsrr-sgg","kind":"named_in_paper","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"a8b4f972e30a11943d2f155d6e5b9dc0cf11ac113e2d7edc6c3de55cbd1bd779","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}