Papers › Class-agnostic Object Detection with Multi-modal Transformer

Class-agnostic Object Detection with Multi-modal Transformer

22 Nov 2021arXiv:2111.11430archive 2025-07-28

Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, Ming-Hsuan Yang

What constitutes an object? This has been a long-standing question in computer vision. Towards this goal, numerous learning-free and learning-based approaches have been developed to score objectness. However, they generally do not scale well across new domains and novel objects. In this paper, we advocate that existing methods lack a top-down supervision signal governed by human-understandable semantics. For the first time in literature, we demonstrate that Multi-modal Vision Transformers (MViT) trained with aligned image-text pairs can effectively bridge this gap. Our extensive experiments across various domains and novel objects show the state-of-the-art performance of MViTs to localize generic objects in images. Based on the observation that existing MViTs do not include multi-scale feature processing and usually require longer training schedules, we develop an efficient MViT architecture using multi-scale deformable attention and late vision-language fusion. We show the significance of MViT proposals in a diverse range of applications including open-world object detection, salient and camouflage object detection, supervised and self-supervised detection tasks. Further, MViTs can adaptively generate proposals given a specific language query and thus offer enhanced interactability. Code: https://git.io/J1HPY.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

mmaaz60/mvits_for_class_agnostic_od officialmentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection (arXiv:2207.03482; that paper's own run record: 3 ran (of which 1 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Class-agnostic Object DetectionObjectObject DetectionObject Proposal GenerationOpen World Object Detectionobject-detection

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Object Detection PASCAL VOC 10% DETReg (MDef-DETR) AP 58.78 #1 of 2 Archive leaderboard report
Object Detection PASCAL VOC 10% DETReg (MDef-DETR) AP50 80.46 #1 of 2 Archive leaderboard report
Object Detection PASCAL VOC 10% DETReg (MDef-DETR) AP75 65.65 #1 of 2 Archive leaderboard report
Object Detection PASCAL VOC 2007 DETReg (MDef-DETR) AP50 84.16 #3 of 30 Archive leaderboard report
Object Detection PASCAL VOC 2007 DETReg (MDef-DETR) MAP 84.16% #3 of 30 Archive leaderboard report
Object Proposal Generation COCO (Common Objects in Context) MDef-DETR (Off-the-shelf evaluation) Average Recall 0.6503 #1 of 1 Archive leaderboard report
Object Proposal Generation PASCAL VOC 2012, 60 proposals per image MDef-DETR Average Recall 0.9126 #1 of 3 Archive leaderboard report
Open World Object Detection COCO 2017 (Electronic, Indoor, Kitchen, Furniture) ORE (MDef-DETR) MAP 31.66 #1 of 2 Archive leaderboard report
Open World Object Detection COCO 2017 (Outdoor, Accessories, Appliance, Truck) ORE (MDef-DETR) A-OSE 5212 #1 of 2 Archive leaderboard report
Open World Object Detection COCO 2017 (Outdoor, Accessories, Appliance, Truck) ORE (MDef-DETR) MAP 46.19 #1 of 2 Archive leaderboard report
Open World Object Detection COCO 2017 (Outdoor, Accessories, Appliance, Truck) ORE (MDef-DETR) Unknown Recall 49.54 #1 of 2 Archive leaderboard report
Open World Object Detection COCO 2017 (Outdoor, Accessories, Appliance, Truck) ORE (MDef-DETR) WI 0.0251 #1 of 2 Archive leaderboard report
Open World Object Detection COCO 2017 (Sports, Food) ORE (MDef-DETR) A-OSE 4117 #1 of 2 Archive leaderboard report
Open World Object Detection COCO 2017 (Sports, Food) ORE (MDef-DETR) MAP 36.75 #1 of 2 Archive leaderboard report
Open World Object Detection COCO 2017 (Sports, Food) ORE (MDef-DETR) Unknown Recall 50.89 #1 of 2 Archive leaderboard report
Open World Object Detection COCO 2017 (Sports, Food) ORE (MDef-DETR) WI 0.0179 #1 of 2 Archive leaderboard report
Open World Object Detection PASCAL VOC 2007 ORE (MDef-DETR) A-OSE 7322 #1 of 2 Archive leaderboard report
Open World Object Detection PASCAL VOC 2007 ORE (MDef-DETR) MAP 64.03 #1 of 2 Archive leaderboard report
Open World Object Detection PASCAL VOC 2007 ORE (MDef-DETR) Unknown Recall 50.13 #1 of 2 Archive leaderboard report
Open World Object Detection PASCAL VOC 2007 ORE (MDef-DETR) WI 0.0474 #1 of 2 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Introduced by this paper: MAVL

MAVLMViT

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections