Papers › Improving Visual Object Tracking through Visual Prompting

Improving Visual Object Tracking through Visual Prompting

27 Sep 2024arXiv:2409.18901archive 2025-07-28

Shih-Fang Chen, Jun-Cheng Chen, I-Hong Jhuo, Yen-Yu Lin

Learning a discriminative model to distinguish a target from its surrounding distractors is essential to generic visual object tracking. Dynamic target representation adaptation against distractors is challenging due to the limited discriminative capabilities of prevailing trackers. We present a new visual Prompting mechanism for generic Visual Object Tracking (PiVOT) to address this issue. PiVOT proposes a prompt generation network with the pre-trained foundation model CLIP to automatically generate and refine visual prompts, enabling the transfer of foundation model knowledge for tracking. While CLIP offers broad category-level knowledge, the tracker, trained on instance-specific data, excels at recognizing unique object instances. Thus, PiVOT first compiles a visual prompt highlighting potential target locations. To transfer the knowledge of CLIP to the tracker, PiVOT leverages CLIP to refine the visual prompt based on the similarities between candidate objects and the reference templates across potential targets. Once the visual prompt is refined, it can better highlight potential target locations, thereby reducing irrelevant prompt information. With the proposed prompting mechanism, the tracker can generate improved instance-aware feature maps through the guidance of the visual prompt, thus effectively reducing distractors. The proposed method does not involve CLIP during training, thereby keeping the same training complexity and preserving the generalization capability of the pretrained foundation model. Extensive experiments across multiple benchmarks indicate that PiVOT, using the proposed prompting method can suppress distracting objects and enhance the tracker.

PaperPDFCode

Syntology This paper's arXiv id is not in Syntology's Atlas id list (549,449 ids, as this build read them; the read time is in the build record), so this page links to no Syntology page for it. The list is not the whole graph; Syntology's feed shows papers as they enter the graph.

Code

chenshihfang/GOT officialmentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to GOT-Edit: Geometry-Aware Generic Object Tracking via Online Model Editing (arXiv:2602.08550; that paper's own run record: 16 ran (of which 2 constructed an object rather than computing a result; 13 with no instrument failure: 4 honoured, 7 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified) and GOT-JEPA: Generic Object Tracking with Model Adaptation and Occlusion Handling using... (arXiv:2602.14771; that paper's own run record: 0 ran · 1 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

ObjectObject TrackingVisual Object TrackingVisual PromptingVisual Tracking

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Visual Object Tracking AVisT PiVOT-L Success Rate 62.2 #1 of 7 Archive leaderboard report
Visual Object Tracking LaSOT PiVOT-L AUC 73.4 #13 of 46 Archive leaderboard report
Visual Object Tracking LaSOT PiVOT-L Normalized Precision 84.7 #13 of 46 Archive leaderboard report
Visual Object Tracking LaSOT PiVOT-L Precision 82.1 #13 of 46 Archive leaderboard report
Visual Object Tracking NeedForSpeed PiVOT-L AUC 0.682 #3 of 10 Archive leaderboard report
Visual Object Tracking OTB-2015 PiVOT-L AUC 0.712 #6 of 18 Archive leaderboard report
Visual Object Tracking OTB-2015 PiVOT-L Precision 0.946 #6 of 18 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

AttentionCLIPDINODense ConnectionsLayer NormalizationLinear LayerMulti-Head AttentionResidual ConnectionSoftmaxVision Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections