Papers › Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers

Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers

2 Apr 2020arXiv:2004.00849archive 2025-07-28

Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, Jianlong Fu

We propose Pixel-BERT to align image pixels with text by deep multi-modal transformers that jointly learn visual and language embedding in a unified end-to-end framework. We aim to build a more accurate and thorough connection between image pixels and language semantics directly from image and sentence pairs instead of using region-based image features as the most recent vision and language tasks. Our Pixel-BERT which aligns semantic connection in pixel and text level solves the limitation of task-specific visual representation for vision and language tasks. It also relieves the cost of bounding box annotations and overcomes the unbalance between semantic labels in visual task and language semantic. To provide a better representation for down-stream tasks, we pre-train a universal end-to-end model with image and sentence pairs from Visual Genome dataset and MS-COCO dataset. We propose to use a random pixel sampling mechanism to enhance the robustness of visual representation and to apply the Masked Language Model and Image-Text Matching as pre-training tasks. Extensive experiments on downstream tasks with our pre-trained model show that our approach makes the most state-of-the-arts in downstream tasks, including Visual Question Answering (VQA), image-text retrieval, Natural Language for Visual Reasoning for Real (NLVR). Particularly, we boost the performance of a single model in VQA task by 2.17 points compared with SOTA under fair comparison.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

microsoft/xpretrain mentioned on GitHubpytorchNOASSERTIONSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment (arXiv:2209.06430; that paper's own run record: 4 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified) and Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning (arXiv:2210.06031; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Image-text RetrievalImage-text matchingLanguage ModellingQuestion AnsweringRetrievalSentenceText MatchingText RetrievalVisual Question AnsweringVisual Question Answering (VQA)Visual Reasoning

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

Introduced by this paper: Pixel-BERT

Pixel-BERT

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections