Papers › Learning a Text-Video Embedding from Incomplete and Heterogeneous Data

Learning a Text-Video Embedding from Incomplete and Heterogeneous Data

7 Apr 2018arXiv:1804.02516archive 2025-07-28

Antoine Miech, Ivan Laptev, Josef Sivic

Joint understanding of video and language is an active research area with many applications. Prior work in this domain typically relies on learning text-video embeddings. One difficulty with this approach, however, is the lack of large-scale annotated video-caption datasets for training. To address this issue, we aim at learning text-video embeddings from heterogeneous data sources. To this end, we propose a Mixture-of-Embedding-Experts (MEE) model with ability to handle missing input modalities during training. As a result, our framework can learn improved text-video embeddings simultaneously from image and video datasets. We also show the generalization of MEE to other input modalities such as face descriptors. We evaluate our method on the task of video retrieval and report results for the MPII Movie Description and MSR-VTT datasets. The proposed MEE model demonstrates significant improvements and outperforms previously reported methods on both text-to-video and video-to-text retrieval tasks. Code is available at: https://github.com/antoine77340/Mixture-of-Embedding-Experts

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

antoine77340/Mixture-of-Embedding-Experts officialmentioned in papermentioned on GitHubpytorchApache-2.0Syntology: not harvested report
caijincen712/CE mentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Use What You Have: Video Retrieval Using Representations From Collaborative Experts (arXiv:1907.13487; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
escorciav/Mixture-of-Embedding-Experts mentioned on GitHubpytorchApache-2.0Syntology: not harvested report
hbzhang/cvpr2020 mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Use What You Have: Video Retrieval Using Representations From Collaborative Experts (arXiv:1907.13487; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
jayleicn/TVRetrieval mentioned on GitHubpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval (arXiv:2001.09099; that paper's own run record: 19 ran (of which 0 constructed an object rather than computing a result; 13 with no instrument failure: 4 honoured, 0 violated, 9 with no contract checked; 6 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

RetrievalText RetrievalVideo RetrievalVideo to Text Retrieval

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Video Retrieval LSMDC MoEE text-to-video Median Rank 27 #33 of 38 Archive leaderboard report
Video Retrieval LSMDC MoEE text-to-video R@1 10.1 #33 of 38 Archive leaderboard report
Video Retrieval LSMDC MoEE text-to-video R@10 34.6 #33 of 38 Archive leaderboard report
Video Retrieval LSMDC MoEE text-to-video R@5 25.6 #33 of 38 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections