Papers › Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning

Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning

18 Mar 2021ICCV 2021 10arXiv:2103.10211archive 2025-07-28

Mandela Patrick, Yuki M. Asano, Bernie Huang, Ishan Misra, Florian Metze, Joao Henriques, Andrea Vedaldi

The quality of the image representations obtained from self-supervised learning depends strongly on the type of data augmentations used in the learning formulation. Recent papers have ported these methods from still images to videos and found that leveraging both audio and video signals yields strong gains; however, they did not find that spatial augmentations such as cropping, which are very important for still images, work as well for videos. In this paper, we improve these formulations in two ways unique to the spatio-temporal aspect of videos. First, for space, we show that spatial augmentations such as cropping do work well for videos too, but that previous implementations, due to the high processing and memory cost, could not do this at a scale sufficient for it to work well. To address this issue, we first introduce Feature Crop, a method to simulate such augmentations much more efficiently directly in feature space. Second, we show that as opposed to naive average pooling, the use of transformer-based attention improves performance significantly, and is well suited for processing feature crops. Combining both of our discoveries into a new method, Space-Time Crop & Attend (STiCA) we achieve state-of-the-art performance across multiple video-representation learning benchmarks. In particular, we achieve new state-of-the-art accuracies of 67.0% on HMDB-51 and 93.1% on UCF-101 when pre-training on Kinetics-400.

PaperPDFConference PDFCodeCode Syntology ranMethod as code

In Syntology View this paper on Syntology, its page in Syntology's graph. That page adds which of its functions are also present in other papers' code Syntology holds, with dates, and its dated run record as JSON and a README badge. It also lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

Syntology 23 code samples harvested from 1 repository linked to this paper: 12 ran (of which 11 constructed an object rather than computing a result; 12 with no instrument failure: 0 honoured, 0 violated, 12 with no contract checked; 0 where Syntology's instrument failed) · 11 unverified. “Where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code; here there are none.

By repository: official repository: 23 samples from 1 repository, 12 ran (12 with no instrument failure; 0 where Syntology's instrument failed). The run record, sample by sample. “Ran” means executed on a synthesized input, not that the code is correct or reproduces the paper.

Levels written between 2026-09-22 and 2026-09-24 (UTC): the dates Syntology's graph wrote each sample's verification level, not when the code executed.

facebookresearch/GDT officialmentioned in papermentioned on GitHubpytorchSyntology: 23 samples, 12 ran (0 instrument-failed) · 11 unverified report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

23 samples harvested: 12 ran (of which 11 constructed an object rather than computing a result; 12 with no instrument failure: 0 honoured, 0 violated, 12 with no contract checked; 0 where Syntology's instrument failed) · 11 unverified. “Where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Syntology's record for this page has not changed since , the first build that kept a record date for it; when this build read Syntology's graph is in the build record. That date is not when the samples ran.

12ran
11unverified

Licence: 0 of the 23 samples are pointer only, meaning Syntology does not serve that copy's text. This page shows no code text for any sample; each one links to its file in the repository.

Harvested from facebookresearch/GDT. “Ran” means the sample executed on a synthesized input. It does not mean the output is correct, and nothing here reproduces the paper's results. A line beginning “fixture:” under a sample that ran gives the arguments Syntology's fixture called the sample with, each as name:type/rank/element kind (? where none was recorded), and the type of what the fixture's run returned, exactly as the harness recorded them; the arguments and the returned type were recorded by two separate runs of the same fixture. No output value is shown, and a returned type says nothing about whether the output was right. “Level recorded” gives the UTC date when Syntology's graph wrote the sample's verification level, not when the code executed; where the level was written before Syntology's graph kept that date, the row says “level recorded, date not recorded”. For 4 samples that ran, the graph records a signature this page does not show: for a class the fixture built without calling, the harness called the object with inputs it made up itself, so the recorded signature is the harness's probe, not the fixture's arguments. “Honoured” and “violated” refer to a contract Syntology drafted from the code itself. “Our draft was wrong” and “the contract check could not run” are failures of Syntology's instrument, not of the code: the sample ran on Syntology's synthesized input, and then the drafted contract did not describe it, or the check itself raised an error (RAISES), lacked a dependency (DEP_MISSING) or ran out of time (TIMEOUT); hover the label for which. A metamorphic tier, where shown, is evidence of the output's shape, not that the sample implements the method.

Each sample ends with its code_sha256. A sample's code_sha256 is Syntology's identity for that exact code. It is printed in full (16 hexadecimal characters for every sample in this build), and get_code(code_sha256="…") takes it exactly as printed. An agent calls it on Syntology's MCP service (how to connect); click an identity to copy that call.

Repository labels, per sample. official repository: The archive marks this repository official for the paper. named in the paper: The archive records that the paper mentions this repository; it is not marked official. community (archive-listed): In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper. found in paper text by Syntology: Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. community: Not in the archive's code links for this paper; a community repository Syntology harvested. Samples are grouped by repository: a repository marked official first, then by how directly the repository is tied to the paper, then in the archive's order; never by how many ran. Licence labels name the repository's licence as recorded at harvest. “Pointer only” means Syntology does not serve that copy's text, for one of six reasons: no licence file was found; the licence was not identified; the licence is recorded as permissive but that copy's record is not marked cleared; the licence is outside the permissive list Syntology serves text under (MIT, Apache-2.0, BSD and similar); this paper's copy was not located, so its licence is not recorded; or this paper's copy is cleared, but the stored copy of the same code that Syntology serves is not. Some licences outside that list permit redistribution, such as WTFPL, and GPL-3.0 under its conditions; they are simply not on the list. Hover a licence label for the reason. File links open the file on GitHub at the default branch, which may have changed since the harvest.

A function or class name links to its page when the name occurs in the harvested code of at least 5 papers; the name is a string, not an identity. A census tag appears when Syntology's census ran implementations with that name on one shared input per bucket: it gives how many were compared, from how many papers, and how many distinct outputs they produced, and names this sample's bucket when this sample was one of them.

facebookresearch/GDT official repository 23 samples, 12 ran (0 instrument-failed) · 11 unverified
Base in 25 papers model.py ran Apache-2.0 (permissive) · 9d7e4b24b5660ac5 · report fixture: returned Base · level recorded 2026-09-22
BasicBlock in 126 papers
census: 6 implementations from 6 papers, 5 distinct outputs on one shared input
model.py ran Apache-2.0 (permissive) · 249b5820589b96d3 · report fixture: returned BasicBlock · level recorded 2026-09-22
Bottleneck in 98 papers model.py ran Apache-2.0 (permissive) · 346198401060aa08 · report fixture: returned Bottleneck · level recorded 2026-09-22
Conv2Plus1D model.py ran Apache-2.0 (permissive) · 4ab7412ae6131e98 · report fixture: returned Conv2Plus1D · level recorded 2026-09-22
FFN in 37 papers model.py ran Apache-2.0 (permissive) · ba5523a1a6748c82 · report fixture: returned FFN · level recorded 2026-09-22
MLP in 375 papers model.py ran Apache-2.0 (permissive) · 66c25c20b741db7d · report fixture: returned MLP · level recorded 2026-09-22
MultiHeadAttention in 124 papers
census: 9 implementations from 7 papers, 9 distinct outputs across 3 buckets, one shared input per bucket
model.py ran Apache-2.0 (permissive) · 025f425a305c99a1 · report fixture: signature input:Tensor/3/float, mask:Tensor/2/float, kv:NoneType/0/?, cache:NoneType/0/?, head_mask:NoneType/0/? · returned tuple · level recorded 2026-09-24
R2Plus1dStem model.py ran Apache-2.0 (permissive) · dfb98feede0d7132 · report fixture: returned R2Plus1dStem · level recorded 2026-09-22
Transformer in 173 papers model.py ran Apache-2.0 (permissive) · a45dddd3d20ddb5b · report fixture: returned Transformer · level recorded 2026-09-22
TransformerPooling model.py ran Apache-2.0 (permissive) · 784b7c6f51d0dc00 · report fixture: returned TransformerPooling · level recorded 2026-09-22
TransposeSqueeze model.py ran fingerprinted Apache-2.0 (permissive) · bd97e7875c7f6d01 · report fixture: returned TransposeSqueeze · level recorded 2026-09-22
VideoResNet model.py ran Apache-2.0 (permissive) · 1c92b909d0cdece1 · report fixture: returned VideoResNet · level recorded 2026-09-22
AudioBaseNetwork model.py unverified Apache-2.0 (permissive) · 05373243575c5edd · report
ResNet in 139 papers model.py unverified Apache-2.0 (permissive) · 6840fa8301c4edba · report
Stica_TransformerFMCrop model.py unverified Apache-2.0 (permissive) · 85fbc7f944b570fc · report
VideoBaseNetwork model.py unverified Apache-2.0 (permissive) · cf7d2feaca7a7e0e · report
_resnet in 27 papers model.py unverified Apache-2.0 (permissive) · f8df10b38e4d9f0d · report
_video_resnet model.py unverified Apache-2.0 (permissive) · 471f1e36d731e3a0 · report
get_audio_feature_extractor model.py unverified Apache-2.0 (permissive) · cf0d617f6919c7db · report
get_video_feature_extractor model.py unverified Apache-2.0 (permissive) · dfcc255237250347 · report
r2plus1d_18 in 6 papers model.py unverified Apache-2.0 (permissive) · 9982977c79b6bef4 · report
r2plus1d_34 model.py unverified Apache-2.0 (permissive) · dcfa82af3e6afc58 · report
resnet9 model.py unverified Apache-2.0 (permissive) · f78b8e83121feab2 · report

Method as code Syntology

What Syntology's graph records as this paper's claimed method, lifted into code: each assembly names its bricks (functions and classes), each with its own role and its own verification level. An assembly is not a check that the code implements the method correctly or reproduces the paper's results. Levels are each brick's own verification level in Syntology's graph (level 0: no recorded run), listed brick by brick and never combined into one level; “ran” means the brick executed on a synthesized input, not that its output is correct, and the instrument figure counts failures of Syntology's instrument, not of the code. Where a model read the code against the paper's claim, each assembly's label verdict is one model's judgement, kept in this page's JSON twin; it is not calibrated, so no card prints it.

1 assembly on this paper: 1 harvested, 0 generated.

harvested Assembly for Space-Time Crop & Attend (STiCA) (graph method: Space-Time Crop & Attend (STiCA)). 29 bricks; individually on a fixture: 17 ran (of which 13 constructed an object rather than computing a result; 15 with no instrument failure: 0 honoured, 0 violated, 15 with no contract checked; 2 where Syntology's instrument failed) · 12 unverified. The assembly has not been run end-to-end.

  1. Stica_TransformerFMCrop entry · level 0 · unverified · 85fbc7f944b570fc
  2. AudioBaseNetwork helper · level 0 · unverified · 05373243575c5edd
  3. MLP helper · level 1 · ran · 66c25c20b741db7d
  4. TransformerPooling helper · level 1 · ran · 784b7c6f51d0dc00
  5. TransposeSqueeze helper · level 1 · ran · bd97e7875c7f6d01
  6. VideoBaseNetwork helper · level 0 · unverified · cf7d2feaca7a7e0e
  7. get_audio_feature_extractor model · level 0 · unverified · cf0d617f6919c7db
  8. Flatten helper · level 1 · ran · a4f923bd5136361f
  9. Unsqueeze helper · level 1 · ran · c7c103d872864926
  10. Base helper · level 1 · ran · 9d7e4b24b5660ac5
  11. Transformer helper · level 1 · ran · a45dddd3d20ddb5b
  12. get_video_feature_extractor model · level 0 · unverified · dfcc255237250347
  13. Identity helper · level 1 · ran · cd43c253a0e41d2f
  14. resnet9 helper · level 0 · unverified · f78b8e83121feab2
  15. FFN helper · level 1 · ran · ba5523a1a6748c82
  16. MultiHeadAttention helper · level 1 · ran · 025f425a305c99a1
  17. r2plus1d_18 helper · level 0 · unverified · 9982977c79b6bef4
  18. r2plus1d_34 helper · level 0 · unverified · dcfa82af3e6afc58
  19. random_weight_init helper · level 0 · unverified · 2d01475f13deb6ce
  20. BasicBlock helper · level 1 · ran · 249b5820589b96d3
  21. _resnet helper · level 0 · unverified · f8df10b38e4d9f0d
  22. Conv2Plus1D helper · level 1 · ran · 4ab7412ae6131e98
  23. R2Plus1dStem helper · level 1 · ran · dfb98feede0d7132
  24. _video_resnet helper · level 0 · unverified · 471f1e36d731e3a0
  25. conv3x3 helper · level 1 · ran · our draft was wrong · 29df79c9fdb0cee8
  26. ResNet helper · level 0 · unverified · 6840fa8301c4edba
  27. VideoResNet helper · level 1 · ran · 1c92b909d0cdece1
  28. Bottleneck helper · level 1 · ran · 346198401060aa08
  29. conv1x1 helper · level 1 · ran · our draft was wrong · 158bf4c3a5f11f04

Each brick ends with its code_sha256, Syntology's identity for that exact code; click one to copy the get_code call that returns it. For agents, on Syntology's MCP service: assembly(arxiv_id="2103.10211") returns these assemblies with each brick's role and wiring (how to connect).

Tasks

Representation LearningSelf-Supervised Learning

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections