Papers › Where to put the Image in an Image Caption Generator

Where to put the Image in an Image Caption Generator

27 Mar 2017arXiv:1703.09137archive 2025-07-28

Marc Tanti, Albert Gatt, Kenneth P. Camilleri

When a recurrent neural network language model is used for caption generation, the image information can be fed to the neural network either by directly incorporating it in the RNN -- conditioning the language model by `injecting' image features -- or in a layer following the RNN -- conditioning the language model by `merging' image features. While both options are attested in the literature, there is as yet no systematic comparison between the two. In this paper we empirically show that it is not especially detrimental to performance whether one architecture is used or another. The merge architecture does have practical advantages, as conditioning by merging allows the RNN's hidden state vector to shrink in size by up to four times. Our results suggest that the visual and linguistic modalities for caption generation need not be jointly encoded by the RNN as that yields large, memory-intensive models with few tangible advantages in performance; rather, the multimodal integration should be delayed to a subsequent stage.

PaperPDFCode

Syntology This paper's arXiv id is not in Syntology's Atlas id list (549,449 ids, as this build read them; the read time is in the build record), so this page links to no Syntology page for it. The list is not the whole graph; Syntology's feed shows papers as they enter the graph.

Code

mtanti/where-image2 officialtfSyntology: not harvested report
VinitSR7/Image-Caption-Generation mentioned on GitHubtfSyntology: not harvested report
bmy4415/DMLAB-intern mentioned on GitHubtfSyntology: not harvested report
bprasannakumar/image_captioning_using_keras mentioned on GitHubSyntology: not harvested report
jishubasak/Punny-Caption--Exploring-Image-Captioning mentioned on GitHubSyntology: not harvested report
kahotsang/image-captioning mentioned on GitHubtfSyntology: not harvested report
mrityunjayojha10/Image-Caption-Generation mentioned on GitHubSyntology: not harvested report
mtanti/rnn-role mentioned on GitHubtfSyntology: not harvested report
nicolafan/image-captioning-cnn-rnn mentioned on GitHubtfMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Show and Tell: A Neural Image Caption Generator (arXiv:1411.4555; that paper's own run record: 28 ran (of which 0 constructed an object rather than computing a result; 20 with no instrument failure: 2 honoured, 2 violated, 16 with no contract checked; 8 where Syntology's instrument failed) · 6 unverified); nothing checks that they implement this paper's method. report
shan1322/Neural-Style-Captioning mentioned on GitHubSyntology: not harvested report
simnyatsanga/image-caption-generator mentioned on GitHubtfSyntology: not harvested report
thomasvengal/deeplearning mentioned on GitHubtfSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Caption GenerationLanguage ModelingLanguage Modelling

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections