\section{Introduction}%
\label{sec:intro}%
Large-scale manually annotated datasets~\cite{deng2009imagenet,zhou2017places} have been fueling the rapid development of deep learning-based methods in computer vision.
Training supervised models over such datasets not only leads to state-of-the-art results, but also enables networks to learn useful image representations that can be exploited on downstream tasks.
However, this approach has major limitations.
First, the cost and complexity of annotating datasets is considerable, especially when the class taxonomy is fine-grained requiring expert knowledge~\cite{deng2009imagenet,parkhi12cats,wah2011CUB}.
Second, retraining from scratch dedicated models for every new task is inefficient.

Some alternative approaches address these issues and require less curated or fewer annotations~\cite{mahajan2018ECCV,yalniz2019billion}.
At the other extreme of visual representation learning, self-supervised learning methods~\cite{caron19unsupervised,doersch2015context,doersch2017mtself,fernando17self,zhang2016colorful,zhang2017splitbrain}
do not require annotations and fabricate proxy labels from the data itself.
They induce regularities of the data itself, decorrelated from any specific downstream task annotations.
Unfortunately, recent findings show that these approaches are not data efficient, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~they require either extremely large training sets (up to a hundred million images)~\cite{caron19unsupervised,goyal19scaling} or need to be trained much longer with larger networks to express their full potential~\cite{chen20simclr,he20moco}.
Hence they demand huge computational resources.

Interestingly, data often comes with informative metadata for free.
For instance, user tags associated with images can be used as image labels~\cite{joulin2016learning,mahajan2018ECCV}.
Even richer, companion text for images, is sometimes available for free.
Using recent sanitation procedures~\cite{qi2020imagebert}, high-quality large-scale captioned datasets can automatically be constructed.

In this paper, we argue that learning visual representations with captions should significantly reduce the scale of the training sets required for pretraining visual representations.
If no text is available, in some context it is still easier to acquire short captions than expert-quality-level fine-grained class labels over thousands of categories like in ImageNet~\cite{deng2009imagenet}.
Yet, caption annotations have rarely been used to train visual representations from scratch.
Notable exceptions are~\cite{gomez17self,joulin2016learning,quattoni07learning} which learn image features by training to predict words in a caption or topic probabilities estimated from an associated text.
However, none of these approaches use the structure of the entire sentences, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~they treat words individually.
Recent studies~\cite{devlin2019bert,peters18deep} have shown the superiority of word representations which are conditioned by their surrounding, where the same word has different representations depending on the sentence.
We believe such caption representations should also be beneficial for learning image representations.

This paper focuses on the following research questions.
{\em Can we train transferable visual representations from limited sets of image-caption pairs?}
If so, {\em how should we formulate the interaction between images and captions?}
To address these questions, we propose several proxy tasks involving images and their captions which we use to train visual representations from scratch.
The first one (Sec.~\ref{sec:method_tp}) is intuitive and requires only extracting {\em image tags} from captions.
We propose several ways to do so, and we show that predicting image tags is already competitive compared to other pretraining strategies.
Then, to utilize the captions more effectively, and inspired by the recent advances in natural language processing~\cite{devlin2019bert}, we propose a second proxy task (Sec.~\ref{sec:method_icmlm}) which employs masked language modeling to learn visual representations.
Similar to the first proxy task, it also leverages both images and captions, but it additionally allows visual representations to learn to {\em localize semantic concepts} in captions.
Qualitative results show that the architecture proposed to tackle this second proxy task effectively leverages the text and attends to relevant image regions (see~\fig{fig:front}).

Our contributions are threefold.
First, we empirically validate that simple tag prediction tasks, where tags are obtained from captions, already learn transferable visual representations.
Second, in an attempt to benefit from captions more, we introduce a new task called {\bf i}mage-{\bf c}onditioned {\bf m}asked {\bf l}anguage {\bf m}odeling (ICMLM) and propose two multi-modal architectures to solve this task.
Third, we show that solving ICMLM leads to useful visual representations as a by-product.
These visual representations, which we obtain using only a hundred thousand captioned images, are competitive with recent self-supervised approaches leveraging a hundred million images, and, in some cases, even fully-supervised approaches showing how powerful a cue text is.

\section{Related Work}%
\label{sec:relwork}%
Pretraining CNNs on an external dataset has become standard practice in computer vision~\cite{chen18deeplab,gordo2017end,ren2015faster,sariyildiz2019CVPR}, especially for domains or tasks for which data is scarce.
The most common strategy is to train a CNN for the ImageNet-1K classification task~\cite{ilsvrc15} and then to use it as a feature extractor or to fine-tune it on a target task or domain.
Although this scheme has proven to be quite useful, designing fully-annotated datasets represents a significant effort requiring prior knowledge and domain expertise~\cite{deng2009imagenet}.
Thus, alternative research directions have gained interest.
We review the ones closest to our work.

\mypar{Weakly/Webly-supervised learning}
Two main research lines have prospered recently.
The first line focuses on using {\em metadata} associated to web data, such as tags or captions for images or videos~\cite{thomee15yfcc100m}.
Although the signal-to-noise ratio of samples crawled from the web may arguably be lower than carefully-constructed datasets, significant progress has been made leveraging this type of data to pretrain models~\cite{chen2015webly,hong2017wsseg}.
Among those, to learn visual representations,~\cite{joulin2016learning} extracts
the most common hashtags and words from the captions and titles of 99 million
images in the YFCC100M dataset~\cite{thomee15yfcc100m} and train to predict these words using CNNs.
Similarly,~\cite{mahajan2018ECCV} uses hashtags associated with images from Instagram to construct datasets containing up to 3.5 billion images.

The second line upscales ImageNet. Leveraging ImageNet labels, those approaches
produce {\em pseudo-labels} for additional unlabeled images~\cite{xie20self,yalniz2019billion}.
We note that these methods require initial annotations and extremely large-scale sets of images.
In contrary, our models need far less images, 118 thousand images at most, but companion captions to learn visual representations.

\mypar{Unsupervised representation learning}
Self-supervised approaches build a {\em pretext task} to learn image representations which are decorrelated from any downstream task and they do not require any manual annotations.
Often, {\em proxy tasks} consist in predicting missing pieces in manipulated
images, for instance context prediction~\cite{doersch2015context},
colorization~\cite{deshpande15learning,larsson17colorization,zhang2016colorful},
inpainting of missing portions~\cite{pathak2016contextenc}, prediction of image rotations~\cite{gidaris2018ICLR}, spotting artifacts~\cite{jenni18self}, or cross-channel prediction~\cite{zhang2017splitbrain}.
Besides, recently, contrastive learning-based unsupervised methods~\cite{bachman19learning,he20moco,oord2018cpc,wu18unsupervised} have showed significant improvements.
However, computational and data efficiency of these methods are still inferior to supervised models.

It is important to note that most unsupervised approaches are trained on {\em curated datasets} such as ImageNet for which images were carefully selected to form a well-balanced collection for a diverse set of fine-grained categories.
Although these approaches do not directly use ImageNet labels, they implicitly benefit from this careful selection and the resulting underlying structure of the dataset.
Indeed,~\cite{caron2018ECCV,doersch2015context} show that the feature quality drops when raw data are used instead of ImageNet.
Yet, assuming that a curated dataset such as ImageNet is readily available is a strong assumption.
Consequently, some works~\cite{caron19unsupervised,goyal19scaling,mahendran18cross} have evaluated unsupervised methods trained on {\em uncurated data}~\cite{thomee15yfcc100m}.
They have concluded that large amounts of raw data (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot~96 millions images) is required to express the full potential of these approaches.
In this work, we focus on learning from a much smaller set of images by leveraging textual information.

\mypar{Vision and language}
Vision and language (VL) have been jointly leveraged to learn cross-modal representations for various VL tasks, such as cross-modal retrieval~\cite{gomez19self,wang16learning}, visual question answering~\cite{goyal17making}, captioning~\cite{sun2019vidbert} or visual grounding~\cite{deng18visual,hu2016natural}.
Building on the recent advances in natural language processing~\cite{devlin2019bert,Vaswani2017AttentionIA}, several works have fine-tuned BERT~\cite{devlin2019bert} to fuse visual and textual information~\cite{lu2019vilbert,su2020vlbert,sun2019vidbert,tan2019lxmert,zhou20unified} for VL tasks.
However, while learning cross-modal representations, such approaches rely on pretrained feature extractors, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~they use visual features pooled from regions of interest produced by a state-of-the-art detector such as Faster-RCNN~\cite{ren2015faster}.
Therefore, their objectives are formulated under the assumption that discriminative visual features are readily available for a list of relevant objects.
We note that such feature extractors are already state-of-the-art for most vision tasks, requiring {\em expensive bounding box annotations} to train.
Our approach follows a different path.
We focus on learning visual representations {\em from scratch} for purely visual tasks by leveraging captions.

\mypar{Learning visual features using text}
Only few works have taken advantage of the companion text to learn image representations.
\cite{quattoni07learning} creates and solves auxiliary prediction tasks from images with associated captions.
\cite{li2017ngrams} constructs label sets out of caption $n$-grams, and trains
CNNs by predicting these labels.
\cite{gomez17self} extracts topic models for Wikipedia pages using latent Dirichlet allocation and trains a CNN to embed their associated images in this topic space.
\cite{gordo17beyond} uses captions to learn image representations for the specific task of semantic retrieval.

We argue that language has a complex structure which cannot be reduced to computing n-grams statistics in a text.
Motivated by this, we differ and propose to use a pretrained language model -
which can be trained in an unsupervised manner for large text corpora - to represent captions and individual words in them.
In our experiments, we show that by doing so it is possible to learn visual representations that are useful for a broad range of tasks.

\section{Method}%
\label{sec:method}%
We argue that captions associated with images can provide semantic information about some {\em observable concepts} that can be captured by image representations.
Such concepts can be objects, attributes, or actions that visually appear in images.
With this motivation, given a dataset composed of image-caption pairs, we want to formulate non-trivial proxy tasks conditioned on both images and captions such that solving these tasks produce generic visual representations as a by-product.
In particular, we want such tasks to properly use the structure of caption sentences, and not only treat them as orderless sets of words.

To this end, we propose two proxy tasks focusing on two distinct objectives to train CNNs to recognize a predefined set of concepts in images.
The first proxy task captures {\em global} semantics in images by predicting image-level tags and is presented in \sect{sec:method_tp}.
The second proxy task, the image-conditioned masked language modeling task, focuses on {\em local} semantics in images and is detailed in \sect{sec:method_icmlm}.
Experiments show that both proxy tasks are complementary.

\mypar{Notations}
We assume that our dataset $\mathcal{D} = \{ (I^i, c^i)\}_1^N$ is composed of $N$ image-caption pairs.
We denote by $\C = \{o_i\}_1^K$ the set of concepts to be recognized in images.
As there can be multiple concepts in an image, we use binary label vectors $\vec{y} \in \{0, 1\}^{K}$ to denote the presence of concepts in images, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~$\vec{y}_k = 1$ if concept $o_k$ appears in image $I$ and $0$ otherwise.
We define two parametric functions $\phi$ and $\psi$ which respectively embed images and text.
More precisely, $\phi: I \rightarrow \X \in \Real{H \times W \times \dx}$ takes an image $I$ as input and produces $\X$  which is composed of $\dx$-dimensional visual features over a spatial grid of size $H \cdot W$.
Similarly, $\psi: c \rightarrow \W \in \Real{T \times \dw}$ transforms a caption (a sequence of $T$ tokens) into a set of $\dw$-dimensional vectors, one for each token.
In our models, we train only $\phi$, which is a CNN producing visual representations, and we use a pretrained language model as $\psi$ that we freeze during training.

\begin{figure}[t!]
	\centering
	\includegraphics[width=\linewidth]{res/model_diagram.pdf}
	\caption{\textbf{Modules used in our models.}
             {\bf (1)} a CNN to extract visual features;
             {\bf (2)} a language model to extract token features;
             {\bf (3)}, {\bf (4)} and {\bf (5)} respectively correspond to our \texttt{tfm}, \texttt{att} + \texttt{fc} and \texttt{tp} modules.
             Our {TP}$_\star${}, {ICMLM}$_\text{\tfm}${} and {ICMLM}$_\text{\att-\fc}${} models combine these modules: {\bf (1)} + {\bf (5)}, {\bf (1)} + {\bf (2)} + {\bf (3)} and {\bf (1)} + {\bf (2)} + {\bf (4)}, respectively.
             Trainable (and frozen) components are colored in {\color{NavyBlue} blue} (and black).
             Only the CNN is used during target task evaluations.
             }
    \label{fig:icmlm}
\end{figure}

\subsection{Capturing image-level semantics}%
\label{sec:method_tp}%

A straightforward way to build a proxy task given image-caption pairs is to formulate a multi-label image classification problem, where, according to its caption, multiple concepts may appear in an image~\cite{joulin2016learning,quattoni07learning}.
For this setup, we create a label vector $\vec{y} \in \{0, 1 \}^{K}$ for each image $I$ such that $\vec{y}_j = 1$ if concept $o_j$ appears in the image, and $0$ otherwise.
We denote these labels as {\em tags}, and name this task as {\em tag prediction} (TP), illustrated in \fig{fig:icmlm} (modules {\bf (1)} + {\bf (5)}).

One of the contributions of this work is to consider different ways to define concept sets $\C$ from captions.
Ground-truth concept vectors can be easily obtained by considering the most frequent bi-grams~\cite{joulin2016learning} or even n-grams~\cite{li2017ngrams} in captions.
More sophisticated ways to obtain artificial labels include using LDA~\cite{blei2003latent} to discover latent topics in captions~\cite{gomez17self}.
In addition to these existing methods, we look for ways to exploit semantics of tokens in captions.

\mypar{{TP\textsubscript{Postag}}{}}
As a first approach, we simply propose to construct label sets by taking into account the {\em part-of-speech} (POS) tags of tokens in captions.
Concretely, we use the off-the-shelf language parser~\cite{spacy} to determine POS tags of tokens in captions and gather three label sets of size $K$, including (i) only nouns, (ii) nouns and adjectives, (iii) nouns, adjectives and verbs.
These three label sets are used to train three separate {TP\textsubscript{Postag}}{} models.

\mypar{{TP\textsubscript{Cluster}}{}}
As mentioned above, we believe it would be beneficial to use the structure of the full caption and not just treat it as an orderless set of tokens as the previously proposed {TP\textsubscript{Postag}}.
To this end, we use the pretrained BERT$_\text{base}${}~\cite{devlin2019bert} model to extract sentence-level caption representations.
We do this by feeding the caption into BERT$_\text{base}${} and taking the representation for the [CLS] token, which is used as a special token to encode sentence-level text representations in BERT$_\text{base}${}.
Then, we cluster the sentence-level representations of all captions using the $k$-means{} algorithm and apply hard cluster assignment.
This way, the labels are the cluster indices and we train $\phi$ by learning to predict the cluster assignments of captions from their associated image.
$K$-means learns $K$ cluster centroids $\cent^\star \in \Real{\dw \times \nk}$ in the caption representation space by minimizing:
\begin{equation}
\cent^\star, \{ \vec{y}^{i\star} \}_{i=1}^{N} = \argmin_{
\substack{
\cent \in \Real{\dw \times \nk},\\
\{ \vec{y}^i \in \{0,1\}^{\nk},\,\, \mathbf{1}_{\nk} ^\top \vec{y}^i = 1 \}_{i=1}^{N}}}\quad
\sum_{i=1}^N
\| \psi(c^i)_\text{[CLS]} -  \cent \vec{y}^i \|_2^2,
\end{equation}
where $\psi(c)_\text{[CLS]}$ and $\vec{y}^\star$ denote the [CLS] representation of the caption $c$ and of the one-hot cluster assignment vector obtained for $c$.
Note that $\vec{y}^\star$ is used as the label for image $I$.
In case there are multiple captions for an image, we simply aggregate the cluster labels of all captions associated to that image.

\mypar{Training {TP}$_\star${} models}
Once we have crafted image labels over a chosen set of concepts (either using POS tags or cluster assignments), following~\cite{mahajan2018ECCV}, we normalize the binary label vectors to sum up to one, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~$\vec{y}^\top \mathbf{1}_{\nk} = 1, $ for all samples.
Then we train models by minimizing the categorical cross-entropy:
\begin{equation}~\label{eq:tp_loss}
\ell_\text{tp} = - \E_{(I, c) \in \D} \Bigg[ \sum_{k=1}^{\nk} \vec{y}_k \log(p(\hat{\vec{y}}_k \, | \, I)) \Bigg],
\end{equation}
where $p(\hat{\vec{y}}_k \, | \, I) = \frac{\exp(\hat{\vec{y}}_k)}{\sum_j \exp(\hat{\vec{y}}_j)}$, $\hat{\vec{y}}_k = \text{\tp{}}(\phi(I))_k$, and $\text{\tp{}}: \Real{H \times W \times \dx} \rightarrow \Real{\nk}$ is a parametric function performing tag predictions.

\subsection{Capturing localized semantics}%
\label{sec:method_icmlm}%
The previous section presents a cluster prediction task where the structure of the sentence is leveraged through the use of the [CLS] output of the pretrained BERT$_\text{base}${}.
Yet, this has a major limitation: token-level details may largely be ignored especially when captions are long~\cite{bahdanau2015neural}.
Our experiments also support this argument, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~{TP\textsubscript{Cluster}}\ performs on par with or worse than {TP\textsubscript{Postag}}.
To address this issue, we propose a second learning protocol that learns to explicitly {\em relate} individual concepts appearing in both an image and its caption.

To this end, we extend the natural language processing task known as Masked Language Model (MLM)~\cite{devlin2019bert} into an {\em image-conditioned} version.
The MLM task trains a language model by masking a subset of the tokens in an input sentence, and then by predicting these masked tokens.
Inspired by this idea, we introduce the Image-Conditioned Masked Language Model (ICMLM) task.
Compared to MLM, we propose to predict masked tokens in a caption by using the visual information computed by $\phi$.
This way, we learn visual representations that should be informative enough to reconstruct the missing information in captions.

For this task, for each image-caption pair $(I, c)$, we assume that there is at least one concept appearing in the caption $c$.
Since $c$ describes the visual scene in $I$, we assume that concepts appearing in $c$ are observable in $I$ as well.
This allows us to define \icmlm\ as a concept set recognition problem in images.
More precisely, we use the pretrained BERT$_\text{base}${} model~\cite{devlin2019bert} as the textual embedding function $\psi$ and define the learning protocol as follows.
First, we segment the caption $c$ into a sequence of tokens $(t_{1}, \ldots, t_{T})$, and mask one of the tokens $t_{m}$, which belongs to the concept set.
Masking is simply done by replacing the token $t_{m}$ with a special token reserved for this operation, for instance BERT$_\text{base}${}~\cite{devlin2019bert} uses ``[MASK]''.
Then, {\em contextualized} representations of the tokens are computed as $\W = \psi((t_{1},\ldots,t_{T}))$.
Meanwhile, the visual representation of the image $I$ is computed by $\phi(I) = \X$.
Since our goal is to predict the masked token by using both visual and textual representations, we need to merge them.
A naive way to accomplish that is to (i) pool the representations of each modality into a global vector, (ii) aggregate (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~concatenate) these vectors, (iii) use the resulting vector to predict the label of the masked token.
However, the representations obtained in this way could only focus on the global semantics, and the local information for both modalities might be lost during the pooling stage.
To address this concern, we describe two possible designs for ICMLM relying on individual visual (in the spatial grid) and textual (in the sequence) features.

\mypar{{ICMLM}$_\text{\tfm}${}}
Here, we contextualize token representations among visual ones by fusing them in a data-driven manner (similar to~\cite{lu2019vilbert}).
Concretely, we spatially flatten and project $\X$ to the token embedding space,
concatenate it with $\W$ and apply a transformer encoder module~\cite{Vaswani2017AttentionIA}, \texttt{tfm}{}, on top of the stacked representations.
Finally, as done in BERT$_\text{base}${}~\cite{devlin2019bert}, the label of the masked token $t_{m}$ can be predicted by feeding the representation of the {\em transformed} masked token into the pretrained token classification layer of BERT$_\text{base}${}.
We call this ICMLM flavor {ICMLM}$_\text{\tfm}$ (modules {\bf (1)} + {\bf (2)} + {\bf (3)} in \fig{fig:icmlm}).

\mypar{{ICMLM}$_\text{\att-\fc}${}}
Transformer networks employ a self-attention mechanism with respect to their inputs.
Therefore they can learn the pairwise relationships of both the visual and the textual representations.
This allows them, for instance, to fuse different domains quite effectively~\cite{lu2019vilbert,sun2019vidbert}.
We also verify this powerful aspect of the transformers in our experiments, \emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot~even a single-layered transformer network is enough to perform significantly well at predicting masked tokens on the MS-COCO dataset~\cite{lin14coco}.
However, the rest of the caption is already a powerful cue to predict the masked token and this transformer-based architecture might rely too much on the text, potentially leading to weaker visual representations.
As an alternative, we propose to predict the label of the masked token by using the visual features alone.
Since the masked token is a concept that we want to recognize in the image, we divide the prediction problem into two sub-problems: localizing the concept in the image and predicting its label.
To do that we define two additional trainable modules: \texttt{att}\ and \texttt{fc}\ modules that we describe in detail below.
This ICMLM flavor is referred to as {ICMLM}$_\text{\att-\fc}${} (modules {\bf (1)} + {\bf (2)} + {\bf (4)} in Fig.~\ref{fig:icmlm}).

The goal of the \texttt{att}\ module is to create a 2D attention map on the spatial grid of the visual feature tensor $\X$ such that high energy values correspond to the location of the concept masked in the caption $c$.
It takes as input the spatially-flattened visual features $\X \in \Real{H \cdot W \times \dx}$ and the textual features $\W$.
First, $\X$ and $\W$ are mapped to a common $\dz$-dimensional space and then pairwise attention scores between visual and textual vectors are computed:
\begin{equation}~\label{eq:att_norm}
\tilde{\X} = \lfloor \text{norm}(\X \Sigma_x) \rfloor_+ ,
\hspace{1cm}
\tilde{\W} = \lfloor \text{norm}(\W \Sigma_w) \rfloor_+ ,
\hspace{1cm}
\vec{S} =  \frac{\tilde{\X} {\tilde{\W}}^\top}{\sqrt{\dz}},
\end{equation}
where $\Sigma_x \in \Real{\dx \times \dz}$ and $\Sigma_w \in \Real{\dw \times \dz}$ are parameters to learn, $\text{norm}$ is LayerNorm~\cite{ba16layer} and $\lfloor . \rfloor_+$ is ReLU operator.
Note that $\vec{S}_{i,j}$ denotes the attention of visual vector $i$ (a particular location in the flattened spatial-grid of the image) to textual vector $j$ (a particular token in the caption).
To be able to suppress attention scores of vague tokens such as ``about'' or ``through'', we compute {\em soft maximum} of the textual attentions for each visual feature:
\begin{equation}~\label{eq:logsumexp}
\vec{s}_i = \log \sum_{j=1}^{T} \exp\left(\vec{S}_{i, j}\right).
\end{equation}
We note that operations in Eqs.~(\ref{eq:att_norm}) and~(\ref{eq:logsumexp}) are performed for a single attention head and the multi-headed attention mechanism~\cite{Vaswani2017AttentionIA} can easily be adopted by learning a weighted averaging layer:
$ \vec{s}_i = \left[ \vec{s}_i^1 | \cdots | \vec{s}_i^H \right] \Sigma_h + b_h $,
where $\Sigma_h \in \Real{H}$ and $b_h \in \Real{}$ are the parameters of the averaging layer, $\vec{s}_i^h$ is the aggregated textual attention score for the $i$\textsuperscript{th} visual feature coming from the $h$\textsuperscript{th} attention head, and $[.|.]$ denotes concatenation.
Finally, attention probabilities are obtained by applying softmax, and used to pool $\X$ into a single visual feature $\hat{\vec{x}}$:
\begin{equation}
{\vec{p_\text{\att{}}}}_i = \frac{\exp(\vec{s}_i)}{\sum_{j=1}^{H \cdot W} \exp(\vec{s}_j)},
\hspace{1cm}
\hat{\vec{x}} = \X^\top \vec{p_\text{\att{}}},
\end{equation}
where $\vec{p_\text{\att{}}} \in [0, 1]^{H \cdot W}$ such that $\vec{p_\text{\att{}}}^\top \mathbf{1}_{H \cdot W} = 1$.

After localizing the concept of interest in image $I$ by means of pooling $\X$ into $\hat{\vec{x}}$, we feed $\hat{\vec{x}}$ into the \texttt{fc}{} module, which consists in a sequence of fully-connected layers, each composed of linear transformation, LayerNorm and ReLU operator.
Finally, we map the output of the \texttt{fc}{} module to the BERT$_\text{base}${}'s token vocabulary $\V$  and compute prediction probabilities as follows:
\begin{equation} \label{eq:probbert}
p_\V\left(k | I, c, t_{m}\right) = \frac{ \exp(\hat{\vec{v}}_{k}) }{ \sum_j \exp(\hat{\vec{v}}_{j})},
\end{equation}
where $\hat{\vec{v}}_{k} = \text{\texttt{fc}}(\hat{\vec{x}})^\top \V_k$ and $\V_k \in \dw$ are the prediction score and the pretrained distributed representation of the $k$\textsuperscript{th} token in the pretrained candidate lexicon of BERT$_\text{base}${}.
As we compute dot-products between post-processed $\hat{\vec{x}}$ and the pretrained representations of the tokens in BERT$_\text{base}${}'s vocabulary, it is possible to leverage the structure in BERT$_\text{base}${}'s hidden representation space.
Indeed, we observe that such probability estimation of a candidate token is more effective than learning a fully connected layer which projects $\text{\texttt{fc}}(\hat{\vec{x}})$ onto the vocabulary.

\mypar{Training \icmlmstar{} models}
To train both model flavors, for each masked token $t_m$ in all ($I$, $c$) pairs in $\D$, we minimize the cross-entropy loss between the probability distribution over the BERT$_\text{base}${}'s vocabulary as computed in Eq.~(\ref{eq:probbert}) and the label of the masked token $t_m$ (index of $t_m$ in $\V$):
\begin{equation}~\label{eq:mlm_loss}
\ell_\text{mlm} = - \E_{(I, c) \in \D} \left[ \E_{t_m \in c} \Big[ \log ( p_\V (k | I, c, t_{m} ) ) \Big] \right],
\end{equation}
where $k$ is the index of $t_m$ in BERT$_\text{base}${}'s vocabulary.
The expectation over captions implies that there can be multiple concepts in a caption and we can mask and predict each of them separately.
For {ICMLM}$_\text{\tfm}$, $\hat{\vec{x}}$ is computed by the \texttt{tfm}\ module, and it corresponds to the representation of the masked token.
For {ICMLM}$_\text{\att-\fc}$, $\hat{\vec{x}}$ corresponds to the output from the \texttt{fc}\ module.

We also note that $\ell_\text{tp}$ and $\ell_\text{mlm}$ are complementary, enforcing $\phi$ to focus on global and local semantics in images, respectively.
Therefore, in both {ICMLM}$_\text{\att-\fc}$\ and {ICMLM}$_\text{\tfm}$\ we minimize the weighted combination of $\ell_\text{tp}$ and $\ell_\text{mlm}$:
\begin{equation}~\label{eq:icmlm_loss}
\ell_\text{icmlm} =  \ell_\text{mlm} + \lambda \ell_\text{tp} .
\end{equation}

\section{Experiments}%
\label{sec:exps}%

This section evaluates
\begin{enumerate*}[(i),font=\bfseries]
    \item how the performance on the masked language modeling (MLM) proxy task translates to target tasks (\sect{sec:exp_mlm_ablation}),
    \item how several types of supervision associated to a set of images (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot full, weak and self-supervision) compare to each other (\sect{sec:exp_fws_comp}),
    \item if the gains of \icmlmstar{} models are consistent across backbone architectures (\sect{sec:exp_resnet50}),
    \item if \icmlmstar{} models attend to relevant regions in images (Figs \ref{fig:front} and \ref{fig:r50_vis_coco}).
\end{enumerate*}
First, we introduce our experimental setup (remaining details are in the
supplementary material).

\mypar{Datasets}
We train our models on the image-caption pairs of either the 2017 split of MS-COCO~\cite{lin14coco} (COCO) or the Visual Genome~\cite{krishna17vg} (VG) datasets.
COCO has 123K images (118K and 5K for train and val) and 5 captions for each image while VG has 108K images (we randomly split 103K and 5K for train and val) and 5.4M captions.
We remove duplicate captions and those with more than 25 or less than 3 tokens.
We construct several concept sets using the captions of COCO or VG, to be used as tags for {TP\textsubscript{Postag}}{} and as maskable tokens for \icmlmstar{} models (an ablative study is provided in the supplementary material).
Note that depending on the concept set, the number of tags and the (image, caption, maskable token) triplets vary, therefore, we specify which concept set is used in all {TP\textsubscript{Postag}}{} and \icmlmstar{} experiments.

\mypar{Networks}
To be comparable with the state-of-the-art self-supervised learning method DeeperCluster~\cite{caron19unsupervised}, we mainly use VGG16~\cite{simonyan15VGG} backbones.
We also evaluate \icmlmstar{} models using ResNet50~\cite{he2016resnet} in \sect{sec:exp_resnet50}.
Note that \icmlmstar{} models operate on a set of visual tensors, therefore, for {TP}$_\star${} and \icmlmstar{} models we remove the FC layers from VGG16.
To compensate, we use 4-layered CNNs combined with global average pooling and linear layer for tag predictions as $\texttt{tp}$ modules.
For $\texttt{tfm}$, $\texttt{att}$ and $\texttt{fc}$ modules, we cross-validated the number of hidden layers and attention heads on the validation set of Pascal VOC-07 dataset, and found that 1 hidden layer (in \texttt{tfm}{} and \texttt{fc}{}) and 12 attention heads (in \texttt{tfm}{} and \texttt{att}{}) works well.
While training \icmlmstar{} models we set $\lambda = 1$ in \eq{eq:icmlm_loss}.

\mypar{Target task}
Once a model is trained, we discard its additional modules used during training (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot all but $\phi$) and evaluate $\phi$ on image classification tasks, to test how well pretrained representations generalize to new tasks.
To do that, following~\cite{caron19unsupervised}, we train linear logistic regression classifiers attached to the last three convolutional layers of the frozen backbones $\phi$ with SGD updates and data augmentation.
We perform these analyses on the Pascal-VOC07 dataset~\cite{pascal-voc-2007} (VOC) for multi-label classification, and ImageNet-1K (IN-1K)~\cite{deng2009imagenet} and Places-205~\cite{zhou2014learning} datasets for large-scale categorization, using the publicly available code of~\cite{caron19unsupervised} with slight modifications: We apply heavier data augmentations~\cite{chen20simclr} and train the classifiers for more iterations, which we found useful in our evaluations.

\mypar{Additional {TP}$_\star${} models}
We note that the TP model defined in \sect{sec:method_tp} can be used for predicting any type of image tags, with slight modifications.
We use it to predict topics as proposed in~\cite{gomez17self} and denote this approach as {TP\textsubscript{LDA}}.
To do so, we only modify \eq{eq:tp_loss} to minimize binary cross-entropy loss instead, where $K$ denotes the number of hidden topics.
Similarly, we denote TP\textsubscript{Label}\ as the supervised approach which uses the annotated image labels as tags.

\subsection{Ablative study on the proxy task}%
\label{sec:exp_mlm_ablation}%

We first study the interplay between ICMLM and target tasks.
To do so, we train several \icmlmstar{} models, and monitor their performance on both the proxy and target tasks, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~we report masked token prediction (MTP) scores on VG and COCO, and mAP scores on VOC, respectively.
For reference, we also report MTP scores obtained by a single BERT$_\text{base}${} model, where masked tokens are predicted using only the remainder of the captions.
In this study, we used the 1K most frequent nouns and adjectives in the captions as maskable tokens.

\noindent {\bf{Results}} are shown in \tab{tab:icmlm_ablation_vgg16}.
We observe that \icmlmstar{} models significantly improve MTP scores compared to BERT$_\text{base}${} model, showing that visual cues are useful for MLM tasks.
Moreover, {ICMLM}$_\text{\tfm}${} is better than {ICMLM}$_\text{\att-\fc}${} on the proxy task, indicating that blending visual and textual cues, which is effectively done by the \texttt{tfm}{} module,  is beneficial for MLM.
However, {ICMLM}$_\text{\att-\fc}${} generalizes better than {ICMLM}$_\text{\tfm}${} to VOC.
We believe that, as {ICMLM}$_\text{\att-\fc}${} predicts masked tokens using visual cues only, it learns semantic concepts from the given training set better than {ICMLM}$_\text{\tfm}${}.
A similar study which uses ResNet50 backbones~\cite{he2016resnet} leads to similar observations (see the supplementary material).

\subsection{Comparison of fully-, weakly- and self-supervised methods}%
\label{sec:exp_fws_comp}%

Next, we compare the visual representations learned by different state-of-the-art fully-, weakly- and self-supervised learning (SSL) models.
We do this by training the models explained below on COCO or VG, then using their backbones $\phi$ to perform the target tasks, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot image classification on VOC, IN-1K and Places-205.

\input{tab_vgg16_lineareval.tex}

\mypar{Supervised}
For reference, we report the results obtained by three supervised classifiers trained on different subsets of IN-1K:
\begin{enumerate*}[(i),font=\bfseries]
    \item ``ImageNet'' on the full IN-1K,
    \item ``$\mathcal{S}$-ImageNet with 1K classes'' on randomly-sampled 100 images per class,
    \item ``$\mathcal{S}$-ImageNet with 100 classes'' on 1K images for each of 100 randomly sampled classes.
\end{enumerate*}
The latter two contain 100K images each \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~the same order of magnitude as COCO or VG.
For the models trained on these three subsets, we repeat the sampling 4 times and report their mean target task results.
We also report TP\textsubscript{Label}{} which is trained to predict ground-truth labels.

\mypar{Weakly-supervised}
We compare {TP\textsubscript{LDA}}{}, {TP\textsubscript{Cluster}}{}, {TP\textsubscript{Postag}}{} and \icmlmstar{} methods, for which image-level tags are extracted from the captions of COCO or VG.
For {TP\textsubscript{LDA}}{} we use the publicly-available code of~\cite{gomez17self} to find 40 latent topics among all captions (the number of topics was validated on the validation set of VOC).
Then, probabilities over caption topics define the tag labels for each image.
For {TP\textsubscript{Cluster}}{}, we cluster the captions (finding 1K or 10K clusters) and assign the cluster IDs of the captions associated to images as their tag labels.
For {TP\textsubscript{Postag}}{}, the tag labels are the most frequent 1K or 10K nouns, adjectives and verbs in the captions.
For \icmlmstar{} models the maskable tokens are the most frequent 1K nouns, adjectives and verbs in the captions.

\mypar{Self-supervised}
For reference, we also provide results for two self-supervised approaches: RotNet~\cite{gidaris2018ICLR} and DeeperCluster~\cite{caron19unsupervised}.
We train RotNet models from scratch on COCO or VG. For DeeperCluster, we use a model pretrained on the large-scale YFCC-100M dataset~\cite{thomee15yfcc100m} (96M images).

\myparnodot{Results} are reported in \tab{tab:vgg16_lineval}.
We observe the following.
\begin{enumerate*}[(i),font=\bfseries]
    \item We see that the good results of ``ImageNet'' are mostly due to its scale.
    Reducing it to 100K images, either by reducing the number of classes or the number of images per class significantly hurt the performance.
    Similarly, the supervised TP\textsubscript{Label}{}, which uses an order of magnitude fewer categories and images performs far worse than ImageNet.
    \item The proposed {TP\textsubscript{Cluster}}{} outperforms the current state of the art for training with captions, {TP\textsubscript{LDA}}~\cite{gomez17self}, for all three datasets.
    Exploiting both the structure and the semantics of captions with the BERT$_\text{base}${}
    language model, it improves over a topic model.
    However, we see that {TP\textsubscript{Cluster}}{} performs on par with or worse than {TP\textsubscript{Postag}}{}, suggesting that the importance of individual tokens might be suppressed in global caption representations.
    This validates our motivation for proposing ICMLM in
    \sect{sec:method_icmlm}: models should leverage both global and local semantics in captions.
    \item We see that both {ICMLM}$_\text{\tfm}${} and {ICMLM}$_\text{\att-\fc}${} improve over all {TP}$_\star${} baselines by significant margins.
    Moreover, on VOC evaluations {ICMLM}$_\text{\att-\fc}${} outperforms {ICMLM}$_\text{\tfm}${} while on IN-1K and Places it performs on par with or worse than {ICMLM}$_\text{\tfm}${}.
    Note that we observe a similar outcome with ResNet50 backbones (\sect{sec:exp_resnet50}).
    \item Surprisingly, for VOC and Places-205, at least one ICMLM flavor outperforms the full ImageNet pretrained model which we believe is a significant achievement.
    For IN-1K, such comparison does not make sense as, in this setting, the proxy and the target datasets are the same.
    Training on the target set clearly confers an unfair advantage w.r.t. other approaches.
\end{enumerate*}

\input{tab_r50_coco.tex}

\subsection{Additional results with ResNet50}%
\label{sec:exp_resnet50}%

Some self-supervised proxy tasks might favor certain network architectures (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot see \cite{kolesnikov2019revisiting}).
This section provides additional results where \icmlmstar{} models use
ResNet50~\cite{he2016resnet}  backbone architectures.
To this end, we train TP\textsubscript{Label}{}, {TP\textsubscript{Postag}}{} and \icmlmstar{} models on COCO and perform image classification on VOC and IN-1K.
To reduce computational costs, following~\cite{goyal19scaling}, we train linear SVMs (on VOC) and logistic regression classifiers (on IN-1K) using image features pre-extracted from frozen backbones.
Note that ResNet50 is a fully-convolutional network being more expressive compared to VGG16 (thanks to its residual connections and higher number of parameters).
Consequently, in this analysis, we use a 2-layered MLPs as $\texttt{tp}$ module, a single attention head, and $\lambda = 0.1$ in \eq{eq:icmlm_loss}.
We also move to a bigger concept set for {TP\textsubscript{Postag}}{} and \icmlmstar{} models, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~the 5K most frequent nouns, adjectives and verbs.

\myparnodot{Results} are shown in \tab{tab:r50_linclf_coco}.
We observe larger improvements of {TP\textsubscript{Postag}}{} over TP\textsubscript{Label}{} and of \icmlmstar{} over {TP\textsubscript{Postag}}{}.
\icmlmstar{} outperforms {TP\textsubscript{Postag}}{} by at least $\mathbf{4.7\%}$, $\mathbf{4.0\%}$ and TP\textsubscript{Label}{} by at least $\mathbf{7.1\%}$, $\mathbf{13.9\%}$ on VOC and IN-1K.
These results indicate that more complex CNNs are better at suppressing noise in weak labels and at learning cross-modal representations.
Besides, similar to our previous analyses, we see that {ICMLM}$_\text{\att-\fc}${} learns semantic concepts from the training set slightly better (see the VOC results).
However, {ICMLM}$_\text{\tfm}${} performs better on IN-1K, suggesting that the ResNet50 backbone learns more discriminative features when guided by the same language model.

\mypar{Qualitative results}
Our goal in \icmlm{} is to perform MLM task by {\em looking at} images.
To see if they can attend to relevant parts in images, we visualize {\em attention maps} corresponding to the attention weights of visual features to masked tokens.
Figs \ref{fig:front} and \ref{fig:r50_vis_coco} present such visualizations produced by our {ICMLM}$_\text{\tfm}${} model with ResNet50 backbone trained on COCO.
We see that not only the model is able to detect possible concepts of interest,
it can also understand which concept is asked in the captions (see the supplementary for more visualizations).

\section{Conclusion}%
\label{sec:conc}%
Until recently, carefully collected and manually annotated image sets have provided the most efficient way of learning general purpose visual representations.
To address the annotation cost, weakly-, webly-, and self-supervised learning approaches have traded quality -- a clean supervisory signal -- with quantity, requiring up to hundreds of million images.
Although, in some cases, large quantities of unlabeled data are readily available, processing such large volumes is far from trivial.
In this paper, we seek for a cheaper alternative to ground-truth labels to train visual representations.
First, starting from the observation that captions for images are often easier to collect compared to \emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot~fine-grained category annotations, we have defined a new proxy task on image-caption pairs, namely image-conditioned masked language modeling (ICMLM), where image labels are automatically produced thanks to an efficient and effective way of leveraging their captions.
Second, we have proposed a novel approach to tackle this proxy task which produces general purpose visual representations that perform on par with state-of-the-art self-supervised learning approaches on a variety of tasks, using a fraction of the data.
This approach even rivals, on some settings, with a fully supervised pretraining on ImageNet. Such results are particularly relevant for domains where images are scarce but companion text is abundant.

\section{Label sets \vs~target task performances}%
\label{sec:exp_ablation_label_sets}%

As we mention in the main paper, for {TP\textsubscript{Postag}}{} and \icmlmstar{} models, we can construct multiple concept (or label) sets from captions, \emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot~the most frequent $K$ nouns, adjectives or verbs in captions can be used as tags for {TP\textsubscript{Postag}}{} and as maskable tokens for \icmlmstar{} models.
In this section, we investigate the impact of learning from such label sets on target task performances.
To do so, we compare learning visual representations using annotated labels of images \vs~tags derived from captions, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~TP\textsubscript{Label}{} \vs~{TP\textsubscript{Postag}}{} with various label sets.

For this analysis, we first train ResNet50 backbones, and then, once a model is trained, we extract image representations from the frozen backbones.
To test generalization capabilities of the representations, we train linear SVMs on VOC~\cite{pascal-voc-2007} and linear logistic regression classifiers on IN-1K~\cite{ilsvrc15}.
Additionally, to understand how effectively models can learn from the training set, we also train linear SVMs on COCO~\cite{lin14coco}.

\myparnodot{Results} are presented in \tab{tab:tp_ablations}.
All {TP\textsubscript{Postag}}{} models trained for the ablation improve over TP\textsubscript{Label}{}, suggesting that a caption describing an image can provide more comprehensive supervision compared to labeling it with a small set of classes.
It is surprising that gaps are more significant on IN-1K, indicating that a large vocabulary of tags allows backbones to encode more discriminative patterns.
{TP\textsubscript{Postag}}{} obtained by using the most frequent 5K nouns, adjectives and verbs in captions improves TP\textsubscript{Label}{} by $\mathbf{2.4\%}$, $\mathbf{9.9\%}$ and $\mathbf{2.0\%}$ on VOC, IN-1K and COCO.
In Sec.~4.3 of the main paper, we report results of {TP\textsubscript{Postag}}{} and \icmlmstar{} models trained with this label set.

\input{tab_r50_tp_ablations.tex}
\input{tab_r50_icmlm_ablations.tex}

\section{\icmlm{} \vs~target task performances}%
\label{sec:exp_ablation_icmlm}%
This section extends the analysis reported in Sec.~4.1 of the main paper, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~we study how the masked language modeling (MLM) performance (the proxy task) translates to target tasks. This time we use ResNet50 backbones instead of VGG16 (as in Sec.~4.1 of the main paper).
To do so, we train {ICMLM}$_\text{\tfm}${} (and {ICMLM}$_\text{\att-\fc}${}) models with different numbers of hidden layers and attention heads in \texttt{tfm}{} (and, in \texttt{fc}{} and \texttt{att}{}, respectively) modules, and monitor both proxy and target task results.
While training \icmlmstar{} models, we set $\lambda = 0$ in Eq.~8 of the main paper: for this ablation study the training solely depends on $\ell_\text{mlm}$ defined by Eq.~7 in the main paper.
Similar to the previous analysis, we perform target tasks using pre-extracted image features on VOC, IN-1K and COCO.
We also report top-1 masked token prediction (MTP) scores on COCO.

\myparnodot{Results} are reported in \tab{tab:icmlm_ablations}.
We observe that having more hidden layers or attention heads improve the MLM performance at the expense of reduced target task results.
We believe that as the complexity of \texttt{tfm}{}, \texttt{att}{} or \texttt{fc}{} modules increase, they can learn more interconnections between visual and textual cues, and this, in turn, lifts some of the burden of capturing the semantics of the caption off the visual model itself, and leads $\phi$ to learn weaker visual features.
Moreover, similar to the observations we made in Secs.~4.1 and~4.2 of the main paper, {ICMLM}$_\text{\tfm}${} significantly outperforms {ICMLM}$_\text{\att-\fc}${} on MLM and IN-1K, however, {ICMLM}$_\text{\att-\fc}${} is slightly better than {ICMLM}$_\text{\tfm}${} on VOC and COCO.
The fact that IN-1K performance of {ICMLM}$_\text{\att-\fc}${} increases when \texttt{fc}{} module has two hidden layers also supports the hypothesis that {ICMLM}$_\text{\att-\fc}${} tends to overfit to the concepts present in the training set (hence it performs better on VOC and COCO).

Comparing \tab{tab:tp_ablations} and \tab{tab:icmlm_ablations}, we see overall that \icmlmstar{} (when \#L and \#H are 1) improves {TP\textsubscript{Postag}}{} by at least $\mathbf{3.1\%}$, $\mathbf{3.3\%}$, $\mathbf{2.1\%}$ and TP\textsubscript{Label}{} by at least $\mathbf{5.5\%}$, $\mathbf{13.2\%}$, $\mathbf{4.1\%}$ on VOC, IN-1K and COCO.

\mypar{A note for \icmlmstar{} models with a VGG16 backbone}
We tried these settings for VGG16 backbones: one attention head in \icmlmstar{} models and $\lambda = 0.$ (Eq.~8 of the main paper) but this lead to inferior models.
We believe that this is due to the absence of residual connections in the backbone architecture, which leads to overfitting to MLM tasks (a similar behavior is observed in~\cite{kolesnikov2019revisiting} for self-supervised learning methods trained with VGG16 architecture).

\mypar{Importance of $\lambda$ in Eq.~8 of the main paper}
We discuss in Sec.~3 of the main paper that global \vs~localized semantics in images can and should be captured separately.
To this end, in Eq.~8 of the main paper, we propose to optimize a combination of $\ell_\text{tp}$ and $\ell_\text{mlm}$ losses to effectively train backbones by providing supervision for both global and localized semantics.
In our \icmlmstar{} experiments, we validated the coefficient $\lambda$ combining these loss terms by monitoring the $\ell_\text{tp}$ loss on the validation sets of COCO or VG.
We tried three values for $\lambda \in \{0.0, 0.1, 1.0\}$ and found that $\lambda = 0.1$ and $\lambda = 1.0$ minimize $\ell_\text{tp}$ loss on the validation sets with ResNet-50 and VGG16 backbones respectively, and moreover improve target task results.
This finding supports our claim that $\ell_\text{tp}$ and $\ell_\text{mlm}$ loss terms are complementary.

\section{Implementation details}%
\label{sec:exp_imp}%

This section provides technical details of both training model for proxy tasks and evaluating them on target tasks.

\subsection{Training for proxy tasks}

\mypar{With VGG16 backbones}
We start training VGG16 networks on the Visual Genome (VG) or MS-COCO datasets by solving the rotation prediction task~\cite{gidaris2018ICLR}.
Note that we do not use any of the existing RotNet~\cite{gidaris2018ICLR} pretrained models as they all have processed millions of images.
Contrarily, we want to restrict all the training steps of our pipeline to access only a small dataset of images (103K and 118K training images of VG and COCO respectively).
For that, first, we train separate VGG16 networks on VG or COCO for 120K iterations using RAdam~\cite{liu20radam} with batches of size 128, initial learning rate $1e-3$, weight decay $1e-5$, and the learning rate is decayed by $0.1$ after 100K and 110K iterations.
Once the networks are trained for the rotation prediction task, we remove the fully-connected layers from the networks and fine-tune the CNN backbones by solving the proxy tasks we defined in Secs.~3.1 and~3.2 of the main paper.

We train {TP}$_\star${} models for 100K iterations using RAdam optimizer~\cite{liu20radam} with batches of size 128, initial learning rate 1e-4, weight decay 1e-3, and the learning rate is decayed by 0.1 after 80K and 90K iterations.
For {TP}$_\star${} models, the number of data samples is equal to the number of images in the training sets (103K in VG and 118K in COCO).
The number of unique triplets (image, caption, masked token) that we use during training \icmlm{} models varies from 2.5M to 13M depending on the dataset and the label set used, because we design the triplets in a way that for each (image, caption) pair, there is only one masked token so many triplets are built for a single (image, caption) pair.
To reduce the training time, we train them for 200K iterations using batches of size 896 (distributed over 4 NVIDIA V100 GPUs).
We note that in early \icmlm{} trainings, attention heads (\texttt{att}{} modules in {ICMLM}$_\text{\att-\fc}${} and self-attention attention heads in {ICMLM}$_\text{\tfm}${}) produce almost uniform attention distributions over the spatial grid of visual features.
Therefore, in {ICMLM}$_\text{\att-\fc}${} models, we find that warming up the attention heads for 50K iterations while freezing VGG16 backbones prevents noisy gradients to flow through backbones.

\mypar{With ResNet50 backbones}
We train TP\textsubscript{Label}{} and {TP\textsubscript{Postag}}{} models from scratch for 100K iterations using SGD with momentum (0.9) optimizer with batches of size 128, initial learning rate 3e-2, weight decay 1e-4, and the learning rate is decayed by a cosine-based schedule.
We initialize ResNet50 backbones in \icmlmstar{} models with pretrained {TP\textsubscript{Postag}}{} checkpoints then train \icmlmstar{} models for 500K iterations using the same optimizer configuration except that batch size is 512.

We validate all hyper-parameters and design choices on the validation sets of VG and COCO.
As we note in Sec.~3.2 of the main paper, while training \icmlmstar{} models, we freeze the pretrained BERT$_\text{base}${} model available in HuggingFace repository\footnote{\url{https://github.com/huggingface/transformers}}.
We use PyTorch~\cite{paszke19pytorch} and the mixed-precision functionality provided by NVIDIA Apex\footnote{\url{https://github.com/NVIDIA/apex}} to perform all experiments.

\subsection{Evaluation on target tasks}

We follow two different evaluation practices to compare models:
\begin{enumerate}[(i),font=\bfseries]
    \item Probing linear logistic regression classifiers after various layers in VGG16 backbones and training them with SGD updates and data augmentation.
          For this evaluation, we use the publicly-available code of \cite{caron19unsupervised} and slightly modify it such that heavier data augmentation is applied and classifiers are trained for more iterations.
          We will share the training configuration for each setting.
          For the details of the evaluation practice, please refer to the code repository of~\cite{caron19unsupervised}\footnote{\url{https://github.com/facebookresearch/DeeperCluster}}.
    \item Extracting image features from the last convolutional layer of ResNet50 backbones and training linear SVMs and logistic regression classifiers using these pre-extracted features.
\end{enumerate}
Note that in both cases, backbones are frozen.

\mypar{Feature extraction}
To extract image features, we resize images such that their smallest dimension is $224$ pixels, then apply central-crops of size $224 \times 224$.
This gives us $7 \times 7 \times 2048$-dimensional visual tensors output for ResNet-50 backbones.
For training SVMs on VOC and COCO, following~\cite{goyal19scaling}, we apply $2 \times 2$ spatial average pooling and flattening to obtain 8192-dimensional visual features, then $\ell_2$-normalize the features.
However, storing and training classifiers on 8192-dimensional features for the 1.28M images of the IN-1K dataset was computationally challenging.
Therefore, for training logistic regression classifiers on IN-1K, we apply global average pooling and obtain 2048-dimensional visual features.

\mypar{SVM classifiers}
Following the convention of~\cite{goyal19scaling}, we train linear SVMs to evaluate visual representations on the 2007 split of Pascal-VOC and the 2017 split of MS-COCO datasets, in a one-\vs-all manner.
Please refer to~\cite{goyal19scaling} for details in training binary SVMs.
Different from~\cite{goyal19scaling}, we tune the cost parameter of SVMs by sampling 40 cost values log-uniformly between $10^{-5}$ and $10^5$ and find the optimal value by Optuna~\cite{optuna2019}.

\mypar{Logistic regression classifiers}
We train linear logistic regression classifiers by performing SGD updates with momentum $0.9$ and batch size $1024$.
We validate the learning rate and weight decay hyper-parameters using Optuna~\cite{optuna2019} over 25 trials.
We log-uniformly sample learning rates between $10^{-1}$ and $10^{2}$, and apply cosine-based learning rate annealing, whereas we uniformly sample weight decays between $0$ and $10^{-5}$.

\section{Additional qualitative results}%
\label{sec:exp_vis}%

In Figs.~1 and~3 of the main paper, we show attention maps produced by our {ICMLM}$_\text{\tfm}${} model (\texttt{tfm}{} module contains 1 hidden layer and 1 attention head) with ResNet-50 backbone trained on COCO.
This section provides additional attention maps obtained by the \texttt{att}{} module in our {ICMLM}$_\text{\att-\fc}${} model (\texttt{fc}{} and \texttt{att}{} modules contain 1 hidden layer and 12 attention heads, respectively) with VGG16 backbone trained on COCO.
These maps are shown in Figs~\ref{fig:attention_1} and~\ref{fig:attention_2}.

\begin{figure*}[t!]
   \includegraphics[width=\linewidth]{res/qualitative_supplementary_1.pdf}
   \caption{\textbf{Qualitative results.}
            For several image-caption pairs of the validation set of the COCO dataset and for a masked token, we show the ground-truth label (GT) together with the top 3 predictions (Pred) and the attention map generated by our {ICMLM}$_\text{\att-\fc}${} model with VGG16 backbone.
            The red parts correspond to higher attentions.}
   \label{fig:attention_1}
\end{figure*}

\begin{figure*}[t!]
   \includegraphics[width=\linewidth]{res/qualitative_supplementary_2.pdf}
   \caption{\textbf{Qualitative results.}
            For several image-caption pairs of the validation set of the COCO dataset and for a masked token, we show the ground-truth label (GT) together with the top 3 predictions (Pred) and the attention map generated by our {ICMLM}$_\text{\att-\fc}${} with VGG16 backbone.
            The red parts correspond to higher attentions.}
   \label{fig:attention_2}
\end{figure*}

First, we see from the figures that the \texttt{att}{} module can successfully localize object categories that have a clear visual appearance.
This is the case for instance of the banana, the baby, the cats, or the sheep from Fig.~\ref{fig:attention_1}.
This is also the case even in cluttered scenes, such as the bed on the second row of Fig.~\ref{fig:attention_1}.

Second, it is interesting to see that even visual concepts that are more abstract than object categories can also be localized, such as the {\em mirror} or {\em glass}.
In the particular case of the {\em glass} category, the versatility of this concept is successfully captured by our model, covering the drinking glass and the material of the table and of the vase.

Third, the model goes beyond nouns and learns the visual appearance associated to colors or textures.
For instance, the concepts {\em blue}, {\em striped} or {\em colorful} are illustrated in Fig.~\ref{fig:attention_2}.

Finally, we show some failure cases.
This is often the case for ambiguous concepts whose visual appearance is not properly defined, such as {\em middle} and {\em open} which are respectively illustrated in the bottom right of Fig.~\ref{fig:attention_1} and Fig.~\ref{fig:attention_2}.
In some extreme cases, the attention maps are meaningless, and the masked word prediction relies on the rest of the caption instead.
An other failure case is the bottom left of Fig.~\ref{fig:attention_2} which shows that grouping several concepts (like the different colors of the three shirts) is still way beyond the capacity of the ICMLM model.

\section{Transformer network in ICMLM}%
\label{sec:model_tfm}%

This section extends Sec.~3.2 of the main paper and describes in detail the transformer encoder layer~\cite{Vaswani2017AttentionIA} in our {ICMLM}$_\text{\tfm}${} model.

In {ICMLM}$_\text{\tfm}${}, we use the multi-headed attention network proposed in~\cite{Vaswani2017AttentionIA} in order to contextualize the token embeddings computed by BERT$_\text{base}${} model, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~$\W_i \in \Real{T \times \dw}$, among the visual features mapped to the token embedding space of BERT$_\text{base}${}, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~$\bar{\X}_i \in \Real{H \times W \times \dw}$, for the $i$-th data sample.
To do so, in our model, we use 1-layer transformer encoder with 8 attention heads which are computed in parallel.
The transformer encoder takes as input the concatenation of $\bar{\X}_i$ and $\W_i$, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~$Z_i = [ \bar{\X}_i; \W_i ] \in \Real{S \times \dw}$, where $S = (H \times W + T)$ denotes the total number of (visual + textual) tokens.

Each attention head $O_h, h \in {1,\cdots,8}$ in the encoder performs the {\em scaled dot-product attention}~\cite{Vaswani2017AttentionIA} on top of $Z_i$ as follows.
First, 3 linear projections of $Z_i$ are computed:
\begin{equation}
\begin{split}
    K_i^h &= Z_i \Sigma_K^h + b_K^h, \\
    Q_i^h &= Z_i \Sigma_Q^h + b_Q^h, \\
    V_i^h &= Z_i \Sigma_V^h + b_V^h,
\end{split}
\end{equation}
where $K_i^h$, $Q_i^h$ and $V_i^h$ are respectively the keys, queries and values $\in \Real{S \times \dw}$ computed by the attention head $h$.
In this formulation, $\Sigma_K^h$, $\Sigma_Q^h$ and $\Sigma_V^h$ $\in \Real{\dw \times \dw}$ are weight; $b_K^h$, $b_Q^h$ and $b_V^h$ $\in \Real{\dw}$ are bias parameters of the projection layers in $O^h$.
Then the output of each head $O^h (Z_i) \in \Real{S \times \dw}$ is computed using the keys, queries and values defined above:
\begin{equation}
    O^h (Z_i) = \text{softmax}\left( \frac{{K_i^h} {Q_i^h}^\top}{\sqrt{D}} \right) V_i^h.
\end{equation}

 Finally all attention heads are merged simply by concatenating the individual head's outputs, and we compute:
 \begin{equation}
     O(Z_i) = \left[ O^1(Z_i) \, | \cdots | \, O^8(Z_i) \right] \Sigma^O + b^O,
 \end{equation}
 where $\Sigma_O \in \Real{8 \times \dw \times \dw}$ and $b^O \in \Real{\dw}$ are learnable parameters, and $[.|.]$ denotes concatenation.
 The output of the multi-headed attention layer is followed by residual connection~\cite{he2016resnet}, dropout~\cite{srivastava14dropout}, LayerNorm~\cite{ba16layer}, ReLU and linear projection layers to obtain the final output of the transformer.

\section{Zero-shot Object Classification}%
\label{sec:exp_zsl}%

We also extend the analysis in Sec.~4.2 of the main paper on an additional target task, zero-shot image classification, on CUB-200-2011 (CUB)~\cite{wah2011CUB} and Attributes with Animals 2 (AWA2) datasets~\cite{xian2018gbu}.
The CUB dataset contains roughly 12K images for 200 types of {\em fine-grained} bird species defined by 312 different semantic {\em attributes}.
The AWA2 dataset has roughly 38K images for 50 {\em coarse-grained} animals defined by 85 different attributes.
The classes in these datasets are split into two subsets called {\em seen} and {\em unseen} classes.
The goal of these benchmarks is to train a classification model on seen classes in a way that the classification model can effectively be used for both seen and unseen classes.
Using the recently proposed splits~\cite{xian2018gbu}, we have 150 (resp. 40) and 50 (resp. 10) classes in the seen and the unseen splits for CUB (and AWA) datasets.
Image samples from seen classes are divided into training and test sets whereas image samples from unseen classes are solely used for testing purposes.

In this analysis, we take the VGG16 backbones trained by {TP}$_\star${} or \icmlmstar{} models on the MS-COCO~\cite{lin14coco} (COCO) or Visual Genome~\cite{krishna17vg} (VG) datasets.
Similar to what we report in Sec.~4.2 from the paper, using the activations from the last three convolutional layers, we train bilinear score functions~\cite{sariyildiz2019CVPR} that measure the {\em compatibility} between the visual features $\vec{x} \in \Real{m}$ (pooled and flattened to have roughly 9K dimension) and class-level attribute vectors $\vec{a} \in \Real{n}$ ($n$ is 312 for CUB and 85 for AWA).
Concretely, we define the score function as
\begin{equation}
f(\vec{x}, \vec{a}) = \vec{a}^\top (\vec{\Sigma} \vec{x} + \vec{b})
\end{equation}
where $\vec{\Sigma} \in \Real{n \times m}$ and $\vec{b} \in \Real{n}$ are parameters of the score function to be learned.
Using the score function, class predictions are simply made by:
\begin{equation}
\hat{y} = \,\, \argmax_{c \in \mathcal{C}} \,\, f(\vec{x}, \vec{\mathcal{A}}_c),
\end{equation}
where $\vec{\mathcal{A}}_c \in \Real{n}$ denotes the class-level attribute vector for class $c$ and $\mathcal{C}$ is the set of all classes.
We train the score function by minimizing the following:
\begin{equation}
\vec{\Sigma}^\star, \vec{b}^\star = \argmin_{\vec{\Sigma}, \vec{b}} \,\,\, - \E_{(\vec{x}, y) \in \mathcal{D}} \Big[  \log \left( p \left(y | \vec{x} , \vec{\mathcal{A}} \right) \right) \Big],
\end{equation}
where $\mathcal{D}$ is a dataset of feature-label pairs $(\vec{x}, y)$ s.t. $y \in \{ 1, \ldots, \mathcal{C} \}$ and
\begin{equation}
p \left( y = c | \vec{x} , \vec{\mathcal{A}} \right) = \frac{ \exp( f(\vec{x}, \vec{\mathcal{A}}_c) )}{\sum_j \exp(  f(\vec{x}, \vec{\mathcal{A}}_j) )}.
\end{equation}

\input{tab_vgg16_zsl.tex}

\mypar{Results}
\tab{tab:vgg16_zsl} reports top-1 prediction accuracies among all classes for both datasets.
We make the following observations.
\begin{enumerate}[(i), font=\bfseries]
    \item We see that {ICMLM}$_\text{\tfm}${} model trained on VG significantly improves {TP}$_\star${} models on CUB, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~up to $\mathbf{1.4\%}$, $\mathbf{1.3\%}$ and $\mathbf{2.2\%}$ with C-11, C-12 and C-13 features.
          On the other hand, {ICMLM}$_\text{\tfm}${} model trained on COCO improves {TP}$_\star${} models on AWA2 up to $\mathbf{1.1\%}$, $\mathbf{0.9\%}$ and $\mathbf{1.0\%}$ with C-11, C-12 and C-13 features.
          In fact, \icmlmstar{} models tend to perform slightly better on AWA2 (particularly with C-13 evaluations), when they are pretrained on COCO indicating that the concepts in COCO are semantically more similar to the concepts in AWA2.
    \item When trained on VG, the C-13 features learned by {ICMLM}$_\text{\att-\fc}${} are inferior to {TP\textsubscript{Postag}}{}, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot~the scores drop up to $0.9\%$.
          This implies that the VGG16 backbone trained by {ICMLM}$_\text{\att-\fc}${} slightly overfits to MLM task.
          However, the opposite is true for the C-11 and C-12 features, suggesting that the network is able to extract richer semantics from the earlier layers.
\end{enumerate}