\section{Introduction}

Reinforcement learning (RL) allows a robot to learn in an environment directly from experience, without knowledge of the environment dynamics or rewards. However, theoretical results indicate that the sample complexity of RL scales poorly unless strong assumptions on the environment are made ~\cite{RL_Hard}. Poor sample complexity is particularly undesirable for robotics and other domains where interaction with the environment may be slow, expensive, difficult, or dangerous.

In contrast, imitation learning provides a more sample efficient alternative to learning from experience, whereby the robot learns to emulate optimal behavior conveyed through expert demonstrations~\cite{IL_Easy}. One approach to learning from demonstrations is behavioral cloning (BC) which is a supervised approach in which the robot learns to predict what action the expert would take at each state~\cite{bain1995framework,daftry2016learning}.

Progress in imitation learning has enabled robots to perform a diverse array of complex tasks~\cite{chen2017end,giusti2015machine,abbeel2010autonomous}. However, most existing methods assume that the robot observes the expert's actions as part of the demonstrations. This assumption imposes a protocol on the process of collecting demonstration data and precludes the use of a large amount of crowd-sourced demonstration data that provides only observations, such as YouTube videos. This has become particularly limiting in recent years due to the advantages and, in turn, the prevalence of neural network-based representation learning, which has proven successful for complex supervised and reinforcement learning tasks \cite{deng2009imagenet,mnih2015human,gu2017deep}. For example, \citet{smith2019avid} uses CycleGAN to transform videos of human demonstrations into robot demonstrations that are easier to imitate.

To take advantage of this information, a more general problem of imitation learning from observation (ILfO) must be considered. \citet{1805.01954} introduced a generalization known as behavioral cloning from observation (BCO) to address this problem. By first learning how to model the dynamics from observation, the robot can infer what actions to take by using the inverse dynamics model to assign control information to the demonstration data set.

The original implementation of BCO requires a large number of prior interactions in the environment to learn the inverse dynamics model. This inefficiency happens because a random policy is used to collect the data. In the first part of this paper, we take a closer look at BCO and show that when interpreted as a semi-supervised learning problem, we can provide guarantees on the ability of BCO to imitate the demonstrations. This suggests a change to the original implementation that we refer to as BCO*. BCO* eliminates the random policy step and replaces it with concurrent training of the inverse dynamics model and policy. In the second part, we provide an empirical examination in both discrete and continuous domains to verify the applicability of our proposed alteration to BCO.

\section{Related Work}

Behavioral cloning from observation is a model-based imitation learning method centered around combining an inverse dynamics model with behavioral cloning to learn an imitation policy. \citet{1805.01954} shows that this approach can match the performance of behavioral cloning using only unlabeled samples. The algorithm works in two stages. First, an inverse dynamics model is learned by allowing the robot to execute a random policy in the environment~\cite{1805.01954}. Next, the inverse dynamics model is used to label the demonstrations. Similar to this \citet{1805.07914} proposed a model-based ILfO method that allows robots to learn a latent policy as a form of unsupervised learning before interacting with the environment which is then mapped to an explicit policy.

Another approach to imitation learning focuses on getting the robot to directly reproduce the state-distribution of the expert. One of the first distribution difference minimization algorithms was generative adversarial imitation learning (GAIL)~\cite{1606.03476}. In analogy with GANs, GAIL focuses on learning a discriminator that is capable of distinguishing expert demonstrations from robot trajectories while simultaneously training the robot to avoid detection by the discriminator~\cite{1406.2661}. Value-DICE and discriminator actor-critic (DAC) built on this work and provide sample-efficient alternatives to minimize the GAIL objective~\cite{kostrikov2019imitation,kostrikov2018discriminator}.

More recently, GAIfO and internal-disagreement minimization have been introduced as an extension of GAIL to the ILfO setting~\cite{Torabi_GAIL_State_Only, yang2019imitation}. \citet{yang2019imitation} also compare their internal-disagreement algorithm against a version of BCO with concurrent training. However, because they start with a random policy the robot fails to perform well. In this work, we show that giving the robot a better initialization leads to better performance. From this result, we propose a method that eliminates the need to execute a random policy and reduces the sample complexity of BCO by a significant amount.

Because the inverse dynamics model is used to assign pseudo-labels to the unlabeled demonstrations dataset, BCO can be seen as a semi-supervised imitation learning algorithm. The advantage of this perspective is that there is a rich literature on improving pseudo-labeling. Recent advances in semi-supervised learning have been able to achieve state-of-the-art results using entropy-based methods and consistency training, which works by forcing classifications to be invariant to noise~\cite{berthelot2019mixmatch,xie2019unsupervised}.

It is also worth mentioning that our basic approach relies on bounding the state distribution difference between the expert and the robot. This is similar in spirit to the noise injection approach introduced by \citet{DART} that is used to alleviate the off-policy problem in imitation learning. In this approach, a noise parameter is tuned so that the demonstrator state distribution maximally overlaps with that of the robot. In this work, we merely assume the existence of a ``noisy'' expert to mitigate the off-policy problem and do not optimize this parameter. However, we are still see significant improvement in the resulting policies.

\section{Preliminaries}

\subsection{Notation}

We briefly review the setup for behavioral cloning from observation. Following \citet{1805.01954}, we work within a Markov decision process (MDP) described as $M = \lbrace S, A, T, r, \gamma \rbrace$, where $S$ is the robot's state space, $A$ is the robot's action space, $T = P(s_{t+1} \vert s_t , a)$ is the transition function that specifies the likelihood of transitioning to state $s_{t+1}$ when executing action $a$ in state $s_t$, $r: S \times A \to \mathbb{R}$ is the reward function, and $\gamma$ is the discount factor. According to this framework, we seek to learn a policy function $\pi: S \to A$ such that the discounted sum of rewards $\sum_{t = 1}^{\infty} \gamma^t r$ is maximized.

In the case of BCO, we're also interested in the inverse dynamics model, $M = P(a \vert s_t, s_{t+1})$ that tries to assign the most likely action given a state transition. In imitation learning, we usually assume that we have no access to the reward function. Instead, we have access to a set of demonstrations
\begin{equation}
    D_\text{demo} = \{ \zeta_1, \zeta_2, \ldots \},
\end{equation}
where $\zeta_i$ is a sequence of state-action pairs $\{ (s_0,a_0), \ldots, (s_N,a_N) \}$. However, imitation learning from observation assumes no access to action labels, and thus$\zeta_i = \lbrace s_0, \ldots, s_N \rbrace$.

\subsection{The Off-Policy Problem}

The goal of imitation learning is to emulate the expert sufficiently so that the robot attains similar or better rewards compared to the demonstrations. In general, this problem can be quite difficult when there are certain actions the robot could take, such as falling off a cliff, that eliminate the chance of any future reward.

Let $J(\pi^*)$ be the cost incurred by the expert policy $\pi^*$, then we can estimate the cost incurred by an empirical policy $\hat \pi$ in terms of it's $0$-$1$ error $\epsilon$. We assume the policy is deterministic and write $\pi_s$ to denote the distribution of actions over the current state $s$. We let $T$ refer to the task horizon. We also define $C(s,a)$ as $C: S \times A \to [0,1]$ to be the immediate cost of taking an action $a$ in the state $s$. In the imitation learning setting, we are generally interested in the quantity $l(s,a) = \mathbbm{1} \{ a \not = \pi^*_s \}$, the occurrence of events where an action is not aligned with the expert's policy. This is a $0$-$1$ loss. This quantity is usually quite difficult to minimize directly so instead, we'll minimize the proxy $C$.

We are interested in the state distribution $d_{\pi}^t$ at time $t$ assuming we follow a policy $\pi$ from the initial time step. We denote the expected $T$-step cost of executing $\pi$ with $J(\pi) = T \mathbb{E}_{s \sim d_{\pi}}[C_{\pi}(s)]$. Given this, we can look at the expected $t$-cost of taking an action under a policy as the Q-function $Q_t^{\pi}(a,s)$. Here we state Theorem~2.2 of \citet{behavior_cloning},

\begin{theorem}
\label{Ross Theorem}
Let $\pi$ be such that $\mathbb{E}_{s \sim d_{\pi}}[l(s,\hat \pi)] = \epsilon$ and $Q_{T-t+1}^{\pi^*}(s,a) - Q_{T-t+1}^{\pi^*}(s,\pi^*) \le \frac{1}{2} u$ for all times and actions, where $d_{\pi}^t(s) > 0$, then $J(\pi) \le J(\pi^*) + \frac{1}{2} u T \epsilon$
\end{theorem}

For convenience, the proof is also provided in the supplementary materials. When there are certain actions that have a permanent impact on the robot's ability to emulate the expert $u \sim T$ and the robot could experience quadratic divergence from the expert policy's over time. On the other hand, if the demonstrations are noisy, but have comparable cost to the perfect expert then it follows that $u \ll T$. Thus, whenever we can inject noise into the expert policy with minimal change to quantities such as total return we infer that $u$ is small.

\section{Theoretical Analysis}

\subsection{Assumptions}

In order to proceed with the analysis, it will be beneficial to explicitly discuss what assumptions are needed. Broadly, we assume that the expert policy and inverse-dynamics models are learnable with reliable sample complexity, that the demonstrator is an $\epsilon$-expert, we have an arbitrarily large number of unlabeled demonstrations, and that mistakes are recoverable.

\subsubsection{Recoverability}

Say we collect a demonstration dataset $D_\text{demo}$ from our expert. The state-action pairs will follow some distribution $d_{\pi}(s,a)$ for any given $\pi$. We assume the experts are \textit{not} perfect.

\begin{definition}

A $\epsilon$-expert policy is an expert policy $\pi^*$ that each action it outputs is replaced to a different random action with probability $\epsilon$.

\end{definition}

If $\epsilon$ is small, then Theorem~\ref{Ross Theorem} would suggest that the performance of the $\epsilon$-expert policy and that of the expert to be close. We can also estimate the relative probability that a state transition $(s,s')$ comes from $\pi^*$ or $\pi$. Since this quantity is bounded between zero and one, relative probability satisfies the conditions of Theorem~\ref{Ross Theorem} as well.

\begin{assumption}
We assume that the demonstrators consist of $\epsilon$-experts.
\end{assumption}

This implies that mistakes can be recovered from. This implies that $u \ll T$ in Theorem~\ref{Ross Theorem}. If the action space is bounded, but continuous, then we further assume that,

\begin{assumption}
Let $\pi$ be such that $\mathbb{E}_{s \sim d_{\pi}}[l(s,\hat \pi)] = \epsilon$. Assume that,
\begin{align}
\mathbb{E}[Q_{T-t+1}^{\pi^*}(s,a) - Q_{T-t+1}^{\pi^*}(s,\pi^*)] \le \frac{1}{2} u \epsilon
\end{align}
for all times and actions where $d_{\pi}^t(s) > 0$.
\end{assumption}

Applying this assumption to Theorem~\ref{Ross Theorem} (Eqn.~\ref{eqn:cost-empirical}) yields a generalized version of the theorem.

\subsubsection{Learnability}

The generalization capability of the robot's policy is directly tied to the model class used to learn the inverse and policy functions. We prefer to use a large function class, such as those learned by neural networks, but the generalization bounds generated using standard VC-dimension arguments are poor~\cite{VC_Bound}. However, recent work has shown that as the width of a neural network's hidden layers grows to infinity, the function approximations converge to a Gaussian process \cite{NTK_Gaussian_Process}. Thus, we focus on kernel methods. Since the specific choice of the kernel is a detail, we'll assume that generalization under the 0-1 loss $L_D^{0-1}(\theta)$ for model parameters $\theta$ can be bounded in terms of an appropriate complexity measure.

\begin{assumption}
\label{NTK Generalization}
Let $\bm{y} = (y_1, \ldots, y_n)^T$. Assume our model class is such that with probability at least $1 - \delta$ the output of our algorithm satisfies,
\begin{align}
\mathbb{E} \left[L_{D}^{0-1}(\bm{\theta}) \right] \le O \left( \sqrt{\cfrac{K(\bm{y})}{n}} + \sqrt{\frac{\log(1/\delta)}{n}}\right)
\end{align}
where $K$ is a kernel-based complexity measure of the data and $n$ is the training set size.

\end{assumption}

For kernel methods, we have $K = \bm{y}^\top (\Theta^{(L)})^{-1} \bm{y}$ where $K$ is a measure of the complexity for the learning problem. Essentially, if our target function $y = f^*(\bm{x})$ has a bounded norm in the induced reproducing kernel Hilbert space (RKHS), then we can expect our algorithm to generalize well.

\begin{assumption}
\label{inverse_learn}
We assume that the true policy and inverse dynamics model being learned have bounded RKHS norm.
\end{assumption}

We denote these quantities by $\lVert \pi^* \rVert_{K}$, $\lVert M^*\rVert_{K}$, or $K$ when it is ambiguous. Finally,

\begin{assumption}
We have access to many more unlabeled demonstrations than labeled demonstrations.
\end{assumption}

All experts are $\epsilon$-experts having an effectively unlimited number of demonstrations. Thus, there will always be unlabeled data showing how to recover from a mistake. A general goal of ILfO is to be able to work with large sets of unlabeled data so we think this simplification is motivated.

\subsection{On-Policy Bound}

If we have a cost-function that measures the relative un-likelihood of state-action pairs $(s,a)$ and we know that the expert is able to recover from mistakes then the amount of time spent off-policy will be relatively small in comparison to the horizon of the task. The assumption that matching the state-action distribution of the expert is enough to guarantee good performance on the task is quite similar to the prevailing assumption underlying model-free adversarial methods to imitation learning \cite{divergence_IL}. Moreover, by assumption \ref{inverse_learn}, looking at state transitions is enough to infer state-action pairs. In this way, we can avoid having to make use of action information in order to evaluate likelihoods.

\begin{theorem}
\label{Sample Ratio}
Say $J(\pi)$ indicates the expected number of samples labeled as ``non-expert" from the states induced by $\pi$ relative to $\pi^*$. Use $S(\pi) = T - J(\pi)$ to indicate the expected number of samples labeled as ``expert''. Then we have,

\begin{align}
    \frac{S(\pi^*)}{S(\pi)} \le  \cfrac{1}{1-u\epsilon}
\end{align}
\end{theorem}

\begin{proof}
First, note that $J(\pi^*) = T/2$ and by theorem \ref{Ross Theorem} we have,
\begin{subequations}
\begin{align}
    & J(\pi) \le J(\pi^*) + \frac{1}{2} u T \epsilon
\\
    & \Rightarrow S(\pi^*) \le S(\pi) + \frac{1}{2} u T \epsilon
\\
    & \Rightarrow S(\pi^*) - \frac{1}{2} u T \epsilon \le S(\pi)
\\
    & \Rightarrow \cfrac{S(\pi^*)}{S(\pi)} \le \cfrac{S(\pi^*)}{S(\pi^*) - \frac{1}{2} u T \epsilon} = \cfrac{T/2}{T/2 - \frac{1}{2} u T \epsilon} = \cfrac{1}{1 - u \epsilon}
\end{align}
\end{subequations}
\end{proof}

Thus, as long as $\epsilon < 1/u$, we can expect to collect samples that are on policy. Under reasonable assumptions, this is enough to show good properties for BCO. However, if $\epsilon \to 1/u$ the multiplier on the sample complexity will diverge. We introduce a definition to distinguish this case,

\begin{assumption}
The $\epsilon$-expert is said to be recurrent if $\epsilon < 1/u$.
\end{assumption}

\subsection{Iterative BCO}

Our modification of BCO works as follows: start with a small data set, learn an initial policy, and then use that policy to collect data and improve the inverse dynamics model. Next, use the inverse dynamics model to label the demonstrations and improve the policy. Iterate until the policy stops improving. This is an extension of the BCO implementation used by \citet{yang2019imitation}.

\begin{algorithm}
\caption{BCO$^*$}\label{BCO_Algorithm}
\begin{algorithmic}[1]
\State $\text{Initialize the inverse dynamics model } M$
\State $\text{Set } D = \lvert D_0 \rvert $ with $\text{buffer-size} = N$
\While {$\text{policy improvement}$} \do \\
\State Improve $M$ by modelLearning($D$)
\State Use $M$ to compute label($D_\text{demo}$)
\State Improve $\pi$ using $D_\text{demo}$ with labels
\For {time-step $t = 1$ to $N$} \do \\
\State Generate samples $(s_t,a_t,s_{t+1})$ using $\pi$
\State Append to $D$
\EndFor
\EndWhile
\end{algorithmic}
\end{algorithm}

Assume that we have labeled demonstrations $D_0$ that we can use to create an initial policy $\pi_0$ and inverse dynamics model $M_0$ such that $\lvert D_0 \rvert \ll \lvert D_\text{demo} \rvert$. In other words, we have an incomplete demonstration dataset that outputs a policy $\pi_0$ that is essentially unusable. More formally, by Theorem~\ref{Sample Ratio}, if the error rate is very close to $1/u$, then $\pi_0$ will be labeled as spending most of the time off-policy. However, as long at the policy is recurrent, it seems reasonable that we could collect enough rollouts from $\pi_0$ to construct a new labeled data-set $D_1$ to improve the inverse dynamics model.

\subsubsection{Convergence to Behavioral Cloning}

Since we can control how many samples we get using $\pi_0$ we can control how close our training data-set for the inverse dynamics model will be to the optimal policy. At each iteration we get a new policy and a new data set,

\begin{equation}
    \pi_{n+1}, D_{n+1} = \text{BCO}_{\pi_n}(D_\text{demo},D_n)
\end{equation}

We will spend the rest of this section trying to make these intuitions precise using Theorem~\ref{NTK Generalization} to analyze convergence of BCO. The basic claim is that for a range of $D_0$, we have as $n \to \infty$,
\begin{equation}
    \text{BCO}^{(n)}(D_\text{demo}) \to \text{BC}(D_\text{labeled \ demo})
\end{equation}

It is worth pointing out the ambiguity regarding whether or not to train the initial policy using BC or BCO. In general, if the policy is easier to learn than the inverse dynamics model, then the initial policy should be made using BC.

\subsection{BCO vs.\ BC}

Although the inverse dynamics model is assumed to be learnable, it might be the case that the inverse dynamics model is simply harder to learn than the policy. Moreover, we need enough initial labeled samples to ensure our initial policy is recurrent. We can formalize this as,

\begin{theorem}
Say we can use $D_0$ to create a recurrent $\epsilon_0$-expert policy. If we create $D_1$ by observing how this policy operates in the environment, we can then learn an $\epsilon_1$-expert policy with probability $1-\delta$ if,
\begin{equation}
    \lvert D_1\rvert \sim O\left( \cfrac{\lVert M^* \rVert_K + \log(1/\delta)}{\epsilon_1^2 (1 - u \epsilon_0)} \right)
\end{equation}
\end{theorem}

In order to make use of the unlabeled data set we will have to create an inverse dynamics model by rolling-out trajectories from the initial policy. We can compare the efficiency of this approach to using BC loosely by comparing error rates,
\begin{subequations}
    \begin{align}
        & \sqrt{\cfrac{2 \lVert M^* \rVert_K}{D_0+D_1 (1-u \epsilon_0)}} = \sqrt{\cfrac{2 \lVert \pi^* \rVert_K}{D_\text{BC}}}\\
        %
        & \Rightarrow D_\text{BC} = (D_0+D_1(1-u\epsilon_0)) \cfrac{\lVert \pi^* \rVert_K}{\lVert M^*\rVert_K}
    \end{align}
\end{subequations}

Thus, the quality of the initial policy and the relative difficulty of learning the policy compared to the inverse dynamics model controls the efficiency of BCO over BC. One limitation of this analysis is that if our initial policy is only barely recurrent then it will take a very large number of samples to train. In fact, \citet{1805.01954} skips the single iteration case and gives a definition of BCO$(\alpha)$ which allows the robot to periodically train during a post-demonstration phase. This suggests that continual learning would be a better strategy. In fact, we'll show that the continual aggregation approach can dampen the impact of a poor initial policy,

\begin{theorem}
    \label{ode}
    Assume the initial policy output by BCO($\alpha$) $\pi_0$ is a recurrent $\epsilon_0$ policy. Following this we then periodically improve the inverse dynamics model as we obtain data $D_1$ from using the policy. Then the sample complexity $\lvert D_0 \rvert + \lvert D_1 \rvert$ to achieve $\epsilon$ error is,

    \begin{equation}
        \lvert D_0 \rvert + \lvert D_1 \rvert \sim O \left( u^2 \lVert M^* \rVert_K \log\left( \cfrac{1/\epsilon - u}{1/\epsilon_0 - u} \right) + \lVert M^* \rVert_K/\epsilon^2 \right)
    \end{equation}
\end{theorem}

The proof is given in the supplementary material, and involves taking a continuum limit and then solving the resulting ODE.

\section{Experiments}

In this section, we try to verify our analysis by seeing whether or not BCO* is able to improve in performance over successive iterations. For each of the simulated environments, we train experts using Soft-Actor-Critic \cite{brockman2016openai, haarnoja2018soft}. The experts are trained using Stable Baselines with the default hyper-parameter settings \cite{stable-baselines}. Using these trained experts, we generate $30$ demonstrations for the Mountain Car, Ant, and Half-Cheetah environments using Gym and the PyBullet physics simulator \cite{gym,coumans2017pybullet}.

The BC and BCO models are two-layer ReLU MLP networks with layer dimensions of 300 and $200$ respectively. We train the model, hold out $30\%$ of the data for validation, and use checkpoints to select the model with the best validation loss. To evaluate BCO*, we first use a small number of expert demonstrations to generate an initial policy. We then evaluate the policy for 25 episodes and use the resulting data to improve the inverse dynamics model. We choose a larger iteration size for BCO* than Theorem \ref{ode} implies is optimal in order to reduce variance in the training. Note this buffer is far smaller than \citet{1805.01954} used in BCO($\alpha$). Each evaluation of BCO* is averaged over five separate runs of the training process, each with different random seeds.

For comparisons, we look at the original implementation of BCO that uses random actions to learn the initial inverse dynamics model. Next, we compare against behavioral cloning (BC) trained using 30 demonstrations. We also compare against GAIL and Value-DICE, a newer algorithm that is meant to address the sample complexity issues of GAIL by directly matching the distribution of the robot with that of the expert \cite{kostrikov2019imitation}. Because BC will learn a policy given enough demonstrations, we train Value-Dice using 3 and 15 episodes from Ant and Half Cheetah respectively to compare distribution-matching with the label inference of BCO*. For each comparison, we average over three runs.

\begin{figure}
  \centering
  \includegraphics[width=1.0\textwidth]{./Figures/BCO_Final_1.png}
  \caption{A visualization the evolution of performance during training for different numbers of demonstrations. Performance is measured as the normalized $[0,1]$ reward, where $0$ and $1$ are the mean rewards of random and expert policies, respectively. Error bars represent standard deviation of performance across multiple runs.}
  \label{fig:BCO_Results}
\end{figure}

\subsection{Mountain Car}

For Mountain Car, we use a noisy expert with $\epsilon = 0.01$. However, we note that there is no significant effect on BCO other than a decrease in expert performance. We collect $25$ rollouts in each iteration step and use them to improve the inverse dynamics model. We train all the models at each step for $5000$ epochs and evaluate using $500$ episodes.

\subsection{Ant / Half Cheetah}

For Ant, we use a noisy expert with isotropic Gaussian noise with variance $0.04$. We collect $25$ rollouts from the robot at each iteration step and use them to improve the inverse dynamics model. Besides the initial model, we train all of the models for $50$ epochs to increase efficiency and to avoid the generalization problems with respect to the inverse dynamics model. We evaluate using $50$ episodes at each iteration.

\subsection{Results}

 Results are shown in Figure~\ref{fig:BCO_Results}. In the case of Mountain Car, the robot converges to near expert-level performance relatively quickly. For Ant, our robot is able to improve over the iterations and to near expert performance by three episodes. In Half Cheetah, the robot is able to perform comparably to the noisy expert. In general, for each of these environments, the performance increases with more iterations and/or demonstrations.

Adding noise to the expert is somewhat essential to our results. If we retrain the robot using noiseless-demonstrations (Fig.~\ref{fig:Perfect_Ant}), the convergence behavior is dampened. Finally, we can compare BCO* against our baselines. We see that BCO* has comparable performance with Value-DICE and BC while GAIL and BCO are too sample inefficient to learn a policy. GAIL, and also GAIfO, can ultimately learn a good policy, however, they are not sample efficient and struggle to learn anything using only 1e6 steps of environment interaction. BCO with a random policy step is also sample inefficient. Typically, we need an order of magnitude more interactions before the random policy will learn a reasonably accurate inverse dynamics model \cite{1805.01954}.

\begin{figure}
    \centering
  \includegraphics[width=\textwidth]{./Figures/Final_BCO_Plot_6.png}
    \caption{On the left we have runs of BCO where the robot learns from an expert without noise. The x-axis indicates the number of trajectories used for training. On the right we have comparisons against other algorithms using labeled data.}
    \label{fig:Perfect_Ant}
\end{figure}

\section{Discussion}

In this work, we proved that performing BCO* with concurrent training leads to faster improvement under relatively mild assumptions. Our experiments confirm that BCO* is capable of self-improvement in both the idealized setting, where the initial policy is close to the expert and realistic settings, where the initial policy is much farther away. On the other hand, we find that the inverse dynamics model is not always learnable to perfect accuracy, which is in line with results found by \citet{yang2019imitation}. However, this is a limitation of any learning from observation approach.

Despite these challenges, we think this improvement to BCO offers an alternative paradigm for robot self-improvement. In the real world, labeled data is scarce and expert demonstrations without mistakes are rarer still. We think that these results show the promise of reformulating imitation learning from observation in terms of semi-supervised learning. Up to this point, BCO required an order of magnitude more environment interactions to learn the inverse dynamics model. In addition to being sample inefficient, it isn't always possible to safely execute a random policy in real environments. By introducing a small amount of labeled data we are able to dramatically reduce the sample complexity and to match the performance of behavioral cloning using more than twice as many labels.

In future work, modifying the inverse dynamics model learning algorithm in order to take advantage of advances in semi-supervised learning could be fruitful~\cite{berthelot2019mixmatch,xie2019unsupervised}. Another avenue for future work is exploring the combination of BCO with RL approaches to ILfO. Methods such as reinforced inverse-dynamics modeling explore this direction with good success~\cite{pavse2019ridm}.

\clearpage
\acknowledgments{The author would like to thank Chip Schaff for fruitful discussion on experimental details and reviewing and Falcon Dai for technical discussion regarding theoretical results.}


\section{Introduction}\label{sec:intro}
Texture is ubiquitous and provides useful cues for a wide range of
visual recognition tasks.
We rely on texture for estimating material properties of surfaces,
for discriminating objects with a similar shape, for
generating realistic imagery in computer graphics applications, and so on.
Texture is localized and can be more easily modeled than shape that is
affected by pose, viewpoint, or occlusion.
The effectiveness of texture for perceptual tasks is also mimicked by
deep networks trained on current computer vision datasets that have
been shown to rely significantly on texture for
discrimination
(\eg,~\cite{geirhos2018imagenet,cimpoi2015deep,brendel2019approximating,hosseini18assessing}).

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{figs/data_example.pdf}

\caption{We introduce the Describable Textures in Detail Dataset
  (\dtdd) consisting of texture images from DTD~\cite{cimpoi14describing} with natural language
  descriptions, which provide rich and fine-grained supervision for
  various aspects of texture such as color compositions, shapes, and
  materials. More examples are in the Supplementary material.
  }
\label{fig:data}
\end{figure}

While there has been significant work in the last few decades on
visual representations of texture, limited work has been done on
describing detailed properties of textures using natural language.
The ability to describe texture in rich detail can enable applications
on domains such as fashion and graphics, as well as to interpret
discriminative attributes of visual categories within a fine-grained taxonomy (\eg, species of
birds and flowers) where texture cues play a key role.
However, existing datasets of texture (\eg,~\cite{cimpoi14describing,bell2015material}) are limited to a
few binary attributes that describe patterns or materials, and do not
describe detailed properties using the compositional nature of language (\eg, descriptions of the
color and shape of texture elements).
At the same time, existing datasets of language and
vision~\cite{VQA, lin2014microsoft, sharma2018conceptual,  plummer2015flickr30k, kazemzadeh2014referitgame, mao2016generation, yu2016modeling} primarily focus on objects and their
relations with a limited treatment of textures (Section~\ref{sec:related}).
Addressing this gap in the literature, we introduce a new dataset containing natural
language descriptions of textures called the Describable Textures in Detail Dataset
(DTD$^2$).
It contains several manually annotated descriptions of each image from the Describable
Texture Dataset (DTD)~\cite{cimpoi14describing}.
As seen in Figure~\ref{fig:data}, these contain descriptions of colors of the
structural elements within the texture (\eg ``circles'' and ``stripes''), their shape, and other
high-level perceptual properties of the texture (\eg ``soft'' and ``protruding'').
The resulting vocabulary vastly extends the 47 attributes present in
the original DTD dataset (Section~\ref{sec:dataset}).

We argue that the domain of texture is rich and poses many
challenges for compositional language modeling that are present in
existing language and vision datasets describing objects and scenes.
For example, to estimate the color of dots in a dotted texture the
model must learn to associate the color to the dots and not to the
background.
Yet the domain of texture is simple enough that it allows us to
analyze the robustness and generalization of existing
vision and language models by synthetically generating variations of a
texture.
We conduct a systematic study of existing visual representations of
texture, models of language, and methods for matching the two domains
on this dataset (Sections~\ref{sec:method}, \ref{sec:exp} and \ref{sec:analysis}).
We find that adopting pre-trained language models significantly improve
generalization.
However, an analysis on synthetically generated variations of each
texture by varying one attribute at a time (\eg, foreground color and
shape) shows that the representations fail to capture detailed properties.

We also present two novel applications of our dataset (Section~\ref{sec:app}).
First, we visualize what discriminative texture properties are learned by
existing deep networks for fine-grained classification on natural domains such as
birds, flowers, and butterflies.
To this end we generate ``maximal images'' for each category by
``inverting'' a texture-based classifier~\cite{lin2018bilinear} and describe
these images using captioning models trained on DTD$^2$.
We find that the resulting explanations tend to be well aligned with the
discriminative attributes of each category (\eg, ``Tiger
Lily'' flower is ``black, red, white, and dotted'' as seen in Figure~\ref{fig:fgvc}-middle).
We also show that models trained on \dtdd offer improvements over
expert-designed binary attributes on the Caltech-UCSD Birds dataset~\cite{WahCUB_200_2011}.
This complements the capabilities of
existing datasets for explainable AI on these domains that focus on
shapes, parts, and their attributes such as color.
Texture provides a domain-independent, albeit incomplete way of
describing interpretable discriminative properties for several domains.

\noindent
In summary, our contributions are:
\begin{itemize}
  \item A novel dataset of
    texture descriptions (Section~\ref{sec:dataset}).
  \item An evaluation of existing models of grounding
    natural language to texture (Section~\ref{sec:method} and
    \ref{sec:exp}).
  \item A critical analysis of these models
    using synthetic, but realistic variations of textures with their
    descriptions (Section~\ref{sec:analysis}).
  \item Application of our models for describing
    discriminative texture attributes and building interpretable models on
    fine-grained domains (Section~\ref{sec:app}).
\end{itemize}
Our dataset and code are at: \url{https://people.cs.umass.edu/~chenyun/texture}.

\section{Related Work}
\label{sec:related}

\para{Language and texture}
Describing textures using language has a long history.
Early works~\cite{amadasun1989textural,tamura1978textural,bajcsy1973computer}
showed that textures can be categorized along a few
semantic axes such as ``coarseness'', ``contrast'', ``complexity'' and
``stochasticity''.
Bhusan~\emph{et al}\onedot~\cite{bhushan1997texture} systematically identified words
in English that correspond to visual textures and analyzed their
relationship to perceptual attributes of textures.
This was the basis of the Describable Texture
Dataset~(\dtd)~\cite{cimpoi14describing} which consolidated a list of 47
texture attributes along with images downloaded from the Internet.
The dataset captures attributes such as ``dotted'', ``chequered'', and ``honeycombed''.
However, it does not capture properties such as the color of
the structural elements (``red and green dots''), or the attributes
that describe the background color.
Our goal is to model the rich space of texture attributes in a compositional manner beyond these attributes.

\para{Datasets of images and text}
The vision and language community has put significant efforts into
building large-scale datasets.
Image captioning datasets such as MS-COCO~\cite{lin2014microsoft},
Flickr30K~\cite{young2014image} and Conceptual
Captions~\cite{sharma2018conceptual} contain sentences describing the
general content of images.
The Visual Question Answering dataset~\cite{VQA} provides language
question and answer pairs for each image, which requires more detailed
understanding of the image content.
In visual grounding datasets such as
RefClef~\cite{kazemzadeh2014referitgame},
RefCOCO~\cite{mao2016generation, yu2016modeling} and Flickr30K
Entities~\cite{plummer2015flickr30k}, detailed descriptions of the
target object instances are annotated to distinguish them from other
objects.
However, these tasks focus on recognizing object categories and
descriptions of pose, viewpoint, and their relationships to
other objects, and have a limited treatment of attributes related to
texture.

\para{Texture representations}
Representations based on orderless aggregations of local features
originally developed for texture has had an significant influence on early
computer vision (\eg,
``Textons''~\cite{leung2001representing},``Bag-of-Visual-Words''~\cite{csurka2004visual},
higher-order statistics~\cite{portilla2000parametric}, and Fisher
vector~\cite{perronnin2010large,sanchez2013image}).
Recent works~(\eg,~\cite{cimpoi2015deep,lin2015bilinear,arandjelovic2016netvlad})
have shown that combining texture representations with deep networks
lead to better generalization on scene understanding and fine-grained categorization tasks.
Even without explicit modeling, deep networks are capable of
modeling texture through convolution, pooling, and non-linear encoding
layers~\cite{gatys2016image}.
Indeed, several works have shown that deep networks trained on
existing datasets tend to rely more on texture than shape for
classification~\cite{geirhos2018imagenet,brendel2019approximating,hosseini18assessing,lin2016visualizing}.
This motivates the need to develop techniques to describe texture
properties using natural language as a way to explain the behavior of deep
networks in an interpretable manner.

\para{Methods for vision and language}
There is a significant literature on techniques for various language and vision tasks.
The Show-and-Tell~\cite{vinyals2015show} model was an early
deep neural net based approach for captioning images that
combined the convolutional image encoder followed by an
LSTM~\cite{hochreiter1997long} language decoder.
Techniques for VQA are based on a
joint encoding of the image and the question to retrieve or generate
an answer~\cite{tan2019lxmert,yu2019deep, kim2018bilinear}.
For visual grounding, where the goal is to identify a region in the image given a ``referring expression'', a common approach is to learn
a metric over expressions and regions~\cite{yu2018mattnet, liu2017referring,plummer2018conditional}.
The basic architectures for these tasks have been improved in a number
of ways such as by incorporating attention
mechanisms~\cite{liu2019improving, anderson2018bottom,wang2019neighbourhood, yu2019deep, gao2019dynamic, kim2018bilinear} and improved
language models~\cite{devlin2018bert,schuster1997bidirectional}.
To model the relation between texture images and their descriptions we investigate a discriminative approach, a metric-learning based approach, and a generative modeling based approach~\cite{xu2015show} on our dataset.

\section{Dataset and Tasks}
\label{sec:dataset}

We begin by describing how we collected \dtdd in
Section~\ref{sec:data-collection}, followed by the tasks and
evaluation metrics in Section~\ref{sec:data-tasks}.
\dtdd contains multiple crowdsourced descriptions for each image
in \dtd.
Each image $I$ contains $k$ descriptions
$\mathbf{S} = \left\{S_1, S_2, \ldots, S_k\right\}$ from $k$ different
annotators who are asked to describe the texture presented in the image.
Instead of providing a grammatically coherent sentence, we found that it more
effective for them to list a set of properties separated by
commas.
Thus each description $S$ can be interpreted as a set of phrases
$\left\{P_1, P_2, \ldots, P_n\right\}$.
Figure~\ref{fig:data} shows some examples of the collected data.
We found that the ordering of phrases in a description is somewhat arbitrary,
which motivates this annotation structure.
Figure~\ref{fig:stats} shows the overall dataset statistics.
\dtdd contains 5,369 images and 24,697 descriptions.
We split the images into $60\%$ training, $15\%$ validation, and
$25\%$ test.
Below we describe details of the dataset collection pipeline and
tasks.

\subsection{Dataset Collection}\label{sec:data-collection}

\para{Annotation}
We present each \dtd image and its corresponding texture category
to $5$ different annotators on Amazon Mechanical Turk, asking them to
describe the texture using natural language with at least $5$ words.
Describable aspects of each image include texture, color, shape,
pattern, style, and material (we provided description examples of several texture
categories in the guidelines).

\para{Verification}
After collecting the raw annotations, we manually verified all of them and
removed annotations that were irrelevant.
For example, a breakfast waffle may have descriptions about the
related food items such as strawberries instead of the texture which is our main goal.
We also removed all images from ``freckled" and ``potholed" categories
because they are primarily of human faces or scenes of roads
with few texture-related terms in their descriptions.
We also excluded images with fewer than $3$ valid descriptions.

\para{Post-processing}
We found that the annotations (as seen in Figure~\ref{fig:data})
describing aspects of texture are often expressed as a set of
phrases separated by commas, instead of a fully grammatical sentence.
We did find some users who provided long unbroken sentences, but
these were few and far between.
Therefore, we represent each description as a set of
phrases indicated by commas (``,") or semicolons (``;'').
For example, the first description of the top-right image in Figure~\ref{fig:data} is: “spiralled, rounded, thick, light colour, rope type”, and it’s split into 5 phrases: “spiralled”, “rounded”, “thick”, “light colour”, “rope type”.
For the purpose of evaluation, we consider words that appear at least
$5$ times and phrases that appear at least $10$ times in the training split of the dataset,
which results in $655$ unique phrases.
Although some long descriptions are lost in the process
and most of the phrases are short (mostly within three words as seen in the lower histogram in Figure~\ref{fig:stats}),
the collection of phrases captures a rich set of describable attributes
for each image.
Modeling the space of phrases poses significant challenges to existing
techniques for language and vision (Section~\ref{sec:analysis}).

\begin{figure}[t]
\centering
\makebox[0pt][c]{\parbox{\textwidth}{%
\begin{minipage}[t]{0.35\hsize}
    \begin{center}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{l|c|c}
        \textbf{Statistics} & overall & frequent\\
        \hline
        \#images & 5369 & -\\
        \hline
        {\#phrases} & 22,435 & 655\\
        \hline
        {\#words} & 7681 & 1673\\
        \hline
        {\#descriptions per image} &  4.60 & -\\
        \hline
        {\#phrases per image} & 16.64 & 11.61 \\
        \hline
        {\#words per description} & 7.13 & 6.69 \\
        \hline
        {\#words per phrase} & 3.93 & 1.19 \\
    \end{tabular}}%
    \end{center}
\end{minipage}
\hfill
\begin{minipage}[t]{0.62\hsize}
    \begin{center}
    \resizebox{\linewidth}{!}{
    \includegraphics[width=\linewidth]{figs_final/stat_n.pdf}
    }
    \end{center}
\end{minipage}
}}
\caption{\textbf{Statistics of \dtdd.}
  The ``overall'' column in the table shows the statistics of all data, while
  the ``frequent" column only considers the phrases (or
words) that occur at least 10 (or 5) times in the training split which
forms our evaluation benchmark.
The cloud of phrases has the font sizes proportional to square-root of
frequencies in the dataset. The vocabulary significantly expands the 47
attributes of DTD.
}
\label{fig:stats}
\end{figure}

\subsection{Tasks and Evaluation Metrics}\label{sec:data-tasks}

The annotation for each image is in the form of a set of descriptions,
with each description in the form of a set of phrases. A phrase is an
ordered list of words.
We consider several tasks and evaluation metrics on this dataset
described next.

\para{Phrase retrieval}
Given an image, the goal is to rank phrases
$p \in {\cal P}$ that are relevant to the image.
Here ${\cal P}$ is the set of all possible phrases, restricted to 655
frequent ones.
For each image, the set of ``true'' relevant phrases are obtained by
taking the union of phrases from all descriptions of the image.
We can evaluate the ranked list using various metrics described as follows:
\begin{itemize}
    \item {Mean Average Precision (MAP):} area under the
      precision-recall curve;
    \item {Mean Reciprocal Rank (MRR):} One over the ranking of first correct phrase;
    \item {Precision at K (P@K):} precision of the top K ranked phrases ($K\in \{5, 20\}$);
    \item {Recall at K (R@K):} recall of the top K ranked phrases ($K\in \{5, 20\}$).
\end{itemize}

\para{Image retrieval from a phrase}  The task is to retrieve images given a
query phrase.
When taking phrases as the query, we consider all phrases $p\in
{\cal P}$ as before and ask the retrieval model to rank all images in the
test or validation set.
The ``true'' list is all images that contain the
phrase (in any of its descriptions).
We consider the same metrics as the phrase retrieval task.

\para{Image retrieval from a description} When using descriptions
the query, we consider all description $s \in {\cal
  S}$ as the input.
Here ${\cal S}$ is the set of all descriptions in the test or
validation set.
We ask the retrieval model to rank all images in the corresponding
set.
We evaluate the rank of the image from which the description was collected
(MRR metric).
This metric allows us to evaluate the compositional properties of texture
over phrases (\eg, ``red dots'' + ``white background'').
While we only quantitatively evaluate phrases and descriptions in the dataset, the ranking models can potentially
generalize to novel descriptions or phrases over the seen words.
We present qualitative results and a detailed study of the models in
Section~\ref{sec:exp} and~\ref{sec:analysis}.

\para{Description generation}
The task is to generate a description for an input image.
Given each image $I$, we compare the generated description against the
set of its collected descriptions $\{S_1, S_2, \ldots, S_k\}$ using standard
metrics for image captioning including
BLEU-1,2,3,4~\cite{papineni2002bleu}, METEOR~\cite{banerjee-lavie-2005-meteor},
Rouge-L~\cite{rouge} and CIDEr~\cite{vedantam2015cider}.
However, we note that the task is open-ended and qualitative
visualizations are just as important as these metrics.

\section{Methods}
\label{sec:method}

We investigate three techniques to learn the mapping between
visual texture and natural languages on our dataset --- a discriminative approach, a metric learning approach, and a language generation approach.
They are explained in detail in the next three sections.

\subsection{A Discriminative Approach}
A simple baseline is to treat each phrase $p \in {\cal P}$ as a binary
attribute and train a multi-label classifier to map the images to
phrase labels.
Given a texture image $I$, let $\psi(I)$ be an embedding computed using
a deep network.
We use activations from layer 2 and layer 4 of
ResNet101~\cite{he2016deep} with mean-pooling over spatial locations
as the image embedding.
A comparison of features from different ResNet layers is included in the supplemental material.
For the classification task, we attach a classifier head $h$ to map the
embeddings to a 655-dimensional space corresponding to each phrase in our
frequent set ${\cal P}$.
The function $h$ is modeled as a two-layer network -- the first is
fully-connected layer with 512 units with BatchNorm and ReLU
activation; the second is a linear layer with 655 units followed by
sigmoid activation.
Given a training set of $\{(I_i, Y_i)\}_{i=1}^N$ where $Y_i$ is the
ground-truth binary labels across 655 classes for image $I_i$, the
model is trained to minimize the binary cross-entropy loss:
$L_{BCE}=\sum_{i} \ell_{bce}(h\circ \psi(I_i), Y_i)$,  where
$\ell_{bce}(y,z) = \sum_i \left(z_i\log(y_i) -
(1-z_i)\log(1-y_i)\right)$.

\para{Training details}
The ResNet101 is initialized with weights pre-trained on ImageNet~\cite{imagenet_cvpr09} and fine-tuned on
our training data
for $75$ epochs using the
Adam optimizer~\cite{kingma2014adam} with an initial learning rate at
$0.0001$. We use image size 224$\times$224 for all our experiments. The hyper-parameters are selected on the validation set.

\para{Evaluation setup}
The classification scores over each
phrase for each image are directly used to rank images or phrases
for phrase retrieval or image retrieval with phrase input.
Retrieving images given a description is more challenging since we
need to aggregate the scores corresponding to different phrases,
and the phrases in input descriptions may not be in ${\cal P}$.
We found the following strategy works well:
Given a description $S = \{P_1, P_2, \ldots, P_n\}$ and an image $I$, obtain the scores
for each phrase $s(P_i) = \sigma(h \circ \psi(I))_k$ where $k$ is the
index of the phrase $P_i \in {\cal P}$. If the phrase is not in the
set, we consider all its sub-sequences that are present in ${\cal P}$ and average the scores of them instead. For example, if the phrase ``red maroon dot'' is not
present in ${\cal P}$, we consider all sub-sequences \{red maroon,
maroon dot, red, maroon, dot\}, score each that is present in ${\cal
  P}$ separately and then average the scores.
By concatenating the top 5 phrases for an image we can also use the
classifier to generate a description for an image.
The key disadvantage of the classification baseline is that it treats
each phrase independently, and does not have a natural way to score
novel phrases
(our baseline using sub-sequences is an attempt to
handle this).

\subsection{A Metric Learning Approach}
The metric learning approach aims to learn a common embedding over
the images and phrases such that nearby image and phrase pairs in the
embedding space are related.
We adopt the standard metric learning approach based on
triplet-loss~\cite{hoffer2015deep}.
Consider an embedding of an image $\psi(I)$ and of a phrase $\phi(P)$ in
$\mathbb{R}^d$. Denote $||\psi(I) - \phi(P)||_2^2$ as the squared
Euclidean distance between the two embeddings.
Given an annotation $(I, P)$ consisting of a positive (image, phrase) pair, we
sample from the training set a negative image $I'$ for $P$, and a negative phrase $P'$ for $I$.
We consider two losses; one from the negative phrase:
$$L_p(I, P, P')=\max(0, 1 + ||\psi(I) - \phi(P)||_2^2 - ||\psi(I) -
\phi(P')||_2^2)$$
and another from the negative image:
$$L_i(P, I, I')=\max(0, 1 + ||\psi(I) - \phi(P)||_2^2 - ||\psi(I') -
\phi(P)||_2^2)$$
The metric learning objective is to learn embeddings $\psi$ and $\phi$
that minimize the loss $L=\mathbb{E}_{(I,P), (I', P')} \left(L_p+L_i\right)$ over the
training set.

For embedding images, we use the same encoder as the
classification approach with features from layer 2 and 4 from ResNet101.
We add an additional linear layer
with 256 units resulting in the embedding dimension
$\psi(I) \in
\mathbb{R}^{256}$.
One advantage of the metric learning approach is that it allows us to
consider richer embedding models for phrases.
Specially we consider the following encoders:
\begin{itemize}
\item \textbf{Mean-pooling}:
$\phi_{mean}(P)=\frac{1}{N_w}\sum_{w\in \texttt{tokenize}(P)}\texttt{embed}(w)$,
where $\texttt{tokenize}(\cdot)$ splits the phrase into a list of
words, $\texttt{embed}(\cdot)$ encodes each token into $\mathbb{R}^{300}$.

\item \textbf{LSTM}~\cite{schuster1997bidirectional}:
$\phi_{lstm}(P)=
\texttt{biLSTM}[\texttt{embed}(w) \text{ for } w \text{ in } \texttt{tokenize}(P)]$,
with the same $\texttt{tokenize}(\cdot)$ and  $\texttt{embed}(\cdot)$
as above. $\texttt{biLSTM}(\cdot)$ is a bi-directional LSTM with a single layer and hidden dimension 256 that returns the
concatenation of the outputs on the last token from both directions.

\item \textbf{ELMo}~\cite{Peters:2018}:
$\phi_{elmo}(P)=\texttt{ELMo}(P)$,
where $\texttt{ELMo}(\cdot)$ uses pre-trained ELMo model~\cite{elmo_wt}
with its own tokenizer,
and outputs the average embedding of all tokens in the phrase $P$.

\item \textbf{BERT}~\cite{devlin2018bert}:
$\phi_{bert}(P)=
\texttt{BERT}(P)$,
where $\texttt{BERT}(\cdot)$ uses pre-trained BERT model~\cite{bert_wt}
with its own tokenizer,
and outputs the average of last hidden states of all tokens in the
phrase $P$.
\end{itemize}
To compute the final embedding of the phrase $\phi(P)$, we add a linear
layer to map the embeddings to 256 dimensions
compatible with the image embeddings.

\para{Training details}
We trained this model on our training split using the Adam
optimizer~\cite{kingma2014adam} with an initial learning rate at
$0.0001$. We found this model to be more prone to over-fitting than the classifier.
Stopping the training when the image retrieval and phrase retrieval MAP on the validation set stops improving was effective.
Same as the classifier, ResNet101 is initialized with ImageNet~\cite{imagenet_cvpr09} weights and fine-tuned on our data.
$\texttt{embed}(\cdot)$ in $\phi_{mean}$ and $\phi_{lstm}$ was initialized with FastText embeddings~\cite{bojanowski2017enriching, fasttext_emb} and tuned end-to-end.
Pre-trained encoders $\phi_{elmo}$ and $\phi_{bert}$ were fixed in our training.

\para{Evaluation setup}
Given the joint embedding space, one can retrieve phrases for each image
and images for each phrase based on the Euclidean distance.
Similar to the classifier we concatenate the top 5 retrieved phrases
as a baseline description generation model.
We also investigate a metric learning approach over descriptions
rather than phrases where the positive and negative triplets are
computed over (image, description) pairs.
The language embedding models are the same since they can handle
descriptions of arbitrary length.

\subsection{A Generative Language Approach}
We adopt the Show-Attend-Tell
model~\cite{xu2015show}, a widely used model for image captioning.
It combines a convolutional network to encode input images with an attention-based LSTM
decoder to generate descriptions.
Following the default setup, we encode images into
the spatial features from the 4-th layer of ResNet101 (initialized
with ImageNet~\cite{imagenet_cvpr09} weights).
The word embeddings are initialized from FastText~\cite{bojanowski2017enriching, fasttext_emb}.
The entire model is then trained end-to-end on the training set, using the Adam
optimizer~\cite{kingma2014adam} with initial learning rate
$0.0001$ for the image encoder and $0.0004$ for the language decoder.
We apply early stopping based on the BLEU-4 score of generated descriptions on the validation images.

This model is primarily for the description generation task.
For evaluation, we apply beam search with a beam-size of 5 to compute the best description.

\section{Experiments and Analysis}

\label{sec:exp}
\begin{table}[t]
\caption{Phrase retrieval and image retrieval on \dtdd.
    Metric learning models are trained with phrase input.
    Among the language encoders BERT works the best.}
  \resizebox{\linewidth}{!}{
    \begin{tabular}{c|l||*{6}{c}|*{6}{c}}
    \hline
    \multicolumn{1}{r}{} & \multicolumn{1}{c||}{\textbf{Task:}} & \multicolumn{6}{c|}{\textbf{Phrase Retrieval}}  & \multicolumn{6}{c}{\textbf{Image Retrieval}} \\
    \hline
    Data Split & \multicolumn{1}{c||}{Model} & \multicolumn{1}{c}{MAP} & \multicolumn{1}{c}{MRR} & P@5 & \multicolumn{1}{c}{P@20} & \multicolumn{1}{c}{R@5} & \multicolumn{1}{c|}{R@20} & \multicolumn{1}{c}{MAP} & \multicolumn{1}{c}{MRR} & P@5 & \multicolumn{1}{c}{P@20} & \multicolumn{1}{c}{R@5} & \multicolumn{1}{c}{R@20} \\
    \hline\hline

    \multirow{4}[2]{*}{Validation} & MetricLearning: MeanPool & 18.80 & 48.66 & 23.13 & 16.20 & 11.52 & 31.54 & 7.19  & 16.18 & 7.60  & 6.56  & 3.36  & 11.44 \\
          & MetricLearning: biLSTM & 23.53 & 58.78 & 31.85 & 18.73 & 15.83 & 36.31 & 8.31  & 17.46 & 8.15  & 7.06  & 4.21  & 13.40 \\
          & MetricLearning: ELMo & 28.13 & 68.46 & 37.02 & 21.11 & 18.44 & 41.12 & 11.25 & 24.05 & 12.79 & 10.27 & 5.85  & 18.57 \\
          & \textbf{MetricLearning: BERT} & \textbf{31.68} & \textbf{72.59} & \textbf{40.67} & \textbf{22.96} & \textbf{20.23} & \textbf{44.50} & \textbf{15.22} & \textbf{31.39} & \textbf{16.27} & \textbf{12.56} & \textbf{9.07} & \textbf{25.69} \\
    \hline
    \multirow{2}[2]{*}{Test} & Classifier: Feat 2,4 & 27.12 & 61.28 & 33.50 & 21.71 & 16.07 & 41.48 & \textbf{14.75} & \textbf{33.94} & \textbf{18.75} & \textbf{16.02} & \textbf{6.47} & \textbf{19.32} \\
          & MetricLearning: BERT & \textbf{31.77} & \textbf{74.12} & \textbf{41.70} & \textbf{23.60} & \textbf{20.17} & \textbf{45.04} & 13.50 & 31.12 & 16.52 & 14.57 & 5.24  & 17.32 \\
    \hline
    \end{tabular}
    }%

  \label{tab:retrieve}%
\end{table}%

\begin{table}[t]
\begin{center}
\makebox[0pt][c]{\parbox{\textwidth}{%
\begin{minipage}[t]{0.3\hsize}
    \caption{Retrieving texture images with descriptions as input.}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{l|c}
        \textbf{Model} & \textbf{MRR}\\
        \hline
        \hline
        Classifier & 12.40 \\
        \hline
        MetricLearning(phrase) & 12.92 \\
        \hline
        \textbf{MetricLearning(description)} & \textbf{13.95} \\
        \hline
    \end{tabular}}%
    \label{tab:retrieve_desc}%
\end{minipage}
\hfill
\begin{minipage}[t]{0.66\hsize}
    \caption{Description generation on textures. Synthesizing descriptions from phrases
      retrieved by the metric-learning based approach outperforms other baselines.}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{l|*{7}{c}}
        \textbf{Model} & \textbf{Bleu-1} & \textbf{Bleu-2} & \textbf{Bleu-3} & \textbf{Bleu-4} & \textbf{METEOR} & \textbf{Rouge-L} & \textbf{CIDEr}\\
        \hline
        \hline
        Classifier: top~5 & 68.07 & 46.17 & 28.39 & 14.44 & 19.89 & 48.13 & 44.73 \\
        \hline
        \textbf{MetricLearning: top~5} & \textbf{72.99} & \textbf{53.69} & \textbf{34.97} & \textbf{19.39} & \textbf{21.81} & \textbf{49.70} & \textbf{47.34}\\
        \hline
        Show-Attend-Tell & 59.90 & 40.41 & 26.52 & 16.35 & 19.92 & 46.64 & 37.47 \\
        \hline
    \end{tabular}}%
    \label{tab:caption}%
\end{minipage}
}}
\end{center}
\end{table}

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{figs_final/img_retrieve.pdf}
\caption{\textbf{Retrieve \dtdd test images with language input.}
We show top~5 retrieved images from the classifier and the metric learning model (trained with phrase or description input).
From left to right we show examples of
(1) phrases the classifier has been trained on,
(2) phrases beyond the frequent classes, and
(3) full descriptions.
}
\label{fig:lang2img}
\end{figure}

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{figs_final/captions.pdf}
\caption{\textbf{Phrase retrieval and description generation on \dtdd test images.}
For each input image, we list ground-truth descriptions beneath, and generated descriptions on the right.
For the classifier and the metric learning model, we concatenate the top~5 retrieved phrases.
Bold words are the ones included in ground-truth descriptions.
}
\label{fig:img2lang}
\end{figure}

\subsection{Phrase and Image Retrieval}
\label{sec:exp-ret}

Table~\ref{tab:retrieve} and \ref{tab:retrieve_desc} compare the
classifier and the metric learning model on phrase and image retrieval tasks as described in
Section~\ref{sec:data-tasks}.
Figure~\ref{fig:lang2img} and~\ref{fig:img2lang} show examples of the
top 5 retrieved images and phrases.

In Table~\ref{tab:retrieve} we first compare language encoders on
the metric learning model.
The performance of both phrase and image retrieval depends largely on
the language encoder, and BERT performs the best.
The metric learning model is
better at phrase retrieval while the classifier is slightly better at
image retrieval.

Table~\ref{tab:retrieve_desc} shows results of image retrieval from
descriptions and here too the metric learning model outperforms the other two models.
As shown in Figure~\ref{fig:lang2img}-right,
although the models trained on phrases work reasonably well, the
metric learning model trained on descriptions handles long
queries better.

\subsection{Description Generation}
We compare the Show-Attend-Tell
model~\cite{xu2015show} with a retrieval based approach.
From the classifier and the metric learning model we retrieve the top 5
phrases and concatenate them in the order of their score to form a
description. As shown in Table~\ref{tab:caption}, the metric learning model
reaches higher scores on the metrics.
However, notice that in Figure~\ref{fig:img2lang} the
generative model's descriptions are more fluent and covers both the
color and pattern of the images, while the retrieval baselines
(especially the classifier) repeat phrases with similar meanings.

\subsection{A Critical Analysis of Language Modeling}
\label{sec:analysis}

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{figs_final/synthetic_s.pdf}
\caption{\textbf{Retrieval on synthetic images.} Positive images are in dashed
  blue borders, hard negative ones are in dotted red borders.
}
\label{fig:synthetic}
\end{figure}

\begin{table}[t]
\caption{Image retrieval performance of R-Precision on synthetic tasks.}
\centering
\resizebox{0.85\linewidth}{!}{
    \begin{tabular}{l|cccc}
        \textbf{Model} & \textbf{~Foreground~} & \textbf{Background~} & \textbf{Color+Pattern~} & \textbf{Two-colors}\\
        \hline
        \hline
        Classifier & 45.45$\pm$20.34 & \textbf{59.82$\pm$9.63} & 35.95$\pm$21.48 & 26.82$\pm$14.17\\
        \textbf{MetricLearning - phrase}& 46.55$\pm$20.65 & 52.00$\pm$6.32 & \textbf{41.73$\pm$22.77} & \textbf{27.45$\pm$15.13}\\
        MetricLearning - description~ & \textbf{47.64$\pm$18.97} & 53.64$\pm$4.66 & 35.77$\pm$21.12 & 21.59$\pm$13.77\\
        \hline
        Random guess & 50.00 & 50.00 & 7.40 & 5.26\\
        \hline
\end{tabular}}
\label{tab:synthetic}
\end{table}

In this section, we evaluate the proposed models on tasks where we
systematically vary the distribution of underlying texture attributes.
This is relatively easy to do for textures than for natural images (\eg, changing the color
of dots) and allows us to understand the degree to which the models
learn disentangled representations.
We describe four tasks with varying degrees of difficulty to highlight
the strengths and weaknesses of these models.

\para{Automatically generating textures and their descriptions}
To systematically generate textures with descriptions,  we follow this procedure:
\begin{itemize}
\item Take the 11 most frequent colors in \dtdd (white, black, brown,
  green, blue, red, yellow, pink, orange, gray, purple) and set
  their RGB values manually.
\item Take 10 typical two-color images from ten different
  categories. We choose:
  \begin{itemize}
  \item Type A: 5 images with ``foreground on background'': [`dots', `polka-dots',  `swirls',`web', `lines' (thin lines on piece of paper)],
    and
    \item Type B: 5 images with no clear distinction between the
  foreground and background: [`squares' (checkered), `hexagon',
    `stripes' (zebra-like), `zigzagged', `banded' (bands with similar width)].
  \end{itemize}

\item For each of these 10 images, we manually extract masks for the foreground and
background(Type~A), or two foreground colors(Type~B).

\item For each of the 10 images, generate a new image by picking 2
  different colors from the 11 and modify pixel values of the two
  regions using the corresponding RGB value. This results in
  10$\times$11$\times$10=1,100 images.
\item For each synthetic image, we construct
  the ground-truth description with the template as ``[color1] [pattern], [color2]
  background"(such as ``pink dots, white background") for Type~A, and ``[color1]
  and [color2] [pattern]"(such as ``yellow and gray squares") for Type~B.
\end{itemize}

\noindent
\textbf{Experiment 1: Foreground.} On Type~A set we construct:
\begin{itemize}
  \item \textbf{Query:} A query of the form ``[color=c] [pattern=p]'' (\eg ``pink dots'').
  \item \textbf{Positive set:} [color=c]  [pattern=p] on randomly
    colored background (\eg ``pink dots, white background'').
  \item \textbf{Negative set:} Randomly colored ($\neq$ c)
      [pattern=p] on [color=c] background (\eg ``blue dots, pink
      background'').
  \item \textbf{Result:}
  Input the query description, we use the models to rank images from
  both the positive and negative set, and report R-Precision: the
  precision of top $R$ predictions, where $R$ is the number of
  positive images. The results are listed in
    Table~\ref{tab:synthetic} first column.
    Since half the images have
    the right attribute the chance performance is
    50\% and the various models are nearly at the chance level.
    Figure~\ref{fig:synthetic} shows that the model is unable to
    distinguish between ``pink dots'' and ``dots on a pink
    background''. This illustrates that the models are unable to associate color correctly with the foreground shapes.
\end{itemize}

\noindent
\textbf{Experiment 2: Background.} This is similar to Experiment 1 but
we focus on the background instead. On Type~A set we construct:
we know the name of its pattern (such
  as ``dots", ``squares", selected from the more frequent phrases that
  matches the category) and names of two colors (color1 and color2).
  \begin{itemize}
  \item \textbf{Query:} A query ``[color=c] background'' (\eg ``pink background'').
  \item \textbf{Positive set:} Randomly colored pattern on [color=c]
    background (\eg ``red dots on pink background'').
  \item \textbf{Negative set:} Random pattern of [color=c] on any
    [color$\neq$c] background (\eg ``pink dots on white
    background'').
  \item \textbf{Result:} R-precision is shown in
    Table~\ref{tab:synthetic} second column.
    Once again the chance performance is
    50\% and the various models are nearly at the chance level.
    Figure~\ref{fig:synthetic}-middle shows that the model is unable to
    distinguish between ``red background'' and ``red dots on random
    background''.
\end{itemize}

\noindent
\textbf{Experiment 3: Color+Pattern.} On both Type~A and~B images we construct:
\begin{itemize}
  \item \textbf{Query:} A query ``[color=c] [pattern=p]'' (\eg ``pink dots'').
  \item \textbf{Positive set:} [color=c] [pattern=p] on random colored
    background, or with another color (\eg ``pink dots, white background'', ``pink and blue squares'').
  \item \textbf{Negative set:} [color=c] [pattern$\neq$p] or
    [color$\neq$c] [pattern=p]. The negative set
    contains images with the correct pattern but wrong color or the
    wrong pattern with the right color (\eg, ``red dots'' or ``pink stripes'').
    Similar patterns (\eg, ``lines'' \emph{vs}\onedot ``banded'') are not considered negative.
  \item \textbf{Result:} The positive and negative set is unbalanced
    which results in a chance performance of 7.4\%. The models
    presented in the earlier section are able to rank the correct
    color and pattern combinations ahead of the negative set and
    achieve a considerably higher performance.
\end{itemize}

\noindent
\textbf{Experiment 4: Two colors.}  On both Type~A and~B images we construct:
\begin{itemize}
  \item \textbf{Query:} A query ``[color=c1] and [color=c2]'' (\eg ``pink
    and green'').
  \item \textbf{Positive set:} [color=c1] of random pattern on [color=c2] background (\eg ``pink dots on green background''), or [color=c1] and [color=c2] of random pattern (\eg ``pink and green squares'').
  \item \textbf{Negative set:} pattern with one color from
    \{c1, c2\} and another color $\neq$\{c1,c2\} (\eg, ``pink
    dots on yellow background", ``green and blue stripes").
  \item \textbf{Result:} The positive and negative set are unbalanced
    which results in a chance performance of 5.26\%. The models
    once again are able to rank the two color combinations ahead of the negative set and
    achieve a considerably higher
    performance. Figure~\ref{fig:synthetic}-right shows an example.
\end{itemize}

\para{Summary}
These experiments reveal that these models do indeed exhibit some high-level discriminative abilities (Exp. 3, 4), but they fail to disentangle properties such as the color of the foreground elements
from background (Exp. 1, 2). This leaves much room for improvement,
motivating future work, such as those that enforces spatial agreement between
different attributes.

\section{Applications}
\label{sec:app}
\begin{figure*}[t]
    \centering
    \includegraphics[width=\textwidth]{figs/fgvc.pdf}
    \caption{
      Fine-grained categories visualized as their training
      images (top row), maximal texture images (middle row), and
      texture attributes (bottom row). The size of each phrase in the
      cloud is inversely decided by its Euclidean distance to the
      input maximal texture image calculated by the triplet model.}
    \label{fig:fgvc}
\end{figure*}

\para{Describing textures of fine-grained categories}
We analyze how the categories in fine-grained domains can be described by their texture.
We consider categories from Caltech-UCSD Birds (CUB)~\cite{WahCUB_200_2011}, Oxford
Flowers~\cite{Nilsback08}, and FGVC Butterflies and
Moths~\cite{fgvcbutterlies} datasets.
For each category, we follow the visualizing deep
texture representations~\cite{lin2016visualizing} to generate
``maximal textures'' --  inputs that maximize the class probability
using multi-layer bilinear
CNN classifier~\cite{lin2018bilinear}.
These are provided as input to our metric learning model (with BERT encoder
and phrase input) trained on \dtdd to retrieve the top phrases.
Figure~\ref{fig:fgvc} shows several categories with their maximal
textures and a ``phrase cloud'' of the top retrieved phrases.
These provide a qualitative description of each category.

\begin{figure}[t]
\centering
\includegraphics[width=\textwidth]{figs/cub2.pdf}
\caption{\textbf{Classification on CUB dataset with \dtdd texture attributes. }
\emph{Left:} classification accuracy \emph{vs}\onedot number of input features.
Orange and green markers with the same shape are comparable with the same set of CUB attributes with or without the \dtdd attributes.
\emph{Right:} The phrase clouds display important phrases for a few bird categories.
Red phrases correspond to positive weights and blue are negative for a linear classifier for the category. Font sizes represent the absolute value of the coefficient.
}
\label{fig:cub}
\end{figure}

\para{Fine-grained classification with texture attributes}
Here we apply models trained on our \dtdd on the CUB dataset to show that
embedding images into the space of texture attributes allows interpretable models for discriminative classification.
Specifically, we input each image from the dataset to our
phrase classifier (trained on \dtdd and fixed) and obtain the
log-likelihood over the $655$ texture phrases as an embedding.
We train a logistic regression model for the 200-way classification task.
The dataset also comes with $312$ binary attributes that
describe the shape, pattern and color of specific parts of a bird,
such as ``has tail shape squared tail", ``has breast pattern spotted",
``has wing color yellow"
(42 attributes for ``shape", 31 for
``pattern" and 239 for ``color").
We also train a logistic regression classifier on top of these attributes.

Figure~\ref{fig:cub} shows the performance by varying the number of
texture phrases ranked by their frequency on \dtdd as the blue curve.
It also shows a comparison of bird-specific
attributes from CUB with generic texture attributes learned from \dtdd.
Results using CUB attributes are shown in green, while
those using combinations of CUB and texture attributes are shown in orange.
Texture attributes are able to distinguish bird species with an accuracy of $28.5\%$, outperforming CUB shape and pattern
attributes.
However, they do not outperform the part-based color attributes that
are highly effective.
Yet, combining CUB attributes with texture attributes lead to
consistent improvements.
On the right is the visualization of discriminative texture attributes for some categories: we display phrases with the most positive weights in red, and those with the most negative weights in blue.
They provide a basis for interpretable explanations of discriminative features without requiring a category-specific vocabulary.

\section{Conclusion}
We presented a novel dataset of textures with natural language descriptions and analyzed the performance of several language and vision models.
The domain of texture is poses challenges to existing models which fail to learn a sufficiently disentangled representation leading to poor generalization on synthetic tasks.
Yet, the models show some generalization to novel domains and enabling us to provide interpretable models for describing some fine-grained domains.
In particular they are complementary to existing domain-specific attributes on the CUB dataset.

\textbf{Acknowledgements.} We would like to thank Mohit Iyyer for helpful discussions and feedback. The project is supported in part by NSF grants \#1749833 and \#1617917. Our experiments were performed in the UMass GPU cluster obtained under the Collaborative Fund managed by the Mass. Technology Collaborative.

\section*{Supplementary Material}
\setcounter{section}{0}
\setcounter{figure}{0}
\setcounter{table}{0}

\section{Dataset statistics and visualizations}
We provide further details about the \dtdd.
Figure~\ref{fig:stat} shows the long-trail distribution of words and phrases in the dataset.
Descriptions of categories such as ``dots", ``lines", ``crystalline" and ``checkers" are simpler than those of ``spiralled'', ``interlaced'' and ``crosshatched'' as indicated by the number of unique words and phrases for each category.
More examples of images and annotations from the dataset are shown in  Figure~\ref{fig:data1},~\ref{fig:data2}, and~\ref{fig:data3}.

\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{supp/supp_stat.pdf}
\caption{\textbf{More Statics of \dtdd.}
On the top (middle) we show the histogram of word(phrase) frequency, where the x labels display some of the words(phrases) with their rankings in the dataset.
On the bottom we show the number of unique words and phrases used to describe each DTD category, where we only count frequent words(phrases) that occur at least 5(10) times in training subset.}
\label{fig:stat}
\end{figure}

\section{Comparison of ResNet101 features from different layers}
We compare features from layer-block 1 to 4 of ResNet101 in the classifier model, as shown in  Table~\ref{tab:retrieve_supp}.
Higher layer features perform better for phrase retrieval.
For image retrieval, better performance is achieved with the
combination of features from different layers.
We select to use the features from layer 2 and 4 for all classifiers
and metric learning models in all experiments with the classifier and the metric learning model.

\begin{table}[t]
\caption{Performance on phrase retrieval and image retrieval on \dtdd.
  ``Classifier: Feat $x$" stands for the classifier with image
    features from ResNet layer block $x$ (or a concatenation of two layers.)}

  \resizebox{\linewidth}{!}{
    \begin{tabular}{c|l||*{6}{c}|*{6}{c}}
    \hline
    \multicolumn{1}{r}{} & \multicolumn{1}{c||}{\textbf{Task:}} & \multicolumn{6}{c|}{\textbf{Phrase Retrieval}}  & \multicolumn{6}{c}{\textbf{Image Retrieval}} \\
    \hline
    Data Split & \multicolumn{1}{c||}{Model} & \multicolumn{1}{c}{MAP} & \multicolumn{1}{c}{MRR} & P@5 & \multicolumn{1}{c}{P@20} & \multicolumn{1}{c}{R@5} & \multicolumn{1}{c|}{R@20} & \multicolumn{1}{c}{MAP} & \multicolumn{1}{c}{MRR} & P@5 & \multicolumn{1}{c}{P@20} & \multicolumn{1}{c}{R@5} & \multicolumn{1}{c}{R@20} \\
    \hline\hline
    \multirow{7}[1]{*}{Validation} & Classifier: Feat 1 & 13.10 & 37.20 & 16.05 & 10.68 & 4.94  & 13.04 & 10.64 & 25.06 & 11.57 & 9.37  & 6.13  & 17.78 \\

          & Classifier: Feat 2 & 17.65 & 44.91 & 22.41 & 14.59 & 6.85  & 17.60 & 13.00 & 29.24 & 14.60 & 11.08 & 8.52  & 22.54 \\

          & Classifier: Feat 3 & 26.43 & \textbf{60.52} & 32.47 & 20.71 & 9.93  & 25.00 & 15.62 & 31.79 & 17.28 & 13.34 & 9.42  & 28.52 \\

          & Classifier: Feat 4 & 26.51 & 59.24 & \textbf{33.07} & 20.84 & \textbf{10.07} & 25.16 & 15.85 & \textbf{33.06} & 17.83 & 13.02 & 9.94  & 27.28 \\

          & Classifier: Feat 1,4 & 25.78 & 58.28 & 31.58 & 20.31 & 9.55  & 24.44 & 15.85 & 32.35 & \textbf{18.35} & 13.51 & 10.24 & 28.03 \\

          & \textbf{Classifier: Feat 2,4} & 26.57 & 59.19 & 32.65 & 21.11 & 9.99  & 25.50 & \textbf{16.19} & 32.53 & 17.47 & \textbf{13.56} & \textbf{10.63} & \textbf{28.69} \\

          &Classifier: Feat 3,4 & \textbf{26.66} & 60.38 & 32.20 & \textbf{21.22} & 9.81  & \textbf{25.68} & 16.04 & 31.18 & 17.59 & 13.50 & 10.33 & 28.32 \\
    \hline

    \multirow{2}[2]{*}{Test} & Classifier: Feat 2,4 & 27.12 & 61.28 & 33.50 & 21.71 & 16.07 & 41.48 & \textbf{14.75} & \textbf{33.94} & \textbf{18.75} & \textbf{16.02} & \textbf{6.47} & \textbf{19.32} \\
          & MetricLearning: BERT & \textbf{31.77} & \textbf{74.12} & \textbf{41.70} & \textbf{23.60} & \textbf{20.17} & \textbf{45.04} & 13.50 & 31.12 & 16.52 & 14.57 & 5.24  & 17.32 \\
    \hline
    \end{tabular}
    }%

  \label{tab:retrieve_supp}%
\end{table}%

\section{Performance on different categories and phrases}
Here we take a closer look at the performance of our models on different tasks on different phrases (Figure~\ref{fig:ph}) and texture categories (Figure~\ref{fig:cat}). The texture category is the category of the image in \dtd.
As seen in Figure~\ref{fig:ph}-middle, the performance is correlated with the frequency of phrases, especially for image retrieval.
As seen in Figure~\ref{fig:cat}, we have better phrase retrieval performance on simpler categories that come with smaller vocabulary size.

\begin{figure}
\centering
\includegraphics[width=0.9\linewidth]{supp/supp_phrase.pdf}
\caption{\textbf{\dtdd test set performance per phrase on each task with selected models.}
Color of phrases represents the metric performance: blue is better and red is worse, as indicated in the color bars. Font sizes are proportional to square root of phrase frequencies.}
\label{fig:ph}
\end{figure}

\begin{figure}[t]
\centering
\includegraphics[width=0.95\linewidth]{supp/subset_plot_.pdf}
\caption{\textbf{\dtdd test set performance per category on each task with our best model.}
Phrase retrieval: AP on our metric learning model;
Image retrieval with phrase input: AP on our classifier;
Image retrieval with description input: MRR on our metric learning model(description input);
Captioning: METEOR on Show-Attend-Tell model.}
\label{fig:cat}
\end{figure}

\section{Generated textures for analysis}
Figure~\ref{fig:syn} shows the synthetically generated images and descriptions used in the experiments in Section 5.3.
Each image is manually separated into two color regions which are systematically varied.
In each row of the figure we systematically vary different attributes such as the foreground attribute, foreground and background color, etc.
This is a simple yet effective way to generate natural textures.
Note how the texture properties are still maintained in each image, such as shading and winkles of zebra stripes.
This is much more challenging to do the same with natural images of scenes and objects.

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{supp/supp_synthetic.pdf}
\caption{\textbf{Original and generated texture images.}
The first row shows original images and their \dtd categories.
The rest rows are generated descriptions and images by sampling and modifying colors. The left 5 columns are images of Type A (foreground and background); the right 5 are Type B (no obvious foreground/background distinction).
}
\label{fig:syn}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=\linewidth]{supp/supp_data_example_1.pdf}
\caption{More examples from \dtdd.}
\label{fig:data1}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=\linewidth]{supp/supp_data_example_2.pdf}
\caption{More examples from \dtdd.}
\label{fig:data2}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=\linewidth]{supp/supp_data_example_3.pdf}
\caption{More examples from \dtdd.}
\label{fig:data3}
\end{figure}