\section{Introduction}
Researchers have studied the problem of understanding actions communicated through natural language in both simulated \cite{Das2017, IQA, Hermann2017} and real environments \cite{arid, TalkTheWalk, Anderson2017}.
This paper focuses on the latter.
More concretely, we consider the problem in an autonomous driving setting, where a passenger can control the actions of an \textbf{A}utonomous \textbf{V}ehicle (AV) by giving natural language commands.
We hereunder argue why this problem setting is particularly interesting.

\begin{figure*}
\begin{center}
 \centering
\subfloat[You can park up ahead behind \textbf{the silver car}, next to that lamppost with the orange sign on it]{\includegraphics[width=.32\linewidth]{car.png}}
\hfill
\subfloat[\textbf{My friend} is getting out of the car. That means we arrived at our destination! Stop and let me out too!]{\includegraphics[width=.32\linewidth]{man.png}}
    \hfill
\subfloat[Yeah that would be \textbf{my son} on the stairs next to the bus. Pick him up please]{\includegraphics[width=.32\linewidth]{example5.png}}
\hfill
\subfloat[After \textbf{that man in the blue top} has passed, turn left]{\includegraphics[width=.32\linewidth]{example3.png}}
    \hfill
\subfloat[There's \textbf{my mum}, on the right! The one walking closest to us. Park near \textbf{her}, she might want a lift]{\includegraphics[width=.32\linewidth]{mom.png}}
\hfill
\subfloat[Turn around and park in front of \textbf{that vehicle in the shade}]{\includegraphics[width=.32\linewidth]{tree.png}}
\end{center}
   \caption{The \texttt{Talk2Car} dataset adds textual annotations on top of the \texttt{nuScenes} dataset for urban scene understanding. The textual annotations are free form commands, which guide the path of an autonomous vehicle in the scene. Each command describes a change of direction, relevant to a referred object found in the scene (here indicated by the red 3D-bounding box). Best seen in color.}
\label{fig:exampleCommand}
\end{figure*}

\paragraph{}
First, a recent study by \newcite{PublicPerceptionSDC}
has shown that the majority of the public is reluctant to step inside an AV.
A possible explanation for this might be the lack of control which can be unsettling to some.
Providing a way to communicate with the vehicle could help alleviate this uneasiness.
Second, an AV can become hesitant in some situations \cite{self-driving-car-problems}.
By giving a task or command, the passenger could guide the agent in its decision process.
Third, some situations request feedback.
For example, a passenger might
indicate that they want to park in the shade during a sunny day.
Finally, the problem of urban scene understanding is one of practical relevance that has been well studied \cite{CityScape,geiger2013vision}.
We believe all of this makes it an interesting setting to assess the performance of grounding natural language commands into the visual space.
\newline

To perform the requested action, an agent is required to take two steps.
First, the agent needs to interpret the command and ground it into the physical visual space.
Secondly, the agent has to devise a plan to execute the given command.
This paper focuses on this former step, or more concretely:
given an image $I$ and a command $C$, the goal is to find the region $R$ in the image $I$ that the command is referring to.
In this paper, to reduce the complexity of the object referral task we restrict the task to the case where there is only one targeted object that is referred to in the natural language command

To stimulate research on grounding commands into the visual space we present the first object referral dataset, named \texttt{Talk2Car},
that comes with commands formulated in textual natural language for self-driving cars.
A few example commands together with their contextual images can be found in Fig. \ref{fig:exampleCommand}.
Moreover, by using this new dataset we evaluate the performance of several strong state-of-the-art models that recognize the referred object of a command in the visual scene. Here we encounter several challenges. Referred objects are sometimes ambiguous (e.g., there are several cyclists in the scene), but can be disambiguated by understanding modifier expressions in language (e.g., the biker with the red jacket).
These modifier expressions could also indicate spatial information.
Furthermore, detecting the targeted object is challenging both in the language utterance and the urban scene, for instance, when dealing with complex and long sentences which might contain coreferent phrases, and with distant objects in the visual scene, respectively.
Finally, in AV settings the speed of predicting the location of the referred object is of primordial importance.

The contributions of our work are the following:
\begin{itemize}
    \item{We propose the first object referral dataset for grounding commands for self-driving cars in free natural language into the visual context of a city environment.}
    \item{We evaluate several state-of-the-art models that recognize the referred object of a natural language command in the urban scene}.
     \item{We especially evaluate the models 1) for their capabilities to disambiguate objects based on modifying and spatial relationships expressed in language; 2)
     for their capabilities to cope with difficult language and visual context; and 3) with respect to prediction speed, which is important in real-life AV settings.}
\end{itemize}
\section{Related Work}
\label{sect:related-work}

\paragraph{Object Referral}
The \texttt{Talk2Car} dataset considers the object referral task, which requires to retrieve the correct object (region) from an image based on a language expression.
A common method is to first extract regions of interest from the image, using a region proposal network (RPN).
\citet{yu2016modeling, mao2016generation} decode these proposals as a caption using a recurrent neural network (RNN).
The predicted region corresponds to the caption that is ranked most similar to the referring expression.
Other works based on RPN \cite{hu2017modeling} or Faster-RCNN \cite{yu2018mattnet} have integrated attention mechanisms to decompose the language expressions into multiple sub-parts but use tailored modules for specific sub-tasks making them less fit for our object referral task.
\citet{karpathy2014deep} interpret the inner product between region proposals and sentence fragments as a similarity score, allowing to match them in a bidirectional manner.
\citet{hu2016natural} uses an encoding of the global context in addition to the local context from the extracted regions.
\citet{STACK} explore the use of modular networks for this task.
They are comprised of multiple smaller predefined building blocks that can be combined together based on the language expression.
The
last three state-of-the-art models are evaluated on \texttt{Talk2Car} (section \ref{sect:sota}).

\paragraph{Grounding in Human-Robot Interaction}
When giving commands to robots, the grounding of the command in the visual environment is an essential task.
\citet{deits2013clarifying}
use Generalized Grounding Graphs ($G^3$) \cite{tellex2011understanding, kollar2013generalized} which is a probabilistic graphical model based on the compositional and hierarchical structure of a natural language command.
This approach allows to ground certain parts of an image with linguistic constituents.
\citet{shridhar2018interactive} consider the task where a robot arm has to pick up a certain object based on a given command.
This is accomplished by creating captions for extracted regions from a RPN and clustering them together with the original command. If the command is ambiguous and more than one caption indicates the
referring expression, the system will ask a clarifying question in order to be able to pick the right object. Due to its computational complexity during prediction, we did not select the last model in our evaluations.

\paragraph{Visual Question Answering}
The goal of
VQA is to ask any type of question about an image for which the system should return the correct answer.
This requires the system to have a good understanding of the image and the question.
Early work
\cite{kafle2016answer, ZhouTSSF15, FukuiPYRDR16}
tried to solve the task by fusing image features extracted by a convolutional neural network (CNN), together with an encoding of the question.
\cite{Johnson2017, Suarez2018} experimented with modular networks for this task.
\citet{Hudson2018} proposed the use of a network made of recurrent Memory, Attention and Composition (MAC) cells. Similar to modular networks, the MAC model also uses multiple reasoning steps, making it a suitable model in our evaluation (section
\ref{sect:sota}).

\paragraph{Object Referral Datasets}
Over the years, various object referral datasets based on both real world and computer generated images have been proposed.
\citet{ReferIt} introduced the first real-world large-scale object referral dataset named \texttt{ReferIt}.
\citet{yu2016modeling} constructed \texttt{RefCOCO} and \texttt{RefCOCO+} and as the names suggest, these two datasets are based on the \texttt{MSCOCO} dataset \cite{MSCOCO}.
A third dataset also based on \texttt{MSCOCO}, named \texttt{RefCOCOg} \cite{mao2016generation}, contains longer language expressions than the previous two datasets.
This dataset has an average expression length of 8.43 words per expression compared to 3.61 for \texttt{RefCOCO} and 3.53 for \texttt{RefCOCO+}.
The dataset closest to ours is the dataset by \citet{vasudevan2018object}, as it augments \texttt{Cityscapes} \cite{CityScape} with textual annotations. We will henceforth refer to this dataset as \texttt{Cityscapes-Ref}. The main difference with this work, is that the \texttt{Talk2Car} dataset contains commands, rather than descriptions.
A computer generated dataset named \texttt{CLEVR-Ref} was proposed by \citet{STACK} which has been created by augmenting the \texttt{CLEVR} dataset \cite{CLEVR} such that it would include referred objects.
We refer to section \ref{sect:compare-datasets} for a thorough comparison between these datasets and ours.
Some new datasets have recently been proposed where a car has to navigate through a city based on a textual itinerary given by a passenger and locate a target at the final destination
\cite{chen2019touchdown, vasudevan2019talk2nav}.
This can also be seen as an object referral task with the addition of following an itinerary.
While being a very interesting problem, it differs from the task being evaluated in this pape

\section{Dataset}
\subsection{Dataset Collection and Annotation}
\label{subsect:dataset-collection}
The \texttt{Talk2Car} dataset is built upon the \texttt{nuScenes} dataset \cite{nuScenes} which is a large-scale dataset for autonomous driving.
The \texttt{nuScenes} dataset contains 1000 videos of 20 seconds each taken in different cities (Boston and Singapore), weather conditions (rain and sun) and different times of day (night and day).
These videos account for a total of approximately 1.4 million images. Each scene comes with data from six cameras placed at different angles on the car, LIDAR, GPS, IMU, RADAR and 3D bounding box annotations. The 3D bounding boxes discriminate between 23 different object classes.
\begin{figure*}[ht]
\begin{center}
\includestandalone[width=0.7\linewidth]{ds_stats}
\end{center}
\caption{Statistics of the \texttt{Talk2Car} dataset.}
\label{fig:stats}
\end{figure*}

We relied on workers of Amazon Mechanical Turk (AMT) to extend the videos from the \texttt{nuScenes} dataset with written commands.
To create commands, each worker watches an entire 20 second long video from the front facing camera. Afterwards, the worker navigates to any point in the video that is found interesting. Once the worker has decided on the frame, a pre-annotated object from the \texttt{nuScenes} dataset for that frame needs to be selected.
The annotation task is completed when a command referring to the selected object is entered. The workers were free to enter any command, as long as the car can follow a path based on the command.\footnote{
The path to be followed by the car when executing the command, will be added in a later version of the dataset.}

We hired five workers per video who could enter as many commands per video frame as they wanted. To ensure high quality annotations, we manually verified the correctness of all the commands and corresponding bounding boxes. The verification happened in a two-round system, where each annotation had to be qualified as adequate by two different reviewers. To incentivize workers to come up with diverse and meaningful commands, we awarded a bonus every time their work received approval.

\subsection{Statistics of the Dataset}
\label{subsect:statistics}
The \texttt{Talk2Car} dataset contains 11 959 commands for the 850 videos of the \texttt{nuScenes} training set as 3D bounding box annotations for the test set of the latter dataset are not disclosed.
55.94\% and 44.06\% of these commands belong to videos taken respectively in Boston and Singapore.
On average a command consist of 11.01 words, 2.32 nouns, 2.29 verbs and 0.62 adjectives.
Each video has on average 14.07 commands.
In Fig. \ref{fig:stats}(d) we can see the distribution of distance to the referred objects.
Fig. \ref{fig:stats}(b) displays the distribution of commands over the videos.
On average there are 4.27 objects with the same category as the referred object per image and on average there are 10.70 objects per image.
Fig. \ref{fig:stats}(a) shows a heatmap of the location of all referred objects in the images of \texttt{Talk2Car}.
In Fig. \ref{fig:stats}(c) we see the distribution of commands that refer to an object of a certain category.

\subsection{Dataset Splits}
\label{subsect:splits}
We have split the dataset in such a way that the train, validation and test set would contain 70\%, 10\% and 20\% of the samples, respectively.
To ensure a proper coverage of the data distribution in each set, we have taken a number of constraints into account.
First, samples belonging to the same video are part of the same set.
Second, as the videos are shot in either Singapore or Boston, i.e., in left or right hand traffic, the distributions of every split have to reflect this.
Third, we aim to have a similar distribution of scene conditions across different sets, such as the type of weather and the time of day.
Finally, as the number of occurrences of object categories is heavily imbalanced
(see fig. \ref{fig:stats}(c)),
we have ensured that every object category contained in the test set is also present in the training set.
With these constraints in mind we randomly sample the three sets for 10 000 times and optimize for a data distribution of 70\%, 10\% and 20\%.
The resulting train, validation and test sets contain 8 349 (69.8\%), 1 163 (9.7\%) and 2 447 (20.4\%) commands respectively.
We have also identified multiple subsets of the test set, which allow evaluation of specific situations. When the referred object category occurs multiple times in an image,
attributes in modifying expressions in language including spatial expressions might disambiguate the referred object. This has led to test sets with different numbers of occurrences of the targeted category of the referred object. Longer commands might contain irrelevant information or might be more complex to understand, leading to test sets with commands of different length. Finally, referred objects at large distances from the AV might be difficult to recognize. Hence, we have built test sets that contain referred objects at different distances from the AV.
\section{Datasets Comparisons}
\subsection{Quantitative Evaluation of \texttt{Talk2Car}}
\label{sect:compare-datasets}

Table \ref{tab:comparison} compares the \texttt{Talk2Car} dataset with prior object referral datasets.
It can be seen that \texttt{Talk2Car}
contains fewer natural language expressions than the others.
However, although the dataset is smaller, the expressions are of high quality thanks to the double review system we discussed earlier in section \ref{subsect:dataset-collection}.
The main reason for having fewer annotations is that the original \texttt{nuScenes} dataset only discriminates between 23 different categories corresponding to the annotated bounding boxes.
Moreover, the original \texttt{nuScenes} dataset considers the specific setting of urban scene understanding.
This limits the visual domain considerably in comparison to \texttt{MS-COCO}.
On the other hand, \texttt{Talk2Car} contains images in realistic settings accompanied by free language in contrast to curated datasets such as MS-COCO.
Compared to most of the above datasets, the video frames annotated with natural language commands are part of larger videos that contain in total 1 183 790 images which could be exploited in the object referral task.

When we consider the average length of the natural language expressions in \texttt{Talk2Car}, we find that it ranks third, after \texttt{CLEVR-Ref} and \texttt{Cityscapes-Ref}.
We did not put limitations on what the language commands could contain, which benefits
the complexity and linguistic diversity of the expressions (section \ref{subsect:qualitative-evaluation}).

When looking at the type of modalities, the \texttt{Talk2Car} dataset considers RADAR, LIDAR and video.
These modalities are missing in prior work except for video in \texttt{Cityscapes-Ref}.
Including various modalities allows researchers to study a very broad range of topics with just a single dataset.

\begin{table*}[ht!]
\small
\begin{center}
\scalebox{0.9}{
\begin{tabular}{|l|c|c|c|c|c|c|c|c|}
\hline
Dataset & Images & Objects & Expressions & Avg expr length  & Video & Lidar & Radar \\
\hline
\texttt{ReferIt} \cite{ReferIt} &  19,894 &  96,654  & 130,525 & 3.46 &$\times$&$\times$&$\times$ \\
\texttt{RefCOCO} \cite{yu2016modeling} &  26,711 & 50,000 & 142,209 & 3.61 &$\times$&$\times$&$\times$ \\
\texttt{RefCOCO+} \cite{yu2016modeling} & 19,992 & 49,856 & 141,564 & 3.53 &$\times$&$\times$&$\times$ \\
\texttt{RefCOCOg} \cite{mao2016generation}& 26,711 & 54,822 & 85,474  & 8.43 &$\times$&$\times$&$\times$ \\
\texttt{CLEVR-Ref} \cite{STACK} & 99,992 & 492,727 & 998,743 & 14.50 &$\times$&$\times$&$\times$  \\
\texttt{Cityscapes-Ref} \cite{vasudevan2018object} & 4,818 &  29,901  & 30,000  & 15.59 &\checkmark& $\times$ & $\times$ \\
\hline
\textbf{\texttt{Talk2Car} (Ours)} & 9,217 & 10,519 & 11,959 & 11.01 &\checkmark &\checkmark &\checkmark  \\
\hline
\end{tabular}
}
\end{center}
\caption{Statistics of and comparison with existing datasets for object referral.}
\label{tab:comparison}
\end{table*}
\vspace{-.01cm}
\subsection{Qualitative Evaluation of \texttt{Talk2Car}}
\label{subsect:qualitative-evaluation}
To make our discussion more concrete, we compare the textual annotations from Fig. \ref{fig:exampleCommand} with some examples from prior work that are listed below.
\texttt{RefCOCO} contains expressions such as `Woman on right in white shirt' or `Woman on right'.
\texttt{RefCOCO+} on the other hand contains expressions such as `Guy in yellow dribbling ball' or `Yellow shirt in focus'.
Lastly, \texttt{ReferIt} contains `Right rocks', `Rocks along the right side'.
The language used in the above prior work is more simple, explicit and is well structured in comparison to the commands of \texttt{Talk2Car}.
Additionally, the latter
tend to include irrelevant side-information, e.g., 'She might want a lift', instead of being merely descriptive.
The unconstrained free language of \texttt{Talk2Car} introduces different challenges, which involve
co-reference resolution, named entity recognition, understanding relationships between objects, linking attributes to objects and understanding which object is the object of interest in the command.

The commands also contain implicit referrals as can be seen
in the command in Fig. \ref{fig:exampleCommand}(f):
`Turn around and park in front of that vehicle in the shade'.
Similar to
\texttt{CLEVR-Ref}, object referral in \texttt{Talk2Car} requires some form of spatial reasoning.
However, in contrast to the former, there are cases where the spatial description in the command is misleading and truthfully reflects mistakes that people make.
An example is the command in Fig. \ref{fig:exampleCommand}(e), where we refer to the object as being on the right side of the image, while the person of interest is actually located on the left.

Another important difference is the type of images in each dataset. For instance, the urban images in RefCOCO are taken from the viewpoint of a pedestrian. On the other hand, the images in Talk2Car are car centric.
\section{Application of the State-of-the-Art Models and their Evaluation}
\label{sect:sota}
We assess the performance of 7 models to detect the referred object in a command on the \texttt{Talk2Car} dataset.
We discriminate between state-of-the-art methods based on region proposals and non-region proposal methods, apart from simple baselines. \footnote {All parameter values obtained on the validation set are cited in the supplementary material. For both the MAC and STACK-NMN we give the best results after empirically setting the number of reasoning steps. }
\subsection{Region Proposal Based Methods}
\label{subsec: region_proposal_methods}
\paragraph{Object Sentence Mapping (OSM)}
This region proposal based method uses a single-shot detection model, i.e., SSD-512 \cite{liu2016ssd}, to extract 64 interest regions from the image.
We pretrain the region proposal network (RPN) for the object detection task on the train images from the \texttt{Talk2Car} dataset.
A ResNet-18 model is used to extract a local representation for the proposed regions.
The natural language command is encoded using a neural network with Gated Recurrent Units (GRUs).
Inspired by \cite{karpathy2014deep}, we use the inner product between the latent representation of the region and command as a score for each proposal.
The region that gets assigned the highest score is returned as bounding box for the object referred to by the command.

\paragraph{Spatial Context Recurrent ConvNet (SCRC)}
A shortcoming of the above baseline model is that the correct region has to be selected based on local information alone. Spatial Context Recurrent ConvNets \cite{hu2016natural} match both local and global information with an encoding of the command. We reuse the SSD-512 model from above to generate region proposals. A global image representation is extracted by a ResNet-18 model. Additionally, we add an 8-dimensional representation of the spatial configuration to the local representation of each bounding box, $x_{spatial} = \left[ x_{min}, y_{min}, x_{max}, y_{max}, h, w, x_{center}, y_{center} \right]$ with $h$ and $w$ respectively being the height and the width of this bounding box. For more details, we refer to the original work \cite{hu2016natural}.

\subsection{Non-Region Proposal Based Methods}
\paragraph{MAC model} This model \cite{Hudson2018} originally created for the VQA task uses a recurrent MAC cell to match the
natural language command represented with a Bi-LSTM model with a global representation of the image. A ResNet-101 is used to extract the visual features from the image.
The MAC cell decomposes the textual input into a series of reasoning steps, where
the MAC cell attends to certain parts of the textual input to guide the model to look at certain parts of the image.
Between each of these reasoning steps, information is passed to the next cell such that the model is capable of representing arbitrarily complex
reasoning graphs in a soft manner in a sequential way.
The recurrent control state of the MAC cell identifies a series of read and write operations.
The read
unit extracts relevant information from both a given image and the internal memory. The
write
unit iteratively integrates the information into the cells' memory state, producing a new intermediate result.

\begin{table*}[t]
\small
\begin{center}
\begin{tabular}{|l|c|c|c|}
\hline
Method & $IoU_{0.50}$ (\%) & Inference Speed (ms) & Params (M) \\
\hline
RS & 2.05 & - & - \\%45 & \textbf{27.2} \\
BOBB & 3.15 & - & - \\
RNM & 9.70 & - & - \\%&68 & \textbf{27.2} \\
MAC \cite{Hudson2018
& \textbf{50.51} & 5
& 41.59 \\
STACK-NMN \cite{STACK
& 33.71 & 5
& 35.2 \\
OSM \cite{karpathy2014deep} & 35.31 & 7
& 43.0 \\
SCRC \cite{hu2016natural} & 38.70 & 90 & 59.5 \\
\hline
\end{tabular}
\end{center}
\caption{Performance ($IoU_{0.50}$), inference speed (evaluated on a TITAN XP) and number of parameters of the different models.}
\label{tab:experiments}
\end{table*}
\paragraph{Stack-NMN}

The Stack Neural Module Network or Stack-NMN \cite{STACK} uses
multiple modules that can solve a task by automatically inducing a sub-task decomposition, where each sub-task is addressed by a separate neural module.
These modules can be chained together to decompose the natural language command into a reasoning process.
Like the MAC model, this reasoning step is based on the use of an attention mechanism to attend to certain parts of the natural language command, which on their turn guide the selection of neural modules.
The modules are first conditioned with the attended textual features after which they perform sub-task recognitions on the visual features.
The output of these modules are attended parts in the image which are then given to the next reasoning step to continue the reasoning process.
Again, a ResNet-101 model is used to extract the image features and a Bi-LSTM to encode the natural language command.
To predict the referred object this model first splits the given image into a 2D grid.
Then it tries to predict in which cell located in the grid the center of the referred object lies.
Once this has been predicted, the model predicts the offsets of the bounding box relative to the predicted center.

\subsection{Simple Baselines}
\paragraph{Random Selection (RS)}
We reuse the single-shot detection model from section \ref{subsec: region_proposal_methods} to generate 64 region proposals per image of the test set. This model randomly samples one region from the proposals and uses it as prediction for the referred object. This is done 100 times and results are averaged.

\paragraph{Biggest Overlapping Bounding Box (BOBB)}
From the heatmap in Fig. \ref{fig:stats} (a) we can see that there is some bias of the referred objects on the left side.
This model tries to exploit this information by searching a 2D bounding box that optimizes the overlap with all the bounding boxes in the training set.
The algorithm is explained in Section \ref{sec:supplemental} of the supplementary material.

\paragraph{Random Noun Matching (RNM)}
In the test set a dependency parser \cite{honnibal-johnson:2015:EMNLP} is used to extract the set of nouns from a given command. We keep the nouns which are substrings of the category names. Then, we randomly sample an object from the region proposals of the corresponding image.
If the set of category names is empty,
we randomly sample a region from all
region proposals.
We re-use the RPN explained in OSM for the region proposals.
This method is evaluated 100 times before averaging the results.
\subsection{Results and Discussion}

\label{sect:experiments}
\begin{figure*}[t!]
\begin{center}
\includestandalone[width=1\linewidth]{test_plots}
\caption{Test performance $IoU_{0.5}$
of different methods on the challenging sub-test sets.
We discriminate between
A test set for the top-k furthest objects in Fig. (\textbf{a}), the top-k shortest and longest commands in Fig. (\textbf{b}) and Fig. (\textbf{c}) respectively, and in function of the number of objects of the same category in the scene in Fig. (\textbf{d}).}
\label{fig:subset_tests}
\end{center}
\end{figure*}

\paragraph{Overall Results} We evaluated all seven models on the object referral task, using both the test split from subsection \ref{subsect:splits} as well as multiple increasingly challenging subsets from this test set.
To properly evaluate existing models against our baselines we convert the 3D bounding boxes to 2D bounding boxes.
We consider the predicted region correct when the Intersection over Union (IoU), that is, the intersection of the predicted region and the ground truth region over the union of these two, is larger than 0.5.
Additionally, we report average inference speed at prediction time per sample and number of parameters of each model.
We report the results obtained on the test set in Table \ref{tab:experiments}. The results over the challenging test subsets can be seen in Fig. \ref{fig:subset_tests}.

In all results we see the following:
First, it is clear that the simple baselines (RS, BOBB, RNM) do not perform well, which evidences the difficulty of the object referral task in the realistic settings captured in \texttt{Talk2Car}.
Second, MAC performs the best on nearly all tasks and it performs significantly better than STACK-NMN which is the model that resembles MAC the most.

If we compare the two RPN systems
we see that SCRC often outperforms OSM, showing that using spatial information is beneficial.
Third, being able to discriminate between the different object classes in the scene is important. Or put it differently, correct alignment between objects in the image and the category names mentioned in the command is a basic requirement. RNM shows us that concentrating on nouns already gives a big improvement over a purely random strategy. In a separate experiment using ground truth bounding boxes the RNM system obtained an IoU of 54\% showing the importance of the alignment of a found object to the correct category name.
Fourth, the command length has a negative impact on most models as can be seen in \ref{fig:subset_tests}(c).
We argue that when commands get longer there might be more irrelevant information included which the models have difficulty to cope with.
Fifth,
from our experiments we found that the non-RPN systems are roughly two times faster
than the RPN-systems.
This is due to the fact that these RPN-systems have to align every proposed region with the command.
On the other hand, the non-RPN systems only have to encode the full image once and then reason over this embedding.
Lastly, when looking at the ambiguity test in Fig. \ref{fig:subset_tests}(d) we see that all models struggle when the number of ambiguous objects of one category increase except for STACK and MAC, whose performance remains fairly stable.
We believe they benefit from
the multiple reasoning steps before giving an answer where modifier constructions in language disambiguate the referred object.
In a separate experiment we have focused on object referral in extra long commands with ambiguous objects of the same category, where we observe the same trends.
\vspace{-0.18cm}
\paragraph{Influence of Region Proposal Quality}
We consider the case when we pre-train a RPN on all keyframes from the training videos, rather than only on the images with commands.
It is found that the test performance of the OSM model increased from 35.31 to 40.78\%.
Similarly, the test performance of the SCRC model increased from 38.70 to 41.15 \% showing the importance of starting from good region proposals.

\paragraph{Blanking out the commands}
\cite{cirik2018visual} found that some referential datasets have some kind of bias in the dataset when blanking out the question.
We evaluated this with both SCRC and OSM by changing the question vector to a zero filled vector and we respectively got the following results. For SCRC we get 40.37\% $IoU_0.5$ (38.70\% with command), OSM: 21.65\% (35.31\% with command). From these results we can conclude two things. First, global information which was added to the local representation of each region in the SCRC model, contains some kind of bias that the models can learn. Second, if no global information is used, as is the case in OSM, the model $IoU_0.5$ actually decreases dramatically indicating that there is not a high bias in the image itself.

\paragraph{Influence of Using Pre-trained Word Embeddings}
Using pre-trained word GloVe embeddings \cite{pennington2014glove}
had no effect on or even lowered the IoU obtained on the test set.
We argue that words like 'car' and 'truck' are very close to each other in the embedding space but for the model to perform well it should be able to discriminate between them.
We also tested ELMO \cite{Peters:2018} and BERT \cite{devlin2018bert} embeddings but found that they gave only minor improvements for some models.
\section{Conclusions and Future Work}
\label{sect:conclusion}
We have presented a new dataset, \texttt{Talk2Car}, that contains commands in natural language referring to objects in a visual urban scene which both the passenger and the self-driving car can see.
We have compared this dataset to existing datasets for the joint processing of language and visual data and have performed experiments with different strong state-of-the-art models for object referral, which yielded promising results.
The available 3D information was neglected to be able to compare existing models but we believe that it
could help in object referral as it contains more spatial information which, as seen in the experiments, is an important factor.
This 3D information will help to translate language into 3D. Moreover, it will allow to perform actions in 3D based on the given command.
Also, the \texttt{Talk2Car} dataset only allows people to refer to one object at a time. It also doesn't include path annotations for the car to follow, nor does it have dialogues if a command is ambiguous.
In future versions, \texttt{Talk2Car} will be expanded to include the above annotations and dialogues. However,
this first version already offers a challenging dataset to improve current methods for the joint processing of language and visual data and for the development of suitable machine learning architectures. Especially for cases where the ambiguity in object referral can be resolved by correctly interpreting the constraints found in the language commands, \texttt{Talk2Car} offers a natural and realistic environment to study these.

\acknowledgments{
This project is sponsored by the MACCHINA project from the KU Leuven  with grant number C14/18/065.
We would like to thank Nvidia for granting us two TITAN Xp GPUs.
We would also like to thank Holger Ceaser from nuTonomy for providing help with the original nuScenes dataset.
We'd also like to thank Tinne Tuytelaars and Matthew Blaschko for their mentorship and good advices.
}

\newpage
\clearpage


\section*{Introduction}

In this paper, we present a framework for using photogrammetry-based synthetic data generation to create an end-to-end deep learning pipeline for use in industrial applications.

While deep learning techniques have documented great success in many areas of computer vision, a key barrier that remains today with regard to large-scale industry adoption is the availability of data that can be used for model training. While large high-quality datasets are readily available for use with common object categories like animals and household items, this no longer holds in the case of many potential applications (e.g. the products in an industrial warehouse). For such applications, costly and labour-intensive data acquisition and labelling must first be carried out before deep learning can be applied to the task at hand. As deep learning moves out of the academia and into industry, this limitation poses a serious problem for potential users: how can one cheaply and efficiently acquire training data when a large dataset does not already exist?

Working in collaboration with a leading UK online supermarket, we address the problem of dataset generation for fixed-appearance objects: object classes whose appearance have little to no change for all instances within a class. Such objects are ubiquitous in many potential deployment settings and include items such as consumer products, industrial goods, and machine parts, among others.

To that end, we propose combining the use of 3D Modelling and render-based image synthesis to generate a synthetic dataset which can be used to train a deep Convolutional Neural Network (CNN), a type of neural network which is frequently used for computer vision tasks. The inspiration comes from the following insight: for fixed (or low variation) appearance object classes, texture and geometry information can be thought of as unchanging and can therefore be captured from a small number of physical samples without the risk of overfitting to individual specimens. Our approach builds on the existing literature by generating 3D scans of physical objects using photogrammetery. By rendering scenes from these realistic 3D models, we were able to generate a diverse array of synthetic images that were used as training data.

While previous work has made use of computer-generated 3D models to train CNNs, our study extends this concept by successfully demonstrating that this approach can be extended to 3D models acquired using photogrammetry.

In order to demonstrate our approach, we sought to train a deep learning model capable of performing the task of recognizing groceries. 10 products were chosen for a classification task. Our results were promising: using our approach, we were able to successfully train and optimize a CNN that achieves a maximum classification accuracy of 95.8\% on a General Environment Test Set.

Furthermore, our use of automatic training data generation has also enabled the automatic annonation and segmentation of training data. This has allowed us to train an object-detector for the same set of objects that is surprisingly robust.

\section*{Related Work}

\subsection*{3D Modelling in Deep Learning}

3D modelling has long been a mainstay of computer vision research. Nonetheless, it is only more recently that its potential applications to deep learning-based image classification and object detection have been considered, with 3D modelling used in conjunction with CNNs to train networks for use on \textit{real} images.
 \cite{Su2015} demonstrate the use of 3D models for viewpoint estimation. By creating a database of millions of \textit{rendered} training images using CAD models drawn from existing 3D model repositories, they were able to train a CNN that outperformed state-of-the-art methods on test sets containing \textit{real} images. Similarly, \cite{Peng2015} and \cite{Sarkar2006} use a large number of 3D CAD models of objects to render realistic looking training images; the output was used to train a classifier for classifying real world images of the objects. \cite{object_detection_synthetic} extend this approach by applying synthetic data generated from 3D CAD models to the problem of object detection; they demonstrate that using domain randomization (i.e. varying parameters such as lighting, pose and object textures), it is possible to train a compelling object detection network on automatically generated, non-photorealistic, synthetic data.

While previous work has exclusively made use of computer-generated 3D models (CAD software), our study extends this concept by successfully demonstrating that this approach can be extended to 3D models acquired using photogrammetry.

\subsection*{3D Multi-Image Photogrammetry}

3D multi-image photogrammetry is the process of reconstructing the three-dimensional properties of a selected object from a set of multiple two-dimensional images. Using such a system, one is able to accurately recreate the surface geometry, texture, color and shape of the target object in a realistic manner.

The principles and techniques involved in multi-image photogrammetry has been the subject of much active computer vision research and have been comprehensively explored in the literature \cite{luhmann_robson_kyle_harley_2007, linder_2018, faugeras_1992}. The fundamental mathematical model of photogrammetry is central projection imaging \cite{luhmann_robson_kyle_harley_2007}. In this model, every 2D image is first used to generate a spatial bundle of rays; when all the ray bundles from multiple images are intersected, triangulation can be used to simultaneously orientate all images and calculate every three-dimensional object point location.

State-of-the-art commercial systems are currently able to perform multi-image photogrammetry with minimal effort and without the need for calibrated cameras \cite{agisoft}. It is even possible to perform photogrammetry using only a smartphone camera \cite{qlone}. Currently available photogrammetry systems thus give us the ability simply and effectively generate 3D models from real-world objects, thereby paving the way for our object-to-model deep learning paradigm.

\section*{Methods}

\begin{figure*}
    \centering
  \includegraphics[width=0.9\textwidth]{Fig1.png}
  \caption{Overall System Design}
  \label{fig:system_design}
\end{figure*}

Figure \ref{fig:system_design} shows the overall design of our object-to-model training pipeline, which is comprised of four main stages. The first two stages, 3D Modelling and Image Rendering, are used to generate a synthetic dataset from physical samples of the target objects, while the next two stages are used to train and validate a deep learning model trained on the synthetic dataset.
\begin{itemize}
    \item \textbf{3D Modelling: } This stage involves scanning physical products to produce 3D models. These include texture and colour representations of the product and are of high enough quality to produce realistic images in the next stage.
    \item \textbf{Image Rendering: } This stage produces a specified number of synthetic training images for each object which vary object pose, lighting, background and occlusions.
    \item \textbf{Network Training: } For the purposes of this study, several deep CNN architectures were trained to classify grocery items using rendered images generated from the image rendering step.
    \item \textbf{Testing and Evaluation: } The trained classifier is then tested on a relatively small number (100 images per class) of test images collected.
\end{itemize}

These four stages fully define our end-to-end pipeline for generating and evaluating an image recognition network, for a number of specified fixed-appearance objects. In  essence, the full pipeline takes in a set of physical objects and outputs a trained model. Each of the four stages will be described in detail in the sections below.

\subsection*{3D Modelling} \label{3d-modelling}
The goal of the 3D Modelling stage was to demonstrate a means by which 3D models of target objects can be feasibly created to be used as input for the image synthesis stage. The method employed required the constructed models to be of high enough quality to generate photo-realistic and consistent representations of itself, as well as being economical to employ in a research setting.

Photogrammetry was used as a tool of choice. This technique takes 2D images of a 3D surface (photographs captured using a conventional camera) as input, and attempts to reconstruct the surfaces primarily using texture cues on the surfaces by following the steps below:

\begin{itemize}
\item \textbf{Camera calibration} - this is done automatically using matching features in the images, and estimating the most probable arrangement of cameras and features. A sparse point cloud of features on the modelled surface is calculated.
\item \textbf{Mesh generation} -  the point cloud is then triangulated and used to create a structural mesh of the surface. \item \textbf{Texture generation} - texture information on the mesh surface is recovered by combining information from the original images.
\end{itemize}

\begin{figure*}
\centering
  \includegraphics[width=0.9\textwidth]{Fig2.png}
  \caption{Visualization of the steps in photogrammetry: (A) camera calibration and point cloud generation; (B) after mesh generation; (C) after texture generation.}
  \label{fig:agisoft_views}
\end{figure*}

Figure \ref{fig:agisoft_views} shows a visualization of the steps described above as applied to an example object. This approach has allowed us to fully capture all geometry and texture information of all modelled objects. Minimal human effort is required - typically no more than 40 images were required per model, which can be done manually at a rate of approximately 5 minutes per product. In an industrial setting, this can be carried out even faster using various commercial photogrammetry-based photo-capture systems which can be used to automate the 3D modelling process.

\subsection*{Image Rendering}

The goal of the image rendering stage is to produce an infinite supply of high-quality training data. This stage involves using a rendering engine to render a pose image of the 3D Model. The appearance of this image would depend on a user-defined distribution of rendering parameters \(\boldsymbol{\theta}\). This is combined with a background to generate the final training image. These steps are repeated to generate a potentially unlimited number of training images. Figure \ref{fig:rendering_flow_diag} provides an illustration of this process.

\begin{figure}[h]
\centering
\includegraphics[width=0.55\textwidth,height=4.0cm]{Fig3.png}
\caption{Flow diagram for synthetic image rendering}
\label{fig:rendering_flow_diag}
\end{figure}

Scene appearance was fully defined with the following parameters: camera position w.r.t object (defined via an azimuth \(\theta\) and elevation \(\phi\)), camera distance to the object, lighting intensity (equivalently distance), number of lights.

As a simple heuristic, the camera location was defined to be evenly distributed around rings in a spherical coordinate system. This is illustrated in Figure \ref{fig:camera_locations_annot}. The reason for this choice of distribution was due to the fact that it corresponded to common viewpoints of handheld grocery items.

\begin{figure}[h]
\centering
\includegraphics[width=0.8\textwidth,height=5.0cm]{Fig4.png}
\caption{(A) The rings on the shell (red rings) around which random normalized camera locations are sampled from (blue points); (B) The uniform distribution on the sphere of lamp locations.}
\label{fig:camera_locations_annot}
\end{figure}

Lamp locations were distributed evenly on a sphere. Additionally, camera-subject distance and lamp energies were distributed according to truncated normal distributions to ensure no negative energy lamps were generated. Point lamps were used primarily as a simple lighting model.

\subsubsection*{Location Distributions}

We generated location distributions according to the method below. Assuming that one could define Cartesian coordinates (\(x,y,z\)) in terms of spherical coordinates (defined using radius \(\rho\), azimuth \(\theta\) and elevation \(\phi\)) as:
\begin{equation}
    x= \rho \cos \theta \sin \phi \quad y=\rho \cos \theta \sin \phi \quad z = \rho \cos \phi
\end{equation}

Distribution of camera locations were defined by stating:
\begin{align}
	\rho & \sim \mathcal{T}(\mu_\rho, \sigma_\rho, a_\rho, b_\rho) \\
    \phi & \sim \mathcal{T}(0, \sigma_\phi, -\pi/2, \pi/2) \\
    \theta & \sim \mathcal{U}(0, 2\pi)
\end{align}

Where \(\mathcal{T}(\mu, \sigma, a, b)\) is the truncated normal distribution, with mean \(\mu\) and standard deviation \(\sigma\), \(a\) and \(b\) define the limits of the distribution, for which the probability density function (PDF) is zero outside the limits. Therefore if \(X \sim \mathcal{T}(\mu, \sigma, a, b)\), then \(X \sim  \mathcal{N}(\mu, \sigma)\) if \(a \leq X \leq b\). This set of variables define a distribution around a ring in the X-Y plane (with a normal Z), and the width of the ring can be controlled by specifying \(\sigma_\phi\). If \(\mathbf{x}=(x,y,z)\) is drawn from the distribution, one can 'flip' the ring to have normal aligned with the X axis by doing:

\begin{align}
	\mathbf{x'}=\mathbf{R_y}(\pi/2)\mathbf{x}
\end{align}

Where \(\mathbf{R_y}(\pi/2)\) is the rotation matrix that rotates the point about the axis y, \(\pi/2\) radians. This way points can be distributed around multiple rings (specified by their normals). For the purposes of this project, camera was distributed about the rings with the Y and Z axes as normals.

It is also worth mentioning that setting \(\sigma_\phi\) to roughly \(\pi/3\) radians generates a roughly uniform distribution of points around a sphere. This approach was used for generated lamp distributions.

\subsubsection*{Lighting Conditions}

The number of point light sources \(n_L\) was also sampled from a uniform distribution of non-negative integers (the min and max can be user-defined). The lamp energies \(E\) were sampled using a truncated normal distribution:
\begin{align}
    n_L \sim \mathcal{U}\{n_{min}, n_{max}\} \\
    E \sim \mathcal{T}(\mu_E, \sigma_E, 0, +\infty)
\end{align}

\subsubsection*{Background Scene Generation}
The rendered pose image is generated with a transparent background. Alpha composition was used to combine the rendered pose image with a background image. The resulting generated images are highly varied in terms of appearance. While a significant proportion of images seemed to look 'unrealistic' (e.g. a yogurt pot in the International Space Station), the aim was not to achieve a simulation with perfect realism. Instead, the focus was on achieving enough background variation within the training data so that the final trained network would be as robust as possible, while ensuring that the \textit{objects} themselves were rendered as realistically as possible.

Figure \ref{fig:collate_rendered_images} shows examples of synthetic training data generated using the method described above. Note the variety of poses and lighting conditions represented, as well as the multitude of different backgrounds.

\begin{figure}[h]
\centering
\includegraphics[width=0.8\textwidth,height=7.5cm]{Fig5.png}
\caption{Examples of generated synthetic images}
\label{fig:collate_rendered_images}
\end{figure}

\subsection*{Network Training}
The aim of the Network Training stage was to take a generated dataset as an input, and produced a trained Neural Network that could be used to classify products from the dataset. The tool of choice for this task was a Convolutional Neural Network (CNN), a class of neural networks that are frequently used for image classification tasks. CNNs are specialized neural networks that perform transformation functions (called convolutions) on image data. Deep CNNs contain hundreds of convolutions in series, arranged in various different architectures. It is common practice to take the output of the CNN and input it into a regular neural network (referred to as the fully-connected or FC layer) in order to perform more specialised functions (in our context, a classification task). We refer readers to \cite{deepCNNreview} for a comprehensive review of the use of Deep CNNs in Image Classification.

\section*{Experimental Setup}

\subsection*{3D modelling}
    In practice, our 3D Modelling stage involved two main steps:
    \begin{itemize}
        \item Step 1: Image capture of the objects to be modelled.
        \item Step 2: The generation of 3D models from captured images using a 3rd party, free-for-research photogrammetry software.
    \end{itemize}
    The first step involved capturing good images of the modelled object. The key here is to capture images so as to aid in the 3 steps for photogrammetry described in the 3D modelling subsection. Since camera calibration is performed by matching sets of features between groups of images to estimate the extrinsic (geometric) camera parameters, it is paramount we had good overlap between images and a large number of unique features to aid in reconstruction. Figure \ref{fig:3d_acquisition_process} shows an example subset of sample images captured for 3D modelling. \\\\
    For most of our object models, which generally measured no more than 20 cm at the longest, the distance for image capture was set at about 40cm away from the object . Images were captured at 2 levels, roughly 30 and 60 degrees elevation from the centre of volume of the object. The azimuth of the camera about the object was changed with increments of 20 degrees (see Figure \ref{fig:agisoft_views} for a visualization of typical camera placement relative to the object). A base with distinct patterns was also placed below the captured object to aid in feature matching.\\\\
    In the second step of the 3D Modelling stage, AgiSoft Photoscan \textsuperscript{\textregistered} was used with the images from the previous step as input. This is free-for-research software and can be used by requesting a free license \footnote{https://www.agisoft.com/buy/licensing-options/}. The software offers an automated pipeline that performs camera calibration / alignment and point cloud generation simultaneously. Mesh generation is then followed by texture generation. This generates a mesh file in the .obj format, which contains positions of vertices in space, its UV coordinate and polygons in the form of vertex lists, and a texture .jpeg file.

\begin{figure}[h]
\centering
\includegraphics[width=0.95\textwidth]{Fig6.png}
\caption{Example subset of images captured for the modelling of a yogurt pot at different elevations and angles. Notice how a magazine was used as a base to create more keypoints for camera alignment}
\label{fig:3d_acquisition_process}
\end{figure}

\subsection*{Image Rendering}
Blender, an open-source software, was used for the rendering of the synthetic training data. A Python layer was built on top of Blender API allowing for programatic control of rendering parameters, which implemented the rendering algorithm described in the Methods section, and enabled automated generation of training data at scale. The output of this layer was a 300*300 pixels RGBalpha image, which contained the product of interest in random orientation and random lightning, while keeping the alpha values of the remaining pixels as zero. The final image was created by merging it with a random background image from the SUN database of over 80 000 images  \cite{SUN:5539970}. For this purpose a Python wrapper around the open-source Python Imaging Library was built. Both Blender and the PIL wrappers have been open sourced with this paper to facilitate reproducibility and future work. \footnote{https://github.com/921kiyo/3d-dl}
\subsection*{Datasets}

\subsubsection*{Training Data}
Using our 3D modelling and image rendering methods, we generated a training data set which consists of 100,000 training images for 10 different grocery products \footnote{10 grocery products used in our study are 1.Anchor Spreadable, 2.Innocent Coconut Water, 3.Essential Waitrose Low Fat Natural Cottage Cheese, 4.Liberte Honey Greek Style Yogurt, 5.Rachel's Organic Greek Style Lemon, 6.Yeo Valley Organic Strawberry Yogurt, 7.New Covent Garden Slow Roast Tomato Soup, 8.Alpro Longlife Original Soya Milk Alternative, 9.Munch Bunch Strawberry Squashums, 10.Cypressa Traditional Halloumi Cheese
} (10,000 images per class) with manually chosen values for the distributions of rendering parameters. Figure \ref{fig:collate_rendered_images} shows examples of the generated training data.

The durations for generating training images is shown in Table \ref{tab:performance_runtimes}. The training set used in our experiments took approximately 9 hours to generate on our Titan X machine.
\begin{table}[h]
\centering
\begin{tabular}{|l|l|l|llll}
\cline{1-3}
\multicolumn{3}{|l|}{Rendering (100K images)}
&  \\ \cline{1-3}
Samples & Resolution (px) & Runtime (hours)
&  \\ \cline{1-3}
64                & 224             & 4
&  \\ \cline{1-3}
128               & 224             & 6
&  \\ \cline{1-3}
64                & 300             & 5.5
&  \\ \cline{1-3}
128               & 300             & 9               &                       &                                   &                                    &  \\ \cline{1-3}
\end{tabular}
\caption{The runtimes for rendering. Note that rendering runtimes depend heavily on rendering settings. Hardware used was an Nvidia GTX Titan X GPU.}
\label{tab:performance_runtimes}
\end{table}

\subsubsection*{Test Data}
To evaluate the classifier's generalization performance,  a special test and validation set was acquired by conventional image capture. As the aim of our method was to create training data for unique fixed-appearance objects for which no suitable dataset already exists, it was not considered constructive to make use of standard test sets such as ImageNet \cite{imagenet_cvpr09}, given that these test sets generally focus on generic object categories (e.g. car) rather than specific individualized objects. Figure \ref{fig:proprietary_general_test_set} displays some examples of our test set which was manually acquired in a variety of locations. The set contains 1000 images of 10 classes (100 images per class) from a variety of perspectives, at different distances, in different lighting conditions and with and without occlusion. These images were acquired with a number of different devices, including smartphones, DSLR cameras and digital cameras.

\begin{figure}[h]
\centering
\includegraphics[width=0.7\textwidth, height=4cm]{Fig7.png}
\caption{Sample of Proprietary General Environment Test Set}
\label{fig:proprietary_general_test_set}
\end{figure}

\subsection*{Network Training}

For network training, three different CNN architectures were tested using the Keras API \cite{chollet2015keras}. These were Google's InceptionV3 \cite{Szegedy2014}, the Residual Network (ResNet-50) \cite{He2015} architecture, and the VGG-16 \cite{Simonyan2014} architecture.

A fully-connected (FC) layer with 1024 hidden nodes and 10 output nodes for each class was defined, and a standard stochastic gradient descent optimizer with momentum was used. For all of the above architectures, the only parameter that was manipulated was the number of trained convolutional layers (the FC layers are always trained), ranging from zero layers to all layers. The final reported models were trained with all convolutional layers unfrozen for retraining. The weights for the convolutional layers were initialized based on those trained using the ImageNet dataset \cite{imagenet_cvpr09}. All other parameters were kept constant as shown in Table \ref{tab:network_const_params}.
Additionally, for the InceptionV3 model a grid-search was carried out to fine-tune the learning rate used.

All computation was performed on a system using an Intel Xeon E5-1630 v3 3.70GHz CPU with 31GB memory and an NVIDIA GTX TITAN X GPU. A virtual environment was used with Tensorflow 1.4.1 and Keras 2.1.3 installed.

\begin{table}
    \medskip
    \centering
    \begin{tabular}{|l|l|}
        \hline
    	Learning rate (Inception V3)        & 0.00256    \\
    	\hline
    	Learning rate (VGG16, ResNet50)     & 0.0001                            \\
    	\hline
    	Image input size (px,px)            & (224,224)                         \\
    	\hline
    	Batch size                          & 64                                \\
    	\hline
    	Number of fully-connected layers    & 2                                 \\
    	\hline
    	Hidden layer size                   & 1024                              \\
    	\hline
        Optimizer                           & SGD                               \\
        \hline
        Epochs
            & 12 \\
        \hline
    \end{tabular}
    \caption{Constant variables for network training}
    \label{tab:network_const_params}
\end{table}

\section*{Experimental Results}

We evaluated the performance of our system on two common computer vision tasks, namely image classification and object detection.

\subsection*{Image Classification} \label{sec:classific_results}

The first task evaluated was to perform image classification on the images in our General Environment Test Set. We were able to achieve a test accuracy of \textbf{95.8\%} using the InceptionV3 architecture with the optimized learning rate and all convolutional layers being retrained. The confusion matrix for this network is shown in Figure \ref{conf_mat}.

\begin{figure}[h]
\centering
\includegraphics[width=1.1\textwidth]{Fig8.png}
\caption{Confusion Matrix for InceptionV3 using Synthetic Data on General Environment Test Set}
\label{conf_mat}
\end{figure}

The trained network was able to classify almost every single image that it could reasonably have been expected to successfully identify. In fact, the majority of the few missclasified images showed the underside of products. From this point of view, the product is usually seen as a white circle or rectangle. This means that there are few to none distinct features by which the different products can be distinguished, making the classification of products shown from the underside extremely challenging, even for humans.

Experimentation with different CNN architectures yielded favorable results, with the InceptionV3 architecture achieving the best performance. These results are shown in Table \ref{tab:arch_test_results}.

\begin{table}[]
    \medskip
    \centering
    \begin{tabular}{|l|l|l|l|l|} \hline Network & Data Source & Accuracy & A. Precision & A. Recall \\
        \hline
        InceptionV3  & Synthetic Images & 95.8       & 96.0                & 95.8           \\
        \hline
        ResNet50     & Synthetic Images & 93.8       & 93.9                & 93.8           \\
        \hline
        VGG16        & Synthetic Images & 83.2       & 84.6                & 83.2           \\
        \hline
        InceptionV3        & Augmented Real Images & 64.5       & 75.7                & 64.5           \\
        \hline
    \end{tabular}
    \caption{Summary of testing results (Accuracy, Data Source, Average Precision, Average Recall) for different architectures}
    \label{tab:arch_test_results}
\end{table}

These results demonstrate that a classifier can be successfully trained using only synthetic data acquired using photogrammetry techniques. To our knowledge this is the first demonstration of the use of photogrammetry-based synthetic data in successfully training a Convolutional Neural Network.

\subsubsection*{Benchmark Experiment}

\begin{figure}[h]
\centering
\includegraphics[width=1.1\textwidth]{Fig9.png}
\caption{Confusion Matrix for InceptionV3 Benchmark Experiment on General Environment Test Set}
\label{conf_mat_augmented}
\end{figure}

To further demonstrate the value of our approach, we also performed a comparison experiment evaluating the performance of a network trained on real images, as opposed to the synthetic data used in our proposed method. To ensure a meaningful comparison, the real images used were the same images used to create the 3D models for synthetic data generation.

During the 3D modelling stage, we used a DSLR camera to capture 60 real images of each product from all angles, and subsequently used about 30-40 of those images to generate the 3D models for use in synthetic data. For this control experiment, we used transfer learning to train a network on those 60 images per class, using a data generator with image augmentation to produce an arbitrarily large dataset for training. To carry out transfer learning, we initialised an InceptionV3 network with ImageNet weights, further trained all layers using the augmented real images. InceptionV3 with transfer learning achieved convergence. However, this performance failed to generalise adequately, with the trained model achieving only 64.5\% accuracy on the General Environment Test Set; while demonstrating some evidence of learning, this result fell far below the model trained on synthetic images. The confusion matrix for this network is shown in Figure \ref{conf_mat_augmented}.

The results of the control experiment show that in situations where only a small number of training images can be captured, our method of generating synthetic training data from 3D models is more effective than the alternative of using image augmentation.

\subsection*{Object Detection}

Our approach opens up the opportunity to train object detectors and segmentation algorithms given that our pipeline can produce pixel-level annotations of products. This is possible as the placement of the object of interest in the image is fully under our control, and can be logged automatically. For this application, we have logged the object bounding box. This information can then be compared with the detector estimated bounding box. This provides a huge advantage as no manual pixel annotation is necessary as is the case with currently available datasets, e.g. the Microsoft Coco dataset \cite{Lin2014}. First experiments using RCNNs on our dataset show strong results when performing object detection on our General Environment Test Set.

The chosen architecture for this task is the RetinaNet architecture \cite{DBLP:journals/corr/abs-1708-02002}. This is a one-stage architecture that runs detection and classification over a dense sampling of possible sub-regions (or 'Anchors') of an image. The authors have claimed that it outperforms most state-of-the-art one-stage detectors in terms of accuracy, and runs faster than two-stage detectors (such as the Fast-RCNN architecture \cite{DBLP:journals/corr/Girshick15}). A 'detection as classification' (DAC) accuracy metric was calculated by using the following formula:

\begin{equation}
	DAC = \frac{\sum_i^n{TP_i}}{n}
\end{equation}

Where the quantity \(TP_i\) is summed over every image \(i\), and is calculated:
\begin{equation}
	TP_i = \begin{cases} 1 & \mbox{if top 3 detections for i contain correct class} \\
	0  & \mbox{otherwise} \end{cases}
\end{equation}

The same General Environment Test Set introduced above was used to test the accuracy of this learning task, when the same set of rendered training images were used. The rendered images were split into a train and validation set. The validation split of the rendered images is meant to be used to measure the deviation in the network performance over real vs. rendered images. The number of rendered validation images per class used was the same as that of the real test images. Figure \ref{detections_collated} highlights the performance of the detection algorithm when applied to a set of sample test images.

\begin{figure*}[]
\begin{center}
\includegraphics[width=\textwidth]{Fig10.png}
\caption{Detection bounding boxes and confidence scores for classification}
\label{detections_collated}
\end{center}
\end{figure*}

The reported DAC accuracy for the test set was 64\%. Upon closer inspection of the test results, it was discovered that the detection bounding boxes were mostly accurately calculated. However, the main factor driving the score down was incorrect classifications. This classification loss was less pronounced in the rendered image validation dataset, giving a final accuracy of 75\%.

\begin{figure}[h]
\begin{center}
\includegraphics[width=\textwidth]{Fig11.png}
\caption{Real and rendered image validation scores for product detection, logged every 200 training steps. The DAC accuracy for real images deviates from the rendered accuracy at around 20K steps.}
\label{dac_detection}
\end{center}
\end{figure}

The logged validation accuracy for both real and rendered data is shown in Figure \ref{dac_detection}, and indicates a deviation between how the network perceives real images of objects, and rendered images. This is a surprising finding, given near-perfect performance on the classification task as noted above. This shows that our current set of rendering parameters might not be as robust as previously thought, and can be dependent on the learning task. It is likely that further optmization of the rendering pipeline can be done in order to optimize against the detection task, and to make it more robust against a larger variety of learning tasks.

\section*{Discussion}
\subsection*{Novel Contribution}

In the course of our study, we developed an original pipeline that goes from data acquisition, over data generation to training a CNN classifier. This is a novel combination of several tools including
photogrammetry for 3D scanning and the use of a graphics engine for training data generation. The
95.8\% accuracy achieved on the General Environment Test Set demonstrates that photogrammetry-based
3D models can be successfully used to train an accurate classifier for real-world images. To our
knowledge this the first time photogrammetry-based 3D models were used to train a CNN. In addition, we developed an end-to-end software tool for synthetic dataset generation, which includes the development of Blender API, Scene generation library as well as the evaluation of the classifier. All the code used in our study has been open-sourced.\footnote{https://github.com/921kiyo/3d-dl}.

\subsection*{Limitations}

Throughout the research conducted into this approach, two main limitations became apparent.

First, using photogrammetry for 3D modelling results in reconstruction noise in the case of transparent features and large unicolor areas, thereby reducing overall network performance. This could be mitigated by combining photogrammetry with other information sources. For example, products from \cite{structure} use an infrared iPad mount to combine information from a typical camera with information from an infrared sensor to create more accurate 3D models. Industrial-grade RGBD scanners which combine depth and colour information are also readily available on the market, allowing for the creation of 3D models of far greater accuracy.

Second, the process of acquiring 3D models and preparing them for rendering required roughly 15 minutes of manual work per product. While this is manageable for a small product set, this would become an increasingly serious limitation as the number of products to be scanned increases. A deployment-ready solution using our method will require a more sophisticated approach to acquiring the product photos in order to minimize the amount of manual labor required. This could include investments into appropriate hardware such as the above-mentioned industrial grade 3D scanners as well as automating the scanning process. For larger products (e.g. cars) a large room or warehouse could be fitted with multiple movable cameras to capture photographs for photogrammetry.

\section*{Further Work}

The viability of using photogrammetry-based data synthesis for training deep neural networks sets the stage for further research capitalizing on the advantages of this approach.

For example, an area holding great promise is that of \textit{expanded hyperparameter optimization}. Standard neural network pipelines generally allow for the optimization of network hyperparameters, whilst assuming a fixed pre-existing dataset \cite{Bergstra:2012:RSH:2188385.2188395}. The system described here, which performs both data generation and model training, can be thought of as a black box with tunable hyperparameters that include both rendering and network training hyperparameters. This expanded set of hyperparameters provides a more expressive and robust model to maximize performance on real-world computer vision tasks. For example, consider the task of detecting objects in a low-light environment. By measuring performance on  real validation images taken from this environment, an appropriate optimization procedure (e.g. a greedy sequential search, or bayesian optimization [\cite{Snoek:2012:PBO:2999325.2999464, Bergstra:2011:AHO:2986459.2986743}]) may be able to efficiently find a suitable lighting distribution in our rendering framework which maximizes performance of the model by optimizing network training, as well as optimizing the training data itself.

\section*{Conclusion}

The insatiable need for ever-larger quantities of quality training data is one of the main limitations of deep learning. As deep learning is applied more extensively, this limitation will become increasingly evident. Novel applications of deep learning will require novel datasets which will need to be manually created with great cost and effort. In this paper, we have shown that by generating synthetic training data using photogrammetry, we are able to produce training data of sufficiently high-quality for use in deep learning applications, while significantly reducing the amount of work and cost associated with data collection. Furthermore the image generation process allows automatic pixel annotation which allows for the training of detection models. We believe our work holds particular promise for use in industrial applications, where recognition and detection tasks tend to involve a large range of unique objects for which there is likely not to be pre-existing datasets. Our method thus opens up new opportunities for the application of deep learning to fields where large scale data collection and dataset curation has previously been unfeasible or expensive.

\section*{Acknowledgments}
We would like to thank Dr Bernhard Kainz at Imperial College London for his advice and valuable feedback, and to Ocado Group, particularly Luka Milic and David Sharp in the 10x Technology Team, for insightful discussions and suggestions.

\bibliographystyle{ieeetr}