\section{Discussion and Conclusion}

To the best of our knowledge, \nameshort{} is the first algorithm for anytime depth estimation from stereo images. As (low-power) GPUs become more affordable and are increasingly incorporated into mobile computing devices, anytime depth estimation will enable accurate and reliable real-time depth estimation for a large variety of robotic applications.

\section{EXPERIMENTAL RESULTS}
In this section, we empirically evaluate our method and compare it with existing stereo algorithms. In addition, we benchmark the efficiency of our approach on an Nvidia Jetson TX2 computing module.

\begin{figure}[t]
  \centering
 \includegraphics[width=0.8\linewidth ]{figures/kitti.pdf}
\caption{(a)-(d) Disparity prediction from 4 stages of \nameshort{} on KITTI-2015. As a larger computational budget is made available, the prediction is refined and becomes more accurate. (e) shows the ground truth LiDAR image, and (f) shows the left input image.}
  \label{fig:vis}
  \vspace{-0.5cm}
\end{figure}

 \paragraph{\textbf{Implementation Details}}
We implement \nameshort{} in PyTorch~\cite{paszke2017automatic}. See Table~\ref{table:network-detail} for a detailed network description.
Our experiments use an \nameshort{} implementation with four stages, as shown in Figure~\ref{fig:warp} and described in the previous section. The maximum disparity is set to $192$ pixels in the original image, which corresponds to a Stage 1 cost volume depth of $M=192/16=12$. In Stages 2 \& 3 the residual range is $\pm 2$, corresponding to $\pm 16$ pixels in Stage 2 and $\pm 8$ pixels in Stage 3.

All four stages, including the \spnshort{} in Stage 4, are trained jointly, but the losses are weighted differently, with weights $\lambda_{1}=1/4$, $\lambda_{2}=1/2$, $\lambda_{3}=1$ and $\lambda_{4}=1$, respectively.  In total, our model contains 40,000 parameters - this is an order of magnitude fewer parameters than StereoNet~\cite{khamis2018stereonet}, and two orders of magnitude fewer than PSMNet~\cite{zhao2017pyramid}. Our model is trained end-to-end using Adam~\cite{kingma2014adam} with initial learning rate $5e^{-4}$ and batch size 6. On the Scene Flow dataset~\cite{mayer2016large}, the learning rate is kept constant, and the training lasts for 10 epochs in total. For the KITTI dataset we first pre-train the model on Scene Flow, before fine-tuning it for 300 epochs.
The learning rate is divided by 10 after epoch 200. All input images are normalized to be zero-mean with unit variance. All experiments were conducted using original image resolutions. Using one GTX 1080Ti GPU, training on the Scene Flow dataset took 3.5 hours, and training on KITTI took 30 minutes. All results are averaged over five randomized 80/20 train/validation splits.

 Figure~\ref{fig:vis} visualizes the disparity maps predicted at the four stages of our model. As more computation time is made available, \nameshort{} produces increasingly refined disparity maps. The final output from stage 4 is even sharper and more accurate, due to the SPNet post-processing.

\paragraph{\textbf{Datasets}} Our model is trained on the synthetic Scene Flow \cite{mayer2016large} dataset and evaluated on two real-world datasets, KITTI-2012~\cite{Geiger2012CVPR} and KITTI-2015~\cite{Menze2015CVPR}. The Scene Flow dataset contains $22,000$ stereo image pairs for training, and $4,370$ image pairs for testing. Each image has a resolution of $960\times 540$ pixels. As in \cite{kendall2017end}, we train our model on $512\times 256$ patches randomly cropped from the original images. The KITTI-2012 dataset contains 194 pairs of images for training and 195 for testing, while KITTI-2015 contains 200 image pairs for each. All of the KITTI images are of size $1242 \times 375$.

\paragraph{\textbf{Baselines}}
Although state-of-the-art CNN based stereo estimation methods have been reported to reach 60FPS on a TITAN X GPU~\cite{khamis2018stereonet}, they are far from achieving real-time performance on more resource-constrained computing devices such as the Nvidia Jetson TX2.
Here, we present a controlled comparison on a TX2 between our method and four competitive baseline algorithms: PSMNet~\cite{zhao2017pyramid}, StereoNet~\cite{khamis2018stereonet}, DispNet~\cite{mayer2016large}, and StereoDNN~\cite{smolyanskiy2018importance}. The PSMNet model has two different versions: PSMNet-classic and PSMNet-hourglass. We use the former, as it is much more efficient than PSMNet-hourglass while having comparable accuracy. For StereoNet, we report running times using a Tensorflow implementation, which we found to be twice as fast as a PyTorch implementation.

Finally, we also compare \nameshort{} to two classical stereo matching approaches: Block Matching~\cite{konolige1998small} and Semi-Global Block Matching~\cite{hirschmuller2008stereo}, supported by OpenCV~\cite{bradski2000opencv}.

In order to collect meaningful results for these baseline methods on the TX2, we use down-sampled input images for faster inference times. The baseline methods are re-implemented, and trained on down-sampled stereo images - this allows a fair comparison, since a model trained on full-sized images would be expected to suffer a significant performance decrease when given lower-resolution inputs. After obtaining a low-resolution prediction, we up-sample it to the original size using bilinear interpolation.

\subsection{Evaluation Results}
Table~\ref{table:main} contains numerical results for \nameshort{} on the KITTI-2012 and KITTI-2015 datasets. Additionally, Figures \ref{fig:2012_main_result_log_scale} and \ref{fig:2012_main_result_log_scale} demonstrate the evaluation error and inference time of our model as compared to baseline methods. Baseline algorithm results originally reported in \cite{bradski2000opencv,khamis2018stereonet,zhao2017pyramid,dispnet,smolyanskiy2018importance} are shown plotted with crosses. For \nameshort{} as well as the StereoNet and PSMNet baselines, computations are performed across multiple down-sampling input resolutions. Results are generated from inputs at full resolution as well as at $1/4$, $1/8$, and $1/16$ resolution, with lower resolution corresponding to faster inference time as shown on Figs. \ref{fig:2012_main_result_log_scale} and \ref{fig:2015_main_result_log_scale}.
As seen in both plots, only \nameshort{} and StereoNet are capable of rapid real-time prediction at $\geq$30 FPS, and \nameshort{} obtains a drastically lower error rate on both data sets. \nameshort{} is additionally capable of running at over 10 FPS even with full-resolution inputs, and at each possible inference time range, \nameshort{} clearly dominates all baselines in terms of prediction error. PSMNet is capable of producing the most accurate results overall, however this is only true at computation rates of 1 FPS or slower. We also observe that the only non-CNN based approach, OpenCV, is not competitive in any inference time range.

\paragraph{\textbf{Anytime setting}}
We also evaluate \nameshort{} in the \emph{anytime} setting, in which we can poll the model prematurely at any given time $t$ in order to retrieve its most recent prediction. In order to mimic an anytime setting for the baseline OpenCV, StereoNet, and PSMNet models, we make predictions successively at increasingly higher input resolutions and execute them sequentially in ascending order of size. At time $t$ we evaluate the most recently computed disparity map. Figures \ref{fig:2012_main_result_log_scale_anytime} and \ref{fig:2015_main_result_log_scale_anytime} show the three-pixel error rates in the anytime setting. Similarly to the non-anytime results, \nameshort{} obtains significantly more accurate results in the 10-30 FPS range. Furthermore, the times between disparity map completions (visualized as horizontal lines in Figs. \ref{fig:2012_main_result_log_scale_anytime} and \ref{fig:2015_main_result_log_scale_anytime}) are much shorter than for any of the baselines, reducing the amount of wasted computation if a query is issued during a disparity map computation.

\begin{figure}
  \centering
  \vspace{-0.5cm}
  \begin{subfigure}[t]{0.8\linewidth}
   \includegraphics[width=\linewidth ]{figures/plot/2012_main_result_log_scale.pdf}
  \caption{KITTI-2012 results across different down-sampling resolutions}
  \label{fig:2012_main_result_log_scale}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.8\linewidth}
     \includegraphics[width=\linewidth ]{figures/plot/2015_main_result_log_scale_bar.pdf}
    \caption{KITTI-2015 results across different down-sampling resolutions}
      \label{fig:2015_main_result_log_scale}
  \end{subfigure}
  \begin{subfigure}[t]{0.8\linewidth}

   \includegraphics[width=\linewidth ]{figures/plot/2012_main_result_log_scale_anytime.pdf}

  \caption{KITTI-2012 results using the anytime setting}\label{fig:2012_main_result_log_scale_anytime}

  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.8\linewidth}

   \includegraphics[width=\linewidth ]{figures/plot/2015_main_result_log_scale_anytime.pdf}

  \caption{KITTI-2015 results using the anytime setting}\label{fig:2015_main_result_log_scale_anytime}
  \end{subfigure}
  \caption{Comparisons of the 3-pixel error rate (\%)  KITTI-2012/2015 datasets. Dots with error bars show accuracies obtained from our implementations. Crosses show values obtained from original publications.}
  \label{fig:main_results}
  \vspace{-0.5cm}
\end{figure}

\begin{figure}
  \centering
\vspace{-0.5cm}
 \includegraphics[width=0.8\linewidth ]{figures/plot/anytime.pdf}

\caption{Ablation results as three pixel error on KITTI-2015. }
  \label{fig:ablation}
\vspace{-0.5cm}
\end{figure}

\begin{table}
\begin{tabular}{M{1.1cm}M{1.35cm}M{1.35cm}M{1.35cm}M{1.55cm}}
\hline
Dataset &Stage 1 28.9ms &Stage 2  48.8ms &   Stage 3   87.9ms &  Stage 4    97.3ms\\ \hline
 KITTI2012  & $15.1 \pm 1.1$  & $9.9 \pm 0.6$ &  $ 6.7 \pm 0.4$  &  $ 6.1 \pm 0.3$      \\
 KITTI2015  & $14.0 \pm 0.7$  & $9.7 \pm 0.7$ &  $ 6.8 \pm 0.6$  &  $ 6.2 \pm 0.6$     \\ \hline

\end{tabular}

\caption{Three-Pixel error (\%) of \nameshort{} on KITTI-2012 and KITTI-2015 datasets. Lower values are better.}
\label{table:main}
  \vspace{-0.5cm}

\end{table}

\subsection{Ablation Study}
In order to examine the impact of various components of the \nameshort{} architecture, we conduct an ablation study using three variants of our model. The first replaces the U-Net feature extractor with three separated ConvNets without shared weights; the second computes a full-scale prediction at each resolution level, instead of only predicting the residual disparity; while the third replaces the distance-based cost volume construction method with the method in PSMNet~\cite{zhao2017pyramid} that produces a stack of $2 \times M$ cost volumes. All ablated variants of our method are trained from scratch, and results from evaluating them on KITTI-2015 are shown in Fig. \ref{fig:ablation}.

\paragraph{\textbf{Feature extractor}}
We modify the model's feature extractor by replacing the U-Net with three separate 2D convolutional neural networks which are similar to one another in terms of computational cost. As seen in Fig. \ref{fig:ablation} (line \nameshort{} w/o UNet), the errors increase drastically in the first two stages ($20.4\%$ and $7.3\%$). We hypothesize that by extracting contextual information from higher resolutions, the U-Net produces high-quality cost volumes even at low resolutions. This makes it a desirable choice for feature extraction.

\paragraph{\textbf{Residual Prediction}}
We compare our default network with a variant that refines the disparity estimation by directly predicting disparities, instead of residuals, in the second and third stages. Results are shown in Fig. \ref{fig:ablation} (line \nameshort{} w/o Residual). While this variant is capable of attaining similar accuracy to the original model, the evaluation time in the last two stages is increased by a factor of more than six. This increase suggests that the proposed method to predict residuals is highly efficient at refining coarse disparity maps, by avoiding the construction of large cost volumes which need to account for a large range of disparities.

\paragraph{\textbf{Distance-based Cost Volume}}
Finally, we evaluate the distance-based method for cost volume construction, by comparing it to the method used in PSMNet~\cite{zhao2017pyramid}. This method builds multiple cost volumes without explicitly calculating the distance between features from the left and right images.  The results in Fig. \ref{fig:ablation} (line \nameshort{} w/o DBCV) show that our distance-based approach is about $10\%$ faster than this choice, indicating that explicitly considering the feature distance leads to a better trade-off between accuracy and speed.

\section{INTRODUCTION}

Depth estimation from stereo camera images is an important task for 3D scene reconstruction and understanding, with numerous applications ranging from robotics~\cite{mancini2016fast,ye2017self,saxena2007depth,schmid2013stereo} to augmented reality \cite{zenati2007dense,alhaija2018augmented,nguyen2018depth}. High-resolution stereo cameras provide a reliable solution for 3D perception - unlike time-of-flight cameras, they work well both indoors and outdoors, and compared to LiDAR they are substantially more affordable and energy-efficient \cite{luo2016efficient}. Given a rectified stereo image pair, the focal length, and the stereo baseline distance between the two cameras, depth estimation can be cast into a stereo matching problem, the goal of which is to find the disparity between corresponding pixels in the two images. Although disparity estimation from stereo images is a long-standing problem in computer vision~\cite{lucas1981iterative}, in recent years the adoption of deep convolutional neural networks (CNN)~\cite{zbontar2016stereo,mayer2016large,kendall2017end,liang2017learning,pang2017cascade} has led to significant progress in the field. Deep networks can solve the matching problem via supervised learning in an end-to-end fashion, and they have the ability to incorporate local context as well as prior knowledge into the estimation process.

\begin{figure}[t]
  \centering
  \includegraphics[width=\columnwidth]{figures/timeline.pdf}
\caption{Example timeline of \nameshort{} predictions. As time progresses the depth estimation becomes increasingly accurate. The algorithm can be polled at any time to return the current best estimate of the depth map. The initial estimates may be sufficient to trigger an obstacle avoidance maneuver, whereas the later images contain enough detail for more advanced path planning procedures. (3-pixel error rate below time.)  }
  \label{fig:timeline}
 \vspace{-0.4cm}
\end{figure}

On the other hand, deep neural networks tend to be computationally intensive and suffer from significant latency when processing high-resolution stereo images. For example, PSMNet~\cite{zhao2017pyramid}, arguably the current state-of-the-art algorithm for depth estimation, obtains a frame rate below 0.3FPS on the Nvidia Jetson TX2 GPU computing module --- far too slow for timely obstacle avoidance by drones or other autonomous robots.

In this paper, we argue for an \emph{anytime} computational approach to disparity estimation, and present a model that trades off between speed and accuracy dynamically (see Figure~\ref{fig:timeline}). For example, an autonomous drone flying at high speed can poll our 3D depth estimation method at a high frequency. If an object appears in its flight path, it will be able to perceive it rapidly and react accordingly by lowering its speed or performing an evasive maneuver. When flying at low speed, latency is not as detrimental, and the same drone could compute a higher resolution and more accurate 3D depth map, enabling tasks such as high precision navigation in crowded scenes or detailed mapping of an environment.

The computational complexity of depth estimation with convolutional networks typically scales cubically with the image resolution, and linearly with the maximum disparity that is considered~\cite{kendall2017end}. Keeping these characteristics in mind, we refine the depth map successively, while always ensuring that either the resolution or the maximum disparity range is sufficiently low to ensure minimal computation time.
We start with low resolution ($1/16$) estimates of the depth map at the full disparity range. The cubic complexity allows us to compute this initial depth map in a few milliseconds (where the bulk of the time is spent on the initial feature extraction and down-sampling). Starting with this low resolution estimate, we successively increase the resolution of the disparity map by up-sampling and subsequently correcting the errors that are now apparent at the higher resolution. Correction is performed by predicting the residual error of the up-sampled disparity map from the input images with a CNN.
Despite the higher resolution used, these updates are still fast because the residual disparity can be assumed to be bounded within a few pixels, allowing us to restrict the maximum disparity, and associated computation, to a mere $10-20\%$ of the full range.

These successive updates avoid full-range disparity computation at all but the initial low resolution setting, and ensure that all computation is re-used, setting our method apart from most existing multi-scale network structures~\cite{saxena2016convolutional,huang2017multi,ke2016neural}. Furthermore, our algorithm can be polled at any time in order to retrieve the current best estimated depth map. A wide range of possible frame rates are attainable (10-35FPS on a TX2 module), while still preserving accurate disparity estimation in the high-latency setting. Our entire network can be trained end-to-end using a joint loss over all scales, and we  refer to it as \namelong{} (\nameshort{}).

We evaluate \nameshort{} on multiple benchmark data sets for depth estimation, with various encouraging findings: Firstly, \nameshort{} obtains competitive accuracy with state of the art approaches, while having orders of magnitude fewer parameters than the baselines. This is especially impactful for resource-constrained embedded devices. Secondly, we find that deep convolutional networks are highly capable at predicting residuals from coarse disparity maps. Finally, including a final spatial propagation model (SPNet)~\cite{liu2017learning} significantly improves the disparity map quality, yielding state-of-the-art results at a fraction of the computational cost (and parameter storage requirements) of existing methods.

\section{\nameshort{}}

\begin{figure*}[ht]
  \centering
  \includegraphics[width=0.7\paperwidth]{figures/warp_v7.pdf}
  \caption{Network structure of \nameshort{}.}
  \label{fig:warp}
\end{figure*}

Fig.~\ref{fig:warp} shows a schematic layout of the \nameshort{} architecture. An input image pair first passes through the U-Net feature extractor, which computes feature maps at several output resolutions (of scale 1/16, 1/8, 1/4). In the first stage, only the lowest-scale features (1/16) are computed and passed through a disparity network (Fig.~\ref{fig:disp}) to produce a low-resolution disparity map (\emph{Disparity Stage 1}). A disparity map estimates the horizontal offset of each pixel in the right input image w.r.t. the left input image, and can be used to compute a depth map.
Because of the low input resolution, the entire Stage 1 computation requires only a few milliseconds. If more computation time is permitted, we enter Stage 2 by continuing the computation in the U-Net to obtain larger-scale (1/8) features.
Instead of computing a full disparity map at this higher resolution, in Stage 2 we simply correct the already-computed disparity map from Stage 1.
First, we up-scale the disparity map to match the resolution of Stage 2. We then compute a residual map, which contains small corrections that specify how much the disparity map should be increased or decreased for each pixel.
If time permits, Stage 3 follows a similar process as Stage 2, and doubles the resolution again from a scale of 1/8 to 1/4. Stage 4 refines the disparity map from Stage 3 with an SPNet~\cite{liu2017learning}.

In the remainder of this section we describe the individual components of our model in greater detail.

\begin{figure}[t]
  \centering
  \includegraphics[width=\columnwidth]{figures/feature_extractor_v3.pdf}

\caption{U-Net Feature Extractor. See text for details.}
  \label{fig:fe}
\vspace{-0.5cm}
\end{figure}
\paragraph{U-Net Feature Extractor}
 Fig~\ref{fig:fe} illustrates the U-Net~\cite{ronneberger2015u} Feature Extractor in detail, which is applied to  both the left and right image. The U-Net architecture computes feature maps at various resolutions (1/16, 1/8, 1/4), which are used as input at stages 1-3 and only computed when needed. The original input images are down-sampled through max-pooling or strided convolution and then processed with convolutional filters. Lower resolution feature maps capture the global context, whereas higher resolution feature maps capture local details. At scale 1/8 and 1/4, the final convolutional layer incorporates the previously computed lower-scale features.

\paragraph{Disparity Network}
The disparity network (Fig.~\ref{fig:disp}) takes as input the feature maps from the left and right stereo images in order to compute a disparity map. We use this component to compute the initial disparity map (stage 1) as well as to compute the residual maps for subsequent corrections (stages 2 \& 3). The disparity network first computes a disparity cost volume. Here, the cost refers to the similarity between a pixel in the left image and a corresponding pixel in the right image.  If the input feature maps are of dimensions $H\times W$, the cost volume has dimensions $H\times W \times M$, where the $(i,j,k)$ entry describes the degree to which pixel $(i,j)$ of the left image matches pixel $(i,j\!-\!k)$ in the right image. $M$ denotes the maximum disparity under consideration. We can represent each pixel $(i,j)$ in the left image as a vector $\mathbf{p}_{ij}^L$, where dimension $\alpha$ corresponds to the $(i,j)$ entry in the $\alpha^{th}$ input feature map  associated with the left image. Similarly we can define $\mathbf{p}_{ij}^R$. The entry $(i,j,k)$ in the cost volume is then defined as the $L_1$ distance between the two vectors $\mathbf{p}_{ij}^L$ and $\mathbf{p}_{i(j+k)}^R$, i.e. $C_{ijk}=\|\mathbf{p}_{ij}^L-\mathbf{p}_{i(j\!-\!k)}^R\|_1$.

This cost volume may still contain errors due to blurry objects, occlusions, or ambiguous matchings in the input images. As a second step in the disparity network (\emph{3D Conv} in Fig~\ref{fig:disp}), we refine the cost volume with several 3D convolution layers~\cite{kendall2017end} to further improve the obtained cost volume.

The disparity for pixel $(i,j)$ in the left image is $k$ if the pixel $(i,j\!-k\!)$ in the right image is most similar. If the cost volume is exact, we could therefore compute the disparity of pixel $(i,j)$ as $\hat{D}_{ij}=\arg\!\min_{k} C_{i,j,k}$.  However, the cost estimates may be too noisy to search for the hard minimum. Instead, we follow the suggestion by Kendall et al.~\cite{kendall2017end} and compute a weighted average  (\emph{Disparity regression} in Fig~\ref{fig:disp})
\begin{equation}
\hat{D}_{ij} = \sum_{k=0}^{M}k\times \frac{\exp{(-C_{ijk})} } {\sum_{k'=0}^{M}\exp{(-C_{ijk'})}}.
\end{equation}
If one disparity $k$ clearly has the lowest cost (i.e. it is the only good match), it will be recovered by the weighted average. If there is ambiguity, the output will be an average of the viable candidates.

\begin{figure}[t]
  \centering
  \includegraphics[width=\columnwidth]{figures/disparity_net.pdf}
\caption{Disparity network. See text for details.}
  \label{fig:disp}
  \vspace{-0.5cm}
\end{figure}

\paragraph{Residual Prediction}
A crucial aspect of our \nameshort{} architecture is that we only compute the full disparity map at a very low resolution in Stage 1. In Stages 2 \& 3 we predict residuals~\cite{hui18liteflownet}. The most expensive part of the disparity prediction is the construction and refinement of the cost volume.
The cost volume scales $H\times W\times M$, where $M$ is the maximum disparity. In high resolutions, the maximum disparity between two pixels can be very large (typically $M=192$ pixels in the KITTI dataset~\cite{Geiger2012CVPR}).
By restricting ourselves to residuals, i.e. corrections of existing disparities, we can limit ourselves to $M=5$ (corresponding to offsets $-2,-1,0,1,2$) and obtain sizable speedups.

In order to compute residuals in stages 2 \& 3, we first up-scale the coarse disparity map and use it to warp the input features at the higher scale (Fig.~\ref{fig:warp}) by applying the disparity estimations pixel-wise. In particular, if the left disparity of pixel $(i,j)$ is estimated to be $k$, we overwrite the value of pixel $(i,j)$  in each right feature map to the corresponding value of pixel $(i,j+k)$ (using zero if out of bounds). If the current disparity estimate is correct, the updated right feature maps should match the left feature maps. Due to the coarseness of the low resolution inputs, there is typically still a mismatch of several pixels, which we correct by computing residual disparity maps.
Prediction of the residual disparity is accomplished similarly to the full disparity map computation. The only difference is that the cost volume is computed as $C_{ijk}=\|\mathbf{p}_{ij}-\mathbf{p}_{i(j-k+2)}\|_1$, and the resulting residual disparity map is added to the up-scaled disparity map from the previous stage.

\paragraph{\spnlong{}}
To further improve our results, we add a final fourth stage in which we use a \spnlong{} (\spnshort{})~\cite{liu2017learning} to refine our disparity predictions. The \spnshort{} sharpens the disparity map by applying a local filter whose weights are predicted by applying a small CNN to the left input image. We show that this refinement improves our results significantly at relatively little extra cost.

\begin{table}[h!]
\centering
\scriptsize
\begin{tabular}{l|l}
 \hline
 0 & Input image  \\
 \hline
 \multicolumn{2}{c} { \textbf{2-D Unet features} } \\
 \hline
  1 & $3 \!\times\! 3$ conv with 1 filter \\
  2 & $3 \!\times\! 3$ conv with stride 2 and 1 filter \\
  3 & $2 \!\times\! 2$ maxpooling with stride 2  \\
4-5 & $3 \!\times\! 3$ conv with 2 filters \\
  6 & $2 \!\times\! 2$ maxpooling with stride 2  \\
7-8 & $3 \!\times\! 3$ conv with 4 filters \\
  9 & $2 \!\times\! 2$ maxpooling with stride 2  \\
10-11 & $3 \!\times\! 3$ conv with 8 filters  \\
 12 & Bilinear upsample layer 11 (features) into 2x size  \\
 13 & Concatenate layer 8 and 12  \\
14-15 & $3 \!\times\! 3$ conv with 4 filters \\
 16 & Bilinear upsample layer 15 (features) into 2x size  \\
 17 & Concatenate layer 5 and 16  \\
18-19 & $3 \!\times\! 3$ conv with 2 filters \\ \hline
 \multicolumn{2}{c} { \textbf{Cost volume} } \\
 \hline
 20 & Warp and build cost volume from layer 11 \\
 21 & Warp and build cost volume from layer 15 and layer 29  \\
 22 & Warp and build cost volume from layer 19 and layer 36  \\
 \hline
 \multicolumn{2}{c} { \textbf{Regularization} } \\
 \hline
 23-27 & $3 \!\times\! 3 \!\times\! 3$ 3-D conv with 16 filters \\
 28 & $3 \!\times\! 3 \!\times\! 3$ 3-D conv with 1 filter \\
 29 & Disparity regression \\
 30 & Upsample layer 29 to image size: \textbf{stage 1 disparity output}   \\
 31-35 & $3 \!\times\! 3 \!\times\! 3$ 3-D conv with 4 filters \\
 36 & Disparity regression: residual of stage 2 \\
 37 & Upsample 36 it into image size   \\
 38 & Add layer 37 and layer 30 \\
 39 & Upsample layer 38 to image size: \textbf{stage 2 disparity output} \\
 40-44 & $3 \!\times\! 3 \!\times\! 3$ 3-D conv with 4 filters \\
 45 & Disparity regression: residual of stage 3 \\
 46 & Add layer 44 and layer 38   \\
 47 & Upsample layer 46 to image size: \textbf{stage 3 disparity output}  \\\hline
 \multicolumn{2}{c} { \textbf{Spatial propagation network} } \\
 \hline
 48-51 & $3 \!\times\! 3$ conv with 16 filters (on input image)\\
 52 & $3 \!\times\! 3$ conv with 24 filters: affinity matrix\\
 53 & $3 \!\times\! 3$ conv with 8 filters (on layer 47)\\
 54 & Spatial propagate layer 53 with layer 52 (affinity matrix) \\
 55 & $3 \!\times\! 3$ conv with 1 filters: \textbf{stage 4 disparity output}  \\
\hline

\end{tabular}
\caption{Network configurations. Note that a \emph{conv} stands for a sequence of operations: batch normalization, rectified linear units (ReLU) and convolution. The default stride is 1.\label{table:network-detail}}

\end{table}
\section{RELATED WORK}
\paragraph{\textbf{Disparity estimation}} Traditional approaches to disparity estimation are based on matching features between the left and right images \cite{barnard1982computational,scharstein2002taxonomy}. These approaches are typically comprised of the following four steps: (1) computing the costs of matching image patches over a range of disparities, (2) smoothing the resulting cost tensor via aggregation methods, (3) estimating the disparity by finding a low-cost match between the patches in the left image and those in the right image, and (4) refining these disparity estimates by introducing global smoothness priors on the disparity map \cite{scharstein2002taxonomy,hosni2013,hirschmuller2009,hu2012stereo}. Several recent papers have studied the use of convolutional networks in step (1). In particular, Zbontar \& LeCun \cite{zbontar2016stereo} use a Siamese convolutional network to predict patch similarities for matching left and right patches. Their method was further improved via the use of more efficient matching networks \cite{luo2016efficient} and deeper highway networks trained to minimize a multilevel loss \cite{shaked2017improved}.

\paragraph{\textbf{End-to-end disparity prediction}} Inspired by these initial successes of convolutional networks in disparity estimation, as well as by their successes in semantic segmentation \cite{long2015fully}, optical flow computation \cite{fischer2015flownet,ilg2017flownet}, and depth estimation from a single frame \cite{eigen2015single}, several recent studies have explored end-to-end disparity estimation models \cite{mayer2016large,kendall2017end,liang2017learning,pang2017cascade}. For example, in \cite{mayer2016large}, the disparity prediction problem is formulated as a supervised learning problem, and a convolutional network called DispNet is proposed that directly predicts disparities for an image pair. Improvements made by DispNet include a cascaded refinement procedure \cite{pang2017cascade}. Other studies adopt the \emph{correlation layer} introduced in \cite{fischer2015flownet} to obtain the initial matching costs; a set of two convolutional networks are trained to predict and further refine the disparity map for the image pair \cite{liang2017learning}. Several prior studies have also explored moving away from the supervised learning paradigm by performing depth estimation in an unsupervised fashion using stereo images \cite{godard2017unsupervised} or video streams \cite{zhou2017unsupervised}.

Our work is partially inspired by the Geometry and Context Network (GCNet) proposed in \cite{kendall2017end}. In order to predict the disparity map between two images, GCNet combines a 2D Siamese convolutional network operating on the image pair with a 3D convolutional network that operates on the matching cost tensor. GCNet is trained in an end-to-end fashion, and is presently one of the state-of-the-art methods in terms of accuracy and computational efficiency. Our model is related to GCNet in that it has the same two stages (a 2D Siamese image convolutional network and a 3D cost tensor convolutional network), but it differs in that it can perform \emph{anytime prediction} by rapidly producing an initial disparity map prediction and then progressively predicting the residual to correct this prediction. LiteFlowNet~\cite{hui18liteflownet} also tries to predict the residual for optical flow computation. However, LiteFlowNet uses the residual to facilitate large-displacement flow inference rather than computational speedup.

\paragraph{\textbf{Anytime prediction}} There exists a substantial body of work on machine learned models with computational budget constraints at inference time \cite{viola2001robust,grubb2012speedboost,karayev2014anytime,greedymiser,xu2013cost,wang2015acyclic,huang2017multi}. Most of these approaches are based on ensembles of decision-tree classifiers~\cite{viola2001robust,greedymiser,grubb2012speedboost,xu2013cost} which allow for tree-by-tree evaluation, facilitating the progressive prediction updates that are the hallmark of anytime prediction. Several recent studies have explored anytime prediction with image classification CNNs that dynamically evaluate parts of the network to progressively refine their predictions \cite{fractalnet,huang2017multi,veit2017,wu2017blockdrop,mcgill2017,figurnov2017}. Our work differs from these earlier anytime CNN models in that we focus on the structured prediction problem of disparity-map estimation, rather than on image classification tasks. Our models exploit particular properties of the disparity-prediction problem: namely, that progressive estimation of disparities can be achieved by progressively increasing the resolution of the image data within the internal representation of the CNN.

\section*{APPENDIX}

 \section*{ACKNOWLEDGMENT}
 This research is supported in part by grants from the National Science Foundation (III-1618134, III-1526012, IIS1149882, IIS-1724282, and TRIPODS-1740822), the Office of Naval Research DOD (N00014-17-1-2175), and the
 Bill and Melinda Gates Foundation. We are thankful for the
 generous support of SAP America Inc. We thank Camillo J. Taylor for helpful discussion.

	\bibliographystyle{IEEEtranS}


\section{Introduction}

Hyperparameter tuning in modern machine learning systems is critical for building effective models. However, the process--in particular for deep learning architectures--is often unintuitive and time consuming. Due to their effectiveness and wide availability, deep learning architectures are nonetheless popular as developers are willing to struggle with the tuning process. Standardized platforms (e.g., Tensorflow, Keras, PyTorch, etc.) have allowed developers to easily release their models through open-source platforms such as GitXiv and Github. The lack of clarity on which settings to use (and where and when) presents an added challenge to `model consumers.' These down-stream developers must contend with finding ways to \textit{retune} or \textit{fine-tune} models to extend, retrain, or perform new evaluation with new data.

Take for example word embedding models such as word2vec~\cite{mikolov2007language}, GloVe~\cite{pennington2014glove}, or Doc2Vec~\cite{le2014distributed}. These models capture the syntactic (and possibly semantic) relationships between terms by embedding words in a high-dimensional vector space by learning the weights for each vector dimension. The magnitude and direction of the vectors \textit{between} can be used to find related terms (nearest neighbors). Additionally, that can be used for other semantic labeling tasks, such as identifying synonyms or other relationships (e.g., hypernyms, antonyms, etc.). Embedding models, unfortunately, are not always easy to train. Even relatively simple ones, such as word2vec, easily have a dozen hyperparameters that control model construction. Everything from window size (how much text to consider around terms) to learning rate (alpha) to vector size (in what dimensional space) to architecture type (`cbow' or 'skip-gram') to sampling types can be modified. It is difficult to understand which hyperparameter 'knobs' to adjust (and how) in order to achieve a good outcome. Additionally, publicly released models are often trained and evaluated on standard corpora (e.g., Wikipedia and SEMEVAL~\cite{semeval}). Real-world use on new corpora requires identifying both the right ground-truth and the right hyperparameters for that corpora.

\begin{figure}
  \centering
  \includegraphics[width=\linewidth]{figs/fig-lamvi2-screenshot.png}
  \caption{Screenshot of LAMVI-2\xspace. A parallel coordinates visualization (A) provides access to models tuned with different hyperparameters (left) and evaluated on different metrics (right). The central heatmap supports model inspection (rows). In this case we see the distances of various terms (columns) to the query word ``legitimate'' encoded through color. A linked t-SNE visualization (B) provides a dimension-reduced view of the embedding. The user can interact with the visualization to understand the trade-offs between hyperparameters and metric, isolate models that perform well, identify outliers (models and instances), and provide examples for evaluation/supervision.}
	\label{fig:lamvi2-screenshot}
\end{figure}

Automated hyperparameter optimization is one common solution.  The simplest variant, grid-search, progressively varies each parameter to find a good model. More complex schemes use other search mechanisms~\cite{ bergstra2012random, NIPS2011_4443,  greff2016lstm, NIPS2012_4522}. An optimizer will attempt to identify the best hyperparameter set given a specific criteria (e.g., to get the best accuracy, use this configuration). Optimization suffers from two problems. First, each `run' may be expensive so constraining the hyper-parameter search is often useful (e.g., identifying a range). Second, not all real world problems have a single optimization metric and many require trade-offs. For example, one may trade-off precision and recall, or increased model size with increased accuracy, but at the cost of space resources (important on devices such as cell phones). In some situations a generic model may be constructed with multiple uses in mind, and maximum performance across all applications can not be simultaneously achieved. Finally, optimization algorithms often assume `stable' test data. A common scenario is that in adapting a model to a new scenario, the developer will find problems in the training or test data and will add data or labels as part of tuning.

Because of these limitations and, debugging and tuning models often requires human interaction. To support human-in-the-loop model tuning, we introduce LAMVI-2\xspace (LAnguage Model Visual Inspector 2.0). LAMVI-2\xspace is a visualization-driven tool specifically focused on debugging neural embedding models (Figure~\ref{fig:lamvi2-screenshot}). In implementing a variety of prototypes and through task analysis, we identified key decision points in model construction and tuning. Rather than a broad solution, LAMVI-2\xspace uses visual channels to target key features of the data/model that are useful in making decisions. The system allows the developer to focus on hyperparameter interaction and to identify how these parameters impact the model at both a fine level (e.g., specific term embeddings and distances) and in more broad measures of model performance. Through LAMVI-2\xspace, the developer can construct new model instances and add these to the comparison set. Through the interface, developers can find the best hyperparameters for their model by identifying complex correlations and interactions (e.g., increasing parameter X improves metric Y, but only up to the value Z). Outliers, that indicate possible problems in training or evaluation data, are also apparent in the interface. By integrating a labeling system directly into the interface, LAMVI-2\xspace users can also act to `correct' or `extend' model training and evaluation based on their observations.

While previous solutions, include our own work, focused on inspecting \textit{single} trained models LAMVI-2\xspace is specifically constructed for \textit{multiple}-model scenarios use cases. This is achieved by combining multiple visual encodings and interaction patterns for tuning-related tasks. As can be seen in Figure~\ref{fig:lamvi2-screenshot}, we 'anchor' LAMVI-2\xspace on a parallel coordinates style visualization (showing parameters and output metrics) and link it to heatmaps (capturing term distances) and a familiar `projection view' (a t-SNE projection commonly used by model developers). Interactions include adding model instances, filtering models based on multiple criteria, reordering (to find trends and outliers in model instances) and browsing and search features to isolate problems and new labels. LAMVI-2\xspace broadly allows the user to identify relationships and correlations between parameters and output metrics \textit{and} to inspect the difference between the embeddings produced by each model instance.

We demonstrate the utility of LAMVI-2\xspace through a case study that replicates the early word2vec work as well as a performing a realistic user study using a real-world task. Participants were able to more rapidly and effectively tune a model using LAMVI-2\xspace over default tools. As critically, we found that they were also better able to better understand the relationship of different hyperparameters.

In this work we contribute a LAMVI-2\xspace, a novel visualization system for tuning hyperparameters of word embedding models. We believe that this approach can be adapted to other machine learning tasks where tuning is necessary.  Additionally, we demonstrate the usefulness on LAMVI-2\xspace on a realistic task.

\section{Related Work}
Work related to our project broadly falls into two broad lines of research and development (with some significant recent overlaps). The first line emerges from the machine learning and database communities on (non-visual) model management and tuning (e.g., ~\cite{modeltracker}). The second line of work is from the visualization and HCI communities on data-specific (i.e., text) visualizations and interactive learning.

\subsection{Hyperparameter Spaces}

\begin{figure}
  \begin{center}
    \includegraphics[width=0.5\textwidth]{figs/3dplot}
  \end{center}
\caption[hyperparameter space]{3D plot of two hyperparameters.}
\label{fig:3d}
\end{figure}

However, some authors have adopted the use of static~\cite{Young2015} and interactive parallel coordinates~\cite{vizier} to illustrate the selection of hyperparameters. We take these as evidence that LAMVI-2\xspace visualizations can be effective for explaining the relationship between hyperparameters. By augmenting generic parallel coordinates with domain-specific views, LAMVI-2\xspace supports finer grain inspection of models. Many visualizations of hyperparameter spaces utilize static 3D models (see Figure~\ref{fig:3d}).

\subsection{Text and Machine Learning}
There is a large amount of work on use of visualization to inspect text, some with a machine learning focus. For example, AntConc provides concordances and other visual toolkits to support corpus linguistic analysis, such as word frequencies and collocation inspection~\cite{anthony2011antconc}. Kulesza et al. use simple bar graphs of feature importance to provide answers to the user's \textit{why}-questions regarding text message classification in an email client application~\cite{kulesza2011oriented}.

Chuang et al. use association matrices and alignment charts to investigate latent topic coverage of a large collection of model variants~\cite{chuang2012termite,chuang2013topic}.
LDAvis employs interactive visualizations to facilitate the user to interpret the content and inter-relationships of different latent topics learned by a topic model~\cite{sievert2014ldavis}. Other examples related to topic visualization include \cite{cui2014hierarchical,liu2009interactive}. Alexander and Gleicher~\cite{alexander16} built a visualization system to compare different topic models. The tool visualized (in)stability of different topic-document connections but did not integrate views into the hyperparameter space and metrics to enable end-user driven optimizations.

These tools provide inspection capabilities for different text-related modeling tasks. They do not however, directly address the model tuning problem. In most cases, each visualization provides access to at most one model instance at a time.

\subsection{Visual Inspection of Neural Networks}

Visualization support for machine learning engineering, and in particular neural nets, has seen significant recent attention (surveyed in~\cite{hohman2018visual}). While model construction~\cite{8019861}, debugging and model selection have been identified as key tasks, hyperparameter tuning has seen less attention. While the attention on building and comparing new architectures is reasonable given the proliferation of new techniques, tuning of the type we describe here is critical to real-world deployment and adaptation of existing architectures to new contexts.

Model comparison is a critical step in model selection. However, many current approaches focus on comparison of the model output or internal `model states' rather than on the parameters that caused a certain behavior. Some exceptions include work on understanding the impact of input data on performance~\cite{Kulesza:2015:PED:2678025.2701399,Patel:2008:ISM:1357054.1357160}.

Using visualization techniques to improve understandability of neural network (NN) models has also drawn a great deal of attention. However, many NN visualization projects are focused on computer vision~\cite{yosinski2015understanding,samek2015evaluating}. There are also several interactive visualization projects developed for educational purposes, such as the recent Tensorflow Playground (\url{http://playground.tensorflow.org/}) and distill.pub~\cite{olah2018the}. More elaborate neural network architectures, such as convolutional neural networks, have also seen progress in using visualization to understand network behavior~\cite{DBLP:journals/corr/SimonyanVZ13, DBLP:journals/corr/ZeilerF13}.

Compared to images, videos, or quantitative multidimensional datasets, words and sentences lack natural visual representation and can be harder to interpret. The required transformation of text to data that can be visually encoded presents unique challenges. To improve understandability of natural language representations learned by neural network models, existing techniques for visualizing high-dimensional data are often borrowed, such as principal component analysis (PCA)~\cite{jolliffe2002principal}, multi-dimensional scaling~\cite{kruskal1964multidimensional}, t-SNE~\cite{van2008visualizing}, and other multi-view analytics solutions~\cite{conceptvector}. While useful for inspecting particular embeddings, these encodings/systems do not address the problem of hyperparamter comparison and tuning.

Notable work on visualizing specific text-based deep learning include systems by Karpathy et al. that employ multiple views of activation levels to illustrate how recurrent neural networks capture patterns in text sequences, such as long-range dependencies~\cite{karpathy2015visualizing}. Others have demonstrated the use of visualizations to investigate different NN features for both text and images (\cite{ming1understanding,bilal,datacube}), finding patterns in hidden states (\cite{DBLP:journals/corr/StrobeltGHPR16,deepeyes,liushi}) or analyzing classifier output (\cite{activis}). These approaches may be integrated into our own if LAMVI-2\xspace is adapted to other neural architectures or ML models.

\subsection{Interactive Learning}
While most machine learning visualizations (including those focused on neural networks) target individual models, a few systems seek to provide inter-model comparisons. ModelTracker~\cite{modeltracker}, for example, allows the user to track the evolution of a model in terms of accuracy and provides interactive learning features. The tool allows the user to inspect  model-specific classification metrics and add new labels. This notion of interactive learning~\cite{amershi10, amershi2014power, LIU201748} inspired features in LAMVI-2\xspace to add labels through the visualization.

\section{Task Analysis}

Based on our experience with early protoypes (WEVI\xspace, LAMVI-1\xspace) and in producing educational material for word2vec users~\cite{rong2014word2vec} we identified key tasks and guidelines for hyperparameter tuning that motivate our design. We describe our prototypes and tasks below.

\subsection{Prototypes: WEVI\xspace and LAMVI-1\xspace}
[Author note: we have anonymized the references and demo in our video but the `live' versions are not anonymous.]

\label{sec-lamvi:v1v2}

\begin{figure*}[!htbp]
\begin{center}
              \includegraphics[width=.8\textwidth]{figs/fig-wevi}
\caption[Screenshot of WEVI\xspace]{The WEVI\xspace interface}
\label{fig:wevi-screenshot}
\end{center}
\end{figure*}

\begin{figure*}[!htbp]
\begin{center}
              \includegraphics[width=.8\textwidth]{figs/fig-screenshot}
\caption[Screenshot of LAMVI-1\xspace]{The LAMVI-1\xspace interface}
\label{fig:lamvi1-screenshot}
\end{center}
\end{figure*}

LAMVI-2\xspace is the third in a line of visualization systems developed for analysis of word embeddings.  The first, WEVI\xspace~\cite{wevi}, was produced as a teaching tool to work alongside a (now-popular) tutorial on word2vec hyperparameter setting~\cite{rong2014word2vec} (see Figure~\ref{fig:wevi-screenshot}. While popular, WEVI\xspace was focused on extremely small language models and did not support model comparison. To test the impact of hyperparameters, the student would make a change and observe the effect on the neural network weights during training. Though useful for understanding how training happened and embeddings could be used to find close terms, the tool did not support realistic tasks. Nonetheless, we observed that end-users would iteratively vary the hyperparameters to understand their effect on the model.

While not intended for real-world use, WEVI\xspace did clarify for us that students, and likely developers, were not interested in vector weights for individual words. Rather, the relationship of words to each other was critical in determining whether a particular embedding was good. This focus on word \textit{relationships} has become a key guideline for our subsequent implementations.

LAMVI-1\xspace~\cite{lamvidemo,rong2016visual} was built to support real-world developer practice. The LAMVI-1\xspace interface (see Figure~\ref{fig:lamvi1-screenshot}) provided similar controls (panel A) to WEVI\xspace, though it used a full implementation of word2vec (allowing hyperparameters like window size to be controlled). Based on our experience with WEVI\xspace, we allowed LAMVI-1\xspace users to focus their analysis to specific ``anchor'' terms (in the example the input text is Jane Austen's ``Pride and Prejudice'' and the anchor term is ``looked'') and the relationship of this anchor to other terms. LAMVI-1\xspace allowed end-user to create compound vectors (e.g., \textit{elizabeth} $+$ \textit{bennet}) as well as displaying a `tracking list.' Tracked terms were those the end-user believed should (or explicitly should not) be related to the target term (a form of ground-truth labels). During training, LAMVI-1\xspace would display the distance of the two terms (anchor and tracking) in a ranked list that simultaneously allowed the developer to identify convergence. Similarly to WEVI\xspace, the LAMVI-1\xspace end-user could initiate training and observe the various visualizations dynamically change based on hyperparameter configurations.

LAMVI-1\xspace solved a number of problems with WEVI\xspace. By explicitly tracking the interesting aspects of the model--the \textit{interactions} between terms, rather than the terms themselves--the tool supported both easier debugging and also allowed the end-user to dynamically build ground-truth labels based on their observations of the data. In using LAMVI-1\xspace for our own projects and observing others' use, it became clear that one of the most common use-cases, comparing models, was not supported directly. Users of LAMVI-1\xspace would make changes to the hyperparameters to retrain the model but would quickly lose track of the behavior of previous configurations. LAMVI-2\xspace was specifically designed with this problem in mind.

\subsection{Design Guidelines}
Our experience with WEVI\xspace and LAMVI-1\xspace, as well as in model building allowed us to identify three main decision points: \textbf{(D1) decisions on which models to test}; \textbf{(D2) decisions on how models are evaluated}; and \textbf{(D3) decisions on which models to use}. Each decision requires the execution of multiple analytical tasks.

\noindent
\textbf{D1: Model Construction}--Given the large hyperparameter space and cost of learning and testing, a key decision point in model construction is where resources are best expended. This decision is strongly connected to determining which model to \textit{use} (D3) but may involve a broader set of tasks.

\begin{itemize}[noitemsep]
    \item \textbf{T1}: Identify unexplored hyperparameter settings. To avoid overfitting, the developer should be able to identify which portions of the hyperparameter space are unexplored.
    \item \textbf{T2}: Identify the correlation and interaction between hyperparameters and output metrics. A key type of analytical determination is the identification and mental modeling of those interactions that display trends. Specifically, the developer must know if tuning the hyperparameter value(s) \textit{in a specific direction} will produce better results.
    \item \textbf{T3}: Identify satisficing hyperparameter setting ranges. As hyperparameters often lead to tradeoffs, a developer should be able to identify those models that satisfice their need (given evaluation metrics) and focus on those for construction and selection.
\end{itemize}

\noindent
\textbf{D2: Model Evaluation}--Model are principally selected, or trained, based on different metrics (i.e., pre-established benchmarks). These can include temporal performance measures (e.g., speed of training or execution), accuracy metrics (based on model-specific ground truth or an evaluation of the model's contribution to downstream applications), and other model features (e.g., memory footprint, power consumption, etc.). It is our goal for developers to be able to identify satisficing ranges for multiple performance metrics (T3). Additionally, because ground truth test-sets are often specific to the training data or generic (e.g.,~\cite{semeval}), they often require adaptation. When applying the model to new training data, the developer often needs to change the ground-truth. Put another way, the developer should not only be able to define which performance metrics to apply (with their given ground-truth labels), but also dynamically augment ground-truth labels.

\begin{itemize}[noitemsep]
    \item \textbf{T4}: Identify, and mentally-model tradeoffs in hyperparameters given different performance metrics.
    \item \textbf{T5}: Identify satisficing-models given global performance metrics.
    \item \textbf{T6}: Compare low-level model differences (i.e., word or word-cluster specific) to identify new examples to incorporate into performance metrics.
\end{itemize}

While we focus in this work on text embeddings, it is worth acknowledging that many of our ground-truth labels are `second order.' That is, there is not `correct' vector representation for any particular word. Rather, the user might want a pair of words to be similar ('close'), or some order on nearest neighbors, or may want a certain group of words to form a coherent cluster. Conversely, the user might have negative examples and want to exclude certain words from a cluster. The forms of ground-truth can vary significantly depending on task. The visualization interface must make it easy for the user to explore the possible candidates for establishing such ground-truth items (T6), and to manage them by creating, updating, and deleting individual items.

\noindent
\textbf{D3: Model Selection}--A final step in our process is to identify the model to deploy. This involves a number of the tasks described above (T2, T3, T5, and T6). In the act of isolating `good' models, and having understood their trade-offs and limitations, a decision can be made about the `best' model. This step also allows the developer to document their choice in a way that others can understand the decision.

\begin{itemize}[noitemsep]
    \item \textbf{T7}: Document model choice/selection for external audiences.
\end{itemize}
\section{LAMVI-2}
\label{sec-lamvi:system}

To support the tasks described above, LAMVI-2\xspace is composed of a number of interlinked views and interaction patterns. We briefly decompose the main elements (Figure~\ref{fig:lamvi2-screenshot}) in the context of the high-level decision, and low-level analytical tasks.

\subsection{Parallel Coordinates (panel A)}
Understanding hyperparameters and metrics is inherently a multivariate data analysis problem. There are quantitative, ordinal, and categorical hyperparameters, and similarly quantitaitve, ordinal and categorical metrics. The analytical tasks that involve comparing (T1-T5) inherently require multiple comparisons between these variables. Additionally, many key decisions in our workflow require filtering on multiple dimensions simultaneously (T6). The interactivity that is afforded by parallel coordinates fits well for the goal of identifying satisficing models.

Parallel coordinate plots~\cite{inselberg1985plane} naturally fit as good mechanism for encoding multiple dimensions onto a 2-D plot. There are multiple reasons to adopt this form beyond the expressiveness of the encoding given our task. First, parallel coordinate plots are becoming familiar to end-users. Despite the learning curve, they are effective representations and are already being adapted in the context of model comparison. For example, Google's Vizier~\cite{vizier} and the open-source Ray Tune software~\cite{raytune} offer parallel coordinate plots. This use is somewhat unsurprising as one of the original case studies~\cite{detective} describes the use of parallel coordinates to trade-off parameters (of chip production system) against different performance measures. While there are likely better static representations, interactive documentation (T7) can be achieved by providing a pre-populated version of LAMVI-2\xspace.

This view is attractive for our context as the encoding supports rapid identification of positive/negative relations (T2, T4) between data dimensions (based on slope). While a limitation of the technique is that, at any given time, each dimension can have at least two neighboring dimensions, but most parallel coordinate implementations (including our own) allow coordinates to be re-ordered. Since one of our requirements above is to facilitate efficient identification of relationships between model hyperparameters and performance scores, we opt to use parallel coordinate plots to depict both kinds of variables. \textit{To begin,} hyperparameter dimensions are placed on the left and output/performance dimensions are placed on the right (with the heatmap in the center). This reflects the natural ordering of parameter to model to outcome but elements may be reordered if necessary. Unexplored hyperparameter spaces (T1) are salient as `empty' areas in the parameter axes.

Our implementation supports the standard parallel coordinates interaction patterns.  By brushing along a specific axis, the end user can \textbf{filter} the data to narrow down the number of lines being highlighted to a smaller set whose crossing points with the axis are within the brushed range. To focus, the brushed region can be rescaled or moved through cursor interactions. Double clicking an an axis label with cause the data to \textbf{flip} for that dimension. This feature may help the user reduce the number of line crossings between axes and also allows for consistent mental models. Other features such as colors and bundling can be enabled, but are not done so by default. Together, these features enable the filtering necessary for identifying good models (T3, T5).

While LAMVI-2\xspace can support any number of loaded models, realistically we do not expect more than 30-40 (and that depends significantly on model training costs). In part, an appeal of LAMVI-2\xspace is that we can `seed' the system with random points in the hyperparameter space and allow the end-user to narrow this space by visual inspection.

\subsection{Embedding Explorer (panel C)}
Embedding words in a 2D space is a natural way to inspect models. In particular, for task T6, inspecting the words in this way allows the developer to identify possible issues with the model. More specifically, approaches such as t-SNE~\cite{van2008visualizing,wattenberg2016how} and PCA reduce multi-dimensional vectors to two dimensions, allow the developer to inspect the relationship of terms both in magnitude and distance. For better or worse, such representations have become standard for both interactive and static visualizations (e.g., Google's Tensorboard: https://www.tensorflow.org/programmers\_guide/embedding). The benefit of projections is that the viewer is that related terms are placed near each other in the low-dimension projections. Gestalt properties of this projection make it easy to find patterns and detect outliers. For neural-net based embeddings, a further (argued) benefit is that the  \textit{relative directions} of terms has `meaning.'

Traditional use of 2D projections involves the placement of all the words in the model or some large top-$k$. Unfortunately, with many words, the visualizations can become difficult to navigate. More critically, there are more opportunities with large differences for words to be placed near each other as a consequence of the dimensionality-reduction and not due to `true' distance~\cite{chuang12}. Projecting all the terms has the appeal that it follows the `overview first, filter, details on demand' mantra. However, it is unclear to us that the particular `overview' provided benefit for the tasks we outlined.

While LAMVI-2\xspace can project all terms in the model, we employ a different default strategy. Instead of a view of the entire set of words, the LAMVI-2\xspace Embedding Explorer shows a projection anchored on a specific query 'term' and a set of its nearest neighbors. As different models in the parallel coordinate view are selected or loaded, the projection updates. LAMVI-2\xspace uses t-SNE for projection. The interactive explorer supports basic pan-and-zoom functionality to explore the projection. The 'focus' term is made salient by using a larger font size (and is orange). Terms that are labeled (e.g., antonyms or synonyms) are also made salient by color coding and increased font size. The developer can easily find term pairs that are unlabeled as well as the clustering of labeled examples (e.g., are the antonyms all near each other?). In the event that these labeled terms are not part of the k-nearest neighbors (i.e., an antonym \textit{could} be very far), LAMVI-2\xspace will add those terms to the projection.

In LAMVI-2\xspace, the developer can perform tasks such as filtering, sorting, creating new models, or switching to old ones. Terms in the embedded view are animated so the developer can observe the impact of the interaction on the spatial relationship of the terms. A secondary benefit of focusing on a small set of terms is that these animations can be visually tracked more easily. Terms that are no longer `near' to the query are animated out, while those that now meet the criteria are animated in. Label placement also dynamically changes based on the vector representations of the selected model. The t-SNE optimization algorithm randomly initializes the embedding vectors and often the labels temporarily shrink to the center of the space, making it difficult to track terms as they move. To counter this, we first perform 150 iterations of the t-SNE algorithm in the background to find a better starting state for the animation.

As we describe below, the t-SNE projection is in some sense a `dual-encoding' of term distances (though it uniquely displays direction). We have found that a useful interaction pattern is to support a focus on word pairs or sets (the smallest 'unit' of useful analysis for word embedding). By clicking to highlight two or more words, and switch focus between models, the developer can track the relationship of a small set of terms between model variants. Because highlights are preserved between views, the user can both visually track where the words are relocated and acquire additional spatial awareness of changes in the embedding space across different models. This allows them to identify patterns that are consistent (building a mental-model of the models' behavior) as well as finding models that are `outliers.'
.

\subsection{Nearest-Neighbor Heatmap (panel B)}
A unique feature of LAMVI-2\xspace is the Nearest-Neighbor Heatmap (panel B). The heatmap is connected both to the parallel coordinates plot (models are rows) and the t-SNE projection (words are columns). Words are both visually linked (through a line) as well as through linked brushing. Mousing over a word will highlight the appropriate column.

The heatmap is specifically targeted at task T6, comparison of low-level model differences. The encoding allows us to compactly display data with a high information-density summary~\cite{munzner2014visualization}. Each cell corresponds to the distance between a `query' and a sampling of related terms.  For example, in Figure~\ref{fig:lamvi2-screenshot}, the current query term is ``legitimate'' (also displayed as the first column).  The second column corresponds to the term ``dubious.''

As the number of nearest neighbors (i.e., columns) and models increases it becomes difficult to identify patterns and outliers in the data.  We use a combination of visual encodings and interaction techniques to help the user navigate this information. Patterns of particular interest are those columns where models agree on the distance between the query term and the other word.  For example, the models seem to agree that `valid' is similar to 'legitimate.'  This manifests as a column of a single color. A second pattern of interest are those when there is either a single outlier in a column or a bifurcation, where some models place the term 'close' (blue) and some 'far' (red). For example, five of the seven models place 'moderate' (4th column) far and two place it close. Other patterns delineate outliers and clusters at the model level. By looking at multiple rows, the end-user can find `clusters' of models that generate similar embeddings and those that are distinct or anomalous.

To increase the number of columns that can be simultaneously display we designed a two-mode display for columns inspired by Tablelens~\cite{rao1994table}. This increased the amount of data displayed but still supported the pattern finding task. Under the ``normal'' mode, columns in the matrix are compressed to a few pixels (for example the column for 'valid'), minimizing space but still allowing the viewer to identify patterns across all models. Under the ``zoomed-in'' mode, the columns in the matrix are displayed with a larger width with labels displayed below the bottom row. By default, the top-K nearest neighbors of the query word will have their corresponding columns zoomed-in (K starts at 15) . As the user interacts with the other parts of the interface, pertinent columns will be adjusted to zoom-in. This focus switching mechanism optimizes the efficiency of space usage while allowing the user to quickly navigate through the most relevant columns. Other heat-map `modes' can put columns with inter-model agreement in the 'normal' (low-pixel) and low-agreement terms in the zoomed-in state.

Although the heatmap is integrated into the parallel coordinates display, the vertical ordering of the rows does not necessarily correspond to the same high-low ordering. While this breaks the encoding, it also allows us to dynamically reorder both the rows and columns of the heatmap to emphasize patterns. Line segments connect the heatmap rows to the parallel dimensions coordinate on the left and right ensure that the viewer can `follow' a particular model's encoding.  When models are filtered in the parallel coordinate dimensions, the corresponding rows are highlighted (those retained by the filter) or are `grayed out.' In our experience, we have not found the `interruption' of the heatmap in the middle of the parallel coordinates plot to be disruptive (visually or semantically). However, we note again that  parallel coordinates axes that are classically 'dimensions' of the encoding can be grouped together (i.e., hyperparameters and metrics), placing the heatmap to the side. This may be a desirable configuration for some end-users or for specific model types.

LAMVI-2\xspace supports a variety of \textbf{clusters} and \textbf{orderings}. As the primary focus for our heatmap is for the identification of model disagreement and clusters, the clustering of models can simplify the process greatly. Figure~\ref{fig:lamvi2-four-matrix} illustrates four different configurations for the same data.  The default sort order places rows in the order the models were loaded.  To minimize edge-crossings, the heatmap can also be sorted by model hyperparameters (Figure~\ref{fig:lamvi2-four-matrix}c and d). In this case, LAMVI-2\xspace takes into account the value of the hyperparameter dimension immediately before (in this case 'retro') or the performance metric immediately after ('triples') in deciding the order of the rows. Because the coordinates can be reordered, the user can move a metric dimension next to the heatmap and order by this value. This will cause the rows of the heatmap to reorder to correspond with the metric and allow the user to inspect the properties of 'good' versus 'bad' models.

\begin{figure*}[t]
\begin{center}
\centerline{\includegraphics[width=.8\textwidth]{figs/fig-four-matrix}}
\caption[Four sorting modes of LAMVI-2\xspace heatmap.]{Four sorting modes of the LAMVI-2\xspace heatmap: (a) default mode -- models are sorted by loading order; (b) cluster-sort mode -- models and words are sorted according to the dendrogram of hierarchical clustering; (c) hyperparameter-sort mode -- models are sorted by the right-most hyperparameter in the parallel coordinate plot; (d) metric-sort mode -- models are sorted by the left-most performance metric.}
\label{fig:lamvi2-four-matrix}
\end{center}
\vskip -0.2in
\end{figure*}

A more sophisticated order is created through hierarchical clustering. We achieve this by running a hierarchical clustering algorithm on the heatmap data (row-wise). The order for the rows corresponds to the order used to visualize the dendrogram. The effect is that models that have similar distance properties are placed near each other. As illustrated in Figure~\ref{fig:lamvi2-four-matrix}b, patterns become apparent (e.g., the agreement in the last three rows) or that the top row is an outlier. Clustering in this way has the further advantage of making salient specific cells that are different (i.e., within-cluster outliers).

\begin{figure*}[t]
\begin{center}
\centerline{\includegraphics[width=.8\textwidth]{figs/fig-heatmap-filter}}
\caption[Filtering of rows in LAMVI-2\xspace heatmap via parallel coordinate brushing.]{Heatmap rows can be filtered and highlighted by brushing axes on the parallel coordinate plot. This allows a focused comparison among models that meet a user-specified filtering criterion.}
\label{fig:heatmap-filter}
\end{center}
\end{figure*}

We retain the \textbf{filtering} interaction pattern of the parallel coordinates inside of the heatmap (see Figure~\ref{fig:heatmap-filter}). Whenever the user applies filtering or brushing to one of the dimensions in the parallel coordinate plot, the corresponding rows in the heatmap will be highlighted. The non-highlighted rows are still visible but will be applied with a semi-transparent mask. Conversely, the end-user can select rows in the heatmap to filter one or more models.

\subsection{SPLOM}
A fourth panel (not pictured) displays a Scatter Plot Matrix (SPLOM). The grid displays pairwise correlations between all hyperparameters and metrics (each point is a model instance). Because it is sometimes difficult to quantify correlations between dimensions in parallel coordinates directly, the SPLOM view provides a more accurate and comprehensive encoding of this data (further supporting T1-T4).  The SPLOM is interactive and linked to the other displays. As the end-user filters (by drawing selection rectangles), models are dynamically filtered from all the views.

\subsection{Inspecting Models and Training}
Recall that a key goal for LAMVI-2\xspace is to facilitate the creation of training data (T6). The end-user may issue queries through a text-input dialog. This will refresh the embedding view to include the query string and nearby terms. When the user is interested in the similarity and neighborhoods of a \textit{pair} of words simultaneously, she may directly query the two words together. This allows her to more easily mark good cases and bad cases as more words are displayed (neighbors of both terms) that likely match the relationship of the query pair.

Our current implementation also supports inspecting the emergence of linguistic regularity captured by the model. The user may enter queries like ``king --queen woman'' and observe how the desired candidate, ``man'', evolves. The user may also inspect the activation levels of the hidden units given all three words as context. The distances between query and neighbors is shown automatically in the heatmap.

\begin{figure*}[t]
\vskip 0.2in
\begin{center}
\centerline{\includegraphics[width=.8\textwidth]{figs/fig-tooltip}}
\caption[Creating user-customized ground-truth in LAMVI-2.]{Multifunctional tooltip in LAMVI-2 heatmap: (a) compact tooltip shows the similarity of the word pair under focus; (b) expanded tooltip allows the user to assign a label to the word pair, creating user-customized ground truth.}
\label{fig:lamvi2-tooltip}
\end{center}
\vskip -0.2in
\end{figure*}

Clicking on a cell in the heatmap will show a dialog box by which pairs can be `labeled' (see Figure~\ref{fig:lamvi2-tooltip}). Though we can support multiple labels, the default prototype categories include \textit{synonyms} and \textit{antonyms}. As labels are created, these are dynamically fed as labels to downstream evaluation `modules.'

A particular workflow that can be executed is the identification of models that correlate with outliers. When the user explores the embedding and discovers words that are inappropriately placed in the neighborhood of a query she can click on the word label to automatically zoom into the associated column in the heatmap. The user can scan down the entire column and see if the score is consistently spuriously high. She can then filter on those rows to see if certain hyperparameters result in the undesired score.

\subsection{Implementation Details}
The back-end of the LAMVI-2\xspace system is written in Python and was designed with extensability in mind. The front-end was programmed using D3~\cite{bostock2011d3}. A model developer can configure and evaluate models programmatically. These models can then be loaded into the interface. We currently provide basic command line tools that will accept a high-level configuration (e.g., one indicating the space of hyperparameters to explore) and builds a model for each point in this space. A web-based tool allows for generation of a JSON configuration file that can be fed to the command-line pipeline.  The tool is currently configured for multi-scale grid search as well as random search~\cite{bergstra2012random}, but can be readily adapted to other hyperparameter optimizers~\cite{NIPS2011_4443,NIPS2012_4522}. Labeled examples created by the developer are stored in a text-based label file. The entire `state' of a particular experimental run can be preserved and provided to other developers as documentation (T7).

\section{Case Study}
\label{sec-lamvi:casestudy}

To understand how LAMVI-2\xspace could be used in a realistic situation we replicated the work by Mikolov et al.~\cite{mikolov2013efficient} (as implemented by Rehurek for the GenSim package using the \textit{text8} corpus~\cite{rehurek}) using our tools. In part, we selected this case as both the the original Mikolov et al. paper and the implementation by Rehurek report their parameters without specific discussion of trade-offs and optimality. We demonstrate both that it is possible to identify the `optimal' hyperparameters but also that we can more fully describe the effect of these parameters on performance. An advantage of using LAMVI-2\xspace is that it becomes easier for the author to identify and report (task T7) on patterns in the data that may provide guidance for others using new neural architectures. As the architectures, and associated hyperparameter spaces, become more complex, such guidance is critical.

To build a set of comparative models we used hyperparameter ranges that were expansive, but included the `optimal' values reported in the studies above. In the interest of replication, merging with other models and retrofitting were not used during training. Hyper-parameter ranges were: Word Vector Size: 100--400, increments of 50; Skip-Gram architecture: Both with/without; Hierarchical Softmax: Both with/without; Window size: 3--7, increments of 2; Negative samples: 5--20, increments of 5.

\begin{figure*}[t]
\begin{center}
\centerline{\includegraphics[width=\textwidth]{figs/fig-LAMVI-size.png}}
\caption[Inspecting size hyper-parameter in LAMVI-2\xspace]{A view of the LAMVI-2\xspace interface for the word2vec case study. The end-user has targeted the word 'sufficient' and has filtered to those models containing skip grams, a window of 5, and 15 negative samples.}
\label{fig:LAMVI-size}
\end{center}
\end{figure*}

Figure~\ref{fig:LAMVI-size} displays the LAMVI-2\xspace interface with the different models in this space active. The analogy metric is displayed on the right as is training time. When filtering on the maximal accuracy we achieve results comparable to Rehurek, including roughly 28\% accuracy on the analogy task for the model with settings \{size: 200, skip-gram, negative sampling w/ 15 samples, window size: 5\}. More critically, we observe trade-offs that were not described in the original paper, but now understood in the outside literature. For example, the use of both skip-gram and hierarchical sampling results in increased accuracy at the cost of a significant increase in training time.

We find that these optimal parameters are slightly different than those reported by Mikolov (dimension: 300, window size of 7). This may be partially due to the use of \textit{text8} but demonstrates the need to perform algorithmic tuning when changing corpora. Additionally, given the time to train different sets of models, performance improvements can be achieved through more fine-grained tuning of hyperparameters. However, we do find that the granularity of the adjustments correlates with the granularity of the improvement. The SPLOM visualization further shows decreasing returns for more training time relative to analogy performance.

\section{Evaluation}
\label{sec-lamvi:experiment}

Evaluating tools for deep learning models can be difficult, as they require significant participant expertise. Realistic scenarios are similarly complex, as a local evaluation of the embedding may not be sufficient. For example, one might create a classifier for synonyms and use this in a downstream information extraction (IE) system. An optimization in the sub-module (the synonym classifier) may not mean optimal behavior in the IE system. To simulate this, we created a two-stage task for polarity classification of IMDB reviews (i.e., are reviews positive ore negative). Because we may not know the polarity of every word in the document, one approach is to find synonyms to known positive terms and label these as positive and identify antonyms and label them as negative (conversely, synonyms of negative polarity terms can be labeled negative).  This scenario naturally induces two metrics: the accuracy in identifying synonyms/antonyms and the accuracy in classifying reviews.  Our experiment focuses on how well (and quickly) participants can achieve an optimal hyperparameter-configuration using LAMVI-2\xspace versus baseline ``command line'' style tools.

\subsection{Task and Model}
For our task, we used the Cornell IMDB movie review corpus~\cite{pang2002thumbs} as the training corpus (100k reviews, 50k of which have positive/negative sentiment labels). The input feature to this classifier is the average word embedding (i.e., the centroid of all words) in that document.  The intuition is that if the embedding is `good', positive synonyms will be grouped together and negative synonyms will be grouped together in space. If the document has more positive terms, the average vector will be in a `positive' location.  We train a logistic regression classifier for this ``downstream'' application.   We calculate $f_A$, as the accuracy score for the sentiment classifier using 50k documents (equal partition, positive/negative). The participant's goal is to provide the classifier with a good embedding.

As described above, we may only have sentiment scores for \textit{some} of the words in the corpus.  By creating an embedding that places synonyms of a given polarity together, it becomes possible to provide scores for all words.  We challenge our participants to tune a word2vec skip-gram model~\cite{mikolov2013distributed} to provide such an embedding. We provide the participants with `synonym-antonym triples'--three words where one word is an `anchor' (e.g., \textit{ghastly}), one is a synonym (\textit{awful}), and one is an antonym (e.g., \textit{delightful}).  We also provide a few examples (specifically five) to the study participant (we call this the 'training set') with the expectation that they add additional examples.  Thirty-five example triples are withheld from the participant (the 'test set') and we evaluate whatever embedding configuration they pick against this set. The 40 triples (5 train, 35 test) were generated by consulting external lexicons (SentiWordNet~\cite{baccianella2010sentiwordnet}, \textit{Oxford American Writer's Thesaurus}~\cite{lindberg2012oxford}, and an online thesaurus dictionary.\footnote{\url{http://www.thesaurus.com/}.}). We specifically selected words with obvious polarity (top 10\% of SentiWordNet) so it was clear from our examples whether the sense was positive or negative. Only words that occurred between 150 and 250 in the IMDB dataset were used (to ensure sufficient data).

The \textit{triples} score, $f_T$, measures how well a model distinguishes synonyms from antonyms. It is calculated as the average difference between the similarity difference of a set of synonym-antonym triples:
\begin{equation}
f_T = \frac{1}{N}\sum_i^N(\cos(q_{i,A},q_{i,B})-\cos(q_{i,A},q_{i,C}))
\end{equation}
where $q_{i,A}$ and $q_{i,B}$ are the vectors of a pair of synonyms and $q_{i,A}$ and $q_{i,C}$ are the vectors of a pair of antonyms.

We use the average of two metric scores, triples and accuracy, to measure the performance of the model. For both $f_T$ and $f_A$, the user can access the corresponding scores assessed on the training set in the interface (i.e., for each parameter configuration a model was generated, and for each model accuracy was calculated for both the triple task and sentiment analysis task and provided to the participant). The final evaluation of the model was to be assessed on the test set.

\subsubsection{Parameter Space}
To control the scope of the experiment, we set some of the parameters to be non-adjustable by the user (specifically, we fixed the frequency downsampling parameter to be $1\times10^{-4}$, 5 iterations, and 5 negative samples). We allowed the user to select whether to have the vectors pretrained on an external corpus (the `lockf' parameter and whether to use an external lexicon to retrofit the vectors (the 'retro' parameter). Pretraining provides a `starting' point for training, for example based on Wikipedia, and is used when the domain corpus is too small. Retrofitting happens after the model training and is shown to be able to improve the quality of the vectors~\cite{faruqui2014retrofitting}. In our case, the paraphrase database (PPDB)~\cite{ganitkevitch2013ppdb} is used as the semantic lexicon for retrofitting. The database contains 8 million English lexical paraphrases (e.g., \emph{jailed} and \emph{imprisoned}).

The parameters that the participant could vary included word vector \textbf{size}; sliding \textbf{window} length; whether vectors pretrained on Wikipedia should be used (\textbf{lockf}, which could be varied between 0 and 1 to determine the weight of the pretrained vectors on the final vectors, or -1 to be disabled); the initial learning rate for gradient descent (\textbf{alpha}); \textbf{retro}fitting, indicating whether to use an external dictionary to augment the vectors after training (ranging between 0 and 2, where a larger number indicates less external influence and a -1 disabling this step). While these parameters may appear overwhelming, we emphasize that they are realistic decision points encountered by developers.

\subsection{Procedure}
Participants answered a brief questionnaire about their experience and familiarity with machine learning and word embedding models. Each participant was randomly placed into a baseline condition or one in which they used LAMVI-2\xspace.

The \textit{baseline} emulated the command line tools currently available, and allowed the participant to enter model parameters and see the evalution metrics. The baseline also allowed the participant to enter `queries' into their model to see similar terms. Baseline participants could add triples to their own `test' set. While this is a `weak' baseline in that we did not offer any visualization tools, the command-line tools were familiar to many of our participants and were part of the `standard workflow.' Participants in the LAMVI-2\xspace condition were given access to the full interface (with the exception of the SPLOM view that had limited use given the constraints below).

Models covering a broad parameter range (capturing all likely selections by the participant) were pre-generated.  Given the time constraint of the experiment, this would allow a participant to load a range of models without waiting for training.  Two models were preloaded into the experimental interface. Participants could load at most 13 more models at a time (for 15 total). This was done to create some `cost' to loading additional models as otherwise the participant could simply load all models and select the optimal one. In total, 374 models were pre-trained for the experiment.

After a three minute introduction using a prepared script, the participant was allowed five minutes to experiment with the interface and ask questions. Logging of interactions began and recorded which models were loaded, filtering operations, search, etc. The participant selected one model as the final submission. Finally, Each participant answered a questionnaire with questions taken from the system usability scale (SUS) as well as a few open-ended interview questions.

\subsection{Results}
Twelve subjects were recruited and received \$15 for participating. All were graduate students in computer science or a relevant major at a large public university with some machine learning experience. From our pre-survey, 25\% of the subjects reported to be beginner level with regard to machine learning, and 75\% self-identified as intermediate to expert level. One third of the subjects had learned about word embeddings but had not used them in practice; one third used them as a black-box component in a project; and one third had done at least a research project on word embedding models.

Table~\ref{table:lamvi2-eval} summarizes the results for the comparison between LAMVI-2\xspace and the baseline system. On both the training set and the test, the average performance of the models in groups with the baseline and the LAMVI-2\xspace system are reported. LAMVI-2\xspace users consistently outperformed the baseline system under the metrics of both $f_T$ and $f_A$.

\begin{table}[htbp]
\centering
\caption[LAMVI-2 Evaluation Results]{LAMVI-2 Evaluation Results}
\label{table:lamvi2-eval}
\begin{tabular}{c|c|c|c|}
\cline{2-4}
\multirow{2}{*}{} & \multicolumn{3}{c|}{Training Set} \\ \cline{2-4}
 & triples $f_T$ & accuracy $f_A$ & average \\ \hline
\multicolumn{1}{|c|}{Random} & 0.202 & 0.751 & 0.477  \\ \hline
\multicolumn{1}{|c|}{Baseline} & 0.277 & 0.795 & 0.536 \\ \hline
\multicolumn{1}{|c|}{LAMVI-2\xspace} & \textbf{0.295} & \textbf{0.814} & \textbf{0.554}\\ \hline
\multirow{2}{*}{} & \multicolumn{3}{c|}{Test Set} \\ \cline{2-4} \hline
\multicolumn{1}{|c|}{Random} &  0.159 & 0.745 & 0.452 \\ \hline
\multicolumn{1}{|c|}{Baseline} &  0.203 & 0.791 & 0.497 \\ \hline
\multicolumn{1}{|c|}{LAMVI-2\xspace} &  \textbf{0.223} & \textbf{0.810} & \textbf{0.517} \\ \hline

\end{tabular}
\end{table}

\begin{figure*}[htbp]
\begin{center}
\centerline{\includegraphics[width=\textwidth]{figs/fig-curbest-models}}
\caption[Model performance improvement process.]{Model performance improvement against the number of models loaded.}
\label{fig:curbest-models}
\end{center}
\end{figure*}

Figure~\ref{fig:curbest-models} compares LAMVI-2\xspace with the baseline system in terms of the performance score ($\frac{f_T+f_A}{2}$) for the current best performing model at each interaction step (i.e., how many models were tested). We note that LAMVI-2\xspace participants were able to improve model performance at a faster rate.

Participants using LAMVI-2\xspace generated 16 synonyms and 13 antonyms (on average) using the query functionality. We compared the performance of the user's best-selected model on each of their own synonym/antonym selections (simulating ground-truth construction) with the performance of their model choice on our hold-out set. The Pearson correlation across all participants (averaged) .599 and ranged from .509 to .774 (all statistically significant at $p<0.0001$). This indicates that participants `flagged' synonym-antonym triples that exhibit similar similarity/dissimilarity patterns as our preselected triples in the held-out dataset. Additionally, a higher correlation ($r$) generally means that the final user-selected model has a better task performance overall.

\subsection{Additional Observations}

\textit{Navigation Strategies}--Participants in the LAMVI-2\xspace condition developed specific strategies to guide themselves to navigate through the parameter space. They noted that the parallel coordinates plot helped with this search, especially for the parameters that they were unfamiliar with.  For example, during the post-study interview, user P3 mentioned that ``I have a search algorithm. I start with a default setting, and vary each parameter at a time, and see if I can get the biggest improvement. Then I fix the one with the biggest improvement, and vary other parameters one by one.'' User P7 said, ``I first applied the prior knowledge that I have. Then I try the medium values for the options that I don't know, and then test the extreme values deviated from the medium values.'' Similar to these two users, most other users developed a variant of ``greedy search'' or ``gradient descent'' algorithm for parameter searching.

For the LAMVI-2\xspace participants the parallel coordinate plot was important to these strategies. P5 commented, ``It was tremendously helpful to use the parallel coordinates to tune down the scope of searching. I first look right to the pick the performance I want, and look left to confirm the intuition. I confirmed my intuition that lockf and alpha and retro should be locked to a specific value, and window and size can be fine-tuned.''

\textit{Outlier Model Detection}--The nearest-neighbor heatmap appeared useful for helping identify outlier \textit{models}. Whenever the user loads a new model, the system automatically uses the new loaded model to find the nearest neighbor words to the query and updates the heatmap. When the user loads a model that results in significant disagreement with previously loaded models, there is a dramatic change on the heatmap which is very salient. As user P3 mentioned, ``When I turn the lockf up, I suddenly see the heatmap change all over, and this helps me know that something bad happens. So I am able to quickly recognize that and revert the change.''

\textit{Confirming Intuitions}--While all our participants were able to make use of the LAMVI-2\xspace interface we found that those with high familiarity with embedding models used it in part to confirm intuitions. For example, P1, P2, and P7 believed the learning rate (from prior experience) should not be very big. By targeting \textit{alpha} they were able to obtain substantial improvement on both $f_T$ and $f_A$. Participants P2 and P5 mentioned that they knew \textit{window} cannot not be too big or too small and focused on tuning that parameter towards the middle region (4--8) and noted the gradual improvement on the performance (specially with $f_A$). Several participants in the LAMVI-2\xspace group reported that being able to use the parallel coordinates plot helped them confirm such intuitions quickly.

\textit{Requested Features}--Multiple participants requested the ability to remove loaded models. This was surprising to us as we thought end-users would like to see models that did not work so they could avoid that area of the parameter space.  However, participants noted that these discarded models often introduced clutter especially in later stages of the workflow when they had narrowed down their search space and were focusing on debugging specific models (e.g., when they were using the query functionality). In the experiment, multiple parameters had non-linear effects on the performance. For example, the two extremes of \textit{lockf} and \textit{retro} had similar effects, while the middle-range values of the two parameters cause a linear change on the performance metric. P5 commented that this non-linear pattern was hard to identify with the parallel coordinates without any external knowledge. Augmented parallel coordinates, such as those with overlaid histograms, may alleviate this issue. The SPLOM view, which participants did not have access to, may also be useful.

\textit{Interface issues}--Most participants in our study were unfamiliar with parallel coordinates and found the brushing interactions to be unclear. We suspect that this is a common issue when learning parallel coordinates, but training and adding visual hints to the interface may help. P3 and P11 found the interactions of flagging word pairs as synonyms and/or antonyms to be cumbersome. Currently, to flag a word pair, the user has to find the target word in the overhead embedding explorer, and follows the connector to the corresponding column in the heatmap and click, and then in a popped-up window select the associated option. This interaction pattern can be simplified by allowing the user directly click on words in the embedding explorer to flag them.

\section{Discussion and Limitations}
\label{sec-lamvi:discuss}

We constructed LAMVI-2\xspace for extension. However, we focused our implementation in a number of ways for the first iteration.  One specific issue is that only one `architecture' at a time is supported. There are many other embedding models including GloVe~\cite{pennington2014glove}, DeepWalk~\cite{perozzi2014deepwalk}, and LINE~\cite{tang2015line}. Not all utilize the same parameter space and an interesting question is how to allow the end-user to compare models which use very different parameters. A simple solution may be to allow parameters values to be ``null'' for models to which the parameter does not apply. Other, more complex, parallel coordinates configurations might also be an alternative~\cite{claessen11}. We also note that while we have adopted the dimensionality-reduction style visualization for the embeddings these sometimes lead to `tea leaf reading.' Alternative representations may yield more interpretable results~\cite{chuang12}.

LAMVI-2\xspace is currently focused on \textit{word} embeddings. There are many other model types that might benefit from the exploratory features of LAMVI-2\xspace, but various adaptations to the visualizations would be necessary. LAMVI-2\xspace may be most readily adaptable for other embedding types (e.g., document/paragraph or bimodal~\cite{allamanis2015bimodal,le2014distributed}) as we can show other kinds of items in the t-SNE/PCA views. The workflow of inspecting vector similarity, vector interaction, and tracking the ranks of watched candidate items could be the same.  Other text-related architectures, such as those for memory networks~\cite{karpathy2015visualizing} that ``generate'' sentences would require other changes to the visualization panels. Similarly, image or audio related architectures would also require data-specific adaptations but we believe model comparison through the parallel coordinate views may still be useful in these contexts. In these cases the heatmap view may be adapted to show nearest neighbor images, salience maps, or pixel masks.

\section{Conclusion}
\label{sec-lamvi:conclude}

In this paper we describe LAMVI-2\xspace, a visual tool for comparing word embedding models. LAMVI-2\xspace is specifically focused on three key decision points for machine learning model developers: (1) determining good hyperparameters for model testing, (2) find good examples for evaluations, and (3) selecting optimal ('satisficing') models. LAMVI-2\xspace is designed to support these decisions through a focus on hyperparameters and their relationship to the model internals and performance metrics. LAMVI-2\xspace combines different visual panels in an integrated view that supports these tasks. By extending a parallel coordinates display, LAMVI-2\xspace allows the user to `pivot' between tasks to identify satisfactory hyperparameters.

Unlike automated hyperparameter optimization algorithms, the interface allows the user to make decisions about how different metrics should be traded off. Additionally, the tool allows the end-user to isolate patterns in the embedding space and focus their attention on desirable models.  We demonstrate that LAMVI-2\xspace can be used to gain insight into the relationship between hyperparameters.  Using a realistic experiment we also show how LAMVI-2\xspace users can more rapidly identify these optimal configurations.

See \url{http://lamvi.info} for live demos.

\section*{Acknowledgements}
We dedicate this paper to Xin Rong. It is based on an unpublished chapter from Xin Rong's doctoral dissertation. It is an important contribution that we felt should be shared.