\section{Introduction}
\label{sec:intro}

Spatio-temporal action localization, requiring localizing persons while recognizing their actions, is an important task that has drawn increasing attention in recent years \cite{gu2018ava, girdhar2019video, feichtenhofer2019slowfast, wu2019long, sun2018actor}. Unlike object detection which can be accomplished solely by observing visual appearances, activity recognition usually demands for reasoning about the actor’s interactions with the surrounding context, including other people and objects. Take Fig.~\ref{fig:figure1} as an example: to recognize the action ``ride'' of the person in the red bounding box, we need to observe that he is inside a car, and there is a driver next to him. Therefore, recent progresses in spatio-temporal action detection have been driven by the success of relation modeling. Such attempts on relation modeling can be categorized into two types:

(1) \textbf{Modeling relation between actors} \cite{girdhar2019video, wu2019long} focuses on interactions between persons, \textit{e.g.} the relation between the actor in the red bounding box (red actor) and the other in the blue bounding box (blue actor) in Fig.~\ref{fig:figure1}. These methods take feature vectors extracted from cropped actor bounding boxes as inputs. They utilize attention mechanisms to directly infer relations between actor features.
However, this type of methods only use information within cropped boxes, and discard important contextual information, such as spatio-temporal locations as well as contextual objects' dynamics and appearances that are helpful for more accurate reasoning.

\begin{figure*}[!t]
\centering
\includegraphics[width=\textwidth]{fig/figure_1_app.pdf}
\caption{Reasoning actor-context or actor-actor relations may not be sufficient for correctly predicting the action labels of all individuals. Our method does not only reason relations between actors, but also model connections between different actor-context relations. As an example, the relation between the blue actor and the steering wheel (drive) serves as a crucial clue for recognizing the action being performed by the red actor (ride). \label{fig:figure1}
}
\vspace{-4mm}
\end{figure*}

(2) \textbf{Modeling relation between actor and context} \cite{sun2018actor},
on the other hand, leverages spatio-temporal context for recognizing the behavior of an actor. As shown in Fig.~\ref{fig:figure1}, it performs relation reasoning by identifying spatial regions (\textit{e.g.} steering wheel) that have highest correlation with the blue actor to recognize his action. This type of methods do not explicitly model the relationship among actors.
The advantage of these methods is that they preserve structural information which could be important for understanding the entire scene.

However, these two attempts will struggle on recognizing the action ``ride'' being performed by the red actor, since neither of these relations (actor-actor or actor-context) can provide sufficient clues. More concretely, in Fig.~\ref{fig:figure1}, it is difficult to infer the action of the red actor solely given its relation with the blue actor or with the scene context (steering wheel). To overcome this problem, we propose to capture the implicit \textit{high-order} relation between the two actors based on their respective \textit{first-order} relations with the context. In this way, we will be able to identify the action (ride) of red actor by reasoning over the interaction between the blue actor and the context (drive).

Given the need for high-order relation reasoning to understand videos, we propose an Actor-Context-Actor Relation Network (ACAR-Net) that deduces indirect relations between multiple actors and the context while being trained on the action localization task. The ACAR-Net takes both actor and context features as inputs. It first encodes the \textit{first-order actor-context} relations, and then applies a \textbf{High-Order Relation Reasoning Operator} in charge of modeling links established on those first-order relations. Finally, to model long-term high-order relations, we build an \textbf{Actor-Context Feature Bank}, which contains actor-context relations at different time stamps across the whole video.

ACAR-Net learns to reason high-order relations among actors and the context, while preserving the spatial structure of the scene. Partly similar to our approach, there exist works that explicitly model interactions between actors and objects~\cite{wang2018videos, zhang2019structured}. However, in these approaches, when deducing the action of one person, the interactions of other persons with contextual objects are ignored. In other words, they do not explicitly model the higher-order relations built on direct actor-context relations. In contrast, our method emphasizes modeling those indirect relations, and does not need the extra step of object detection.

We conduct extensive experiments on the challenging Atomic Visual Actions (AVA) dataset~\cite{gu2018ava} for spatio-temporal action localization. This dataset contains a large number of complex realistic scenes, and most of its action classes are human-object or human-human interactions.
We show that our proposed ACAR-Net provides a clear advantage over previous methods on this benchmark.

Our contributions are summarized as the following:
\begin{itemize}
\item We propose to model high-order actor-context-actor relations for spatio-temporal action localization. Such relations are mostly ignored by previous methods but crucial for achieving accurate action localization.
\item We propose a novel Actor-Context-Actor Relation Network for improving spatio-temporal action localization by explicitly reasoning about high-order relations between actors and the context.
\item We achieve state-of-the-art performances with significant margins on the AVA dataset.
\end{itemize}

\section{Introduction}
\label{sec:intro}

\begin{figure}[!t]
\centering
\includegraphics[width=\linewidth]{fig/figure_1.pdf}
\caption{We contrast our Actor-Context-Actor relation modeling with existing relation reasoning approaches for action localization.
Reasoning relations between pairs of entities may not always be sufficient for correctly predicting the action labels of all individuals. Our method not only reasons relations between actors, but also models connections between different actor-context relations. As an illustration, the relation between the blue actor and the steering wheel (drive) serves as a crucial clue for recognizing the action being performed by the red actor (ride). \label{fig:figure1}
}
\vspace{-4mm}
\end{figure}

Spatio-temporal action localization, which requires localizing persons and recognizing their actions from videos, is an important task that has drawn increasing attention in recent years \cite{gu2018ava, girdhar2019video, feichtenhofer2019slowfast, wu2019long, sun2018actor, zhang2019structured, xie2018rethinking, yang2019step, ulutan2020actor, peng2016multi, ye2019discovering, hou2017tube, kalogeiton2017action}. Unlike object detection which can be accomplished solely by observing visual appearances, activity recognition usually demands for reasoning about the actors' interactions with the surrounding context, including environments, other people and objects. Take Fig.~\ref{fig:figure1} as an example. To recognize the action ``ride'' of the person in the red bounding box, we need to observe that he is inside a car, and there is a driver next to him.
Therefore, most recent progress in spatio-temporal action detection has been driven by the success of relation modeling. These approaches focus on modeling relationships in terms of pairwise interactions between entities.

However, it is not always the case that relations between elements can be formulated in terms of pairs; often, higher-order relations provide crucial clues for accurate action detection. In Fig.~\ref{fig:figure1}, it is difficult to infer the action of the red actor given only its relation with the blue actor, or only with the scene context (steering wheel). Instead, in order to identify that the red actor performs the action ``ride", one has to reason over the interaction between the blue actor and the context (drive). In other words,  it is necessary to capture the implicit \textit{second-order} relation between the two actors based on their respective \textit{first-order} relations with the context.

There were previous works that employ Graph Neural Networks (GNNs) to implicitly model higher-order interactions between actors and contextual objects~\cite{wang2018videos, zhang2019structured, tang2020asynchronous, zellers2018neural, gao2020drg}. However, in these approaches, an extra pre-trained object detector is required, and only located objects are used as context. Since bounding-box annotations of objects in spatio-temporal action localization datasets are generally not provided, the pre-trained object detector is limited to its original object categories and may easily miss various objects in the scenes. In addition, the higher-order relations in these methods are limited to be inferred solely from contextual objects, which might miss important environmental or background cues for action classification.

To tackle the above issues, we propose an Actor-Context-Actor Relation Network (ACAR-Net)
which focuses on modeling second-order relations in the form of Actor-Context-Actor relation. It deduces indirect relations between multiple actors and the context for action localization. The ACAR-Net takes both actor and context features as inputs. We define actor features as the features pooled from the actor regions of interest, while for context features, we directly use spatio-temporal grid feature maps from our backbone network.
The context that we adopt does not rely on any extra object detector with predefined categories, thus making our overall design much simpler and flexible.
Moreover, grid feature maps are capable of representing scene elements of various levels (\eg instance level and part level) and types (\eg background, objects and object parts), which is useful for fine-grained action discrimination.
The proposed ACAR-Net first encodes first-order actor-context relations, and then applies a \textit{High-Order Relation Reasoning Operator} to model interactions between the first-order relations. The High-Order Relation Reasoning Operator is fully convolutional and operates on first order relational features maps without losing spatial layouts
. For supporting actor-context-actor relation reasoning between actors and context at different time periods, we build an \textit{Actor-Context Feature Bank}, which contains actor-context relations from different time steps across the whole video.

We conduct extensive experiments on the challenging Atomic Visual Actions (AVA) dataset~\cite{gu2018ava, li2020ava} as well as the UCF101-24 dataset~\cite{soomro2012ucf101} for spatio-temporal action localization.
Our proposed ACAR-Net leads to significant improvements on recognizing human-object and human-human interactions. Qualitative visualization shows that our method learns to attend contextual regions that are relevant to the action of interest.

Our contributions are summarized as the three-fold:
\begin{itemize}
\item We propose to model actor-context-actor relations for spatio-temporal action localization. Such relations are mostly ignored by previous methods but crucial for achieving accurate action localization.
\item We propose a novel Actor-Context-Actor Relation Network for improving spatio-temporal action localization by explicitly reasoning about higher-order relations between actors and the context.
\item We achieve state-of-the-art performances with significant margins on the AVA and UCF101-24 datasets. At the time of submission, our method ranks first on the ActivityNet leaderboard \cite{caba2015activitynet}.
\end{itemize}

\section{Related Work}
\label{sec:RelatedWork}

{\flushleft \bf Action Recognition.}
Research works on action recognition generally fall into three categories: action classification, temporal localization and spatio-temporal localization. Early works mainly focus on classifying a short video clip into an action class. 3D-CNN \cite{tran2015learning, carreira2019short, feichtenhofer2019slowfast}, two-stream network \cite{simonyan2014two, wang2016temporal,feichtenhofer2016convolutional} and 2D-CNN \cite{yue2015beyond, donahue2015long, lin2019tsm} are the three dominant network architectures adopted for this task. While progress has been made for short trimmed video classification, the main research stream also moves forward to understand long untrimmed videos, which requires not only to recognize the category of each action instance but also to locate its start and end times. A handful of works \cite{shou2016temporal, xu2017r, Chao_2018_CVPR, zhao2017temporal} consider this problem as a detection problem in 1D temporal dimension by extending object detection frameworks.
{\flushleft \bf Spatio-Temporal Action Localization.}
Recently, the problem of spatio-temporal action localization has drawn considerable attention from the research community, and datasets (such as AVA \cite{gu2018ava, li2020ava}) with atomic actions of all actors in the video being continuously annotated are introduced. It defines the action detection problem into a finer level, since the action instances need to be localized in both space and time. Typical approaches used by early works apply R-CNN detectors on 3D-CNN features \cite{gu2018ava, girdhar2018better, yang2019step, xiao2020audiovisual, li2018recurrent}. Wu \textit{et al.} \cite{wu2020context} show that actor features obtained by running 3D-CNN backbone on top of the cropped and resized actor region from the original video preserve better spatial details than RoI-pooled actor features. Nevertheless, it has the limitation that computational costs and inference time almost increase linearly with the number of actors. Several more recent works have exploited graph-structured networks to leverage contextual information \cite{sun2018actor, girdhar2019video, zhang2019structured, tang2020asynchronous, ulutan2020actor, tomei2019stage}.

{\flushleft \bf Relational Reasoning for Video Understanding.}
Relational reasoning has been studied in the domain of video understanding \cite{wang2018non, wang2018videos, zhou2017temporal, sun2019relational, zhang2019structured, sun2018actor, wu2019long, ji2020action, materzynska2020something, tang2020asynchronous}.
This is natural because recognizing the action of an actor depends on its relationships with other actors and objects.
Zhou \textit{et al.} \cite{zhou2017temporal} extend Relation Network \cite{santoro2017simple} for modeling relations between video frames over time. Non-local Networks \cite{wang2018non} leverage self-attention mechanisms to capture long range dependencies between different entities. Wang \textit{et al.} \cite{wang2018videos} show that representing videos with Space-time Region Graph improves action classification accuracy.
In the context of spatio-temporal localization, there are many traditional approaches that are dedicated to capturing spatio-temporal relationships in videos \cite{zhang2013modeling, morariu2011multi, swears2014complex, intille1999framework}. For deep neural networks based methods, Sun \textit{et al.} \cite{sun2018actor} propose Actor-Centric Relation Network that learns to aggregate actor and scene features.
Girdhar \textit{et al.} \cite{girdhar2019video} re-purpose the Transformer network \cite{vaswani2017attention} for encoding pairwise relationships between every two actor proposals.
Concurrently, Wu \textit{et al.} \cite{wu2019long} use long-term feature banks (LFB) to provide temporal supportive information up to 60s for computing long range interaction between actors.
Zhang \textit{et al.} \cite{zhang2019structured} propose to explicity model interactions between actors and objects.
However, their approach focuses on modeling actor-object and actor-actor relations separately. When deducing the action of a person, the interactions of other persons with contextual objects are ignored. In other words, they do not explicitly model the actor-context-actor relations. In contrast, our method emphasizes modeling those higher-order relations. Perhaps the most similar work to ours is \cite{tang2020asynchronous}, which aggregates multiple types of interactions with stacked units akin to Transformer Networks \cite{vaswani2017attention}. Nonetheless, while this approach also supports actor-context-actor interactions, it treats object detection results as context, which requires extra pre-trained object detectors with fixed object categories and ignores other important types of contexts (such as background, objects not in the predefined categories, and specific parts of some objects).

\section{Method}
\label{sec:Method}
In this section, we provide detailed descriptions of our proposed Actor-Context-Actor Relation Network (ACAR-Net).
Our ACAR-Net aims at effectively modeling and utilizing higher-order relations built upon the basic actor-actor and actor-context relations for achieving more accurate action localization.

\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{fig/pipeline.pdf}
\caption{ \textbf{Action Detection Framework.} Videos are processed with a Backbone Network to produce spatio-temporal context features. For each actor proposal (person bounding box), we extract actor features from the context features by RoIAlign. Given the actor and context features, the ACAR-Net computes second-order relation between every two actors based on their interaction with the context.
\label{fig:pipeline}
}
\vspace{-5mm}
\end{figure*}

\subsection{Overall Framework}
We first introduce our overall framework for action localization, where the proposed actor-context-actor relation (ACAR) modeling is the key module. The framework is designed to detect all persons in an input video clip ($\sim$2s
in our experiments) and estimate their action labels.
As shown in Fig.~\ref{fig:pipeline}, following state-of-the-art methods~\cite{wu2019long,feichtenhofer2019slowfast,xia2019three}, the framework is built based on an {off-the-shelf person detector} (\textit{e.g.} Faster R-CNN~\cite{ren2015faster}) and a {video backbone network} (\textit{e.g.} I3D~\cite{carreira2017quo}). Person and context features are then processed by the proposed ACAR module with a long-term \textit{Actor-Context Feature Bank} for final action prediction.

In details, the person (actor) detector operates on the center frame (\ie key frame) of the input clip and obtains $N$ detected actors. The detected boxes are duplicated to neighboring frames of the key frame in the clip.
In the meantime, the backbone network extracts a spatio-temporal feature volume from the input video clip. We perform average pooling along the temporal dimension to save follow-up computational cost, which results in a feature map $X \in \mathbb{R}^{C\times H\times W}$, and $C, H, W$ are channel, height and width respectively. We apply RoIAlign~\cite{he2017mask} ($7\times 7$ spatial output) followed by spatial max pooling to the $N$ actor features, producing a series of $N$ actor features, $A^1, A^2, \dots, A^N \in \mathbb{R}^C$, each of which describes the spatio-temporal appearance and motion of one Region of Interest (RoI).

The proposed Actor-Context-Actor Relation (ACAR) module is illustrated on the right side of Fig.~\ref{fig:pipeline}. This module takes the aforementioned video feature map $X$ and RoI features $\{A^i\}_{i=1}^N$ as inputs, and outputs the final action predictions after relation reasoning. The ACAR module has two main operations. (1) It first encodes  first-order actor-context relations between actors and spatial locations of the spatio-temporal context. Based on the actor-context relations, we further integrate a \textit{High-order Relation Reasoning Operator} (HR$^2$O) for modeling the interactions between pairs of first-order relations, which are indirect relations mostly ignored by previous methods.
(2) Our reasoning operation is extended with an {Actor-Context Feature Bank} (ACFB). The bank contains actor-context relations at different time stamps, and can provide more complete spatio-temporal context than the existing long-term feature bank \cite{wu2019long} which only consists of features of actors. We will elaborate the two parts in the following sections. Notably, our high-order relation reasoning only requires action labels as supervision.

\subsection{Actor-Context-Actor Relation Modeling}
\label{sec:hro}

\noindent \textbf{First-order actor-context relation encoding.} We adopt the Actor-Centric Relation Network (ACRN) \cite{sun2018actor} as a module for encoding the first-order actor-context relations by combining RoI features $A^1,$ $\dots, A^N$ with the context feature $X$. More specifically, it replicates and concatenates each actor feature $A^i \in \mathbb{R}^{C}$ to all $H\times W$ spatial locations of the context feature $X \in \mathbb{R}^{C \times H \times W}$ to form a series of concatenated feature map $\{\tilde{F}^{i}\}_{i=1}^N \in \mathbb{R}^{2C \times H \times W}$. Actor-context relation features for each actor $i$ can then be encoded by applying convolutions to this concatenated feature map ${\tilde{F}^{i}}$

\noindent \textbf{High-order relation reasoning.}
We now discuss how to compute high-order relations between two actors based on their first-order interactions with the context.
Let $F^{i}_{x,y}$ record the first-order features between the actor $A^i$ and the scene context $X$ at the spatial location $(x,y)$. We propose to model the relationship between first-order actor-context relations, which are high-order relations encoding more informative scene semantics. However, since there are a large number of actor-context relation features, $F^i_{x,y} \in \mathbb{R}^{C \times 1 \times 1}$, $i \in \{1,\dots, N\}, x \in [1,H], y \in [1,W]$, the number of their possible pairwise combinations is generally overwhelming. We therefore design
a \textit{High-order Relation Reasoning Operator} (HR$^2$O) that aims at learning the high-order relations between pairs of actor-context relations at the same spatial location $(x,y)$, \ie,\ $F^i_{x,y}$ and $F^j_{x,y}$. In this way, the proposed relational reasoning operator limits the relation learning to second-order actor-context-actor relations, \ie two actors $i$ and $j$ can be associated via the same spatial context, denoted as $i \leftrightarrow (x,y) \leftrightarrow j$, to help the estimation of their actions.

Our proposed HR$^2$O takes as input a set of first-order actor-context relation feature maps $\{F^i\}_{i=1}^N$.
The operator outputs
$\{H^i\}^{N}_{i=1}=\text{HR}^2\text{O}(\{F^i\}^{N}_{i=1})$
that encode second-order actor-context-actor relations for all actors.
The operator is modeled as stacking several modified non-local blocks~\cite{wang2018non}.
For each non-local block,
convolutions are used to convert the input first-order actor-context relation feature maps $F^i$ into query $Q^i$ , key $K^i$ and value $V^i$ embeddings of the same spatial size as $F^i$. All feature maps are of dimension $d=512$ in our implementation. It is worth noting that the use of convolutions is not only useful for aggregating local information but also makes the operator position and order-sensitive. The attention vectors are computed separately at every spatial location, and the Actor-Context-Actor Relation feature $H^i$ is given by the linear combination of all value features $\{V^j\}_{j=1}^N$ according to their corresponding attention weights $Att^{i,j}$. The overall process can be summarized by the following equations,
\begin{equation}
    \vspace{-3mm}
    \begin{aligned}
    {Q^i, K^i, V^i}&=\mathrm{conv2D}(F^i)\\

    {Att}^{i}_{x,y}&= \mathrm{softmax}_j\left(\frac{\left\langle{Q^{i}_{x,y}},{K^{j}_{x,y}}\right\rangle}{\sqrt{d}}\right),\\

    \tilde{H}^{i}_{x,y}&=\sum_{j}Att^{i,j}_{x,y}{V^{j}_{x,y}}.
    \end{aligned}

\end{equation}
Following \cite{wu2019long}, we also add layer normalization and dropout to our modified non-local block,
\begin{equation}
    \begin{aligned}

    {H^{i}}&=\mathrm{Dropout}(\mathrm{Conv2D}(\mathrm{ReLU}(\mathrm{norm}({\tilde{H}^{i}})))), \\
    {F'}^i &= F^i + H^i,
    \end{aligned}
\end{equation}
where $H^i$ and the input actor-context features $F^i$ are fused via residual addition to obtain the actor-context-actor feature ${F'}^i$, which can be further processed by the following non-local block again.

We also exploit another instantiation, which directly obtains second-order actor-context-actor interaction features from actor features $\{A^i\}_{i=1}^N$ and the context feature $X$ by a Relation Network~\cite{santoro2017simple}. More specifically, we obtain the relation feature between actors $A^i$, $A^j$ and context $V_{x,y}$ as
\begin{equation}
    H^{i,j}_{x,y} = f_\theta([A^i, A^j, V_{x,y} ]),

\end{equation}
where $[\cdot,\cdot,\cdot]$ denotes concatenation along the channel dimension and $f_\theta(\cdot)$ is a stack of two convolutional layers.
The high-order relation of an actor $i$ is calculated as the average of all relation features related to that actor,
\begin{equation}
    H^{i} =  \frac{1}{N} \sum_{j} H^{i,j}_{x,y}.
    \vspace{-3mm}
\end{equation}
It is also fused with the input features to obtain actor-context-actor features via residual addition, \ie ${F'}^i = F^i + H^i$.
This method is computationally expensive when the number of actors $N$ is large, since the number of feature triplets is proportional to $N^2$.

\noindent {\bf Action classifier.} After the actor-context-actor feature maps $\{{F'}^i\}_{i=1}^N$ are obtained for all actors, a final action classifier is introduced as a single fully-connected layer with a non-linearity function to output the confidence scores of each actor belonging to different action classes.

\subsection{Actor-Context Feature Bank}
\label{sec:acfb}

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{fig/bank_operator.pdf}
\caption{Illustration of ACAR-Net equipped with Actor-Context Feature Bank, where ACAR-Net* refers to the first-order relation extraction part of our proposed module.
}
\label{fig:acfb}
\vspace{-5mm}
\end{figure}

In order to support actor-context-actor relation reasoning between actors and context at different time periods in a long video, we propose an Actor-Context Feature Bank (ACFB), in which we store contextual information from both past and future. This is inspired by the Long-term Feature Bank (LFB) proposed in \cite{wu2019long}. Yet instead of providing relational features for long-term higher-order reasoning, the previous LFB only stores actor features for facilitating first-order actor-actor interaction recognition.

As is illustrated in Fig.~\ref{fig:acfb}, clips are evenly sampled (every 1 second) from an input video, and the clips ($\sim$2s) could overlap with each other. We first train a separate ACAR-Net without any feature bank following the descriptions in Section \ref{sec:hro}.
First-order actor-context relation features $F^i$ of each actor in all clips of the entire video would be extracted by the separately pre-trained ACAR-Net and archived as the feature bank. To avoid confusion, we re-denote these acquired first-order features in the bank as $L^i$.

To train a new ACAR-Net with the support of the long-term actor-context feature bank to conduct high-order relation reasoning at some current time step $t$, we retrieve all $M$ archived actor-context relation features $\{L^i\}_{i=1}^M$ from the frames within a time window $[t-w, t+w]$.
Actor-context-actor interactions between short-term features (encoding first-order interactions at current time $t$) and long-term ones from the archived bank can be computed as $\{H^i\}^{N}_{i=1}=\text{HR}^2\text{O}(\{F^i\}^{N}_{i=1},\{L^j\}^{{M}}_{j=1})$.
Note that, the $\text{HR}^2\text{O}$ is the same as before, but the self-attention mechanism is replaced with the attention between current and long-term actor-context relations, where query features $Q$ are still computed from short-term features $\{F^i\}^{N}_{i=1}$, but key and value features, $K$ and $V$, are calculated with the long-term archived features $\{{L}^i\}^{M}_{j=1}$, \ie,
\begin{equation}
    \begin{aligned}
    {Q^i}&=\mathrm{conv2D}(F^i),\\
    {K^j, V^j}&=\mathrm{conv2D}({L}^j).\\
    \end{aligned}
\end{equation}
Consequently, for any actor $i$ at current time $t$, our ACAR-Net is now capable of reasoning about its higher-relations with actors and context over a much longer time span, and thus better captures what is happening in the temporal context for achieving more accurate action localization.
\vspace{-2mm}

\section{Experiments on AVA}
\label{sec:Experiments}

AVA \cite{gu2018ava} is a video dataset for spatio-temporally localizing atomic visual actions. For AVA, box annotations and their corresponding action labels are provided on key frames of 430 15-minute videos with a temporal stride of 1 second. We use version 2.2 of AVA dataset by default.
In addition to the current AVA dataset, Kinetics-700~\cite{carreira2019short} videos with AVA~\cite{gu2018ava} style annotations are also introduced. The new AVA-Kinetics dataset~\cite{li2020ava} contains over 238k unique videos and more than 624k annotated frames. However, only a single frame is annotated for each video from Kinetics-700.
Following the guidelines of the benchmarks, we only evaluate 60 action classes with mean Average Precision (mAP) as the metric, using a frame-level IoU threshold of $0.5$.

\subsection{Implementation Details}

{\flushleft \bf Person Detector.} For person detection on key frames, we use the human detection boxes from \cite{wu2019long}, which are generated by a Faster R-CNN~\cite{ren2015faster} with a ResNeXt-101-FPN~\cite{xie2017aggregated,lin2017feature} backbone. The model is pre-trained with Detectron~\cite{girshick2018detectron} on ImageNet~\cite{deng2009imagenet} as well as the COCO human keypoint images~\cite{lin2014microsoft}, and fine-tuned on the AVA dataset.
\vspace{-2mm}
{\flushleft \bf Backbone Network.} We use SlowFast networks~\cite{feichtenhofer2019slowfast} as the backbone in our localization framework and increase the spatial resolution of res$_5$ by $2\times$. We conduct ablation experiments using a SlowFast R-50 $8\times8$ instantiation (without non-local blocks). The inputs are 64-frame clips, where we sample $T=8$ frames with a temporal stride $\tau=8$ for the slow pathway, and $\alpha T (\alpha=4)$ frames for the fast pathway.
The backbone is pre-trained on the Kinetics-400 dataset\footnote{The pre-trained SlowFast R-50 and SlowFast R-101+NL (in the following section) are downloaded from SlowFast's official repository.
}.

\vspace{-3mm}
\textbf{\flushleft Training and inference.}
 In AVA, actions are grouped into 3 major categories: poses (e.g. stand, walk), human-object and human-human interactions. Given that poses are mutually exclusive and interactions are not, we use softmax for poses and sigmoid for interactions before binary cross-entropy loss for training.
We train all models end-to-end (except for
the feature bank part) using synchronous SGD with a batch size of 32 clips.
We train for 35k iterations with a base learning rate of 0.064, which is then decreased by a factor of 10 at iterations 33k and 34k.
We perform linear warm-up~\cite{goyal2017accurate} during the first 6k iterations. We use a weight decay of $10^{-7}$ and Nesterov momentum of 0.9.
We use both ground-truth boxes and predicted human boxes from \cite{wu2019long} for training.
For inference, we scale the shorter side of input frames to 256 pixels and use detected person boxes with scores greater than 0.85 for final action classification.

\begin{table*}[!t]\vspace{-4mm}
\centering
\subfloat[\textbf{Relation Modeling}\label{tab:ablation:type}]{%
\tablestyle{4.8pt}{1.1}\begin{tabular}{@{}lx{20}@{}}
 & mAP \\
\shline
 {Baseline + STO} \cite{wu2019long} & 26.10\\
 {Baseline + ACRN \cite{sun2018actor} } & 26.71\\
 {Baseline + AIA} \cite{tang2020asynchronous} & 26.79\\
 {Baseline + HR$^2$O} & \textbf{27.83}\\
\end{tabular}}\hfill
\subfloat[\textbf{Component Analysis}\label{tab:ablation:structure}]{%
\tablestyle{4.8pt}{1.1}
\begin{tabular}{@{}lx{20}@{}}
& mAP \\
\shline
{Baseline} & 25.39\\
{Baseline + HR$^2$O} & {27.83}\\
{ACAR} & \textbf{28.84}\\
\multicolumn{2}{c}{~}\\
\end{tabular}}\hfill
\subfloat[\textbf{HR$^\mathbf{2}$O Design}\label{tab:ablation:function}]{%
\tablestyle{4.8pt}{1.1}
\begin{tabular}{@{}lx{25}@{}}
 & mAP\\
\shline
 {Avg \ \ \ \ \ } & 26.97 \\
 {RN} & 27.18 \\
 {NL} & \textbf{27.83}\\
 \multicolumn{2}{c}{~}\\
\end{tabular}}\hfill
\subfloat[\textbf{Relation Order}\label{tab:ablation:order}]{%
\tablestyle{4.8pt}{1.1}\begin{tabular}{@{}lx{20}@{}}
 & mAP \\
\shline
 {Actor First} & 27.62\\
 {Context First} & \textbf{27.83}\\
 \multicolumn{2}{c}{~}\\
\multicolumn{2}{c}{~}\\
\end{tabular}}\hfill
\subfloat[\textbf{Relation Depth}\label{tab:ablation:layer}]{%
\tablestyle{4.8pt}{1.1}
\begin{tabular}{@{}lx{20}@{}}
 & mAP\\
\shline
 {HR$^2$O-1L\ \ \ } & {27.63}\\
 {HR$^2$O-2L} & \textbf{27.83}\\
 {HR$^2$O-3L} & 27.25\\
\multicolumn{2}{c}{~}\\
\end{tabular}}\hfill
\subfloat[\textbf{Feature Bank}\label{tab:ablation:bank}]{%
\tablestyle{4.8pt}{1.1}\begin{tabular}{@{}lx{20}@{}}
 & mAP\\
\shline
 {HR$^2$O} & 27.83\\
 {HR$^2$O + LFB \cite{wu2019long}} & 27.75\\
 {HR$^2$O + ACFB} & \textbf{28.84}\\
\multicolumn{2}{c}{~}\\
\end{tabular}}

\vspace{2mm}
\caption{\textbf{Ablation study on AVA dataset}. The ``Baseline" of our framework only consists of the video backbone, actor detector and one-layer action classifier. HR$^2$O: High-order Relation Reasoning Operator. ACFB: Actor-Context Feature Bank. \label{tab:ablations}}
\vspace{-3mm}
\end{table*}

\subsection{Ablation Study}
We conduct ablation experiments to investigate the effect of different components in our framework on AVA v2.2. The baseline of our framework only consists of the video backbone (SlowFast R-50), the actor detector and the single-layer action classifier (denoted as ``Baseline'' in Table \ref{tab:ablations}).
\vspace{-3mm}
{\flushleft \bf Relation Modeling - Comparison.} In order to show the effectiveness of our actor-context-actor relation reasoning module, we compare against several previous approaches that leverage relation reasoning for action localization based on our baseline. Here we focus on validating the effect of relation modeling only, thus we disable long-term support in this study.
We adapt their reasoning modules such that all methods use the same baseline as our ACAR-Net in order to fairly compare only the impact of relation reasoning. We evaluate
ACRN that focuses on learning actor-context relations; STO \cite{wu2019long} (a degraded version of LFB) that only captures actor interactions within the current short clip; AIA (w/o memory) \cite{tang2020asynchronous} that aggregates both actor-actor and actor-object interactions. As listed in Table~\ref{tab:ablation:type}, our proposed actor-context-actor relation modeling (``Baseline + HR$^2$O'' in Table~\ref{tab:ablation:type}) significantly improves over the compared methods. We observe that AIA with both actor and context relations performs better than ACRN and STO which only model one type of first-order relations, yet our method based on high-order relation modeling outperforms all compared methods by considerable margins.

We further break down the performances of different relation reasoning modules into three major categories of the AVA dataset, which are poses (\textit{e.g.} stand, sit, walk), human-object interactions (\textit{e.g.} read, eat, drive) and human-human interactions (\textit{e.g.} talk, listen, hug). Fig.~\ref{fig:catgain} compares the gains of different approaches with respect to the baseline in terms of mAP on these major categories. We can see that our HR$^2$O gives more performance boosts
on two interaction categories compared to the pose category, which is consistent with our motivation to model indirect relations between actors and context. Once equipped with ACFB, our framework can further improve on the pose category as well.

Finally, we contrast our ACAR with existing relation reasoning approaches in AVA. We visualize attention maps from different reasoning modules over an example key frame in Fig.~\ref{fig:compare_attmap}. Without needing object proposals, ACAR is capable of localizing free-form context regions for indirectly establishing relations between two actors (the actor of interest is listening to the supporting actor reading a report). In comparison, the attention weights of STO as well as AIA are distributed more diversely and do not have a clear focus point. Note that we do not show the attention map of ACRN since it assigns equal weights to all context regions.
{\flushleft \bf Component Analysis.} To validate our design, we first ablate the impacts of different components of our ACAR as shown in Table \ref{tab:ablation:structure}. We can observe that both HR$^2$O and ACFB lead to significant performance gains over baseline.
{\flushleft \bf HR$^\mathbf{2}$O Design.} We test different instantiations of the High-order Relation Reasoning Operator on top of our baseline in Table \ref{tab:ablation:function}.
Our modified non-local (denoted as ``NL'') mechanism works better than simply designing $\text{HR}^2\text{O}$ as an average function (denoted as ``Avg''), \ie $H^i = \frac{1}{N} \sum_{i} F^i,$. In addition, the instantiation with relation network (RN) described in Section \ref{sec:hro} also works
alright. Nonetheless, the modified non-local attention is computationally more efficient than RN with feature triplets and has better performance.
{\flushleft \bf Relation Ordering.} There are two possible orders for reasoning actor-context-actor relations: 1) aggregating actor-actor relations first, or 2) encoding actor-context relations first. Note that our ACAR-Net adopts the latter one. We implement the former order by performing self-attention between actor features with the modified non-local attention before incorporating context features in our baseline. The results in Table~\ref{tab:ablation:order} validate that context information should be aggregated earlier for better relation reasoning.
{\flushleft \bf HR$^\mathbf{2}$O Depth.} In Table~\ref{tab:ablation:layer}, we observe that stacking two modified non-local blocks in HR$^2$O has higher mAP than the one-layer version, yet adding one more non-local block produces worse performance, possibly due to overfitting. We therefore adopt two non-local blocks as the default setting
{\flushleft \bf Actor-Context Feature Bank.} In this set of experiments, we validate the effectiveness of the proposed ACFB. We set the ``window size'' $2w+1$ to 21s due to memory limitations, and longer temporal support is expected to perform better \cite{wu2019long}.
As presented in Table~\ref{tab:ablation:bank}, adding long-term support with ACFB significantly improves the baseline (HR$^2$O's 27.83 $\rightarrow$\ HR$^2$O + ACFB's \textbf{28.84}).
We also test replacing the ACFB in our framework with the long-term feature bank (LFB)~\cite{wu2019long} (denoted as ``HR$^2$O + LFB''). However, LFB even fails to match the baseline performance. This drop might be because LFB encodes only ``zeroth-order'' actor features, which cannot provide enough relational information from neighboring frames for assisting interaction recognition.

\begin{figure}[t]
\centering
\includegraphics[width=1\linewidth]{fig/category_gain.pdf}
\caption{\textbf{Gains of mAP on three major categories of the AVA dataset with respect to Baseline.} Our ACAR consistently outperforms other relation reasoning methods, and achieves larger performance gains on the two interaction categories. \label{fig:catgain}}

\end{figure}

\begin{table}[!t]
\centering
\tablestyle{4.8pt}{1.1}\begin{tabular}{@{}l|x{23}|x{37}@{}}
 model & inputs & val mAP\\
\shline
 {ACRN, S3D} \cite{sun2018actor} & V+F & 17.4 \\
 {Zhang \textit{et al.}} \cite{zhang2019structured}, I3D & V & 22.2 \\
 {Action TX, I3D} \cite{girdhar2019video} & V & 25.0 \\
 {LFB}, R-50+NL \cite{wu2019long} & V & 25.8 \\
 {LFB}, R-101+NL \cite{wu2019long} & V & 27.4 \\
 {SlowFast, R-50, $8\times8$} \cite{feichtenhofer2019slowfast} & V & 24.8 \\
 {SlowFast, R-101, $8\times8$} \cite{feichtenhofer2019slowfast} & V & 26.3 \\
 \hline
 \textbf{Ours}, R-50, $8\times8$ & V & 28.3 \\
 \textbf{Ours}, R-101, $8\times8$ & V & {\bf 30.0} \\
\end{tabular}
\vspace{2mm}
\caption{\textbf{Comparison with state-of-the-arts on AVA v2.1.}
All models are pre-trained on Kinetics-400. V and F refer to visual frames and optical flow respectively. \label{tab:sota_v2.1}}
\end{table}

\begin{table}[!t]
\centering
\tablestyle{4.8pt}{1.1}\begin{tabular}{@{}l|x{51}|x{37}@{}}
 model & pre-train & val mAP \\
\shline
 {SlowFast, R-101+NL} \cite{feichtenhofer2019slowfast} & Kinetics-600 & 29.0 \\

 {AIA, R-101+NL} \cite{tang2020asynchronous} & Kinetics-700 & 32.3 \\
 \hline
 Ours, R-101+NL & Kinetics-600 & 31.4 \\
 \textbf{Ours}, R-101 & Kinetics-700 & {\bf 33.3} \\
\end{tabular}
\vspace{2mm}
\caption{\textbf{Comparison with state-of-the-arts on AVA 2.2.} We do not conduct testing with multiple scales and flips. All models use $T\times\tau=8\times8$. \label{tab:sota_v2.2}}
\end{table}

\begin{table}[!t]
\centering
\tablestyle{4.8pt}{1.1}\begin{tabular}{@{}l|x{37}|x{37}@{}}
 model & val mAP & test mAP\\
\shline
 {AIA++, ensemble \cite{xia2020report}}& - & 32.91 \\
 {MSF, ensemble \cite{Zhu2020report}}& - & 31.88 \\
\hline
 {SlowFast, R-101, $8\times8$ (our impl.)} & 32.98 & - \\
 {Ours, R-101, $8\times8$} & 35.84 & - \\
 {Ours++, R-101, $8\times8$} & 36.36 & - \\
 {Ours++, ensemble}  & \textbf{40.49} & \textbf{39.62}
\end{tabular}
\vspace{2mm}
\caption{\textbf{AVA-Kinetics results}. ``++'' refers to inference with 3 scales and horizontal flips. Models submitted to the test server are trained on both training and validation sets.
\label{tab:sota_ak}}
\vspace{-3mm}
\end{table}

\begin{figure}[t]
\centering
\includegraphics[width=1\linewidth]{fig/compare_att.pdf}
\caption{\textbf{Comparison of attention maps from different approaches of relation modeling for action detection.}
Our method is able to attend the contextual region (some document) that relates the \textcolor{red}{actor of interest} marked in red (performing ``listen to'') and the \textcolor{green}{supporting actors} in the green box (performing ``read''), while other methods fail to achieve similar effects.  \label{fig:compare_attmap}}
\vspace{-4mm}
\end{figure}

\subsection{Comparison with State-of-the-arts on AVA}

We compare our ACAR-Net with state-of-the-art methods on the validation set of both AVA v2.1 (Table~\ref{tab:sota_v2.1}) and v2.2 (Table~\ref{tab:sota_v2.2}). Note that we also provide results with more advanced video backbones, \ie two SlowFast R-101 instantiations (with / without NL).
On AVA v2.1, our framework achieves {30.0 mAP} and outperforms all prior results with pre-trained Kinetics-400 backbone. On AVA v2.2, our ACAR-Net reaches {33.3 mAP} with only single-scale testing, establishing a new state-of-the-art. Note that our method surpasses AIA \cite{tang2020asynchronous} with only $1/3$ of temporal support. The results indicate that with proper modeling of higher-order relations, our approach extracts more informative cues from the context.

\begin{figure*}[t]
\centering
\includegraphics[width=1\linewidth]{fig/attention.pdf}
\caption{\textbf{Visualization of actor-context-actor attention maps on AVA.} \textcolor{red}{Actors of interest} are marked in red and \textcolor{green}{supporting actors} in green. Heat maps illustrate the context regions' attention weights $Att^{i,j}$ from actor-context-actor relation reasoning. We observe that our model has learned to attend to useful relations between actors and context, and the context serves as the bridge for connecting actors.
\label{fig:attn}}
\vspace{-4mm}
\end{figure*}

We present our results on AVA-Kinetics in Table~\ref{tab:sota_ak}. Our baseline is already highly competitive ($\sim$33 mAP). Yet integrating our ACAR modeling still leads to a significant gain of +2.86 mAP. This demonstrates that performance enhancement brought by high-order relation modeling can generalize to this new dataset.
With an ensemble of models, we achieve \textbf{39.62 mAP} on the test set, ranking first in the AVA-Kinetics task of ActivityNet Challenge 2020 and outperforming other entries by a large margin (+6.71 mAP). More details on our winning solution are provided in the technical report \cite{chen20201st}.

\subsection{Qualitative Results}

Our proposed ACAR operates fully convolutionally on top of spatio-temporal features, and this allows us to visualize the actor-context-actor relation maps $\{Att^{i,j}\}$ generated by our High-order Relation Reasoning Operator. As shown in Fig.~\ref{fig:attn}, the first two columns include the key frame as well as the corresponding relation map from the same clip, and the last three columns show the relation map denoting interactions with actors and context from a neighboring clip.
We can observe that the attended regions usually include the actor of interest, supporting actors' body parts (\ie head, hands and arm) and objects being in interaction with the actors. Take the first example on the left as an example. The green supporting actor $A^j$ is taking a package from the red actor of interest $A^i$. Such information is well encoded by our ACAR-Net in the form of actor-context-actor relations: packages, hands and arms of both actors are highlighted.

\section{Experiments on UCF101-24}
\label{sec:Experiments_UCF}
UCF101-24 is a subset of UCF101~\cite{soomro2012ucf101} that contains spatio-temporal annotations for 3,207 videos on 24 action classes. Following the evaluation settings of previous methods \cite{kalogeiton2017action, yang2019step}. We experiment on the first split and report frame-mAP with an IoU threshold of 0.5.
\vspace{-2mm}
{\flushleft \bf Implementation Details.} We also use SlowFast R-50 pre-trained on Kinetics-400 as the video backbone, and adopt the person detector from \cite{kopuklu2019you}. The temporal sampling for the slow pathway is changed to $8\times4$ and the fast pathway takes as input 32 continuous frames.

For training, we train all the models end-to-end for 5.4k iterations with a base learning rate of 0.002, which is then decreased by a factor of 10 at iterations 4.9k and 5.1k.
We perform linear warm-up during the first quarter of the training schedule.
We only use ground-truth boxes for training, and use all boxes given by the detector for inference. Other hyper-parameters are similar to the experiments on AVA.
\vspace{-2mm}
{\flushleft \bf Results.} As shown in Table~\ref{tab:ucf}, ACAR surpasses the strong baseline with a considerable margin, which again indicates the importance of high-order relation reasoning.

\begin{table}[t]
\centering
\tablestyle{4.8pt}{1.1}\begin{tabular}{@{}l|x{23}|x{31}@{}}
 model & inputs & mAP\\
\shline
 {T-CNN} \cite{hou2017tube} & V &67.3 \\
 {ACT} \cite{kalogeiton2017action} & V &69.5 \\
 {STEP, I3D} \cite{yang2019step} & V+F &75.0 \\
 {I3D} \cite{gu2018ava} & V+F & 76.3 \\
 {Zhang \textit{et al.}} \cite{zhang2019structured}, I3D & V & 77.9\\
 {S3D-G} \cite{xie2018rethinking} & V+F &78.8\\
 {AIA, R-50} \cite{tang2020asynchronous} & V &78.8\\
 \hline
 SlowFast R-50, $8\times4$ (ours) & V &\underline{82.4} \\
 \textbf{Ours w/o ACFB}, R50, $8\times4$ & V &{\bf 84.3} \\
\end{tabular}
\vspace{3mm}
\caption{\textbf{Comparison with previous works on UCF101-24.} We evaluate frame-mAP on split 1. V and F refer to visual frames and optical flow respectively.\label{tab:ucf}}
\vspace{-5mm}
\end{table}

\section{Conclusion}
\label{sec:Conclusion}

Given the high complexity of realistic scenes encountered in the spatio-temporal action localization task which involve multiple actors and a large variety of contextual objects, we observe the demand for a more sophisticated form of relation reasoning than current ones which often miss important hints for recognizing actions.
Therefore, we propose Actor-Context-Actor Relation Network for explicitly modeling higher-order relations between actors based on their interactions with the context.
Extensive experiments on the action detection task show our ACAR-Net outperforms existing methods that leverage relation reasoning, and achieves state-of-the-art results on several challenging benchmarks of spatio-temporal action localization.

\noindent {\bf Acknowledgements.} We thank Charlie W., Jiajun T. and Daixin W. for helpful discussions. This work is supported in part by the General Research Fund through the Research Grants Council of Hong Kong under Grants (Nos. 14208417, 14207319, 14202217, 14203118, 14208619), in part by Research Impact Fund Grant No. R5001-18, in part by CUHK Strategic Fund.


\section{Introduction}

Ultrasound (US) imaging has been widely used in interventional applications to monitor and trace target tissue. US possesses many advantages, such as low cost, portable setup and the capability of navigating through patient in real-time for anatomical and functional information. Transrectal ultrasound imaging (TRUS) has been commonly used for guiding prostate cancer diagnosis and can significantly reduce the false negative rate when fused with magnetic resonance imaging (MRI)~\cite{siddiqui2015comparison}. However, 2D US images are difficult to be registered with 3D MRI volume, due to the differences in not only image dimension but also image appearance. In practice, a reconstructed 3D US image volume is usually required to assist such interventional tasks.

A reconstructed 3D US imaging volume visualizes a 3D region of interest (ROI) by using a set of 2D ultrasound frames~\cite{mohamed2019survey}, which can be captured by a variety of scanning techniques such as
mechanical scan~\cite{daoud2015freehand} and freehand tracked scan~\cite{wen2013accurate}. Among these categories, the tracked freehand scanning is the most favorable method in a number of clinical scenarios. For instance, during a prostate biopsy, the freehand scanning allows clinicians to freely move the US probe around the ROI and produces US images with much more flexibility. The tracking device, either an optical or electro-magnetic (EM) tracking system, helps to build a spatial transformation chain between the imaging planes in the world coordinate for 3D reconstruction.

US volume reconstruction from sensorless freehand scans takes a step further by removing the tracking devices attached to the US probe. The prior research on this was mainly supported by the speckle decorrelation~\cite{chen1997determination,tuthill1998automated}, which maps the relative difference of position and orientation between neighboring US images to the correlation of their speckle patterns, \emph{i.e.} higher the speckle correlation, lower the elevational distance between neighboring frames~\cite{chang20033}. By removing the tracking devices, such sensorless reconstruction allows the clinicians to move the probe with less constraint without the concerns of blocking tracking signals. In addition, it also reduces the hardware cost.
Although the speckle correlation carries information of the relative transformation between neighboring frames, relying on the decorrelation alone renders unreliable performance \cite{laporte2011learning,afsham2015nonlocal}.

In the past decade, deep learning (DL) methods based on convolutional neural networks (CNN) have been identified as an important tool for automatic feature extraction. In the field of US volume reconstruction, a pioneer work carried out by Prevost \emph{et~al.}~\cite{prevost20183d} explored the feasibility of using CNN to directly estimate the inter-frame motion between two 2D US scans. A 2D convolutional network takes two consecutive ultrasound frames and a generated optical flow field between them as a stacked input to estimate the relative rotations and translations between these two frames. However, a typical US scanning video contains rich contextual information beyond two neighboring frames. A sequence of 2D US frames can provide a more general representation of the motion trajectory of US probe. Using only two neighboring frames may lose temporal information and thus result in less accurate reconstruction. In addition, optical flow field, which is good at describing in-plane motion, may not help out-of-plane motion analysis. Besides, the prior works on decorrelation suggests that paying more attention to the speckle-rich regions can boost the reconstruction performance, which hasn't been explored.

In this paper, based on the above observations, we propose a novel Deep Contextual Learning Network (DCL-Net) for sensorless freehand 3D ultrasound reconstruction. The underlying framework takes multiple consecutive US frames as input, instead of only two neighboring frames, to estimate the trajectory of US probe by efficiently exploiting the rich contextual information. Furthermore, to make the network focus on the speckle-rich image areas to utilize the decorrelation information between frames, the attention mechanism is embedded into the network architecture. Last but not the least, we introduce a new case-wise correlation loss to enhance the discriminative feature learning to prevent overfitting the scanning style.

\section{Data Materials}
\label{data}
All TRUS scanning videos studied in this work are collected by an EM-tracking device from real clinical cases. The dataset contains 640 TRUS videos all from different subjects acquired by a Philips iU22 scanner in varied lengths.
Every frame corresponds to an EM tracked vector that contains the position and orientation information of that frame. We convert this vector to a 3D homogeneous tranformation matrix $M = [R \; T; 0 \; 1]$,
where $R$ is a 3$\times$3 rotation matrix and $T$ is a 3D translation vector.

The primary task of 3D ultrasound reconstruction is to obtain the relative spatial position of two or more consecutive US frames.
Without loss of generality, here we use two neighboring frames as an example for illustration. Let $I_i$ and  $I_{i+1}$ denote two consecutive US frames with corresponding transformation matrices $M_i$ and $M_{i+1}$, respectively. The relative transformation matrix $M'_i$ can be computed as $M'_i=M_{i+1}M_i^{-1}$.
By decomposing $M'_i$ into 6 degrees of freedom $\theta_i={\left \{t_x,t_y,t_z,\alpha_x,\alpha_y,\alpha_z \right \}}_i$, which contains the translations in millimeters and rotations in degrees, we can use this $\theta_i$ computed from EM tracking as the ground-truth for network training.

 \section{Ultrasound Volume Reconstruction}
\label{method}

\begin{figure}[t]
	\centering
	\includegraphics[width=\textwidth]{figures/dcl.pdf}
	\caption{An overview of the proposed DCL-Net, which takes one video segment as input volume and gives the mean motion vector as the output.}
	\label{fig:framework}
\end{figure}

Fig.~\ref{fig:framework} shows the proposed DCL-Net architecture, which is designed on top of the 3D ResNext model \cite{xie2017aggregated}. Our model consists of 3D residual blocks and other types of basic CNN layers~\cite{he2016deep}. The skip connections help preserve the gradients to train very deep networks. The use of the multiple pathways (cardinalities) enables the extraction of important features.
In our design, 3D instead of 2D convolutional kernels are used, mainly because 3D convolutions can better extract the feature mappings along the axis of channel, which is the temporal direction in our case. Such properties enable the network to focus on the slight displacement of image features between consecutive frames. The network can thus be trained to connect these speckle correlated features to estimate the relative position and orientation.

During the training process, we stack a sequence of $N$ frames with height and width denoted by $H$ and $W$, respectively, to form a 3D input volume in the shape of $N \times H \times W$. Let $\{ \mathbf{\theta}_i | i=1,\ldots,N-1 \}$ denote the relative transform parameters between the neighboring frames. Instead of directly using these parameters as ground-truth labels for network training, the mean parameters
\begin{equation}
\bar{\theta} = \frac{1}{N-1} \sum_{i=1}^{N-1} \theta_i,
\end{equation}
are used for the following two reasons.
Most importantly, since the magnitude of motion between two frames is small, using the mean can effectively smooth the noise in probe motion. Another advantage in practice is that there is no need to modify the output layer every time when we change the number of input frames.
During the test, we slide along the video sequence with a window size $N$. The inter-frame motion of two neighboring frames is the average motion computed in all the batches.

\subsection{Attention Module}

The attention mechanism in the deep learning models makes the CNN to focus on a specific region of an image, which carries salient information for the targeted task \cite{bahdanau2014neural}. It has led to significant improvement in various computer vision tasks such as object classification~\cite{fukui2019attention} and segmentation~\cite{oktay2018attention}.
In our 3D US volume reconstruction task, regions with strong speckle patterns for correlation are of high importance in estimating the transformations.
Thus, we introduce a self-attention block, as shown in Fig.~\ref{fig:framework}, which takes the feature maps produced by the last residual block as input and then outputs an attention map. This helps assign more weights to the highly informative regions.

\subsection{Case-wise Correlation Loss}
\label{sec:correlation}

\begin{figure}[t]
	\centering
	\includegraphics[width=\textwidth]{figures/corr.pdf}
	\caption{Overview of the case-wise correlation loss computation.}
	\label{corrflow}
\end{figure}

The loss function of the proposed DCL-Net consists of two components. The first one is the mean squared error (MSE) loss, which is the most commonly used loss in deep regression problems. However, the use of MSE loss alone can lead to the smoothed estimation of the motion and thus the trained network tends to memorize the general style of how the clinicians move the probe, \emph{i.e.} the mean trajectory of the ultrasound probes. This shortcoming of the MSE loss for network training has been reported before~\cite{yang2018low,johnson2016perceptual}. To deal with problem, we introduce a case-wise correlation loss based on the Pearson correlation coefficient to emphasize the specific motion pattern of a scan.

Fig.~\ref{corrflow} shows the workflow of calculating the case-wise correlation loss. $K$ video segments with each having $N$ frames are randomly sampled from a TRUS video.
The correlation coefficients between the estimated motion and the ground truth mean are computed for every degree-of-freedom and the loss is denoted as
\begin{equation}
L_{corr} = 1-\frac{1}{6}\sum_{d=1}^{6}\frac{Cov\left (\bar{\theta}_{d}^{GT}, \bar{\theta}_{d}^{Out} \right )}{\sigma \left (\bar{\theta}_{d}^{GT}  \right )\sigma \left ( \bar{\theta}_{d}^{Out} \right )},
\end{equation}
where $Cov$ gives the covariance and $\sigma$ calculates the standard deviation.
The total loss is the summation of the MSE loss and the case-wise correlation loss.

\section{Experiments and Results}
\label{experiments}

For the experiments performed in this study, a total of 640 TRUS scanning videos (640 patients) from the Nation Institute of Health (NIH) were acquired from IRB-approved clinical trial. During an intervention, a physician used an end-firing transrectal ultrasound probe to acquire axial images by steadily sweeping through the prostate from base to apex. The positioning information given by an electromagnetic tracking device serves as the ground truth label in our training phase. The dataset is split into 500, 70 and 70 cases as training, validation and testing, respectively. Our network is trained for 300 epochs with batch size $K=20$ using Adam optimizer~\cite{kingma2014adam} with initial learning rate of 5$\times 10^{-5}$, which decays by 0.9 after 5 epochs.
Since the prostate US image only takes a relative small part of each frame, each frame is cropped without exceeding the imaging field and then resized to $224 \times 224$ to fit the design of ResNexts~\cite{xie2017aggregated}.  We implemented the DCL-Net using the publicly available Pytorch library~\cite{pytorch}. The entire training phase of the DCL-Net takes about 4h, taking 5 frames as input. During testing, it takes about 2.58s to produce all the transformation matrix of an US video with 100 frames.

Two evaluation metrics are used for performance evaluation. The first one is the average distance between all the corresponding frame corner-points throughout a video. This distance error reveals the difference in speed and orientation variations across the entire video. The other one is the final drift~\cite{prevost20183d}, which is the distance between the center points of the transformed end frames of a video segment using the EM tracking data and our DCL-Net estimated motion.

\subsection{Parameter Setting}

\begin{figure}[t]

	\subfloat{\includegraphics[width=.45\textwidth]{figures/box2.pdf}}
	\subfloat{\includegraphics[width=.55\textwidth]{figures/attention.pdf}}
	\caption{(Left) Effect of number of input frames. Green curve shows the mean distance error of each box. (Right) Visualization of two attention maps regarding rotation around the Y-axis.} \label{fig:nsat}
\end{figure}

We first performed experiments to determine an optimal $N$ number of frames for each video segment. The left panel of Fig.~\ref{fig:nsat} shows how the overall reconstruction performance varies as the number of consecutive frames changes. There is a decrease then increase in the error, with neighboring frame number equaling to 5 or 6 has similarly the best performance. According to our paired t-test, the calculated p-value is smaller than the confidence level of 0.05, indicating the result using 5 frames is significantly better than that using only 2 frames. The explanation is that the network takes advantage of the rich contextual information along the time-series and produces more stable trajectory estimation.

The right panel of Fig.~\ref{fig:nsat} visualizes two example attention maps. The first image column shows the cropped US images. The second column is the speckle correlation map~\cite{chang20033} between a US image and its following neighboring frame. Inside this speckle correlation map, brighter the area, longer the elevational distance to the next frame. Such pattern with dark areas at the bottom and brighter on the upper part is consistent with our TRUS scanning protocol, as there is less motion around the tip of the ultrasound probe. The third column shows the attention map regarding the rotation $\alpha_y$ around the Y-axis, which also indicates part of the out-of-plane rotation. The attention maps have strong activation at the bright speckle correlation regions, indicating that the attention module helps the network to focus on speckle-rich areas for better reconstruction.

\subsection{Reconstruction Performance and Discussions}
\begin{table}[t]
\caption{Performance of different methods on the EM-tracking dataset.}
\centering
\begin{tabular}{l|cccc|cccc}\hline
\multicolumn{1}{c|}{\multirow{2}{*}{Methods}}                                 & \multicolumn{4}{c|}{Distance Error (mm)}                            & \multicolumn{4}{c}{Final Drift (mm)}                                \\ \cline{2-9}
\multicolumn{1}{c|}{}                                                         & Min           & Median        & Max            & Average       & Min           & Median        & Max           & Average        \\\hline
Linear Motion                                                                 & 7.17          & 19.73         & 60.79          & 22.53         & 12.53         & 37.15         & 114.02        & 42.62          \\
Decorrelation~\cite{chang20033}    & 9.62          & 17.58         & 56.72          & 18.89         & 15.32         & 38.45         & 104.13        & 38.26          \\
2D CNN~\cite{prevost20183d}                                                           & 5.66          & 15.8          & 43.35          & 17.42         & 7.05          & 23.13         & 68.87         & 26.81          \\
3D CNN (NS2)~\cite{xie2017aggregated}                                                                 & 2.38          & 10.14         & 31.34          & 12.34         & 1.42          & 19.08         & 68.61         & 21.74          \\
Our DCL-Net  & \textbf{1.97} & \textbf{9.15} & \textbf{27.03} & \textbf{10.33} & \textbf{1.09} & \textbf{17.40} & \textbf{55.50} & \textbf{17.39}\\\hline
\end{tabular}
\label{tabcompare}
\end{table}

Table~\ref{tabcompare} summarizes the overall comparison of the proposed DCL-Net against other existing methods. The approach of ``Linear Motion'' means that we first calculate the mean motion vector of the training set and then apply this fixed vector to all the testing cases. The approach of ``Decorrelation'' is based on the speckle decorrelation algorithm presented in \cite{chang20033}. ``2D CNN'' refers to the method presented by Prevost \emph{et~al.}~\cite{prevost20183d}. ``3D CNN'' is the vanilla ResNext~\cite{xie2017aggregated} architecture taking only two slices as input.

It can be seen from Table~\ref{tabcompare} that our proposed DCL-Net outperforms all the other methods. Paired t-test was performed and the performance improvement made by DCL-Net is significant in all the cases with $p$-value$<$0.05. It is worth noting that although the average distance error of 10.33mm achieved DCL-Net is still a large error, this is the best performance on real clinical data instead of phamtom studies. The performance of the state-of-the-art 2D-CNN method reproduced in our experiments has consistent performance compared to the accuracy reported in the paper \cite{prevost20183d}. It is a challenging task to reconstruct 3D US volume using these freehand TRUS scans and we have been making significant progress in this important area.

\begin{figure}[t]
	\centering
	\includegraphics[width=\textwidth]{figures/corrloss.pdf}
	\caption{Predicted rotation $\alpha_y$ on one video sequence with different methods. Applying correlation loss makes our prediction (blue line) more sensitive to the strongly varying speed of the ground-truth (green line).}
	\label{fig:corrstudy}
\end{figure}

\begin{figure}[t]
	\centering
	\includegraphics[width=\textwidth]{figures/volume.pdf}
	\caption{Comparison of the US volume reconstruction results of four cases with different qualities.}
	\label{fig:recon}
\end{figure}

Next, we demonstrate the effectiveness of incorporating case-wise correlation loss into the network training. Fig.~\ref{fig:corrstudy} shows the prediction of $\alpha_y$ along a video sequence. We can observe that the network trained with MSE loss can only produce mediocre results (red line) which are nearly constant, showing almost no sensitivity to the change in speed and orientation. Its prediction wonders around the linear motion (black line) which represents the mean value of the probe's trajectory. The correlation coefficients of all testing cases show a mean of $0.09 \pm 0.03$ which represents little correlation. This indicates that using MSE alone makes the network memorizing the general style of the US probe motion trajectory and fails to produce valid prediction based on image contents. By incorporating the correlation loss (CL) into the loss function (blue line), the correlation coefficients of all testing cases have a mean of $0.21 \pm 0.09$, representing weak correlation. Based on a paired t-test with $\alpha=0.05$, this is found to be significantly better than the previous results. Intuitively, the network's prediction reacts more sensitively to the variation of the probe's real translation and rotation (green line).

Last but not the least, we report the volume reconstruction results using four testing cases with different reconstruction qualities as shown in Fig.~\ref{fig:recon}. One good case, one bad case, and two median cases are included to offer a complete view of the performance. To reduce the clutter in the figure, we only show the comparison between our DCL-Net, the 2D-CNN~\cite{prevost20183d} and the ground-truth. While producing competitive performance, the 2D-CNN method is less sensitive to the speed variations of US probe and the estimated trajectory has noisy vibration. The results sometimes even severely deviate from the ground-truth. Our proposed DCL-Net shows a much smoother trajectory estimation thanks to the contextual information provided by video segments.

\section{Conclusions}
\label{Conclusion}
This paper introduced a sensorless freehand 3D US volume reconstruction method based on deep learning. The proposed DCL-Net can well extract the information among multiple US frames to improve the US probe trajectory estimation. Experiments on a well-sized EM-tracked ultrasound dataset demonstrated that the proposed DCL-Net has benefited from the contextual learning and showed superior performance when compared to other existing methods. Further experiments on the ultrasound videos with different scanning protocols will be studied in our future work.

\section{Acknowledgements}
\label{Acknowledgements}
This work was partially supported by National Institute of Biomedical Imaging and Bioengineering (NIBIB) of the National Institutes of Health (NIH) under awards R21EB028001 and R01EB027898, and through an NIH Bench-to-Bedside award made possible by the National Cancer Institute.

\bibliographystyle{splncs04}