\section{Introduction}
\label{sec:introduction}

Due to the popularization of smart phones and social media, video content have been rapidly growing in recent years. Therefore, automatic video understanding~\cite{carreira2017quo,feichtenhofer2016convolutional} is in high demand in both academia and industry. Temporal action detection \textbf{}(TAD)~\cite{CabaHeilbron2016,Escorcia2016DAPsDA,Gao_Shi_Wang_Li_Yuan_Ge_Zhou_2020,caba2015activitynet,xu2020boundary,xu2020g,zhao2019hacs} is a fundamental but challenging task in this area, which requires an intelligent system to predict the start and end time of all  activities/actions in a video, and to identify their categories as well.

Most of the current solutions to temporal action detection follow the two-step framework of \textit{feature encoding + action detection}. Concretely, a feature encoder first represents video snippets as a sequence of feature vectors, then an action detector operates on the feature vectors to produce action detections. Both steps are significantly resource-demanding.  Limited by  GPU memory and  processing ability, most existing works rely on pre-encoded video features obtained from an off-the-shelf action recognition model~\cite{carreira2017quo,lin2019tsm,wang2016temporal}, and only focus their computation on learning an action detector~\cite{Bai2020bcgnn,xu2020g,zhao2021video}.

By foregoing the complex computation in a  feature encoder, a standalone action detector itself is usually not resource-friendly, especially when designed with expensive modules in order to pursue good detection performance.
For example,  to aggregate long-range information, many recent TAD solutions incorporate graph convolutions~\cite{liu2020progressive,xu2020g,zhao2021video}, attentions~\cite{sridhar2021class,qin2021pcmnet}, or transformers~\cite{zhang2022actionformer}, which incur high computation and memory costs to correlate many or even all snippet pairs. Moreover, the action detector tends to exhaustively compute for all candidate actions (referred to as proposals), and consequently the computational complexity increases quadratically with the number of video snippets.

In addition to its efficiency issue, a separate action detector based on pre-encoded features can only obtain a sub-optimal solution, since the features are not optimized for the end TAD task and dataset, and may have lost the sensitivity to action boundaries~\cite{alwassel2020tsp,xu2021low}. Therefore, an ideal solution is to jointly optimize the feature encoding network \emph{and} the action detection network, \textit{i.e. } end-to-end training. However, this is extremely challenging due to the conflict between the large data volume of untrimmed videos on one hand and GPU limitations on the other. Current end-to-end TAD methods densely process all video snippets in the feature encoder~\cite{lin2021learning,liu2020progressive,xu2017r}, and as a result, have to compromise data resolution or batch size, which influences network performance.

In this paper,  in order to build a more efficient detector as well as to enable end-to-end training that can result in  better snippet features, we propose Detector Proposal Sampling (DPS) and Encoder Gradient Sampling (EGS). The idea behind the two sampling modules is to remove the redundant computation in both \textit{action proposal learning} and \textit{video snippet learning}. DPS  generates a much smaller but sufficient set  of proposals for the detector to further process. EGS reduces the number of effective snippets for gradient computation during network backpropagation, which greatly alleviates  the GPU memory and computation cost incurred by the encoder.

With DPS, we design an efficient action detection network, which we make effective by incorporating \textit{LSTM-boosted temporal aggregation} and \textit{cascade proposal refinement} to capture long-range context and progressively improve prediction  precision. With EGS, we are able to  efficiently optimize the video encoder and the action detector end-to-end.  As such, we present a novel unified framework for efficient end-to-end temporal action detection (ETAD), as shown in Fig. \ref{fig:intro}, which also achieves state-of-the-art performance. Concretely, compared with VSGN \cite{zhao2021video}, a state-of-the-art TAD method, Fig.~\ref{fig:intro}(b) shows that ETAD only requires 5\% of VSGN's memory usage, but it improves the average mAP from 35.94\% to 37.78\% on ActivityNet under the same configuration.

\vspace{6pt}\noindent \textbf{Contributions.} {\bf (1)} We propose \textbf{a novel, efficient, and effective temporal action detection network} with  Detector
Proposal Sampling (DPS) as well as LSTM-boosted temporal aggregation and cascaded proposal refinement. The network takes minimal GPU memory and is efficient in training.
{\bf (2)} We propose Encoder Gradient Sampling (EGS), which enables \textbf{an efficient end-to-end framework} to unify the learning of the feature encoder and action detector. Our framework is able to optimize the feature encoder and the action detector jointly on an average GPU memory budget.
{\bf (3)} Our extensive experiments show that both of our feature-based action detector and end-to-end framework reach \textbf{state-of-the-art performance} on two commonly used TAD benchmarks, ActivityNet-1.3 and THUMOS-14.

\section{Related Work}
\label{sec:related_work}

\subsection{Temporal Action Detection}
\label{related_tal}

An action detector can  localize action instances directly from features of video snippets (direct), or merely refine the action boundaries of proposals from a proposal-generation network (refinement). The \textit{direct} solution usually consists of feature enhancement and proposal evaluation~\cite{Bai2020bcgnn,chao2018rethinking,heilbron2017scc,lin2019bmn,Long2019GaussianTA,xu2017r,xu2020g,Yuan2017TemporalAL}, and focuses on improving one or both components. For example, G-TAD~\cite{xu2020g} models the correlations between video snippets for feature enhancement, and  BC-GNN~\cite{Bai2020bcgnn} models the correlations between proposals for proposal evaluation. The \textit{refinement} solution tends to prune off-the-shelf action proposals, which can be from heuristic designs (\textit{e.g. } sliding window), or a well-developed proposal generation method~\cite{buch2017sst,Escorcia2016DAPsDA,Gao2018CTAPCT,Heilbron2016FastTA,Liu2019MultiGranularityGF}, and provide more accurate boundary predictions \cite{lin2018bsn,qing2021temporal,shou2017cdc,Shou2016TemporalAL,Zeng2019GraphCN,zhao2021video,Zhao2017TemporalAD}. PGCN~\cite{Zeng2019GraphCN} is a typical example that  exploits  proposal-proposal relations from the predictions of BSN~\cite{lin2018bsn} and uses graph convolutional networks to re-rank BSN results. TCANet~\cite{qing2021temporal} uses the high-quality proposals generated from BMN~\cite{lin2019bmn} and proposes a cascade structure to progressively refine the actions. Our proposed ETAD belongs to the family of direct solutions, since it  does not rely on any external proposal methods, but interestingly, it is able to surpass the best refinement solution.

\subsection{End-to-end solutions in TAD}

Few methods study the problem of action detection directly from the original untrimmed video. R-C3D~\cite{xu2017r} encodes the frames with 3D filters, proposes action segments, then classifies/refines them. However, its performance is limited due to underlying computational restrictions (\textit{e.g. } low-resolution frames and low-capacity video networks). PBRNet~\cite{liu2020progressive} progressively refines action boundaries with three cascaded detection modules, but it is also plagued by hardware constraints that necessitate low-resolution frame input. ASFD~\cite{lin2021learning} proposes the first anchor-free temporal localization method, but it suffers from a low convergence rate and small batch size. To approximately overcome this issue, some works propose ways of pre-training the feature encoder. Those methods develop new training tasks to close the gap between action recognition and action detection. For example, LoFi~\cite{xu2021low} reduces the mini-batch composition in terms of temporal, spatial or spatio-temporal resolution. BSP~\cite{xu2020boundary} and TSP~\cite{alwassel2020tsp} train the feature encoder on novel tasks to make video features sensitive to the temporal localization of the action. Unfortunately, they all require an extra training stage for training the localization network with the pre-extracted snippet features.

Our ETAD framework is inspired by ideas from these end-to-end training implementations. We manage to train our network with high frame resolution, large batch size, fast convergence rate, and single-stage training, thus, setting a new SOTA performance and outperforming previous methods by a large margin.

\subsection{Sampling in Video Understanding}

Densely sampling snippets  over the entire sequence is effective in understanding short videos, but expensive for long, untrimmed video understanding. An alternative approach is to select only the relevant frames or snippets such as in video summarization~\cite{huang2019novel}. For example, SCSampler~\cite{korbar2019scsampler} selects salient clips from video for efficient action recognition, but it has to train another model for clip sampling. TQN~\cite{zhang2021temporal} designs a stochastically updated feature bank to store intermediate features for action recognition, but the intermediate features in the bank are not always up-to-date. In contrast, ETAD does not need an extra model to train the feature encoder, and all snippet features are computed on-the-fly.

Sampling proposals in TAD is an under-explored topics. Most methods such as~\cite{lin2019fast,lin2019bmn,xu2020g,zhao2021video} choose to exhaustively enumerate the possible locations of activity, leading to redundant computation especially for highly overlapped proposals. On the other hand, ETAD has a proposal sampling module to reduce computation and does not require extra steps to select proposals.

\section{Method}
\vspace{-0.1cm}
\label{sec:method}

\begin{figure}[t]
\begin{center}
\includegraphics[trim={2cm 0cm 3cm 2.5cm},width=12cm,clip]{figures/etad_method.pdf}
\end{center}
\vspace{-0.4cm}
\caption{
\textbf{Detailed Architecture of ETAD.}
(\textit{top}) The action detector of ETAD consists of a feature enhancement module, a boundary evaluation module, and a cascaded proposal evaluation module, along with an efficient DPS strategy. (\textit{bottom}) The EGS enables gradient backpropagation until the video snippets, so that the feature encoder can be trained end-to-end on the TAD task.}
\label{fig:framework}
\vspace{-0.3cm}
\end{figure}

In this section, we will introduce our  Efficient Temporal Action Detection framework (ETAD), consisting of jointly optimized  action detector and feature encoder (illustrated in Fig.~\ref{fig:framework} (bottom)). We propose  \textit{Detector Proposal Sampling} (DPS) to  enhance detection efficiency, and \textit{Encoder Gradient Sampling} (EGS) to enable efficient end-to-end training by alleviating computation redundancy.

\vspace{-0.2cm}
\subsection{TAD Formulation}

Given an untrimmed video, temporal action detection (TAD) aims to predict actions with temporal boundaries denoted as ${{\Psi }}=\left\{\varphi _i\!=\!(t_s,t_e,c)\right\}_{i=1}^N$, where $({t_s},{t_e},c)$ are the start time, end time, and category of the action instance ${{\varphi }_{i}}$, and $N$ is the total number of actions. The feature encoder in TAD is commonly implemented using an off-the-shelf action recognition model, such as  TSN~\cite{wang2016temporal}, I3D~\cite{carreira2017quo}, SlowFast~\cite{slowfast_iccv19}, and R(2+1)D~\cite{tran2018closer}. ETAD is agnostic of the encoder architecture and thus can incorporate any of the common encoders in its framework. We specifically take the feature map before the classification head of the encoder and use global average pooling on the temporal and spatial dimensions to obtain the feature vector $F$, which is input into the action detector.

\vspace{-0.2cm}
\subsection{Efficient Action Detector}

\subsubsection{Architecture.}
The architecture of our action detector is shown in Fig.~\ref{fig:framework} (top). First, \textit{feature enhancement module} processes features of all snippets and  enhances the representation by aggregating temporal information. In this module, we adopt LSTM-boosted temporal aggregation to leverage long-range temporal context. Following BMN \cite{lin2019bmn}, we evaluate the startness and endness of each snippet through two convolution layers in the \textit{boundary evaluation module}. Then, a small number of proposals determined by proposed sampling strategy will be further evaluated and
refined in the following module. Following G-TAD \cite{xu2020g}, we use RoI alignment  to extract the features of each proposal from the enhanced feature sequence. Finally, \textit{proposal evaluation module}  regresses the boundary offset between a proposal and its corresponding ground-truth action instance, as well as the IoU of each proposal by several fully-connected (FC) layers. To progressively improve the accuracy of the regression, we propose cascade proposal refinement, which comprises a sequence of proposal evaluation modules.

\vspace{-0.3cm}
\subsubsection{LSTM-boosted Temporal Aggregation.} Long-range relation modeling is vital for temporal action detection~\cite{buch2017sst,gao2017cascaded,khojasteh2021temporal,qing2021temporal}, considering that local aggregation alone such as convolution is not sufficient when an action is longer than the receptive field or correlations between different action instances are  critical for prediction \cite{xu2020g}. Although transformers or graph convolutions can also capture temporal context, they usually need more data to converge and strong regularization to optimize. In our work, to enrich the feature representation with global and local information, we adopt two LSTM \cite{hochreiter1997long} layers prior to the 1D convolutional network, which aggregate the temporal context in the forward and backward directions respectively, as shown in Fig. \ref{fig:framework} (top). Each LSTM learns the informative context from the past feature or future feature independently. We concatenate the output of the forward and backward LSTM as well as a residual connection, and then use a convolutional layer with kernel size 1 to reduce the output channels. Compared with only local aggregation, our LSTM-based temporal aggregation effectively enhances snippet feature representations in reference to their order, which  provides smoother but more distinctive features

\vspace{-0.3cm}
\subsubsection{Cascade Proposal Refinement.} Refining proposals is essential to achieving higher performance as mentioned in Sec.~\ref{related_tal}. To this end, we propose the cascade strategy that stacks multiple stages of the proposal evaluation module to progressively refine proposal boundaries. In each stage, we use several FC layers to regress the start/end offsets, as well as the center/duration offset of each proposal. We also use an additional branch of FC layers to regress the intersection over union (IoU) (used to rank proposals) and another branch to classify the action startness and endness for regularization. For the cascade design, we sequentially run three stages, where the  proposals generated by the first stage are further refined in the second stage and so forth. We use progressively increasing IoU thresholds for the three stages, namely 0.7, 0.8, and 0.9. As such, the proposal boundaries are expected to become more accurate after each stage. Three cascade proposal evaluation modules can not only refine the action proposal boundaries for proposal localization, but they also provide  a more reliable IoU score for better proposal ranking. It is worth mentioning that our cascade proposal refinement is different from the multi-stage strategy in TCANet~\cite{qing2021temporal}, which relies on an additional  proposal generation network.

\subsubsection{Detector Proposal Sampling (DPS).}
\label{sec:DPS}

\begin{figure}[t]
\begin{center}
\includegraphics[trim={2cm 5.6cm 2cm 5.5cm}, width=11cm, clip]{figures/etad_strategy.pdf}
\end{center}
\vspace{-0.7cm}
\caption{\textbf{Three sampling strategies in DPS and EGS: \textit{random}, \textit{grid}, and \textit{block}.} \textit{Random} strategy samples certain proposals/snippets from a uniform distribution, \textit{grid} strategy  samples with a fixed temporal stride, and \textit{block} strategy only samples a consecutive area in the proposal map/snippet sequence.}
\label{fig:sampling_strategy}
\vspace{-0.7cm}
\end{figure}

With the aforementioned techniques, our detector can produce high-quality action predictions. However, such a high-performance detector usually comes at a substantial computational cost, especially when using the cascade network structure. If end-to-end training is adopted, an efficient action detector is more suitable for general action detection. Based on such insight, it is natural to raise the following question. \textbf{\textit{Is there any redundancy in the proposal learning?}}

To answer this question, we first analyze the main source of the computation complexity of an action detector -- proposal extraction (\textit{e.g. } RoI Alignment) and proposal evaluation (\textit{e.g. } FC layers). If we enumerate all possible combinations of start and end locations for all snippets as in G-TAD \cite{xu2020g}, the complexity of both modules is quadratic in the number of snippets. Specifically, given the number of snippets $T$, there will be $C_{T}^{2} = T \cdot (T-1) / 2 $ proposals, and over \textbf{95\%} of the total network memory is spent on extracting and evaluating these proposals. When cascade refinement with more stages is used, this complexity will be further increased. However, due to the dense enumeration, most of these proposals overlap with each other, and as a result, a large portion of the extracted proposal features are duplicated. Moreover, the proposal evaluation module also refines each proposal's start and end, making it unnecessary to consider proposals that are temporally close. Therefore, the answer is that there is huge redundant computation in the proposal extraction and evaluation modules.

In order to reduce the redundancy while preserving performance, we propose to replace the densely sampled proposal set  with a subset produced by an efficient sampling method, called \textbf{Detector Proposal Sampling (DPS)} as illustrated in Fig. \ref{fig:framework}. This is based on the empirical observation that detector's performance saturates at a low proposal sampling ratio. In practice, to sample a subset from all the $C_{T}^{2}$ proposals, we consider three sampling strategies: \textit{random}, \textit{grid}, and \textit{block}, as shown in Fig. \ref{fig:sampling_strategy}. Given a sampling ratio, the \textit{random} strategy simply samples  proposals randomly following a uniform sampling distribution. The \textit{grid} sampling strategy samples a proposal based on a pre-defined grid on a proposal map.
The \textit{block} sampling strategy only samples the proposals within a block in the proposal map, which  essentially evaluates proposals in a  trimmed segment of a video. We adopt random sampling strategy with ratio 6\% in our action detector due to its simplicity and performance (see Sec. \ref{sec:proposal_sampling_ratio} for a detailed study).

\subsection{Efficient End-to-End training}

\subsubsection{Joint Optimization.}
Another long-standing problem in untrimmed video understanding is the difficulty of end-to-end training. In TAD, common practice dictates the use of an off-the-shelf feature encoder pretrained on a classification task to extract informative video snippet features, and a high-quality detector on top of these features. This paradigm will inevitably lead to a task-discrepancy problem, since the optimization target is not consistent between the classification and detection tasks \cite{alwassel2020tsp,xu2020boundary}. Ideally, to avoid such issues and pursue higher detection performance, end-to-end training, which jointly optimizes the feature encoder and action detector together, is the standard approach. However, this is severely restricted by the conflict between the large data volume of untrimmed videos and the memory constraint of commodity GPUs. To be specific, massive resources are needed for the feature encoder. For example, when considering a large snippet length (\textit{e.g. } 128 snippests), a high frame resolution (\textit{e.g. } $224\times 224$), and a large batch size (\textit{e.g. } 16 videos),  a common encoder (\textit{e.g. } TSM \cite{lin2019tsm}), it requires more than $500$ GB memory to perform end-to-end TAD training, making it unrealistic on most platforms. Some other works also try to deal with end-to-end learning, but must sacrifice either data/network scales \cite{liu2020progressive} or training efficiency \cite{lin2021learning}. In short, the large data consumption makes  end-to-end TAD training impractical in untrimmed video understanding.

\vspace{-0.3cm}
\subsubsection{Encoder Gradient Sampling (EGS).}
In order to achieve the joint optimization of the feature encoder and action detector, we target approaches that avoid sacrificing the data or network scale. Inspired by observed redundancy in detector proposal learning, it is reasonable to suspect that there also exists redundancy in the encoder snippet learning. In fact, the input of the encoder in traditional end-to-end training is multiple video snippets, and the optimizer updates the encoder parameters based on all snippet gradients in each iteration. Instead of reducing the input data or network scale, we can reduce the gradient computation by snippet-level sampling, called \textbf{Encoder Gradient Sampling}, as illustrated in Fig. \ref{fig:framework} (bottom). Formally, given a sampling ratio, we only compute the encoder's gradients on the sampled snippets, while snippets that are not sampled only contribute to the forward propagation of the encoder. This snippet-level gradient-based sampling  saves a large portion of the GPU memory, since gradients do not need to be computed and intermediate activations do not need to be saved for unsampled snippets. It also allows for higher resolution video features, which are subsequently processed by the detector module, since we can exploit  video frames with higher resolution and a denser frame rate.

We also consider different strategies for EGS as shown in Fig. \ref{fig:sampling_strategy}, namely \textit{random}, \textit{grid}, and \textit{block}. \textit{Random} samples snippets randomly to compute the encoder's gradients. \textit{Grid} has the same temporal stride between each sampled snippet. \textit{Block} makes sure consecutive snippets are sampled. The different sampling strategies and ratios are well-studied in Sec. \ref{sec:snippet_sampling_ratio}, and we find that even a 10\% snippet sampling ratio can reach a detection performance that is on-par with using all the snippets, while saving 83\% of the memory. As a default setting, we use the random sampling strategy with ratio 10\% in our action detector.

To implement encoder gradient sampling, we first split (through random sampling) the video snippets into two sets: with gradient snippets $X_a$ and without gradient snippets $X_b$. Then, we feed $X_b$ to the feature encoder without saving gradients to produce the corresponding features $F_b$. For $X_a$, we feed them into the encoder to produce features $F_a$, while preserving the encoder's gradients when backpropagating. We then concatenate $F_a$ and $F_b$ together and arrange them by their original order as feature sequence $F$, which is sent to the efficient action detector. During backpropagation, the encoder parameters are only updated by the gradient given by $X_a$. Also, we did not observe that such sampling harms the stability of the network training. More detailed implementation can be found in the \textbf{supplementary material}.

\vspace{-0.3cm}
\subsubsection{Loss function.} For our end-to-end training, we directly compute our loss function based on the original video frames $V$ (in contrast to snippet features) as well as ground-truth labels $\Psi$, formulated as $\mathcal{L}(V, \Psi)$.  The loss function is composed of losses from multiple tasks, including the ones in the cascade proposal refinement module, as well as, the boundary evaluation in the feature enhancement module. $\mathcal{L}$ is computed as follows:
\begin{equation}
    \mathcal{L} =
    \lambda_a\mathcal{L}_{ce:bd_s} +
     \sum_{i=1,2,3} \left(
    \lambda_b\mathcal{L}_{ce:bd_p}^i + \lambda_c\mathcal{L}_{iou}^i + \lambda_d\mathcal{L}_{rg:secw}^i \right),
\end{equation}
where $\lambda_{a},\lambda_{b},\lambda_{c},\lambda_{d}$ are the weights for the corresponding losses.
$\mathcal{L}_{ce:bd_s}$ in the boundary evaluation module uses batch-balanced cross entropy to supervise the startness and endness of each snippet. The other losses originate from the cascade proposal refinement, which contains three stages of proposal evaluation modules. Each of these stages has the following three losses: $\mathcal{L}_{ce:bd_p}$ computes the cross entropy for the startness and endness of proposal's boundaries;  $\mathcal{L}_{iou}$ contains a classification loss and a regression loss for the IoU between predicted and ground-truth actions; $\mathcal{L}_{rg:secw}$  uses smooth-L1 loss for regressing start/end offset and center/width offset. We follow \cite{lin2019bmn} for the detailed expression and the ground-truth for each loss. More details can be found in the \textbf{supplementary material}.
\section{Experiments}
\label{sec:experiments}

\subsection{Datasets and evaluation metrics}
\subsubsection{Datasets.}
ActivityNet-1.3~\cite{caba2015activitynet} is a large-scale video understanding dataset, consisting of 19,994 videos annotated for the temporal action detection task. The dataset is divided into training, validation and testing sets with a ratio of 2:1:1. THUMOS-14~\cite{jiang2014thumos} contains 200 annotated untrimmed videos in the validation set and 213 annotated untrimmed videos in the testing set. Performing action detection on this dataset is challenging because over 70\% of the untrimmed video is background and each video has more than 15 action instances on average.

\vspace{-0.2cm}
\subsubsection{Evaluation Metrics.} Following previous works, mean Average Precision (mAP) at certain IoU thresholds and average mAP are reported as the main evaluation metrics. On ActivityNet-1.3,  the IoU thresholds are chosen from 0.5 to 0.95 with 10 steps. On THUMOS-14, the threshold is chosen from \{0.3,0.4,0.5,0.6,0.7\}.

\subsection{Implementation details}

\subsubsection{Feature preparation for the standalone detector.} Here, only pre-extracted snippet features are used, \textit{i.e. } no end-to-end-training is performed. For ActivityNet-1.3, we adopted three types of pre-extracted features: TSN (similar to \cite{lin2019bmn,xu2020g,qing2021temporal}), TSM \cite{lin2019tsm}, and  TSP \cite{alwassel2020tsp}. For THUMOS-14, we adopted the TSN features provided by \cite{xu2020g} and I3D features provided by \cite{zhao2020bottom} with a frame rate of 7.5. In ActivityNet, we resize the feature sequences to a fixed length of 128 snippets. For THUMOS-14, we utilize the sliding window approach  with window length 128 and stride 64 to generate one training sample from each video sequence.

\subsubsection{Feature encoding for end-to-end training.}  We choose TSM \cite{lin2019tsm} and R(2+1)D \cite{tran2018closer} as our feature encoder for the end-to-end training. We fix the weights of the first two stacked layers of the backbone network and freeze all batch normalization layers of the feature encoder. For the feature encoders of TSM and R(2+1)D, the image resolution is set to 224 and 112, and clip length is set to 8 and 16, respectively. We only adopt random cropping as data augmentation. Note that the TSM model is only pretrained on Kinetics-400 \cite{kay2017kinetics} and not finetuned on the target datasets, \textit{i.e. } . ActivityNet-1.3, HACS, or THUMOS-14.

\subsubsection{Training and inference.} We implement our framework with the software environment of PyTorch1.10, CUDA11.2, and mmaction2 \cite{2020mmaction2}. The feature-based experiments are conducted on one RTX3090 and we report the training time and GPU memory. The end-to-end experiments with EGS are conducted on 4 V100 GPUs. In training, we use a batch size of 16 and the AdamW optimizer~\cite{loshchilov2017decoupled} with weight decay of $10^{-4}$ in all experiments. The learning rate for the detector is set to $10^{-3}$ on ActivityNet-1.3 and THUMOS-14. In end-to-end training, the learning rate for the encoder is set to $10^{-6}$. The total training epoch is set to 6 and the learning rate decays by 0.1 after 5 epochs. $\lambda_{a},\lambda_{b},\lambda_{c}$ in the loss function are set to 1, while $\lambda_{d}$ is set to 10. For inference, to post-process network outputs, we use the boundary selecting method in \cite{lin2019bmn} to select proposals with high startness and endness, and use the refined boundaries for each proposal generated from the last cascade stage of the detector as the predictions. Note that the proposal sampling ratio is set as 100\% in testing. We rank all predictions for NMS processing by their confidence scores computed as $p=p_s \cdot p_e \cdot p_{iou}$, where $p_s$ and $p_e$ are from $\mathcal{L}_{ce:bd_s}$ standing for the start and end probabilities of a proposal respectively, and $p_{iou}$ is the score of the proposal from $\mathcal{L}_{iou}$. Following \cite{lin2019bmn,qing2021temporal}, we apply the video-level classification scores from \cite{zhao2017cuhk} and \cite{wang2017untrimmednets} on ActivityNet-1.3 and THUMOS-14. For completeness, we also test our methods on the HACS dataset~\cite{zhao2019hacs} and achieve state-of-the-art performance. More details can be found in the \textbf{supplementary material}.

\subsection{Comparison with State-of-the-Art TAD Methods}

\begin{table}[t]
\begin{minipage}{0.5\linewidth}
\captionsetup{font=scriptsize}
\scriptsize
\centering
\caption{Comparison in terms of mAP on ActivityNet 1.3 (val) \textbf{with pre-extracted features}.}
\begin{tabular}{c|c|c|ccc|>{\columncolor[gray]{0.9}}c}
	\toprule
	Method          & Feat.   & O.F.       & 0.5   & 0.75  & 0.95  & Avg.  \\
	\midrule
    BMN             & TSN     & \checkmark & 50.07 & 34.78 & 8.29  & 33.85 \\
    G-TAD           & TSN     & \checkmark & 50.36 & 34.60 & 9.02  & 34.09 \\
    VSGN            & TSN     & \checkmark & 52.38 & 36.01 & 8.37  & 35.07 \\

    TCANet          & TSN     & \checkmark & 52.27 & 36.73 & 6.86  & 35.52 \\
    \textbf{ETAD}            & TSN     & \checkmark & \textbf{52.49} & \textbf{37.07} & \textbf{9.47}  & \textbf{36.06} \\
	\midrule
    G-TAD           & TSP     & $\times$   & 51.26 & 37.12 & 9.29  & 35.81 \\
    CSA             & TSP     & $\times$   & 52.64 & 37.75 & 7.94  & 36.25 \\
    \textbf{ETAD}            & TSP     & $\times$   & \textbf{54.26} & \textbf{38.94} & \textbf{10.43} & \textbf{37.78} \\
    \bottomrule
    \end{tabular}
\label{tab:sota_anet}
\end{minipage}
\begin{minipage}{0.5\linewidth}
\captionsetup{font=scriptsize}
\scriptsize
\centering
\caption{Comparison in terms of mAP on test set of THUMOS-14 \textbf{with pre-extracted features}.}
\begin{tabular}{c|c|cc>{\columncolor[gray]{0.9}}ccc}
	\toprule
	Method          & Feat. & 0.3  & 0.4  & 0.5  & 0.6  & 0.7  \\
	\midrule
    BMN             & TSN     & 56.0 & 47.4 & 38.8 & 29.7 & 20.5    \\
    G-TAD           & TSN     & 57.3 & 51.3 & 43.0 & 32.6 & 22.8    \\
    TCANet          & TSN     & 60.6 & 53.2 & 44.6 & 36.8 & 26.7    \\

    VSGN            & TSN     & 66.7 & 60.4 & 52.4 & 41.0 & 30.4    \\
    \textbf{ETAD}            & TSN     & \textbf{68.26} & \textbf{63.63} & \textbf{55.65} & \textbf{45.83} & \textbf{35.13} \\
	\midrule

    AFSD            & I3D     & 67.3 & 62.4 & 55.5 & 43.7 & 31.1    \\
    MUSES           & I3D     & 68.9 & 64.0 & 56.9 & 46.3 & 31.0    \\
    \textbf{ETAD}            & I3D     & \textbf{70.29} & \textbf{65.26} & \textbf{57.81} & \textbf{47.88} & \textbf{36.25} \\
    \bottomrule
\end{tabular}
\label{tab:sota_thu}
\end{minipage}
\end{table}

\vspace{-0.3cm}
\noindent \textbf{Efficient Standalone Action Detector.}
We compare our action detection network with state-of-the-art detectors on ActivityNet-1.3. The results are the average after training 10 times. As shown in Tab.~\ref{tab:sota_anet}, our method reports the highest average mAP on this large-scale dataset with pre-extracted features. Notably, without expensive optical flow (O. F.), our localization network can reach an average mAP of 37.78\%, significantly outperforming all  other  methods. We also show the advantage of our method on THUMOS-14 in Tab.~\ref{tab:sota_thu}, which surpasses both TSN-based VSGN~\cite{zhao2021video} and I3D-based AFSD~\cite{lin2021learning} and MUSES~\cite{liu2021multi}. Interestingly, ETAD achieves remarkable results especially on high IoU threshold on two datasets, indicating the precision of the generated action boundary.

\begin{table}[t]
    \vspace{-0.6cm}
    \centering
    \captionsetup{font=scriptsize}
	\scriptsize
	\caption{Comparison in terms of mAP on ActivityNet-1.3 validation set \textbf{with end-to-end training}. Sampling ratio means the EGS ratio. Total GPU memory are reported under the batch size 16.}
	\begin{threeparttable}
	\begin{tabular}{p{1.2cm}<{\centering}p{1.4cm}<{\centering}p{1.1cm}<{\centering}p{1.4cm}<{\centering}p{1.9cm}<{\centering}p{0.9cm}<{\centering}p{0.8cm}<{\centering}p{0.9cm}<{\centering}>{\columncolor[gray]{0.9}}p{1.7cm}}
	\toprule
	Method & \makecell{Encoder\\Backbone}     & \makecell{End To\\ End} & \makecell{Sampling\\Ratio}   & \makecell{GPU\\Memory(GB)}        & 0.5   & 0.75  & 0.95  & Avg. mAP  \\
	\toprule
	AFSD \tnote{$\dag$}   & I3D & \checkmark & N.A. & N.A. & 52.40 & 35.30 & 6.50 & 34.40 \\
    LoFi   & TSM & \checkmark & N.A. & 4$\times$32 \tnote{$\ddag$} & 50.91 & 35.86 & 8.79 & 34.96 \\
    PBRNet \tnote{$\dag$} & I3D & \checkmark & N.A. & N.A. & 53.96 & 34.97 & 8.98 & 35.01 \\
	\midrule
	\multirow{4}{*}{\textbf{ETAD}}  & \multirow{4}{*}{\makecell{TSM- \\ResNet50}}
	       & $\times$   &    N.A.    & 1.23        & 53.01 & 36.78 & 9.77  & 35.98  \\
	&      & \checkmark & 100\%      & 8$\times$68  & 53.88 & 37.73 & 10.65 & \textbf{36.91}\scriptsize{($\uparrow$0.93)} \\
	&      & \checkmark & 30\%       & 4$\times$47\scriptsize{($\downarrow$65\%)}  & 53.79 & 37.59 & 10.56 & \textbf{36.79}\scriptsize{($\uparrow$0.81)} \\
	&      & \checkmark & 10\%       & 4$\times$23\scriptsize{($\downarrow$83\%)}  & 53.85 & 37.60 & 10.48 & \textbf{36.75}\scriptsize{($\uparrow$0.77)}\\
	\midrule
	\multirow{3}{*}{\textbf{ETAD}}  & \multirow{3}{*}{\makecell{TSP-\\R(2+1)D34}}
	       & $\times$   &    N.A.    & 1.23         & 54.26 & 38.94 & 10.43 & 37.78 \\
	&      & \checkmark & 100\%      & 8$\times$31  & 55.34 & 39.21 & 10.56 & \textbf{38.27}\scriptsize{($\uparrow$0.49)} \\
	&      & \checkmark & 30\%       & 4$\times$22\scriptsize{($\downarrow$65\%)}  & 55.31 & 39.04 & 10.57 & \textbf{38.25}\scriptsize{($\uparrow$0.46)} \\
	&      & \checkmark & 10\%       & 4$\times$18\scriptsize{($\downarrow$70\%)}  & 55.27 & 39.06 & 10.56 & \textbf{38.21}\scriptsize{($\uparrow$0.43)} \\
    \bottomrule
    \end{tabular}
\begin{tablenotes}
\scriptsize
\item[$\dag$] Additional optical flow is used for late fusion.
\item[$\ddag$] 4$\times$32 means that 4 V100 are used and each GPU takes 32GB memory.
\end{tablenotes}
\end{threeparttable}
\vspace{-0.7cm}
\label{tab:sota_anet_e2e}
\end{table}

\vspace{2pt}\noindent \textbf{Efficient End-To-End Framework.}
Tab.~\ref{tab:sota_anet_e2e} compares our method with other detection methods that adopt end-to-end training with different encoder backbones. Firstly, joint optimization of the feature encoder and action detector can bring a large performance gain but requires more than $500$GB of GPU memory\footnote{Following \cite{2020mmaction2}, we use \texttt{torch.cuda.max\_memory\_allocated()} to compute the GPU memory.} on the GPU devices, which is usually not feasible for most platforms. However, enabling encoder gradient sampling (EGS) with sampling ratio 10\% can reduce the memory usage by up to $83\%$, while still maintaining the similar detection performance. Particularly, using our EGS method with the R(2+1)D encoder can achieve an even higher performance of 38.21\% average mAP with only 4.5GB memory per video, thus outperforming previous state-of-the-art methods.

\subsection{Ablation Study on Action Detector}

\begin{table}[t]
	\centering
	\scriptsize
	\caption{\textbf{Ablations on the architecture} of proposed action detector on ActivityNet with TSN features.}
	\begin{subtable}[t]{0.51\linewidth}
        \begin{tabular}{p{1.7cm}<{\centering}p{1.3cm}<{\centering}p{1.2cm}<{\centering}>{\columncolor[gray]{0.9}}p{1.5cm}<{\centering}}
            \toprule
            \makecell{Feature \\ Module} & Time & Memory & Avg. mAP \\
            \midrule
            Conv          & 2min 35s & 0.47GB & 34.80 \\
            Transformer   & 3min 05s & 0.68GB & 34.82 \\
            LSTM          & 3min 23s & 0.53GB & \textbf{35.25} \\
            \bottomrule
        \end{tabular}
        \caption{Study of different feature modules.}
    \end{subtable}
    \begin{subtable}[t]{0.48\linewidth}
        \begin{tabular}{p{1.4cm}<{\centering}p{1.3cm}<{\centering}p{1.2cm}<{\centering}>{\columncolor[gray]{0.9}}p{1.5cm}<{\centering}}
            \toprule
            \makecell{Cascade \\ Stages} & Time & Memory & Avg. mAP \\
            \midrule
            1  & 3min 23s & 0.53GB  & 35.25 \\
            2  & 4min 33s & 0.88GB  & 35.81 \\
            3  & 5min 58s & 1.23GB  & \textbf{36.06} \\
            \bottomrule
        \end{tabular}
        \caption{Study of different cascade stages.}
    \end{subtable}
	\label{tab:ablation_architecture}
\vspace{-0.9cm}
\end{table}

\noindent {\bf{LSTM-boosted Temporal Aggregation.}}
We study the effectiveness of our  feature enhancement module with LSTM-boosted temporal aggregation by comparing it to a convolutional network (Conv) and a vanilla Transformer. The cascade refinement is not used in this ablation. In Tab. \ref{tab:ablation_architecture} (a), Conv shows the lowest run-time and memory usage, but also the lowest performance due to its inability to capture long-range context. Vanilla Transformer takes the most resources without bringing mAP gains. Comparatively, our LSTM-boosted method requires a moderate amount of memory, while achieving high performance without the requirements of more data to converge and strong regularization to optimize as with Transformers.

\vspace{2pt}\noindent {\bf{Cascade Proposal Refinement.}} We also study the impact of the number of stages used for the cascade proposal refinement. With a single proposal evaluation stage, the average mAP is  35.25\%, and it  consistently improves as we use  more stages. Considering efficiency as well, we choose to use three stages in total and achieve an average mAP of 36.06\%.

\vspace{2pt}\noindent {\bf{Detector Proposal Sampling Ratio}}.
\label{sec:proposal_sampling_ratio}
One important study is the impact of different DPS ratios on the action detection performance. As shown in Fig.~\ref{fig:proposal_sample}, we manage to reduce $94\%$ of the proposals with a minimal drop in detection performance. In terms of efficiency, the training time drops from 45 mins to 6 mins, while 92\% of memory is saved. This is due to the proposal redundancy and a small number of the proposals is sufficient for discriminating  positive and negative actions. In fact, the average mAP will drop sharply only if the sampling ratio is less than 2\%. These results show that DPS is a simple and effective way to alleviate the proposal redundancy problem.

\vspace{2pt}\noindent {\bf{Detector Proposal Sampling  Strategy}}. We test different proposal sampling strategies on ActivityNet, as shown in Tab.~\ref{tab:strategy}. The random sampling strategy and grid sampling strategy both have high detection performance, since these sampling methods capture all scales of candidate proposals and preserve the original proposal distribution. Unsurprisingly, block-wise sampling shows quite low performance. This is mainly because such sampling changes the proposal distribution and cannot cover proposals with different scales. In practice, we use random sampling for simplicity.

\begin{figure}[t]
    \centering
    \includegraphics[trim={1.2cm 0cm 1cm 1cm},width=12cm,clip]{figures/proposal_sample_ratio.pdf}
    \vspace{-0.2cm}
    \caption{\textbf{Effect of sampling ratio in Detector Proposal Sampling.} We apply the random sampling strategy in DPS and report the model performance, GPU memory, and training time. Note that the x-axis is not uniformly distributed.}
    \label{fig:proposal_sample}
    \vspace{-0.5cm}
\end{figure}

\subsection{Ablation Study on End-to-End Training}

\label{sec:snippet_sampling_ratio}
\vspace{4pt}\noindent {\bf{Encoder Gradient Sampling Ratio.}} Here, we study the impact of the EGS ratio on detection performance as shown in Fig. \ref{fig:snippet_sampling}. The sampling ratio of 0 means that there is no gradient backpropagating to the feature encoder, which essentially is the same as detection using pre-extracted features. Particularly, ratio 0* means only adopting data augmentation on video snippets with frozen video encoder, which can increase the mAP from 35.98\% to 36.33\%. When the ratio is 100\%, the network is trained in the completely end-to-end way, with 0.93 mAP gains but tremendous compute cost. However, the performance also saturates if the ratio is larger than 10\%. This result also shows that certain redundancy exists in dense snippet learning. In our experiment, we observe that a 10\% EGS ratio is a good trade-off, which still provides 0.77 mAP gains but reduces GPU memory usage by more than 83\%. This reduction makes end-to-end training of long-form video understanding practical on more platforms.

\vspace{4pt}\noindent {\bf{Encoder Gradient Sampling Strategy}}. As shown in Tab. \ref{tab:strategy}, the grid and random sampling strategies in EGS achieve better detection mAP than the block-wise strategy. This is a similar conclusion to that of  DPS. All three sampling methods outperform the baseline of 35.98\% with pre-extracted features, which also proves the effectiveness of end-to-end training. Moreover, we find that the loss value in grid sampling is slightly better than that of random sampling. We expect that with more training epochs and larger sampling ratios, the gap between these two methods would be smaller.

\begin{figure}[t]
	\begin{minipage}[m]{0.56\linewidth}
    \centering
    \includegraphics[trim={0.5cm 0cm 0.5cm 0.5cm},width=7.2cm,clip]{figures/snippet_sample_ratio.pdf}
	\vspace{-0.8cm}
    \captionof{figure}{\textbf{Effect of sampling ratio in Encoder Gradient Sampling.} We apply random sampling with TSM-R50 and report the mAP and GPU memory on ActivityNet.}
    \label{fig:snippet_sampling}
	\end{minipage}\hfill
	\begin{minipage}[m]{0.4\linewidth}
    \centering
	\vspace{0.3cm}
	\begin{tabular}{p{1.5cm}<{\centering}|p{1.2cm}<{\centering}|p{1.2cm}<{\centering}}
	\toprule
	\makecell{Sampling \\ strategy} & \makecell{DPS \\ (6\%)} & \makecell{EGS \\ (10\%)} \\
	\midrule
    block       & 33.42 & 36.54 \\
    grid        & \textbf{36.11} & \textbf{36.80} \\
    random      & 35.98 & 36.75 \\
    \bottomrule
	\end{tabular}
	\vspace{0.6cm}
    \captionof{table}{\textbf{Effect of sampling strategies in DPS and EGS}. We adopt the TSM-R50 and report the mAP on ActivityNet.}
	\label{tab:strategy}
	\end{minipage}
\end{figure}

\vspace{4pt}\noindent {\bf{Discussions about Different End-To-End Strategies}}. To verify the effectiveness of our EGS method, we also compare with other \textit{nearly end-to-end} approaches. This refers to methods that use certain techniques to reduce the computation to approximate the real end-to-end training. To meet the requirements of memory, LoFi \cite{xu2020g} introduces three stages for encoder pretraining including a T-LoFi stage, which downsamples the temporal resolution from 128 to 32, then extracts features and detects actions in full-fidelity. We adopt T-LoFi in our detector and find that the improvement is not obvious (refer to Tab. \ref{tab:ablation_e2e}). Inspired by TQN \cite{zhang2021temporal}, we design a similar feature bank to store snippet features as cache, and stochastically update only a small proportion of the snippet features. Such a design also has low computation cost, but the mAP is still not promising. Our simple but effective EGS achieves a good trade-off between memory and time while visibly improving the detection performance. Other ablations regarding frame resolution, pretraining, and frozen backbones are detailed in the \textbf{supplementary material}.

\begin{table}[t]
\vspace{-0.5cm}
	\centering
	\caption{Comparison with other end-to-end TAD strategies. Total GPU memory are reported under the batch size 16.}
	\small
	\begin{tabular}{p{1.8cm}<{\centering}p{2.8cm}<{\centering}p{2.9cm}<{\centering}p{1.5cm}<{\centering}p{2.3cm}<{\centering}}
	\toprule
	Detector    & \makecell{End-To-End \\ strategy}    & GPU Memory & Time  & \makecell{avg. \\mAP} \\
	\midrule
    \multirow{4}{*}{ETAD}        &  $\times$       & 4$\times$12GB      & 0.1h    & 35.98 \\
            &  T-LoFi \cite{xu2020g}               & 4$\times$32GB      & 3h      & 36.36 \\
            &  TQN \cite{zhang2021temporal}        & 4$\times$23GB      & 6h    & 36.54 \\
            &  \textbf{EGS (ours)}                 & 4$\times$23GB      & 10h     & \textbf{36.75} \\
    \bottomrule
	\end{tabular}
	\label{tab:ablation_e2e}
\vspace{-0.7cm}
\end{table}

\section{Conclusion}
In this paper, we systematically studied the efficiency in temporal action detection.
We identify that computation redundancy is a major bottleneck that hinders efficient training of an action detector. Therefore, we propose detector proposal sampling (DPS) and encoder gradient sampling (EGS), which reduce resource requirements for the detector and the feature encoder training, respectively.  Using both sampling strategies, the total computation can be reduced by up to 83\%.
In addition, we design an action detector with LSTM-boosted temporal aggregation and cascaded boundary regression to achieve more effective detections.  Our proposed methods enables  a unified framework for efficient  joint optimization of the feature encoder and action detector on an affordable GPU budget. This  makes end-to-end training tractable in real-world applications. It will encourage the community to carry out more research on end-to-end training  for various untrimmed video understanding tasks besides TAD, such as video language grounding, active speaker detection, and video captioning.
\label{sec:conclusion}