\section{Introduction}
As Deep Neural Networks (DNNs) have shown a remarkable performance in various fields, many studies have attempted to resolve the basis of network predictions. However, it still lacks clear transparency about the myriad of components and the complex inner structure of DNNs. The problem of attribution, also called relevance, seeks the most relevant factors with respect to the predictions of DNNs and characterizes them as a supporting basis for the decision.
\begin{figure}
  \centering
  \includegraphics[width=1\linewidth]{Figure/Fig1.png}
    \caption{Relative Sectional Propagation (RSP) aims to fully decompose the network predictions with taking advantages of i) strong objectness, ii) class-discriminativeness, and iii) detailed descriptions of neuron activations.}
  \label{fig:intro}
\end{figure}
Grad-CAM \cite{selvaraju2017grad} is the most popular and widely used method in the field of weakly supervised segmentation and detection due to the easily applicable property and its high performance for localizing the primary objects. Despite these advantages, there is a limitation that the feature extraction stage of DNNs cannot be decomposed \cite{rebuffi2020there}. Also, detailed information about the neuron activation is lost due to the interpolation of the class activation map.
To fully interpret the network, many studies based on the modified backpropagation algorithm ~\cite{bach2015pixel,kindermans2017patternnet, montavon2017explaining, zhang2018top, nam2019relative} attempted to identify the significant parts in the input image with each of their own perspectives by decomposing the output predictions in a backward manner. By visualizing attributions as saliency maps, the important objects are highlighted as the basis for the predictions. Despite many studies on such methods, it is challenging to address the class-discriminativeness, incorrect distribution of attributions that are derived in unrelated objects, and model sensitivity. Furthermore, the different mechanisms among various types of DNNs and the variations of feature attention according to the layer depth make the interpretation of the network more difficult.

The class discriminative issue was addressed in recent papers, and the contrastive perspective was presented as a countermeasure \cite{zhang2018top, gu2018understanding}. The idea of contrasting is erasing the duplicated attributions among classes by backpropagating twice from the target and all other classes. By efficiently removing the duplicate relevance of the activated neurons, it is possible to obtain the attributions of the target class within the object area. However, unreasonable positive or negative attributions in irrelevant regions, such as background and watermark, are easily found in the results of this concept.

In this paper, we propose an attribution method, Relative Sectional Propagation (RSP), by analyzing the relative gradient activation maps between the target and hostile classes and propagating corresponding relevance according to the sectional influence of individual neurons. We carefully investigate the reasons for the non-suppressed characteristics of neuron activations according to the different classes and address these issues by assigning the bi-polar relevance scores: from highly related with target to highly relevant to hostile classes. Inspired by the traits of \textbf{winner always wins} ~\cite{zhang2018top} among activated neurons, the main idea is to separately compute the relative gradient activation maps, which contain the neuron importance, and purge the conflicting units to the channel attributions along the channel-axis, thereby preventing the decrement of the gap between bi-polar relevance scores. Our method preserves the conservation rule ~\cite{bach2015pixel} to prevent degeneracy problem, and allocates the relevance scores aligned with contributions.

Fig.~\ref{fig:intro} illustrates the samples that summarize the advantages of RSP. The attributions are fully decomposed from the output to input with the characteristics of i) the detailed visualizations of neuron activations, ii) strong objectness to output predictions, and iii) discriminativeness among classes with bi-polar relevance scores (positive and negative). As these characteristics are the trade-off relations in previous attribution literature, we mainly focus on overcoming the limitations during decomposition.
The main contributions of this work are as follows:
\begin{itemize}
\item We propose a new method for decomposing the output predictions with the relative gradient activation maps and backward sectional propagation according to the individual influences of neurons. By hostilely changing the priority of attributions corresponding to the non-target, it is possible to properly distribute the bi-polar relevance scores between the predicted classes while maintaining the irrelevant attributions as negative.

\item
We carefully address the phenomenon of non-suppressed characteristics of activated neurons and the contrary influences of the conflicting units to the channel attribution map, both of which prevent the attributions from being distinctive. We present a purging process to account for these neurons and to sustain the gap between positive and negative attributions.
\item
For evaluation, we apply Pointing Game \cite{lapuschkin2016analyzing}, sanity check with Model Sensitivity \cite{adebayo2018sanity}, and mIOU to assess the quality of attributions. We report the performance in two cases of model decision (either only correct or all labels) to confirm the efficacy of interpreting the models. The evaluation demonstrates that our method outperforms other backpropagation-based attribution methods in circumstances of complete decomposition, including the advantages of strong objectness, class-discriminativeness, and detailed descriptions of activations.

\end{itemize}

\section{Related Work}
As DNNs are applied to a variety of traditional computer vision methodologies \cite{roh2007accurate,roh2010view,yang2007reconstruction,bulthoff2003biologically}, there are many attempts to improve the transparency issues of DNNs. As the manner of interpreting a DNN model itself, intermediate features are visualized by maximizing the activated neurons in intermediate layers ~\cite{erhan2009visualizing} or generating saliency maps \cite{simonyan2013deep,zeiler2014visualizing,mahendran2016visualizing,zhou2016learning,dabkowski2017real,zhou2018interpreting}. ~\cite{ribeiro2016should} proposed LIME, which explains the black-box models, by locally approximating them as simpler interpretable models.

A perturbation-based approach directly analyzes the variations of decision when distorting the input of the network. ~\cite{zeiler2014visualizing, petsiuk2018rise} investigate the variations of the output as applying occlusions to images with specified patterns. ~\cite{fong2019understanding} introduced the concept of extremal perturbation to understand network behavior with theoretically based masking.

From the viewpoint of decomposing the network decision, ~\cite{bach2015pixel} proposed several kinds of Layer-wise Relevance Propagation (LRP) rules with the concept of relevance and conservation. As a theoretical foundation, ~\cite{montavon2017explaining} proposed Deep Taylor decomposition by utilizing Taylor expansion among neurons of intermediate layers. \cite{selvaraju2017grad} proposed Grad-CAM to generate class-discriminative activation maps by computing gradients with respect to the last convolutional units of the feature extraction stage. Guided BackProp~\cite{springenberg2014striving} is based on gradient backpropagation that considers only positive values. Integrated Gradients \cite{sundararajan2017axiomatic} addressed the gradient saturation problem by computing the average partial derivatives of the output. DeepLIFT \cite{shrikumar2017learning} decomposes the differences in relevance scores between the activation and its reference. \cite{ancona2018towards} approached with the theoretical perspective to attributions and formally proved the conditions of equivalence of the previous methods. \cite{lundberg2017unified} unifies some explaining methods and approximate with the shapley value. ~\cite{zhang2018top} proposed Excitation Backprop (EB) by modeling the probabilistic winner-take-all process and addressed the class-discriminative issue with contrastive top-down attention. ~\cite{nam2019relative} pointed out the overlapping phenomenon of positive and negative relevance and utilized the influential perspective to separate relevant and irrelevant attributions. \cite{lapuschkin-ncomm19} discussed the spurious correlations among the objects in the input, (such as tags in pictures) and present the necessity of comprehending the network decision to unmask ``Clever Hans'' phenomenon.

We mainly focus on attribution methods based on backpropagation. Although there are many studies related to this, the interpretation of DNNs remains a trade-off between the objectives of each attribution method. Therefore, our method aims to overcome the main issues: class-discriminativeness, the details of neuron activations with full decomposition, and objectness which can separate the main objects and background.
\begin{figure}[!t]
  \centering
  \includegraphics[width=1\linewidth]{Figure/Fig2_hor.png}
    \caption{The shortcomings of some attribution methods. Results of Grad-CAM and Integrated Gradients are lack of detail. Gui, LRP, and RAP are not class discriminative. Some attributions of CEB are distributed in unrelated parts.}
  \label{fig:2}
\end{figure}

\section{Revisiting Attribution Methods}
In a multi-classification task, it is clear that objects in the input are not learned antagonistically during the training procedure, because most of the networks consider the correct predictions among output logits simultaneously, not competitively. When we investigate the saliency maps of intermediate layers, highly activated neurons always have the lion's share of relevance.

Fig.~\ref{fig:2} illustrates the motivational examples of trade-off among attribution methods (the same images in Fig.~\ref{fig:intro}) and presents the result of Grad-CAM, LRP, CEP, and RAP, which are based on a modified backpropagation algorithm with their individual purposes. As is widely known, Grad-CAM shows impressive localization performance for finding attributions of DNN predictions. However, because it utilizes the end layer of the feature extraction stage and interpolation, there are many losses in describing the details of pixel-level granularity.

The other methods in Fig.~\ref{fig:2} could fully decompose DNNs in a backward manner, including the feature extraction stage. The attributions from Integrated gradient provide the responsibility for a target label by computing the gradient w.r.t the features of the image. However, it is difficult to intuitively judge the quality of interpretation in a human-view due to the scattered and overlapped positive and negative attributions. Guided Backprop and LRP represent the output logit as the relevance scores in a pixel-level. However, although there is a minor difference in values, there is no visual difference in attributions between classes. The role of the CNN feature extractor stage goes from low-level features (edge or color) to high-level features (object or texture) as the layer becomes deeper. It is inevitable that the low-level features activated in the front stage, which turns out to be irrelevant in the latter part, are assigned the positive relevance in the backward layer-wise propagation. In conjunction with the above problem, unrelated parts to the target class, e.g. corner and watermark, tend to be attributed as positive.

While RAP approaches with the influential perspective to separate relevant/irrelevant attributions and shows strong advantages of objectness, it is not distinguishable among the predicted classes. As a countermeasure, \cite{zhang2018top} proposed a contrastive perspective, which contrasts the relevance for one class with the ones of all the others. However, the positive or negative relevance is distributed in the background or other parts not related to its origin. As this method shows a relatively activated area compared to other classes, there is a probability of having positive or negative relevance scores in irrelevant parts.

In this paper, the concept of hostile refers to an element that could have a negative influence on finding the attribution corresponding to the target. For example, when we decompose the output of the right image in Fig.~\ref{fig:intro} from the bottle class, the hostile class would be a person. The relevance of the hostile class is represented as hostile attributions. By assigning the negative relevance scores to the hostile attributions, we thwart the characteristics of ``winner always wins'', resulting in assigning bi-polar relevance scores among the target (positive), hostile (negative) attributions. We also present our method with a contrastive perspective for the hostile class. In this case, all other classes except the target are set hostile.

\section{Relative Sectional Propagation}
Inspired by the above problems, our method has two main streams: (i) Relative Gradient Activation Map and purging process, and (ii) Sectional propagation according to influence with gradient and uniform shifting.

\subsubsection{Relative Gradient Activation Map}
Letter \(y\) denotes the value of the network output before passing through the final layer of the classification stage. $\{t, o_1,\dots o_n, b\}$ is class notation where each notation represents a target, other predictions, and irrelevant classes, respectively. First, we obtain the gradient activation map $G$ by backpropagating the gradient of \(y\) until the end convolution layer $X$ of the feature extraction stage.
\begin{align}
    \begin{gathered}
        G^{(t)}_{ijk} = \lambda * ReLu\left(x_{ijk} * \frac{1}{Z} \sum_{i}\sum_{j}\frac{\partial y^t}{\partial x_{ijk}}\right)\\
        F_{ijk}^{(t)} = n*G^{(t)}_{ijk} - \sum_{q=1}^n G^{(h_q)}_{ijk}
    \end{gathered}
\end{align}
Here, $x_{ijk}$ denotes the neuron in the $k^{th}$ feature map of layer $X$, indexed by width $i$ and height $j$ dimensions, and $\lambda$ is a normalization factor to keep a maximum value as $1$. The equation for computing $G_{ijk}^{(t)}$ is the same process in Grad-CAM except for the last linear combination between the feature map $X$ and partial linearization. $h$ represents the hostile classes $\{o_1, \dots, o_n\}$.

The comparative gradients of the target against hostile class are contained in $F_{ijk}^{(t)}$. The attributions that are conflict to the channel attribution map ~\cite{fong2019understanding}, generated by summing over the channel dimension $k$, still exist along the channel. Conflicting attributions refer to the units in $F_{ijk}^{(t)}$ with opposite sign to the channel attribution. To preserve the gap between target and hostile attributions during propagation, it is necessary to eliminate these conflicting attributions as follows.
\begin{equation}
F^{\prime(t)}_{ijk} =
\begin{cases}
 F_{ijk}^{(t)}  &,    \text{sign}\left(F_{ijk}^{(t)}\right) = \text{sign}\left(\sum_{k}F_{ijk}^{(t)}\right) \\
0 &,    \text{sign}\left(F_{ijk}^{(t)}\right)  \neq \text{sign}\left(\sum_{k}F_{ijk}^{(t)}\right)
\end{cases}
\end{equation}
\begin{figure}
  \centering
  \includegraphics[width=1\linewidth]{Figure/Fig_purge.png}
    \caption{An illustration of generating the relative gradient activation map. The elements marked with red and blue color represent the target: \emph{Horse} and hostile: \emph{Person} attributions, respectively. Eq.~(1) computes the comparative gradients of the target. Eq.~(2) is the purging process to eliminate the conflicting attributions to the channel attribution map.}
  \label{fig:purge}
\end{figure}

Fig.~\ref{fig:purge} and Fig.~\ref{fig:inter} show the overview of generating the relative gradient activation map and the effect of the purging process to attributions in the intermediate layer of ResNet-50, respectively. In Fig.~\ref{fig:inter}, the first row illustrates the channel activation maps of intermediate layers, and most activations are concentrated in the dog region. After applying Eq.~(1) in case \{$t$=person, $h$=dog\}, the channel attribution map is shown as the first column. Without the purging process, although the dog regions are negative in the channel attribution map, there are still conflicting units (positive) along the channel. When we backpropagate each attribution in this state, it is inevitable to have an adverse effect on the backward step due to the nature of ``winner always wins''. Thus, they are canceled out and the gap between bi-polar relevance is decreased during the propagation procedure. As shown in the second row, instead of exact positions of hostile attributions, irrelevant parts, such as corners, are emphasized as negative. The phenomenon that the attribution is visualized as tiles is due to skip-connection operations in ResNet.

After, we have non-overlapping positive and negative attributions along the channel in $F^{\prime(t)}_{ijk}$ through channel-wise purging. Here, we set the positive and negative sections as \(\mathcal{P}_{ijk}^{(t)} = \{i,j,k| F_{ijk}^{\prime(t)}>0\}\) and \(\mathcal{N}_{ijk}^{(t)} = \{i,j,k| F_{ijk}^{\prime(t)}<0\}\). We normalize these values to make the sum of positive values have twice as many values as the sum of negative values: $\mathcal{P}_{ijk}^{(t)} \leftarrow 2*\mathcal{P}_{ijk}^{(t)}$, $R^{(t)}_{ijk} = \mathcal{P}_{ijk}^{(t)} \cup \mathcal{N}_{ijk}^{(t)}$. This normalization is necessary to preserve the conservation rule and prevent degeneracy problems.

We report the difference between the two cases of obtaining the relative activation map $R^{(t)}_{ijk}$, (i) from Eq.~(1), and (ii) applying the contrastive perspective instead of Eq.~(1). For the latter case, we obtain $F^{(t)}_{ijk}$ by applying the contrastive excitation backprop \cite{zhang2018top} until the same convolution layer $X$ without channel-wise sum and ReLu. This is conceptually the same in terms of computing the relative gradient map for target class with the ones of all other (hostile) classes. The purging process and normalization are equally applied. Since the configuration and working mechanism are different according to the type of network, we report the results of both perspectives. The detailed equation of the latter case is described in the supplementary material.
\begin{figure}[t!]
  \centering
  \includegraphics[width=1\linewidth]{Figure/Fig_inter.png}
    \caption{The difference between the channel attributions of intermediate layers with/without the purging process.}
  \label{fig:inter}
\end{figure}
\subsubsection{Sectional Relevance Propagation with Gradient}
The forward process between the current layer $l+1$ and the layer $l$ to which attributions are propagated is denoted as $f(x,w^{(l,l+1)})$. Also, we notate the boolean masks of \(\mathcal{P}_{ijk\in(l+1)}^{(t)}\) and \(\mathcal{N}_{ijk\in(l+1)}^{(t)}\) as $\mathcal{B}_{ijk\in(l+1)}^{+(t)}$, $\mathcal{B}_{ijk\in(l+1)}^{-(t)}$, respectively. Since $w^{(l,l+1)}$ are not directly influenced to $R^{(t)}_{ijk\in(l+1)}$, it is necessary to compute the gradient between $R^{(t)}_{ijk\in(l+1)}$ and the sectional influence of individual neurons $f(x,w^{(l,l+1)})*\mathcal{B}_{ijk\in(l+1)}^{\pm(t)}$ with respect to weight $w^{(l,l+1)}$. This gradient contains the correlation between the individual contributions of each neuron in a forward pass and the attributions in $R^{(t)}_{ijk\in(l+1)}$.
\begin{equation}
\begin{gathered}
    \nu^{+} = \frac{f(x,w^{(l,l+1)}) * \mathcal{B}_{ijk\in(l+1)}^{+(t)}}{\partial w^{(l,l+1)}} \mathcal{P}_{ijk\in(l+1)}^{(t)}\\
    \nu^{-} = \frac{f(x,w^{(l,l+1)}) * \mathcal{B}_{ijk\in(l+1)}^{-(t)}}{\partial w^{(l,l+1)}} \mathcal{N}_{ijk\in(l+1)}^{(t)}
\end{gathered}
\end{equation}
Here, Eq.~(3) could be represented as easily readable format: $\nu=f^*(f(x,w)*B, w, R)$ (vector-Jacobian product), which is implemented in many deep learning libraries and highly optimized. Through this gradient $\nu$, we backpropagate the attributions in $R^{(t)}_{ijk\in(l+1)}$ to the previous layer $l$ with an influential perspective \cite{nam2019relative} of individual neurons.
\begin{equation}
\begin{split}
\hat R_{ijk\in(l)}^{\{t\}} & = x\odot f^{*}(f(x,\nu^{+}), \nu^{+}, \mathcal{P}_{ijk\in(l+1)}^{(t)} \oslash f(x, \nu^{+}))\\
& + x\odot f^{*}(f(x,\nu^{+}), \nu^{-}, \mathcal{N}_{ijk\in(l+1)}^{(t)} \oslash f(x, \nu^{-}))
\end{split}
\end{equation}

\begin{figure*}[t!]
  \centering
  \includegraphics[width=1\linewidth]{Figure/Fig_all_int.png}
    \caption{Comparison of the conventional attribution methods and RSP applied to VGG-16. The class names on the left side represent the predictions of DNNs among labels in input image. Each method shows the attribution results per predicted class.}
  \label{fig:all}
\end{figure*}

Here, $\odot$ and $\oslash$ denote the element-wise multiplication and division, respectively. The influence perspective represents that the importance of each neuron is an order of the absolute value, not the sign of the value. To be more specific, the attributions of the highest (positive) and lowest (negative) relevance scores denote a large amount of influence on the target and hostile classes, respectively. Attributions with a relevance score near-zero mean having a relatively small influence. From this influential perspective, it is possible to assign the relevance scores to activated neurons in order of importance among intermediate layers.

Now, the whole relevance sum of $\hat R_{ijk\in(l)}^{(t)}$ is the same as the sum of $R^{(t)}_{ijk\in(l+1)}$. We utilize uniform shifting ~\cite{nam2019relative} to change the irrelevant attributions, in which relevance scores are near zero, into negative. Let \(\Gamma\) be the number of activated neurons in $l$ and the sum of $\hat R_{ijk\in(l)}^{(t)}$ is $S$, this sum value is evenly divided and subtracted. To preserve the entire relevance sum as $S$ for each layer, we double the values of attributions in $\hat R_{ijk\in(l)}^{(t)}$.
\begin{equation}
\ddot R_{ijk\in(l)}^{(t)} =
\begin{cases}
2*\hat R_{ijk\in(l)}^{(t)} - S * \frac{1}{\Gamma}  &,    \text{ $x_{ijk}>0$ } \\
2*\hat R_{ijk\in(l)}^{(t)} &,    \text{ $x_{ijk}=0$ }
\end{cases}
\end{equation}
In case that a neuron $x_{ijk}$ is not activated, $\hat R_{ijk\in(l)}^{(t)}$ is equal to zero. Relatively unimportant attributions, which are near zero, would be converted into the negative during the propagation procedure, thereby the irrelevant attributions, e.g. background, have the negative relevance scores in the final output.
$\ddot R_{ijk\in(l)}^{(t)}$ is the input attributions for the next previous layer $l-1$ and propagated by repeating this process from the purging step. This procedure is repeated until the first layer $l=1$ of the model. For the final propagation between the input and first layer, we adopt the \(Z^{\beta}\) rule \cite{bach2015pixel} which is commonly used for propagating to the input layer, resulting in clear visualizations without distorting the priority of attributions. Detail expansion of each equation is described in the supplementary material.

\begin{table*}[t]
\RawFloats
  \centering
    \resizebox{1\columnwidth}{!}{
    \large
    \begin{tabular}{cc|cc|cc|cc|cc|cc|cc|cc|cc}
        \specialrule{2.0pt}{1pt}{1pt}
     & &  \multicolumn{8}{c|}{\textbf{PASCAL VOC 2007}} & \multicolumn{8}{c}{\textbf{COCO 2014}} \bigstrut[b]\\
          & &\multicolumn{4}{c|}{VGG-16}  &\multicolumn{4}{c|}{ResNet-50} &\multicolumn{4}{c|}{VGG-16}  &\multicolumn{4}{c}{ResNet-50}  \bigstrut[b]\\
          & &\multicolumn{2}{c}{ALL}&\multicolumn{2}{c|}{DIF}  &\multicolumn{2}{c}{ALL}&\multicolumn{2}{c|}{DIF} &\multicolumn{2}{c}{ALL}&\multicolumn{2}{c|}{DIF}  &\multicolumn{2}{c}{ALL}&\multicolumn{2}{c}{DIF}  \bigstrut[b]\\
    METHOD &T& PG & mIOU  & PG & mIOU & PG & mIOU  & PG & mIOU  & PG & mIOU  & PG & mIOU  & PG & mIOU  & PG & mIOU\\
    \hline

    \multirow{2}[2]{*}{Grad-CAM} & L & .866  & .43/.49
    & .740  & .39/.48
    & .903  & .56/.57
    & .823  & .47/.57
    & .542  & .35/.46
    & .490  & .33/.43
    & .573  & .44/.51
    & .523  & .40/.48      \bigstrut[t]\\

    & P & .945  & .41/.50
    & .924  & .33/.54
    & .953  & .55/.58
    & .932  & .44/.59
    & .727  & .30/.49
    & .689  & .25/.45
    & .705  & .39/.52
    & .674  & .32/.47 \bigstrut[b]\\
    \hdashline
    \hdashline

    \multirow{2}[2]{*}{Gradient} & L & .762  & \textcolor[gray]{0.5}{.00}/.47
    & .568  & \textcolor[gray]{0.5}{.00}/.41
    & .723  & \textcolor[gray]{0.5}{.00}/.45
    & .568  & \textcolor[gray]{0.5}{.00}/.40
    & .355  & \textcolor[gray]{0.5}{.00}/.39
    & .289  & \textcolor[gray]{0.5}{.00}/.37
    & .319  & \textcolor[gray]{0.5}{.00}/.39
    & .262  & \textcolor[gray]{0.5}{.00}/.37      \bigstrut[t]\\

    & P&  .858  & \textcolor[gray]{0.5}{.00}/.49
    & .716  & \textcolor[gray]{0.5}{.00}/.50
    & .734  & \textcolor[gray]{0.5}{.00}/.44
    & .605  & \textcolor[gray]{0.5}{.00}/.43
    & .547  & \textcolor[gray]{0.5}{.00}/.44
    & .492  & \textcolor[gray]{0.5}{.00}/.40
    & .455  & \textcolor[gray]{0.5}{.00}/.42
    & .405  & \textcolor[gray]{0.5}{.00}/.38  \bigstrut[b]\\
    \hdashline

    \multirow{2}[2]{*}{DeconvNet} & L & .675 & \textcolor[gray]{0.5}{.00}/.41
    & .441  & \textcolor[gray]{0.5}{.00}/.31
    & .686  & \textcolor[gray]{0.5}{.00}/.43
    & .447  & \textcolor[gray]{0.5}{.00}/.33
    & .241  & \textcolor[gray]{0.5}{.00}/.35
    & .164  & \textcolor[gray]{0.5}{.00}/.32
    & .273  & \textcolor[gray]{0.5}{.00}/.35
    & .192  & \textcolor[gray]{0.5}{.00}/.33       \bigstrut[t]\\

    & P  & .802  & \textcolor[gray]{0.5}{.00}/.46
    & .573  & \textcolor[gray]{0.5}{.00}/.37
    & .789  & \textcolor[gray]{0.5}{.00}/.44
    & .595  & \textcolor[gray]{0.5}{.00}/.39
    & .469  & \textcolor[gray]{0.5}{.00}/.36
    & .372  & \textcolor[gray]{0.5}{.00}/.31
    & .429  & \textcolor[gray]{0.5}{.00}/.36
    & .338  & \textcolor[gray]{0.5}{.00}/.31  \bigstrut[b]\\
    \hdashline

    \multirow{1}[2]{*}{Guided} & L & .758 & \textcolor[gray]{0.5}{.00}/\textcolor[rgb]{ .0,0,1}{\textbf{.49}}
    & .530  & \textcolor[gray]{0.5}{.00}/.43
    & .771  & \textcolor[gray]{0.5}{.00}/.51
    & .594  & \textcolor[gray]{0.5}{.00}/.46
    & .365  & \textcolor[gray]{0.5}{.00}/.41
    & .288  & \textcolor[gray]{0.5}{.00}/.39
    & .410  & \textcolor[gray]{0.5}{.00}/.43
    & .340  & \textcolor[gray]{0.5}{.00}/.41       \bigstrut[t]\\

    \multirow{1}[2]{*}{BackProp}& P & \textcolor[rgb]{ .0,0,0}{{.880}} & \textcolor[gray]{0.5}{.00}/\textcolor[rgb]{ .753,0,0}{\textbf{.52}}
    & .784  & \textcolor[gray]{0.5}{.00}/.54
    & .857  & \textcolor[gray]{0.5}{.00}/.53
    & .756  & \textcolor[gray]{0.5}{.00}/.53
    & .600  & \textcolor[gray]{0.5}{.00}/.47
    & .536  & \textcolor[gray]{0.5}{.00}/.43
    & .573  & \textcolor[gray]{0.5}{.00}/.47
    & .519  & \textcolor[gray]{0.5}{.00}/.44 \bigstrut[b]\\
    \hdashline

    \multirow{1}[2]{*}{Excitation} & L & .735 & \textcolor[gray]{0.5}{.00}/.46
    & .520  & \textcolor[gray]{0.5}{.00}/.45
    & .785  & \textcolor[gray]{0.5}{.00}/.46
    & .623  & \textcolor[gray]{0.5}{.00}/.45
    & .377  & \textcolor[gray]{0.5}{.00}/.42
    & .304  & \textcolor[gray]{0.5}{.00}/.40
    & .437  & \textcolor[gray]{0.5}{.00}/.43
    & .374  & \textcolor[gray]{0.5}{.00}/.41      \bigstrut[t]\\

    \multirow{1}[2]{*}{BackProp} & P & .856 & \textcolor[gray]{0.5}{.00}/.47
    & .742  & \textcolor[gray]{0.5}{.00}/.53
    & .864  & \textcolor[gray]{0.5}{.00}/.47
    & .768  & \textcolor[gray]{0.5}{.00}/.50
    & .573  & \textcolor[gray]{0.5}{.00}/.47
    & .505  & \textcolor[gray]{0.5}{.00}/.45
    & .582  & \textcolor[gray]{0.5}{.00}/.46
    & .533  & \textcolor[gray]{0.5}{.00}/.44 \bigstrut[b]\\
    \hdashline

    \multirow{1}[2]{*}{c*Exitation} & L & .766 & .38/.45
    & \textcolor[rgb]{ .0,0,1}{\textbf{.634}}  & .34/.50
    & .857  & .49/.49
    & .741  & \textcolor[rgb]{ .0,0,1}{\textbf{.45}} /\textcolor[rgb]{ .753,0,0}{\textbf{.56}}
    & \textcolor[rgb]{ .0,0,0}{{.472}}  & .32/.46
    & \textcolor[rgb]{ .0,0,0}{{.417}}  & .30/.45
    & .536  & \textcolor[rgb]{.753,0,0}{\textbf{.41}}/\textcolor[rgb]{.753,0,0}{\textbf{.49}}
    & .485  & \textcolor[rgb]{.753,0,0}{\textbf{.37}}/\textcolor[rgb]{.753,0,0}{\textbf{.48}}      \bigstrut[t]\\

    \multirow{1}[2]{*}{BackProp}& P & .856 & .40/.42
    & \textcolor[rgb]{ .0,0,0}{{.784}}  & .39/.55
    & \textcolor[rgb]{ .0,0,1}{\textbf{.945}}  & .52/.49
    & \textcolor[rgb]{ .0,0,1}{\textbf{.887}}  & \textcolor[rgb]{ .0,0,1}{\textbf{.51}}/\textcolor[rgb]{ .753,0,0}{\textbf{.62}}
    & \textcolor[rgb]{ .0,0,0}{{.659}}  & .37/.49
    & \textcolor[rgb]{ .0,0,0}{{.620}}  & .34/.50
    & .671  & \textcolor[rgb]{.0,0,1}{\textbf{.47}}/\textcolor[rgb]{.753,0,0}{\textbf{.53}}
    & .636  & \textcolor[rgb]{.0,0,1}{\textbf{.42}}/\textcolor[rgb]{.753,0,0}{\textbf{.53}} \bigstrut[b]\\
    \hdashline

    \rowcolor[rgb]{ .906,  .902,  .902}     \multirow{2}[2]{*}{RSP}  & L
    & \textcolor[rgb]{ .753,0,0}{\textbf{.849}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.51}}/\textcolor[rgb]{ .753,0,0}{\textbf{.51}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.712}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.43}}/\textcolor[rgb]{ .753,0,0}{\textbf{.54}}

    & \textcolor[rgb]{ .0  ,0,1}{\textbf{.859}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.49}}/\textcolor[rgb]{ .0,0,1}{\textbf{.51}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.749}}
    & \textcolor[rgb]{ .0,0,0}{.39}/\textcolor[rgb]{ .0,0,0}{.49}

    & \textcolor[rgb]{ .753,0,0}{\textbf{.540}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.43}}/\textcolor[rgb]{ .753,0,0}{\textbf{.49}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.479}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.37}}/\textcolor[rgb]{ .753,0,0}{\textbf{.47}}

    & \textcolor[rgb]{ .753,0,0}{\textbf{.558}}
    & \textcolor[rgb]{ .0,0,0}{{.39}}/\textcolor[rgb]{ .0,0,0}{{.46}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.504}}
    & \textcolor[rgb]{ .0,0,0}{{.35}}/\textcolor[rgb]{ .0,0,0}{{.43}}
    \bigstrut[t]\\

    \rowcolor[rgb]{ .906,  .902,  .902}
     \multirow{-2}{*}{RSP} & P
    & \textcolor[rgb]{ .753,0,0}{\textbf{.946}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.56}}/\textcolor[rgb]{ .0,0,1}{\textbf{.51}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.903}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.54}}/\textcolor[rgb]{ .753,0,0}{\textbf{.63}}

    & \textcolor[rgb]{ .0,0,0}{.909}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.54}}/\textcolor[rgb]{ .0,0,1}{\textbf{53}}
    & \textcolor[rgb]{ .0,0,0}{.836}
    & \textcolor[rgb]{ .0,0,0}{.44}/\textcolor[rgb]{ .0,0,0}{54}

    & \textcolor[rgb]{ .753,0,0}{\textbf{.725}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.51}}/\textcolor[rgb]{ .753,0,0}{\textbf{56}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.680}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.45}}/\textcolor[rgb]{ .753,0,0}{\textbf{54}}

    & \textcolor[rgb]{ .0,0,1}{\textbf{.688}}
    & \textcolor[rgb]{ .0,0,0}{{.44}}/\textcolor[rgb]{ .0,0,0}{{.51}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.654}}
    & \textcolor[rgb]{ .0,0,0}{{.38}}/\textcolor[rgb]{ .0,0,0}{{.48}}
     \bigstrut[t] \\
     \hdashline

    \rowcolor[rgb]{ .906,  .902,  .902}     \multirow{2}[2]{*}{c*RSP}  & L
    & \textcolor[rgb]{ .0,0,1}{\textbf{.785}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.46}}/\textcolor[rgb]{ .0,0,0}{.47}
    & \textcolor[rgb]{ .0,0,0}{{.627}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.42}}/\textcolor[rgb]{ .0,0,1}{\textbf{.52}}

    & \textcolor[rgb]{ .753,0,0}{\textbf{.891}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.52}}/\textcolor[rgb]{ .753,0,0}{\textbf{.52}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.777}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.46}}/\textcolor[rgb]{ .0,0,1}{\textbf{.54}}

    & \textcolor[rgb]{ .0,0,1}{\textbf{.475}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.39}}/\textcolor[rgb]{ .0,0,1}{\textbf{.47}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.418}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.36}}/\textcolor[rgb]{ .0,0,1}{\textbf{.45}}

    & \textcolor[rgb]{ .0,0,1}{\textbf{.545}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.41}}/\textcolor[rgb]{ .0,0,1}{\textbf{.47}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.488}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.37}}/\textcolor[rgb]{ .0,0,1}{\textbf{.44}}
    \bigstrut[t]\\

    \rowcolor[rgb]{ .906,  .902,  .902}
     \multirow{-2}{*}{c*RSP} & P
    & \textcolor[rgb]{ .0,0,1}{\textbf{.881}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.49}}/\textcolor[rgb]{ .0,0,0}{.46}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.791}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.51}}/\textcolor[rgb]{ .0,0,1}{\textbf{.60}}

    & \textcolor[rgb]{ .753,0,0}{\textbf{.949}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.56}}/\textcolor[rgb]{ .753,0,0}{\textbf{.53}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.893}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.53}}/\textcolor[rgb]{ .0,0,1}{\textbf{.61}}

    & \textcolor[rgb]{ .0,0,1}{\textbf{.675}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.46}}/\textcolor[rgb]{ .0,0,1}{\textbf{.51}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.634}}
    & \textcolor[rgb]{ .0,0,1}{\textbf{.42}}/\textcolor[rgb]{ .0,0,1}{\textbf{.52}}

    & \textcolor[rgb]{ .753,0,0}{\textbf{.697}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.47}}/\textcolor[rgb]{ .0,0,1}{\textbf{.52}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.659}}
    & \textcolor[rgb]{ .753,0,0}{\textbf{.42}}/\textcolor[rgb]{ .0,0,1}{\textbf{.49}}
     \bigstrut[t] \\

    \specialrule{2.0pt}{1pt}{1pt}
    \end{tabular}%
    }
    \caption{The performance of Pointing Game and mIOU over Pascal VOC 2007 test set and COCO 2014 validation set. $T$ denotes the different circumstance of each step for testing: $P$: Only predicted classes, $L$: All labels. ALL and DIF represent the full data and the subset of difficult images, respectively. For the mIOU results, left/right values means the performance when applying without/with the threshold. The threshold is set as the mean value of positive attributions. \textcolor{myred}{RED} and \textcolor{myblue}{BLUE} represent the most and second highest numbers excluding Grad-CAM, respectively. All attribution methods, except Grad-CAM, is fully decomposed from the output to the \textbf{first} layer of each network.} \label{table:tab1}
\end{table*}%
\begin{figure}[t!]
  \centering
  \includegraphics[width=1\linewidth]{Figure/Fig_res_c.png}
    \caption{Comparison of CEB, RSP and RSP with contrastive perspective in ResNet-50.}
  \label{fig:res}
\end{figure}
\section{Experimental Evaluations}
\subsection{Implementation Details}
We utilize the popular CNN architectures: VGG-16 and ResNet-50. Each model is trained on Pascal VOC 2007 \cite{everingham2010pascal} and MS COCO 2014 dataset \cite{lin2014microsoft}, which are widely employed and easily accessible. For a fair comparison, the models that we used are available online with TorchRay package \cite{fong2019understanding}. We implement our method with PyTorch and visualize the attributions as a heatmap represented by seismic colors, where red and blue colors denote positive and negative values, respectively. We utilize the implementation introduced in \cite{fong2019understanding} for the other attribution methods. All conditions in experiments are the same except for setting the saliency layer. For the fully decomposed environment, the saliency layer for visualizing the attributions is set as the first convolution layer of each model. Each type of evaluation is described in subsequent sections.

\subsection{Qualitative Assessment}
To qualitatively evaluate the attributions of each method, we report the visual differences to compare how the high-rated points are gathered in the bounding box. As the goal of attribution methods are the same for seeking the most important factors, we could assess the consistency of positive relevance among methods. Among many researches related to attribution methods that are described in related work, we compare the methods that can be visualized with class-discriminativeness and contain detailed information of neuron activations. The other methods are not qualitatively compared due to the reasons related to Fig.~\ref{fig:2} (shown in supplementary material). The comparison methods are Grad-CAM, Integrated Gradients, Contrastive Excitation Backprop, and RSP. Fig.~\ref{fig:all} and Fig.~\ref{fig:res} illustrate the heatmaps of each method for the output predictions of VGG-16 and ResNet-50, respectively. Compared to other methods, RSP shows detailed descriptions of activated neurons and clear separations between the target object and other objects (including background), resulting in much clearer visualizations with strong objectness. More qualitative comparisons are given in the supplementary material.

\begin{figure}[h!]
  \centering
  \includegraphics[width=1\linewidth]{Figure/Fig_saliency.png}
    \caption{Model weights of VGG-16 are progressively initialized from the end to beginning.}
  \label{fig:sal}
\end{figure}
\subsubsection{Sanity Check}
\cite{adebayo2018sanity} addresses the non-sensitivity problem of some saliency methods when the parameters of the model are randomly initialized in a cascading fashion from the end layer. It is crucial to support the basis that our explanation and the model decision are dependent on each other. Fig.~\ref{fig:sal} illustrates the variations of our method when applying weight randomization progressively. Attributions from each label are extremely distorted compared to the original explanations.
\subsection{Evaluating Quality of Attributions}
\begin{table}[t!]
\begin{small}
    \centering
    \begin{tabular}{@{}l|c@{}}
        \toprule[0.4ex]

        Method & mIOU\\
        \midrule
         Grad-CAM (threshold: mean) + CRF & 52.14\\
         Segmentation Prop~\cite{guillaumin2014imagenet} & 57.30\\
         DeepMask~\cite{pinheiro2015learning} &  58.69\\
         RAP~\cite{nam2019relative} &59.46\\
         DeepSaliency~\cite{li2016deepsaliency} &  62.12\\
         Pixel Objectness~\cite{xiong2018pixel} & 64.22\\

        \midrule
         RSP & 60.81\\
         \textcolor[rgb]{ .753,0,0}{\textbf{RSP + CRF}} & \textcolor[rgb]{ .753,0,0}{\textbf{64.51}}\\
         \bottomrule[0.4ex]
    \end{tabular}
    \end{small}
    \caption{Segmentation mIoU results on the ImageNet Segmentation task. Our method is highly comparable to those methods without using the additional supervision.}
    \label{imagenet_seg}
\end{table}
\subsubsection{Pointing Game}
Pointing Game \cite{zhang2018top} assesses the attribution methods by computing the matching scores of localization between the highest relevance point and the semantic annotations of object categories in the image. However, in previous literature, this metric does not consider the performance of the model itself because the assessed attributions are decomposed from the label, not the predicted classes. It is necessary to consider the context of the predictions because performing the decomposition on a label that has not been identified by DNNs is likely to lead to incorrect interpretation. Therefore, we report both cases of the decomposition from (i) P: only predicted labels and (ii) L: all labels in Tab.\ref{table:tab1}.

To compare the attribution methods in completely decomposed circumstances, all methods except Grad-CAM do backward propagation until the first convolution layer of each network. Methods with notation c* denote a contrastive perspective to set comparative classes to target. In our case, c* represents the case of computing the relative gradient activation map from the contrastive perspective. As shown in the results, our method shows the superior performance of localization in both cases compared to the fully decomposed attribution methods. In VGG-16, the relative gradient maps between the target and the rest of the predicted classes show much superior performance compared to the contrastive initialization. On the contrary, ResNet-50 shows much better results in the class setting of contrasting.

\subsubsection{Objectness and Weakly Supervised Segmentation}
Interpretation of the attribution methods and objectness is closely related in terms of aiming to find the pixels corresponding to the target object. Based on this concept, many studies in weakly supervised segmentation (image-label level supervision) \cite{ahn2019weakly, Lee_2019_CVPR, huang2018weakly} start from the seeds extracted from the attribution methods. We report the mean Intersection of Union (mIoU) to measure the false positives of attributions that are distributed in irrelevant parts (other objects or background). Instead of comparing it with the segmentation mask, we computed the accuracy of mIOU between the bounding box and positive attributions to allow the error range in both datasets. Since some methods: Guided Backprop, DeconvNet, and Excitation Backprop, have all positive values in output results, the mean value of attributions is taken as the threshold. In Tab.\ref{table:tab1}, the left (right) values for each method represent the without (with) the threshold. Our method outperforms in almost all cases, especially when the target is in the set of correct predictions.

Furthermore, we represent the segmentation performance on the ImageNet segmentation dataset  ~\cite{Guillaumin2014ImageNetAW}, which consists of 4,276 images with segmentation mask. Tab.\ref{imagenet_seg} compares other saliency based objectness methods and RSP. The results of Grad-CAM, RAP, and RSP are from the ImageNet pretrained model in PyTorch. Although some methods use additional supervision, (i.e. bounding box, optical flow), RSP shows comparable performance with only image-label level supervision. First row in Fig.~\ref{fig:seg_dis} shows the results of Grad-CAM (mean threshold) and RSP with Dense-CRF \cite{krahenbuhl2011efficient} to refine the attributions, resulting in superior performance compared to other methods.
\begin{figure}[t!]
  \centering
  \includegraphics[width=1\linewidth]{Figure/Fig_seg_dis.png}
    \caption{The first and second rows demonstrate the weakly supervised segmentation results with CRF and Misconception of ResNet-50 about a single object, respectively.}
  \label{fig:seg_dis}
\end{figure}
\section{Discussions}
To the best of our knowledge, there is still no exact elucidation of the internal mechanisms except for the structural and conceptual differences between VGG-Net and ResNet. Some differences could be inferred by investigating the failure attributions seen only in ResNet. ResNet tends to classify objects independently between classes. For example in Fig.~\ref{fig:seg_dis} second row, a single object is misclassified as two classes by focusing on different features, resulting in overlap of the relative gradient activation map from Eq.~(1). Furthermore, there is a clear effect of unlabeled objects in ResNet. In these cases, the contrastive hostile setting shows better performance to find the target attributions. More discussion and analysis are described in the supplementary materials.
\section{Conclusion}
In this paper, we propose a new attribution method for decomposing the output of DNNs by assigning the bi-polar relevance score among the target and hostile classes. From the antagonistic perspective among objects, it is possible to allocate the bi-polar relevance scores to neuron activations, resulting in distinguishable and attentive decomposition. We assess our methods in quantitative and qualitative ways with i) Pointing Game, ii) mIOU, and iii) Model randomization to confirm the quality of attributions. The results demonstrate that the attributions from RSP have the properties of strong objectness, class-discriminativeness, and detailed descriptions of the neuron activations.

\section{Acknowledgments}
This work was supported by Institute for Information \& communications Technology Planning \& Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-01779, A machine learning and statistical inference framework for explainable artificial intelligence \& No. 2019-0-01371, Development of brain-inspired AI with human-like intelligence \& No. 2019-0-00079,  Artificial Intelligence Graduate School Program, Korea University).

{\small
\bibstyle{aaai21}


\section{Introduction}
\label{section:introduction}
Video action classification models have achieved remarkable performance improvements in recent years.
The main innovations have stemmed from the introduction of large-scale video dataset like Kinetics~\cite{kinetics} and Sports-1M~\cite{KarpathyCVPR14} and the development of powerful network architectures using 3D convolutional neural networks (3D CNNs).

As the architecture of video models become deeper and more complex, overfitting and the resulting loss of generalizability become greater concerns. For example, models trained on large-scale video dataset still suffer from the object and scene biases: models rely heavily on specific discriminative objects and scene elements~\cite{sevilla2019only,li2019repair,weinzaepfel2019mimetics}. This has led to sub-optimal generalization performances and the decrease in localization abilities of video action classifiers. Ideally, a model should extract cues for recognition from diverse sources to enhance generalizability and the robustness to missing features.

The above problems are already identified and studied in static image recognition tasks, especially in the context of modern models based on 2D CNN architectures. For 2D image recognition, data augmentation has proven to be effective, while not requiring extra annotation or training time~\cite{singh2017hide,choe2019attention,cutmix}.

In contrast, there is a lack of extensive studies on the data augmentation strategies for video recognition tasks. In this paper, we examine the efficacy of image-domain data augmentation strategies on video data, especially the ones based on feature erasing that are known to improve model robustness and generalizability~\cite{devries2017cutout,zhong2017randomerase}.
In particular, we consider a generalization of CutMix~\cite{cutmix} to video sequences.
We show experiments to decide which axis (spatial or temporal) CutMix needs to be extended for the best performance on video sequences.
As a result of our analysis, we introduce the \textbf{VideoMix} augmentation strategy. A new training video sample is constructed by cutting and pasting a random video cuboid patch from one video to the other. The ground truth label for this video is a volume-proportional combination of the source video labels.

\input{arxiv_videomix/main/tables/title}

We show the effectiveness of VideoMix with extensive evaluations on various 3D CNN architectures, datasets, and tasks.
Table~\ref{table:intro} summarizes the improvements by VideoMix.
VideoMix consistently improves video classification models without any additional parameter and a significant amount of computation.

\section{Related Works}
\label{section:related_works}

We briefly discuss the related works of the video classification task and the data augmentation in this section.

\subsection{Video Classification}

The key difference between video and image classifications is that the former must capture temporal information.
Some prior works~\cite{simonyan2014two,feichtenhofer2016convolutional,carreira2017quo} have extracted temporal cues explicitly (e.g. via optical flow).
With the development of deep learning and large-scale video datasets~\cite{kinetics,sports1M}, 3D convolutional neural networks (3D CNNs)~\cite{tran2015learning_spatiotemporal,hara2018can_spatiotemporal,carreira2017quo,tran2019CSN} and non-local modules~\cite{wang2018non}
are proposed to learn the temporal cues automatically.
Recently proposed SlowFast network~\cite{feichtenhofer2019slowfast} and CSN~\cite{tran2019CSN} show the state-of-the-art performances on the video classification using 3D CNNs.
SlowFast proposed a dual-branch architecture to combine a slow path for static spatial features and a fast path for dynamic motion features, and CSN utilizes depth-wise convolution for lightweight 3D CNN architecture.
While the advances in video classification have been focused on the architectural axis, we explore the orthogonal \textbf{data} axis, which is seldom explored in the context of video recognition tasks.
We show the effectiveness of VideoMix by conducting experiments on top of the state-of-the-art SlowFast and CSN networks.

\subsection{Data Augmentation}

\paragraph{Data augmentation for image classification.}
There are many augmentation strategies for static image classification tasks.
Horizontal flipping, random resizing, and cropping have been used in training ImageNet classifiers and are now considered the standard set of augmentation strategies~\cite{Inceptionv3}.
There have been regional dropout methods~\cite{devries2017cutout,zhong2017randomerase} which remove random regions of an image to enhance robustness and generalization.
Other single-image augmentation strategies include RandAugment~\cite{randaugment} and AutoAugment~\cite{autoaugment}. They consider the combination of extensive pixel-level image augmentation types, such as rotation, shear, translation, and color jittering. RandAugment and AutoAugment train classifiers with above operations via random selection and learned policy, respectively.
Augmentation strategies that combine more than one image include Mixup~\cite{zhang2017mixup} and CutMix~\cite{cutmix}. Mixup averages the RGB values of two images and the ground truth labels to create new samples.
CutMix~\cite{cutmix} has improved upon regional dropout by filling in image patches from other images in the dropped-out region, thereby maximizing the pixel efficiency during training.
The labels are mixed among the source images as in Mixup. We discuss and experiment with the above static-image augmentation strategies on video classification tasks in our analysis.

\paragraph{Data augmentation for video classification.}
There do exist a few attempts to apply data augmentation strategies on videos. On the spatial side, the standard single-image augmentation strategies for image classification have been considered: horizontal flipping and random cropping~\cite{wang2015towards}.
Along the temporal axis, a widely-used augmentation strategy is to randomly sub-sample a shorter video clip from the full sequence.
However, there has been an overall lack of extensive studies on video augmentation methods. This work contributes the first studies on the impact of video augmentation strategies on the generalization, localization, and transfer learning capabilities.
\section{VideoMix}
\label{section:videomix}

In this section, we analyze existing data augmentations, describe the VideoMix algorithm, and discuss the effectiveness of VideoMix.

\subsection{Revisiting Data Augmentation Techniques}

\input{arxiv_videomix/main/tables/baseline}

We review existing data augmentation methods that partially remove regions in an image~\cite{devries2017cutout}, add pixel-level noises~\cite{autoaugment,randaugment}, or manipulate both input pixels and labels~\cite{zhang2017mixup,cutmix}.
We first evaluate their effectiveness by simply extending the two-dimensional methods to the three spatio-temporal dimensions.

Table~\ref{table:method:analysis} compares the augmentation strategies including Mixup~\cite{zhang2017mixup}, RandAugment~\cite{randaugment}, Cutout~\cite{devries2017cutout}, and CutMix~\cite{cutmix}.
It is straightforward to extend Mixup and RandAugment to videos: they apply global operations over the pixels. Mixup averages the RGB values of two images and RandAugment applies rotation, shear, and uniform perturbations on the images. We apply the same operation over the spatio-temporal frames.
For Cutout and CutMix, we choose the full spatio-temporal extension where sub-cuboids in videos are randomly selected to be removed or replaced.
We set the hyperparameter $\alpha$ of Mixup and CutMix to $1.0$, the mask size for Cutout to $112\times112$, the magnitudes $M$ for RandAugment to $9$.
Table~\ref{table:method:analysis} shows that even the naive extension of the baselines lead to improvements in video classification against the vanilla model. For example, Mixup achieves 77.0\% top-1 accuracy on Mini-Kinetics, an $+\mathbf{1.8}\%$ improvement against the vanilla SlowOnly-34 ($8\times8$) model.

In the rest of the section, we seek ways to boost the video recognition performances further by studying the design choices in augmentation strategies in greater depth.

\subsection{VideoMix Algorithm}
\label{section:videomix:algorithm}
We introduce the \textbf{VideoMix} algorithm.
Let $x \in \mathbb{R}^{T \times W \times H}$ be a video sequence, where $T$, $W$, and $H$ are the number of frames, width, and height of video sequences, respectively\footnote{We omit the input channel dimension (3 for RGB) for simplicity.}. Let $y$ be the corresponding ground truth label, represented as a one-hot vector.
VideoMix generates a new training sample $(\Tilde{x},\Tilde{y})$ by combining two training samples $(x_{A}, y_{A})$ and $(x_{B}, y_{B})$.
The generated $(\Tilde{x},\Tilde{y})$ is used for training the model with its original loss function.

More precisely, VideoMix first defines a binary tensor mask $\mathbf{M} \in \{0,1\}^{T \times W \times H}$ signifying the cut-and-paste locations in two video tensors. The new video is generated by the below procedure:
\begin{equation}
\begin{split}
    \Tilde{x} & =  \mathbf{M} \odot x_{A} + (\mathbf{1}- \mathbf{M}) \odot x_{B} \\
    \Tilde{y} & =  \lambda_\mathbf{M} y_A + (1-\lambda_\mathbf{M}) y_B
\end{split}
\label{eq:cutmix}
\end{equation}
where $\odot$ is the element-wise multiplication and $\lambda_\mathbf{M} := \frac{1}{TWH}\sum_{t,w,h}\mathbf{M}_{t,w,h}$
denotes the proportion of the volume occupied by $\mathbf{M}$.

The binary tensor mask $\mathbf{M}$ is decided by drawing the 3D cuboid coordinates $\mathbf{C}=(t_1,t_2,w_1,w_2,h_1,h_2)$ at random. More specifically,
\begin{align}
\small
    \mathbf{M}_{t,w,h}:=
    \begin{cases}
        1,&\text{ if }\quad t_1\leq t \leq t_2\text{, }w_1\leq w \leq w_2\text{, and } h_1\leq h \leq h_2 \\
        0,&\text{ otherwise}
    \end{cases}
\end{align}
We will investigate the design choices for the random selection of coordinates in the next part.

\subsection{Investigation of spatial and temporal axes for VideoMix}
\begin{figure}[!t]
\centering
\includegraphics[width=\linewidth]{arxiv_videomix/main/figures/videomix.pdf}
\caption{\textbf{VideoMix variants.} Illustrations of Spatial (S-), Temporal (T-), and Spatio-temporal (ST-) VideoMix.
We omit the channel (color) dimension for clarity. A sub-cuboid of Video B (blue cube) is inserted into Video A (white cube).}
\label{fig:videomix:videomix_types}
\end{figure}
\begin{table}[!t]
\centering
\begin{tabular}{@{}lccc@{}}
\toprule
Methods              & top1  & top5  \\ \midrule
Spatial VideoMix       &  \textbf{77.6}    &  \textbf{93.5} \\
Temporal VideoMix       &   75.6   & 92.5 \\
Spatio-temporal VideoMix       & 76.7  & 92.9\\
\midrule
\end{tabular}
\caption{\textbf{Performances of VideoMix variants.} Performances of Spatial, Temporal, and Spatio-temporal VideoMix. SlowOnly-34 ($8\times8$) network and Mini-Kinetics dataset are used.
}
\label{table:method:videomix_types}
\end{table}

We identify three types of VideoMix: Temporal, Spatial, and Spatio-temporal VideoMix.
Temporal VideoMix samples the cuboid coordinates $\mathbf{C}$ only along the temporal axis ($(t_1,t_2)$ are sampled) and fixes spatial coordinates at $(w_1,w_2,h_0,h_1)=(0,W,0,H)$. Spatial VideoMix samples the spatial coordinates $(w_1,w_2,h_1,h_2)$, while fixing $(t_1,t_2)=(0,T)$.
Spatio-temporal VideoMix samples all the coordinates $(t_1,t_2,w_1,w_2,h_1,h_2)$.
The VideoMix variants are illustrated in Figure~\ref{fig:videomix:videomix_types}.
Note that the Spatio-temporal VideoMix is the same as the CutMix in Table~\ref{table:method:analysis}.

Table~\ref{table:method:videomix_types} compares the performances of the VideoMix variants. Spatial VideoMix is the best among them. We hypothesize that the video sub-cuboid must secure a sufficient number of frames to represent the semantic information for the video category. Temporal VideoMix or Spatio-temporal VideoMix is limited in terms of the semantic content in cut and pasted video cuboids.
Spatial VideoMix, on the contrary, retains the full temporal semantics.

Based on this observation, we define \textbf{Spatial VideoMix} as our default VideoMix setting for video classification task.
The random coordinate selection strategy follows that of CutMix~\cite{cutmix}.
The spatial ratio $\lambda$ is sampled from the Beta distribution Beta($\alpha$,$\alpha$), where we set $\alpha:=1$\footnote{When $\alpha=1$, it is the uniform distribution $\text{Unif}(0,1)$.}.
The center coordinates $(w_c,h_c)$ are sampled from $\text{Unif}(0,W)\times\text{Unif}(0,H)$. Other cuboid endpoints are determined by
\begin{equation}
\begin{split}
    w_{1} = w_c - \frac{W\sqrt{\lambda}}{2}~~~ & w_{2} = w_c + \frac{W\sqrt{\lambda}}{2} \\
    h_{1} = h_c - \frac{H\sqrt{\lambda}}{2}~~~ & h_{2} = h_c + \frac{H\sqrt{\lambda}}{2}
\end{split}
\label{eq:sampling}
\end{equation}
and fixed $(t_1,t_2)=(0,T)$.
Codes for VideoMix variants are presented in Appendix~\ref{appendix:algorithm}.

\paragraph{VideoMix for temporal localization.}
Under certain application scenarios, it is difficult to directly manipulate the input videos. For example, features may have already been extracted from the original frames and the raw frames are unavailable because of the storage limits or legal issues~\cite{abu2016youtube}.
As we will cover in the experiments of temporal action localization, \textbf{Temporal VideoMix} is a good alternative that improves the localization ability of a video classifier.

\begin{figure}[!t]
\centering
\includegraphics[width=\linewidth]{arxiv_videomix/main/figures/valloss.pdf}
\caption{\textbf{Training curves.} Validation loss (left) and top-1 accuracy (right) plots for Mini-Kinetics action recognition. VideoMix achieves lower validation loss and higher top-1 accuracy than the vanilla model.}
\label{fig:videomix:loss_plot}
\end{figure}

\subsection{Discussion}

\paragraph{Effect on preventing overfitting.}

To see the effect of VideoMix on preventing overfitting and stabilizing the training process, we compare validation loss and validation accuracy of SlowOnly-34 models during the training over Mini-Kinetics action recognition dataset in
Figure~\ref{fig:videomix:loss_plot}.
We confirm that VideoMix enables video models to obtain lower validation loss and higher validation accuracy than the baseline.
After about 200 training epochs, the baseline performance saturates and the loss does not decrease further.
Applying VideoMix lets the model overcome the barrier and improve further beyond this point. The training samples generated by VideoMix allow the video classifiers to generalize better.

\begin{figure*}[!t]
\centering
\includegraphics[width=0.80\linewidth]{arxiv_videomix/main/figures/cam2.pdf}
\caption{\textbf{Class activation mapping (CAM) on VideoMix sample.} We show the spatio-temporal CAM~\cite{zhou2016CAM} score maps on a VideoMix sample. It is a combination of the ``playing harmonica'' and ``passing American football'' videos. The CAM score maps are visualized with respect to the two action classes separately.
}
\label{fig:videomix:cam_visualize}
\end{figure*}

\paragraph{What does the model learn with VideoMix?}

We expect VideoMix to let an action classifier simultaneously recognize multiple actions present in the mixed videos. To verify that this is achieved, we visualize the spatio-temporal attention of a video on the synthetic video generated by VideoMix with the class activation maps (CAM)~\cite{zhou2016CAM}.
We use a Kinetics-400 pre-trained SlowOnly-50 model.
We extend the original CAM proposed for static images to its spatio-temporal version that generates the $T$ sequential score maps.
Detail description of spatio-temporal CAM and more examples are in Appendix~\ref{appendix:algorithm}.
Figure~\ref{fig:videomix:cam_visualize} shows VideoMix samples and corresponding class activation maps with respect to the two action classes, ``playing harmonica'' and ``passing American football''.
The CAM results show that VideoMix guides a model to see multiple actions at once (e.g., the CAM for ``playing harmonica'' highlights the player's mouth and hands, and the CAM for ``passing American football'' emphasizes the kid's hands and the football object).
Furthermore, VideoMix reduces scene contexts of videos, as the background scene of ``passing American football'' are partially removed, and also hides some object cues, as the football
of ``passing American football'' and the hands holding a harmonica of ``playing harmonica'' are blocked in some frames, which leads to learning more robust and generalized cues beyond the object and scene for action recognition.

\subsection{Kinetics Action Recognition}
\label{section:experiments:kinetics}

\paragraph{Dataset.}
Kinetics-400~\cite{kinetics} is a widely used large-scale action recognition benchmark consisting of 240k training videos and 20k validation videos in 400 human action classes.
The performances are evaluated with the top-1 and top-5 accuracies.
Note that about 10\% of the Kinetics-400 videos are not available on YouTube to be downloaded. It is not possible to reproduce the exact accuracies reported in the original paper; we train the action classifier over the available subset and treat this result as the baseline.

\paragraph{Training and evaluation.}
We follow the original training recipes of the baseline architectures in~\cite{feichtenhofer2019slowfast}.
We train models from scratch using the stochastic gradient descent optimizer for 250 epochs with batch size $64$ and initial learning rate $0.1$, which is decayed by cosine annealing.
For a training video clip, $64$ consecutive frames are randomly sampled from a video and $T$ frames are sub-sampled with $\tau$ temporal stride as an input for video models.
For every model, random resize crop and random horizontal flip are applied on training video clips as standard augmentations.
For evaluation, we use $3\times10$ view ensembles as in~\cite{feichtenhofer2019slowfast}, where $10$ clips are uniformly sampled along the temporal dimension from the entire video sequences and $3$ spatial regions are uniformly sampled along the longer side of the frames.

\paragraph{Network architecture.}
We use the SlowOnly, SlowFast~\cite{feichtenhofer2019slowfast}, and interaction preserved CSN (ip-CSN)~\cite{tran2019CSN} to show the impact of VideoMix on the video classification task.
Every model is based on the ResNet architecture~\cite{resnet}. We denote the specific ResNet type with the suffix ``-(\#\ignorespaces depth)''.
We also denote each video model with frame length $T$ and temporal stride $\tau$ in the trailing bracket $(T\times\tau)$.
For example, SlowOnly-50 ($4\times16$) is based on the ResNet-50 architecture, considers $T=4$ input frames sub-sampled from the original $64$ frames with temporal stride $\tau=16$.
SlowFast-50 ($8\times8$) takes two separate input streams, the slow and fast pathways, each with $8$ and $32$ total number of input frames ($T$) with temporal strides ($\tau$) $8$ and $2$, respectively.

\paragraph{Kinetics-400 results.}

\input{arxiv_videomix/main/tables/kinetics400}

We evaluate VideoMix on Kinetics-400 with SlowOnly and SlowFast networks~\cite{feichtenhofer2019slowfast} as the base network architectures.
SlowFast combines two branches: the slow branch for static spatial features and the fast branch for dynamic motion features.
SlowOnly only has the slow branch that is similar to the ResNet~\cite{resnet} architecture with 3D convolutional kernels.
The experimental results are shown in Table~\ref{table:experiment:kinetics400}.
The accuracies in the table are reproduced results\footnote{The original paper has reported the top-1 accuracies for {SlowOnly-50 ($4 \times 16$), SlowOnly-50 ($8 \times 8$), and SlowFast-50 ($8 \times 8$)} as $72.6$, $74.8$, and $77.0$, respectively. The difference is due to the 10\% unavailable videos in Kinetics-400 and smaller batch sizes due to GPU limitations.}.
We also report the inference cost (GFlops) of a single view (a temporal clip with spatial crop) as well as the number of views for the prediction of a single video.
We observe that VideoMix consistently improves the accuracy of baseline models.
VideoMix achieves the top-1 accuracy of $72.7\%$, $74.9\%$, and $76.6\%$ for SlowOnly-50 ($4 \times 16$), SlowOnly-50 ($8 \times 8$), and SlowFast-50 ($8 \times 8$) with improvements of $+\mathbf{0.9}\%$, $+\mathbf{1.3}\%$, and $+\mathbf{0.7}\%$, respectively.
We also show that VideoMix-augmented SlowFast recognizer achieves a competitive performance ($76.6\%$) against other methods such as I3D~\cite{carreira2017quo} ($72.1\%$), Two-Stream I3D~\cite{carreira2017quo} ($75.7\%$), and Nonlocal-ResNet50~\cite{wang2018non} ($76.5\%$) which require more computational costs (GFlops) than the SlowFast architecture.

\paragraph{Mini-Kinetics results.}

\input{arxiv_videomix/main/tables/mini-kinetics}

We also evaluate VideoMix on Mini-Kinetics as shown in Table~\ref{table:experiment:mini-kinetics}.
We observe that VideoMix improves the performance of various baseline architectures: ip-CSN-50~\cite{tran2019CSN}, SlowOnly-50 ($4\times16$), SlowOnly-50 ($8\times8$), and SlowOnly-50 ($8\times8$) with $\mathbf{75.9}\%$ ($\mathbf{+1.1}\%$), $\mathbf{76.0}\%$ ($\mathbf{+1.6}\%$), $\mathbf{79.2}\%$ ($\mathbf{+1.7}\%$) and $\mathbf{81.9}\%$ ($\mathbf{+2.4}\%$), respectively.

\paragraph{Ablation Studies.}

We conduct ablation studies on the Mini-Kinetics dataset.
SlowOnly-34 is used as the running baseline, where the BasicBlock is used as in ResNet-34~\cite{resnet}.
The results are shown in Table~\ref{table:experiment:albation}.

\input{arxiv_videomix/main/tables/ablation}

We first examine the impact of the mixture area hyperparameter $\alpha \in \{0.2,0.5,$ $1.0\text{(ours)},2.0\}$.
VideoMix at various $\alpha$ values exhibits stable performances, uniformly outperforming the vanilla baseline. VideoMix is not sensitive to the hyperparameter $\alpha$.
When we reduce the chance of applying VideoMix on a minibatch to $prob$=$0.5$ (default is $prob$=$1.0$), the top-1 accuracy drops by $0.6$ percent point. The simple strategy of applying VideoMix on every video in the batch is a better choice.
When we increase the number of mixing videos (``\#videos") more than two, the performance significantly drops, which indicates that two videos are enough for VideoMix and mixing more than two videos may hinder the training convergence.

Temporal VideoMix and Spatio-temporal VideoMix are not as effective as the VideoMix (Spatial VideoMix), with only $+0.4\%$ and $+1.5\%$ boosts over the baseline.
Per-frame VideoMix independently applies VideoMix at every frame, ruining the temporal continuity of the VideoMix operation. It shows an even worse accuracy of 74.8\%, lower than the original model. Temporal continuity is an important ingredient for the video augmentation.

More ablation studies to see the effect of dataset sizes and applying other data augmentation methods together with VideoMix are provided in Appendix~\ref{appendix:more_baseline}.
\subsection{Something-Something-V2}
\label{section:experiments:something}

\input{arxiv_videomix/main/tables/something}

\paragraph{Dataset.}
Something-Something-V2 dataset~\cite{goyal2017something} contains 169k training and 25k validation videos with 174 action classes.
We evaluate the performances with top-1 and top-5 accuracies.
Something-Something-V2 is known for the fine-grainedness of actions, the diversity of contexts. It poses new challenges for action recognition not covered by Kinetics~\cite{kinetics}.

\paragraph{Implementation details.}
We use SlowOnly-50 ($8\times8$) and SlowFast-50 ($8\times8$) models.
The models are pre-trained on Kinetics-400 with the standard training strategy, and the final fully-connected layer is replaced with the new one with $174$ output dimensions.
The entire models are then fine-tuned for the Something-Something-V2 dataset for $40$ epochs with the batch size $64$ and learning rate $0.01$, which is decayed by a factor of 10 after $26$ and $33$ epochs.
Other implementation details are in Appendix~\ref{appendix:somethingv2}.

\paragraph{Results.}
We investigate how well VideoMix improves the generalizability of action recognition models in the challenging benchmark beyond the Kinetics.
To separate the fine-tuning effects of VideoMix, it is applied only during the fine-tuning stage. The pretrained model is the same as the baseline.
Table~\ref{table:experiment:something} shows the results.
We observe that VideoMix improves the top-1 accuracies of SlowOnly-50 ($8\times8$) and SlowFast-50 ($8\times8$) by $+\mathbf{0.9}\%$ and $+\mathbf{0.8}\%$ against the baselines, respectively.
VideoMix is effective on Something-Something-V2 as well.
\subsection{Weakly Supervised Temporal Action Localization}
\label{section:experiments:wstal}

The goal of weakly supervised temporal action localization (WSTAL) is to localize actions in untrimmed videos with a classifier trained using video-wise class labels only.
Given a video input sequence, WSTAL model predicts the sequence's class label and also generates one-dimensional temporal proposals to localize actions in the video.
WSTAL models do not exploit temporal action annotations during training and the generated temporal proposals are evaluated on validation videos with validation ground truth annotations.
To localize the action instances well, a video model recognizes action categories from full video sequences and not focus on small discriminant frames of the action.
Through the WSTAL experiments, we verify that VideoMix improves the temporal localization ability of an action recognition models by guiding them to attend on wider frames of action.
To evaluate the temporal localizability, we apply VideoMix over the baseline WSTAL methods.

\paragraph{Dataset.}
We conduct weakly supervised temporal action localization (WSTAL) task on THUMOS’14 dataset~\cite{THUMOS14}.
THUMOS’14 dataset originally consists of 13,320 trimmed videos for training and 2,584 untrimmed videos for validation with 101 action categories.
We follow previous WSTAL methods' setting~\cite{lee2020bas,nguyen2018weakly,shou2018autoloc}.
We train WSTAL models with the 20 class subset of the untrimmed videos, which consists of 200 untrimmed videos without temporal annotations.
The temporal localization performance of a model is evaluated by 212 untrimmed videos with temporal annotations.
WSTAL on THUMOS’14 dataset is a challenging task since the length of untrimmed video could be quite long (up to 26 minute) and multiple actions could exist in a video.

\paragraph{Training and inference.}
For training, we first extract I3D~\cite{carreira2017quo} features from the training videos as done in~\cite{lee2020bas}.
We sample $750$ video segments from a training video and RGB frames and optical flows are separately fed into dual-path I3D network.
Each RGB and optical flow frame results in $1024$-dimensional feature, thus the dimension of extracted feature for a video is $750\times1024$ for RGB input, and $750\times2048$ for both using RGB and optical flow input (RGB+flow).
The WSTAL model, which consists of two $3\times3$ convolutional layers followed with LeakyReLU activation and a $1\times1$ convolutional layer, takes the extracted features and predicts its class label.
Since the network is trained with pre-computed features, we do not consider spatial VideoMix, but utilize \textbf{temporal VideoMix} on the extracted features.
Implementation details are in Appendix~\ref{appendix:wstal}.
For evaluation, temporal class activation mapping (T-CAM)~\cite{zhou2016CAM,nguyen2018weakly} is utilized to localize action instances along the temporal dimension.
We threshold T-CAM below 50$\%$ of the highest value, and all 1-dimensional continuous segments are considered as action instance proposals as in~\cite{singh2017hide}.
Evaluation metric is average precision (AP) of intersection over union (IoU) thresholds from $0.1$ to $0.9$, and we report their mean value (mAP) as in~\cite{singh2017hide,nguyen2018weakly,lee2020bas}.

\input{arxiv_videomix/main/tables/wstal}

\paragraph{Results.}
Table~\ref{table:experiment:wstal} shows the WSTAL performances with and without optical flow features.
We compare VideoMix against the baselines including Hide-and-Seek~\cite{singh2017hide}, which has been reported to improve weakly supervised object localization on static images and WSTAL on videos.
We also compare against Mixup~\cite{zhang2017mixup}.
We observe that VideoMix improves the temporal localization accuracy mAP of baseline by $+\mathbf{0.9}\%$ and $+\mathbf{1.5}\%$ with and without optical flows, respectively.
VideoMix also outperforms the Hide-and-Seek and Mixup in both scenarios showing its superior temporal localization ability.
We also conducted VideoMix with a stronger baseline, W-TALC~\cite{paul2018w}, and confirmed that VideoMix improves the performance of W-TALC from 31.1 to \textbf{32.3 (+1.2)} mAP, which is at the state-of-the-art level.

\subsection{AVA Action Detection}
\label{section:experiments:ava}

Kinetics pretraining is a widely-used practice for many video recognition tasks \cite{gu2018ava,sun2018actor,feichtenhofer2019slowfast}.
We validate whether VideoMix pretrained models bring better performance on the downstream task of detecting actions in videos.

\paragraph{Dataset.}
AVA v2.2 dataset~\cite{gu2018ava} consists of 235 training and 64 validation videos of human actions in videos.
Each video is 15 minutes long, and the action locations are densely annotated as bounding boxes in space and time.
We follow the protocol in~\cite{gu2018ava} to train and evaluate the detection of 60 human action classes.
We use the mean average precision (mAP) metric to measure the performance of video action detection using a frame-level intersection-over-union (IoU) threshold $0.5$.

\paragraph{Detection framework.}
Our detector is based on the Faster R-CNN~\cite{ren2015faster} architecture, which is modified as in~\cite{feichtenhofer2019slowfast} to adapt to the video action detection task.
The spatial stride of the final convolutional block is reduced from $2$ to $1$ to increase the feature map size. The 2D RoIAlign layer~\cite{he2017mask} is replaced by the 3D RoIAlign.
SlowOnly-50 ($8\times8$) and SlowFast-50 ($8\times8$) have been used as the backbone network for the detection framework.
We use the human bounding box proposals provided by~\cite{feichtenhofer2019slowfast} computed by an off-the-shelf human detector fine-tuned on AVA persons.

\paragraph{Training and inference.}

We initialize two detectors with the weights pretrained on Kinetics-400 with or without VidoeMix. We apply the same fine-tuning strategy afterwards to separate the effect of VideoMix on pretraining.
We train detectors for $20$ epochs using the SGD optimizer with initial learning rate $0.1$ decayed by factor $0.1$ at $10$ and $15$ epoch.
The spatial dimension of the shorter side is resized to 256 pixels while maintaining the aspect ratio.
$64$ consecutive frames are extracted for training.
Further details are in Appendix~\ref{appendix:ava}.

\input{arxiv_videomix/main/tables/ava}

\paragraph{Results.}
Table~\ref{table:experiment:ava} shows the performances of our detector on the AVA benchmark.
Pretraining the detector with VideoMix improves the performance of SlowOnly-50 and SlowFast-50 to $\mathbf{20.4}$ ($+\mathbf{1.3}$) and $\mathbf{24.9}$ ($+\mathbf{1.7}$) mAP, respectively. Switching the pretrained weights to the VideoMix version introduces the gain in detection performance for free.
The weights will be published in the future.

\section{Experiments}
\label{section:experiments}

We evaluate VideoMix in terms of the improved generalization performances as well as the transfer-learning performances of VideoMix pre-trained models on multiple tasks.
We first verify the effect of VideoMix on video classification tasks: Kinetics action recognition~\cite{kinetics} and Something-Something-V2.
We show the temporal action localization performance via weakly-supervised temporal action localization experiments.
Finally, VideoMix is evaluated in terms of the transfer-learning performances on the video action detection task.
We also provide additional experiments on HMDB-51~\cite{kuehne2011hmdb} and UCF-101~\cite{soomro2012ucf101} action recognition benchmarks in Appendix~\ref{appendix:more_baseline}.
All experiments were implemented and evaluated on NAVER Smart Machine Learning (NSML)~\cite{nsml} platform
with PyTorch~\cite{paszke2017automatic}

\input{arxiv_videomix/main/4-1-Kinetics}
\input{arxiv_videomix/main/4-2-Something}
\input{arxiv_videomix/main/4-3-WSTAL}
\input{arxiv_videomix/main/4-4-AVA}

\section{Conclusion}
\label{section:conclusion}

We have analyzed the augmentation strategies for video action classification task. We have introduced VideoMix, a simple, efficient, and effective augmentation method.
On Kinetics action recognition, VideoMix improves the top-1 accuracies of SlowOnly-50 and SlowFast-50 by $+1.3\%$ and $+0.5\%$, respectively.
On Something-Something-V2 dataset, VideoMix brings $+0.9\%$ and $+0.8\%$ gains in top-1 accuracies on SlowOnly-50 and SlowFast-50, respectively.
VideoMix also improves the localization ability of the classifiers: on weakly supervised temporal action localization (WSTAL), VideoMix consistently improves the localization accuracy over the baselines.
Finally, VideoMix improves the Kinetics-pretrained model for the transfer-learning task of video action detection. VideoMix, as well as VideoMix pretrained weights, provide a simple and cheap solution to boost up the video recognition performances across diverse tasks.

\section{VideoMix Algorithm}
\label{appendix:algorithm}

\algnewcommand\Input{\item[\textbf{Input:}]}%
\algnewcommand\Output{\item[\textbf{Output:}]}%
\begin{algorithm*}[!t]
\small
  \caption{Pseudo-code of VideoMix}
  \begin{algorithmic}[1]
    \For {each iteration}
    \State input, target = get\_minibatch(dataset)
    \Comment{input: $(\text{N}\times C \times T \times W\times H)$} %
    \Statex $ $ \Comment{target: $(\text{N}\times \#\text{Classes})~~~~~~~~~$}
        \If{mode $==$ training}
            \State input$_\text{shuff}$, target$_\text{shuff}$ = shuffle\_minibatch(input, target)
            \State $\lambda$ = $\text{Unif}(0,1)$
            \If{VideoMix\_mode $==$ Spatial}
                \State $w_c$, $w_h$ = $\text{Unif}(0,W)$, $\text{Unif}(0,H)$
                \State $w_1$, $w_2$ = $\text{clip}(w_c-W\frac{\sqrt{\lambda}}{2})$, $\text{clip}(w_c+W\frac{\sqrt{\lambda}}{2})$
                \State $h_1$, $h_2$ = $\text{clip}(h_c-H\frac{\sqrt{\lambda}}{2})$, $\text{clip}(h_c+H\frac{\sqrt{\lambda}}{2})$
                \State $t_1$, $t_2$ = $0$, $T$
            \ElsIf {VideoMix\_mode $==$ Temporal}
                \State $t_c$ = $\text{Unif}(0,T)$
                \State $w_1$, $w_2$ = $0$, $W$
                \State $h_1$, $h_2$ = $0$, $H$
                \State $t_1$, $t_2$ = $\text{clip}(t_c-T\frac{{\lambda}}{2})$, $\text{clip}(t_c+T\frac{{\lambda}}{2})$
            \ElsIf {VideoMix\_mode $==$ Spatio-Temporal}
                \State $w_c$, $w_h$, $t_c$ = $\text{Unif}(0,W)$, $\text{Unif}(0,H)$, $\text{Unif}(0,T)$
                \State $w_1$, $w_2$ = $\text{clip}(w_c-W\frac{\sqrt[3]{\lambda}}{2})$, $\text{clip}(w_c+W\frac{\sqrt[3]{\lambda}}{2})$
                \State $h_1$, $h_2$ = $\text{clip}(h_c-H\frac{\sqrt[3]{\lambda}}{2})$, $\text{clip}(h_c+H\frac{\sqrt[3]{\lambda}}{2})$
                \State $t_1$, $t_2$ = $\text{clip}(t_c-T\frac{\sqrt[3]{\lambda}}{2})$, $\text{clip}(t_c+T\frac{\sqrt[3]{\lambda}}{2})$
            \EndIf

            \State input[:, :, $t_1$:$t_2$, $w_1$:$w_2$, $h_1$:$h_2$] = input$_\text{shuff}$[:, :, $t_1$:$t_2$, $w_1$:$w_2$, $h_1$:$h_2$]
            \State $\lambda$ = $\frac{(t2-t1)\times(w2-w1)\times(h2-h1)}{(T\times W\times H)}$ \Comment{Adjust lambda to the exact fraction ratio.}
            \State target = $\lambda$ * target + (1 - $\lambda$) * target$_\text{shuff}$%
        \EndIf
        \State output = model\_forward(input)
        \State loss = compute\_loss(output, target)
        \State update\_model(model, loss)
    \EndFor
  \end{algorithmic}
  \label{alg:videomix_algorithm}
\end{algorithm*}

We describe code-level algorithm of VideoMix variants in Algorithm~\ref{alg:videomix_algorithm}.
The input video of a minibatch is $(N\times C \times T \times W \times H)$-size tensor, where $N$, $C$, $T$, $W$, and $H$ denote the size of a minibatch, the channel size, the the width, and the height of a frame.
VideoMix first shuffles the order of the minibatch along the first dimension of the tensor.
Next $\lambda$ is sampled from the uniform distribution.
Then the cuboid coordinate $\mathbf{C}=(t_1,t_2,w_1,w_2,h_1,h_2)$ are sampled corresponding to the type of VideoMix.
Note that `clip' function truncates the coordinates to fit in the frame space (e.g., $\text{clip}(w)$=$\text{min}(\text{max}(w,W),0)$).

\textbf{Spatial VideoMix} ({line 7-10}) is the same as we described in the main paper.
For \textbf{Temporal VideoMix} (line 12-15), we only samples $t_1$ and $t_2$.
For \textbf{Spatio-temporal Videomix} (line 17-20), $t_1$, $t_2$, $w_1$, $w_2$, $h_1$, and $h_2$ are simultaneously sampled.
After sample a cuboid $\mathbf{C}$, we combine two videos by cutting and inserting the cuboid region, and $\lambda$ is adjust by computing the exact fraction ratio of the cuboid.
The target label is also blended by interpolation manner.
Note that VideoMix is simple and easy to implement with few lines, but it is very effective on various video tasks.

\section{Additional Experiments}
\label{appendix:more_baseline}

\subsection{HMDB-51 and UCF-101}

We evaluated VideoMix on HMDB-51~\cite{kuehne2011hmdb} and UCF-101~\cite{soomro2012ucf101} benchmarks. Table~\ref{hmdb51} and Table\ref{ucf101} shows the results.
Our method consistently boosts the top-1 accuracy against the baseline models.

\begin{table}[h]
\centering
\begin{tabular}{@{}ll@{}}
\toprule
Methods              & top1 acc.  \\ \midrule
I3D (Baseline)	& 66.0 \\
I3D + VideoMix	&\textbf{66.9 (+0.9)}   \\ \midrule
\end{tabular}
\caption{HMDB-51 benchmark results.}
\label{hmdb51}
\end{table}

\begin{table}[h]
\centering
\begin{tabular}{@{}ll@{}}
\toprule
Methods              & top1 acc.   \\ \midrule
VGG16 (Baseline)	& 79.8 \\
VGG16 + VideoMix & \textbf{81.7 (+2.1)}\\
I3D (Baseline)	& 93.3 \\
I3D + VideoMix	& \textbf{93.4 (+0.1)} \\ \midrule
\end{tabular}
\caption{UCF-101 benchmark results.}
\label{ucf101}
\end{table}

\subsection{Additional ablation study}

We subsample the mini-Kinetics dataset (10$\%$, 25$\%$, 50$\%$, and 100$\%$) and train the SlowOnly-34 model on the sampled datasets.
Table~\ref{ablation:datasize} shows the results.
Results show that VideoMix consistently improves accuracies against the baseline for all the subsets of the Mini-Kinetics dataset.

\begin{table}[h]
\centering
\begin{tabular}{@{}lcccc@{}}
\toprule
Dataset size &	10$\%$	& 25$\%$	& 50$\%$	& 100$\%$ \\ \midrule
Baseline	& 31.2 &	47.2	& 67.6 & 	75.2 \\
VideoMix	& \textbf{33.3}	& \textbf{49.6}	& \textbf{68.0}	& \textbf{77.6} \\ \midrule
\end{tabular}
\caption{\textbf{Impact of dataset sizes.} Top-1 accuracy is reported.}
\label{ablation:datasize}
\end{table}

To see the complementarity among the data augmentation methods (e.g., Cutout, Mixup, RandAug, and VideoMix), we conduct various combinations of data augmentation methods.
Starting from the standard augmentation (flip and random resize), we stack up VideoMix, Mixup, Cutout, and RandAugment.
Table~\ref{ablation:adding} shows the results.
VideoMix boosts performance against the standard augmentation, a relatively weak augmentation.
Combining VideoMix with other strong augmentations (Cutout or RandAugment) degrades performance since the combination leads to an excessive amount of regularization.
Mixup also shows similar tendency that combining with other augmentations leads to degraded performance.

\begin{table}[h]
\tabcolsep=0.05cm
\centering
\begin{tabular}{@{}cccccc@{}}
\toprule
Standard Aug. &	VideoMix	& Mixup & Cutout	& RandAug	& top1  \\ \midrule
\checkmark	& & & &	&		75.2 \\
\checkmark	& \checkmark & &	& &		\textbf{77.6} \\
\checkmark	& \checkmark & & \checkmark & &		76.3 \\
\checkmark	& \checkmark & &	& \checkmark &	76.4 \\
\checkmark	& \checkmark & & \checkmark & \checkmark &	73.5 \\
\checkmark	& & \checkmark &  &	&	77.0 \\
\checkmark	& & \checkmark & \checkmark &	&	74.2 \\
\checkmark	& & \checkmark & \checkmark & \checkmark &	76.7  \\
 \midrule
\end{tabular}
\caption{\textbf{Complementarity among data augmentations.}}
\label{ablation:adding}
\end{table}
\section{Something-Something-V2 Action Recognition}
\label{appendix:somethingv2}
We describe the implementation details for Something-Something-V2 Action Recognition here.

We train models on Something-Something-V2 dataset for 40 epochs with the batch size 64 and learning rate 0.01.
The learning rate is decayed by a factor of 10 after 26 and 33 epochs.
Other training details are the same as Kinetics (Section 4.1 of the main paper), except for the standard data augmentation.

The standard data augmentation of Kinetics experiments is that horizontal flipping, random cropping, and temporal uniform sampling.
Temporal uniform sampling samples a random clip of the entire sequences with a uniform frame interval.

For Something-Something-V2, we do not use horizontal flipping augmentation since the action's direction is critical for this dataset (e.g., there is  a `pushing something from left to right' action category).
Also, we sample frames with temporally perturbed interval instead of temporal uniform sampling.
In detail, we first split the entire frames with $T$ bins ($T$ is the number of sampled frames), and we select a frame from each bin and aggregate $T$ frames.

For inference, we use 3 spatial crops and single temporal crop.
\section{THUMOS'14 Weakly Supervised Temporal Action Localization}
\label{appendix:wstal}

We describe the implementation details for THUMOS'14 Weakly Supervised Temporal Action Localization (WSTAL) task.

We utilize the codebase\footnote{https://github.com/Pilhyeon/BaSNet-pytorch} of \cite{lee2020bas} for I3D~\cite{carreira2017quo} baseline.
We extract I3D~\cite{carreira2017quo} features from training video using this repository\footnote{https://github.com/piergiaj/pytorch-i3d}.
We sample $750$ video segments from a training video.
Each segment has $16$ frames and the segments are not overlapped.

The input video has RGB frames and also optical flows, and they are separately fed into dual-path I3D network.
Each segment of RGB and optical flow frames results in 1024-dimensional feature.
Thus the dimension of extracted feature for a video (i.e., $750$ segments) is $750\times1024$ for RGB input, and $750\times2048$ for both using RGB and optical flow input (RGB+flow).
We apply \textbf{Temporal VideoMix} along the temporal dimension (i.e., the first axis of the feature $750\times2048$) on the extracted feature.

To see the effectiveness of VideoMix with more stronger baseline, we conducted VideoMix with W-TALC~\cite{paul2018w} using the official pytorch codebase\footnote{https://github.com/sujoyp/wtalc-pytorch}.
We follow the original codebase's setting and we apply \textbf{Temporal VideoMix} along the temporal dimension as in the I3D experiment.

\section{AVA Action Detection}
\label{appendix:ava}

Our action detector is based on the Faster R-CNN~\cite{ren2015faster} architecture, which is modified as in~\cite{feichtenhofer2019slowfast} to adapt to the video action detection task.
We use PySlowFast\footnote{https://github.com/facebookresearch/SlowFast} and Detectron2\footnote{https://github.com/facebookresearch/detectron2} codebases.
The spatial stride of the final convolutional block is reduced from 2 to 1
to increase the feature map size.
We extend 2D RoIAlign layer~\cite{he2017mask} to 3D RoIAlign layer, which extracts RoI features spatially and then aggregate via global average pooling.
We use the human bounding box proposals provided by~\cite{feichtenhofer2019slowfast} computed by an off-the-shelf human detector fine-tuned on AVA persons, which is a Faster RCNN with a ResNeXt-101-FPN~\cite{xie2017resnext} backbone.
The person region proposals are detected by human detector with a confidence threshold of $0.8$.

We train detectors for 20 epochs using the SGD optimizer with initial learning rate 0.1 decayed by factor 0.1 at 10 and 15 epoch.
The spatial size of the input video is $224\times224$, and $64$ consecutive frames are extracted for training.
For inference, the spatial dimension of the shorter side is resized to $256$ pixels while maintaining the aspect ratio.

\section{Spatio-temporal Class Activation Mapping}
\label{appendix:st-cam}

We describe the spatio-temporal class activation mapping (ST-CAM) which is extended from spatial CAM of the original paper~\cite{zhou2016CAM}.
We use a SlowOnly-50 ($8\times8$) network~\cite{feichtenhofer2019slowfast} which is pretrained on Kinetics-400~\cite{kinetics} dataset.

To obtain a ST-CAM of a given video input, we first extract the final feature map of the SlowOnly-50 network before the global average pooling layer.
The temporal dimension of the feature map is $8$ which is the same as the number of input frames.
We reduce the spatial stride of the last convolution from $2$ to $1$ of the original SlowOnly-50 network, so that the spatial dimension of the feature map is $14 \times 14$ to clearly see the CAM heatmap, while the original size is $7 \times 7$.
The number of channels of the feature map is $2048$.
Then, similar to the original paper~\cite{zhou2016CAM}, the extracted final feature map $(2048\times8\times14\times14)$ is multiplied with the fully-connected layer's weight corresponding to the target class $(2048\times1)$, resulting in $(8\times14\times14)$-dimensional spatio-temporal class activation mapping.
We sub-sampled $4$ frames of the ST-CAM in the main paper due to the limit of page width.

We present additional ST-CAM visualizations in Figure~\ref{supp:cam1},~\ref{supp:cam2},~\ref{supp:cam3}, and~\ref{supp:cam4} which show the VideoMix samples and corresponding class activation maps with respect to the two action classes.

\begin{figure*}[!t]
\centering
\includegraphics[width=0.77\linewidth]{arxiv_videomix/supplementary/figs/cam_supp1.pdf}
\caption{Spatio-temporal class activation mapping (ST-CAM) on VideoMix sample of the ``clean and jerk'' and ``kicking soccer ball'' videos.}
\label{supp:cam1}
\end{figure*}
\begin{figure*}[!t]
\centering
\includegraphics[width=0.77\linewidth]{arxiv_videomix/supplementary/figs/cam_supp2.pdf}
\caption{Spatio-temporal class activation mapping (ST-CAM) on VideoMix sample of the ``golf putting'' and ``juggling balls'' videos.}
\label{supp:cam2}
\end{figure*}
\begin{figure*}[!t]
\centering
\includegraphics[width=0.77\linewidth]{arxiv_videomix/supplementary/figs/cam_supp3.pdf}
\caption{Spatio-temporal class activation mapping (ST-CAM) on VideoMix sample of the ``mowing lawn'' and ``jumping into pool'' videos.}
\label{supp:cam3}
\end{figure*}
\begin{figure*}[!t]
\centering
\includegraphics[width=0.77\linewidth]{arxiv_videomix/supplementary/figs/cam_supp4.pdf}
\caption{Spatio-temporal class activation mapping (ST-CAM) on VideoMix sample of the ``high kick'' and ``pull ups'' videos.}
\label{supp:cam4}
\end{figure*}

\section*{Acknowledgement}
We would like to thank NAVER AI LAB team members, especially Jung-Woo Ha for his helpful feedback and discussion.
}

{\small
\bibliographystyle{ieee_fullname}