\section*{A: Details for COT-GAN}
The family of cost functions $\mathcal{C}^\mathcal{K}(\mu,c)$ is given by
\begin{align*}
\mathcal{C}^\mathcal{K}(\mu,c) :=\Bigg\{ & c(x,y)+\sum_{j=1}^J \sum_{t=1}^{T-1} h^j_t(y)\Delta_{t+1}M^j(x): \\
& J\in\mathbb{N}, (h^j,M^j)\in\mathcal{H}(\mu) \Bigg\},
\end{align*}
where $\Delta_{t+1}M(x) := M_{t+1}(x_{1:t+1}) - M_t(x_{1:t})$ and $\mathcal{H}(\mu)$ is a set of functions depicting causality:
\begin{align*}
\mathcal{H}(\mu):=\{ & (h,M) : h=(h_t)_{t=1}^{T-1},\ h_t\in\mathcal{C}_b(\mathbb{R}^{n\times t}), \\
 &  M=(M_t)_{t=1}^{T}\in\mathcal{M}(\mu),M_t\in \mathcal{C}_b(\mathbb{R}^{n\times t})\},
\end{align*}
with $\mathcal{M}(\mu)$ being the set of martingales on $\mathbb{R}^{n\times T}$ w.r.t. the canonical filtration and the measure $\mu$, and  $\mathcal{C}_b(\mathbb{R}^{n\times t})$ the space of continuous, bounded functions on $\mathbb{R}^{n\times t}$.

Moreover, in the implementation of COT-GAN, the dimensionality of the sets of  $\mathbf{h}:=(h^j)_{j=1}^J$ and $\mathbf{M}:=(M^j)_{j=1}^J$ is bounded by a fixed $J\in\mathbb{N}$. The discriminator in COT-GAN is formulated by parameterizing $\mathbf{h}_{\varphi_1}$ and $\mathbf{M}_{\varphi_2}$ in the cost function $c^\mathcal{K}$ as two separate neural networks that respect causality,
\begin{align}
c^\mathcal{K}_{\varphi}(x,y) = c(x,y)+\sum_{j=1}^J \sum_{t=1}^{T-1} h^j_{\varphi_1, t}(y)\Delta_{t+1}M^j_{\varphi_2}(x),
\end{align}
where $\varphi := (\varphi_1, \varphi_2)$ and $J$ corresponds to the output dimensionality of the two networks. Thus, we update the parameters based upon the loss given by \eqref{eq:sup_OT} between the empirical distributions of two mini-batches,

Given a mini-batch of size $m$ from training data $\{x^d_{1:T}\}_{i=1}^m$ we define the empirical measure for the mini-batch as
\begin{align*}
& \hat{\mu} :=\frac{1}{m}\sum_{d=1}^m\delta_{{x}^d_{1:T}}.
\end{align*}

As the last piece of the puzzle, \citet{Xu2020} enforced $\mathbf{M}$ to be close to a martingale by a regularization term to penalize deviations from being a martingale on the level of mini-batches.
\[
{p}_{\mathbf{M}}(\widehat{\mu}):=\frac{1}{mT}\sum_{j=1}^J\sum_{t=1}^{T-1}\Bigg|\sum_{d=1}^m \frac{M^j_{t+1}(x^d_{1:t+1}) - M^j_t(x^d_{1:t})}{\sqrt{\text{Var}[M^j]} + \eta}\Bigg|,
\]
where $\text{Var}[M]$ is the empirical variance of $M$ over time and batch, and $\eta>0$ is a small constant.

\section*{B: Training details}

\begin{table}
\centering
\caption{Generator architecture.}
\resizebox{8.4cm}{!}{
 \begin{tabular}{||c c||}
 \hline
 Generator & Configuration  \\
 \hline\hline
 Input &  $z \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ \\
 \hline
 0 & LSTM(state size = 64), BN \\
 \hline
 1 & LSTM(state size = 128), BN \\
 \hline
 2 & Dense(8*8*256), BN, LeakyReLU \\
 \hline
 3 & reshape to 4D array of shape (m, 8, 8, 256) \\
 \hline
 4 & DCONV(N256, K5, S1, P=SAME), BN, LeakyReLU \\
 \hline
 5 & DCONV(N128, K5, S2, P=SAME), BN, LeakyReLU \\
 \hline
 6 & DCONV(N64, K5, S2, P=SAME), BN, LeakyReLU \\
 \hline
 7 & DCONV(N1, K5, S2, P=SAME) \\
 \hline
\end{tabular}}
\label{table:g_structure}
\end{table}

\begin{table}
\caption{Discriminator architecture.}
\centering
\resizebox{8.4cm}{!}{
 \begin{tabular}{||c c||}
 \hline
 Discriminator & Configuration  \\
 \hline\hline
 Input &  \\
 \hline
 0 & CONV(N64, K5, S2, P=SAME), BN, LeakyReLU\\
 \hline
 1 & CONV(N128, K5, S2, P=SAME), BN, LeakyReLU\\
 \hline
 2 & CONV(N256, K5, S2, P=SAME), BN, LeakyReLU\\
 \hline
 3 & reshape to 3D array of shape (m, T, -1)\\
 \hline
 4 & LSTM(state size = 256), BN \\
 \hline
 5 & LSTM(state size = 64) \\
 \hline
\end{tabular}}
\label{table:d_structure}
\end{table}

We used a smaller size of model with the same network architectures as COT-GAN to train all three datasets.  The architectures for generator and discriminator are given in Tables \ref{table:g_structure} and \ref{table:d_structure}.

Hyperparameter settings are as follows:  the Sinkhorn regularizer $\epsilon=0.8$, Sinkhorn iteration $L=100$, the lengthscale $l=20$ and martingale penalty $\lambda=1.5$.  We used Adam optimizer with learning rate $0.0001$, $\beta_1=0.5$ and $\beta_2=0.9$. All models are trained for $60,000$ iterations.

\section*{C: Evaluation metrics}

To compute our three metrics, let us first assume that we have a set of real data samples ($\mathcal{P}$) and synthetic data samples ($\mathcal{S}$).
EMD is defined as:
\begin{equation}
    EMD(\mathcal{P},\mathcal{S}) = \min_{\phi: \mathcal{P} \rightarrow \mathcal{S}} \sum_{p \in \mathcal{P}} \| p - \phi (p)\|
\end{equation}
where $\phi: \mathcal{P} \rightarrow \mathcal{S}$ is a bijection.
MMD is defined as:
\begin{equation}
\begin{aligned}
    \widehat{MMD}^{2}(\mathcal{P},\mathcal{S}) = \frac{1}{n(n - 1)} \sum k(p,p) +  \\
    \frac{1}{n(n - 1)} \sum k(s,s) -  \frac{2}{n^2} \sum k(p,s)
\end{aligned}
\end{equation}
where $k$ denotes a positive-definite kernel (e.g. RBF kernel) and $n$ is the number of (real or synthetic) samples.

Lastly, to compute the KNN score, we first split our real and synthetic samples $\mathcal{P}$ and $\mathcal{S}$ into training and test datasets $\mathcal{D}_{tr}$ and $\mathcal{D}_{te}$ so that $\mathcal{D} = \mathcal{D}_{tr} \cup \mathcal{D}_{te}$. We train the KNN classifier $f: \mathcal{X}_{tr} \rightarrow [0,1]$ using training data. The accuracy of the trained classifier is then obtained using test samples $\mathcal{D}_{te}$ and given as:
\begin{equation}
    \hat{t} = \frac{1}{n_{te}} \sum_{(z_{i},l_{i}) \in \mathcal{D}_{te}} \mathbb{I} \left[\left(f(z_{i}) > \frac{1}{2}\right) = l_{i}\right]
\end{equation}
where $f(z_{i})$ estimates the conditional probability distribution $p(l=1|z_{i})$. A classifier accuracy approaching random chance (50\%) indicates better synthetic data. As suggested by \citet{Lopez-Paz2019}, we use a $1$-NN classifier to obtain the score.

\section*{D: More figures}
In this section, we provide more results in larger figures for visual comparisons.

\begin{figure*}[h!]
    \centering
    \includegraphics[width=0.98\textwidth]{figures/embs1.png}
    \caption{Larger version of Figure 2 for the purpose of visual comparison.}
    \label{fig:eweather_metrics}
\end{figure*}

\begin{figure*}[h!]
    \centering
    \includegraphics[width=0.98\textwidth]{figures/lcgp_full1.png}
    \caption{More selected samples for log-Gaussian Cox process (LGCP) dataset.}
    \label{fig:lgcp_full}
\end{figure*}

\begin{figure*}[h!]
    \centering
    \includegraphics[width=0.98\textwidth]{figures/ew_full1.png}
    \caption{More selected samples for extreme weather (EW) dataset.}
    \label{fig:eweather_full}
\end{figure*}

\begin{figure*}[h!]
    \centering
    \includegraphics[width=0.98\textwidth]{figures/tf_full1.png}
    \caption{More selected samples for turbulent flow (TF) dataset.}
    \label{fig:tf_full}
\end{figure*}

\subsection{Baselines and evaluation metrics}

We use COT-GAN \cite{Xu2020} and GAN proposed by \cite{GPC}, which we name as SinkGAN, as base models. We augment both models with our new embedding loss, using SPATE with $k$, $kw$ and $ksw$ configurations. We refer to all models using a COT-GAN backbone in combination with our new embedding loss as SPATE-GAN. We further denote the SinkGAN models corresponding to three SPATE settings as $\text{SinkGAN}^{k}$, $\text{SinkGAN}^{kw}$ and $\text{SinkGAN}^{ksw}$.
To compare our approach to a non-time-sensitive embedding, we also deploy models using the Moran's I metric using the same embedding loss procedure,
denoted as $\text{COT-GAN}^{M}$ and $\text{SinkGAN}^{M}$.

To compare our GAN output to real data samples, we use three different metrics: Earth Mover Distance (EMD), Maximum Mean Discrepancy (MMD) \cite{Borgwardt2006} and a classifier two-sample test based on a k-nearest-neighbor (KNN) classifier with $k=1$ \cite{Lopez-Paz2019}. All these measures are general purpose GAN metrics. While GAN metrics specialized on video data exist, they rely on extracting features from models
pre-trained on three-channel RGB video data. As we are working with single-channel, non-image data, these methods are not applicable in our case.

\subsection{Experimental Setting}

We compare SPATE-GAN to a range of baseline configurations. We use the same GAN architecture for all these settings to ensure comparability.
Our GAN generators feed the noise input through two LSTM layers to obtain time-dependent features. These are then mapped into the desired shape for deconvolutional operations using a fully-connected layer with a leaky ReLU activation. Lastly, 4 deconvolutional layers map the output into video frames, all also with leaky ReLU activations.
Our discriminators initially feed video input through three convolutional layers with leaky ReLU activations. The outputs from the convolutional operations are then reshaped and fed through two LSTM layers to create the final discriminator outputs.

All our models are implemented in PyTorch \cite{Paszke2019} and optimized using the Adam algorithm \cite{Kingma2015}. Our experiments are conducted on a single Geforce 1080Ti or RTX 3090 GPU.  Further training details can be found in Appendix B.

\subsection{Results}
Results from our experiments are shown in Table \ref{tab:compare1}. Visual comparisons between real and generated data from the different models are shown in Figures \ref{fig:lgcp},\ref{fig:eweather}, and \ref{fig:tf}. For larger figures including results from all tested model configurations, please see the Appendix D. Through all experiments we can observe that $\text{SPATE-GAN}^{ksw}$ consistently outperforms the competing approaches, achieving the best scores across all datasets and evaluation metrics.

\begin{table}[!h]
\centering
\caption{Evaluations for LGCP, EW and TF datasets. Lower values in EMD and MMD indicate better sample quality, while values close to 0.5 are more desirable for KNN.}
\resizebox{8.4cm}{!}{%
\begin{tabular}[!t]{lcccccc}
\hline
\hline
\bf{LGCP} &  EMD  &  MMD  &  KNN  \\
\hline
SinkGAN    & 12.46 (0.02)   &  0.38 (0.001)   &  0.14 (0.001) \\
$\text{SinkGAN}^M$     & 12.46 (0.02)   &  0.38 (0.001)  & 0.14 (0.001)  \\
$\text{SinkGAN}^{k}$   &  12.65 (0.03)  &  0.38 (0.001)  & 0.15 (0.001) \\
$\text{SinkGAN}^{kw}$    &   10.60 (0.01)  &   0.63 (0.008)  & 0.30 (0.002)   \\
$\text{SinkGAN}^{ksw}$   &  13.33 (0.01)  &  0.36 (0.001)  &  0.38 (0.003) \\
\hline
COT-GAN          &  12.38 (0.02) &  \bf{0.30 (0.001)}  &  0.20 (0.004) \\
$\text{COT-GAN}^M$   &  12.38 (0.02)   & \bf{0.30 (0.001)}   &  0.20 (0.004) \\
$\text{SPATE-GAN}^k$          &  11.56 (0.02) &  0.32 (0.01) & 0.31 (0.01) \\
$\text{SPATE-GAN}^{kw}$   &  10.92 (0.03) &  0.64 (0.035)   &  0.15 (0.006)  \\
$\text{SPATE-GAN}^{ksw}$     &  \bf{10.47 (0.02)}  &  \bf{0.30 (0.001)}  &  \bf{0.39 (0.01)} \\
\hline
\hline
\multicolumn{2}{l}{\bf{Extreme Weather}}   &     &     \\
\hline
SinkGAN    &  29.40 (0.05)  &  0.49 (0.001)  &  0.41 (0.004) \\
$\text{SinkGAN}^M$   & 29.27 (0.05)  &  0.72 (0.002)  & 0.22 (0.01)  \\
$\text{SinkGAN}^{k}$   &  32.57 (0.03)   &  0.81 (0.001)   &  0.16 (0.004) \\
$\text{SinkGAN}^{kw}$   &  32.78 (0.05) &  0.81 (0.001) & 0.18 (0.004) \\
$\text{SinkGAN}^{ksw}$  &  30.00 (0.04)  &  0.50 (0.001)   &  0.41 (0.004) \\
\hline
COT-GAN       &  26.66 (0.09)  & 0.43 (0.002)  &  \bf{0.42 (0.002)} \\
$\text{COT-GAN}^M$    &  36.42 (0.14)   &  0.65 (0.002)  &  0.09 (0.01) \\
$\text{SPATE-GAN}^k$    &  33.58 (0.07)  &  0.73 (0.002)  &  0.15 (0.01)\\
$\text{SPATE-GAN}^{kw}$  &  33.36 (0.09)  & 0.72 (0.002) &  0.13 (0.003)  \\
$\text{SPATE-GAN}^{ksw}$ &  \bf{26.24 (0.07)} &  \bf{0.42 (0.002)}  &  \bf{0.42 (0.002)} \\
\hline

\hline
\hline
\multicolumn{2}{l}{\bf{Turbulent Flows}}     &     &     \\
\hline
SinkGAN    &  26.52 (0.007)   &  1.23 (0.001)  &   0.15 (0.001) \\
$\text{SinkGAN}^M$    &  28.02 (0.005)   &  1.22 (0.0002)  & 0.01 (0.002)   \\
$\text{SinkGAN}^{k}$   &   28.14 (0.002)   &    1.32 (0.002)  & 0.08 (0.001) \\
$\text{SinkGAN}^{kw}$    &  30.98 (0.001)  &    1.50 (0.001)&  0.03 (0.001) \\
$\text{SinkGAN}^{ksw}$    &  25.47 (0.008)   &  1.24 (0.0002)  &  0.13 (0.002)   \\
\hline
COT-GAN  &  27.03 (0.01) &  1.22 (0.001)   & \bf{0.16 (0.002)} \\
$\text{COT-GAN}^M$   &  24.93 (0.01)   &1.19 (0.001)  &  0.09 (0.002)  \\
$\text{SPATE-GAN}^k$   &  25.70 (0.02)   &     1.21 (0.001) &  0.12 (0.003)  \\
$\text{SPATE-GAN}^{kw}$   &  24.30 (0.002)  &  1.42 (0.001)  &  0.13 (0.004) \\
$\text{SPATE-GAN}^{ksw}$    &  \bf{22.98 (0.01)}   & \bf{1.16 (0.001)}    &  \bf{0.16 (0.002)}  \\
\hline
\hline
\end{tabular}}
\label{tab:compare1}
\end{table}
\begin{figure}[h]
    \centering
    \includegraphics[width=0.49\textwidth]{figures/lgcp2.png}
    \caption{Selected samples for LGCP dataset.}
    \label{fig:lgcp}
\end{figure}

This finding is interesting as the $ksw$ setting theoretically looses information over the $k$ and $kw$ approaches, which both have access to future time steps when calculating the SPATE metric. Nevertheless, this result underlines the strong synergies between $\text{SPATE}^{ksw}$ and the COT-GAN backbone: The metric is calculated in sequential fashion and thus respects the same causality constraints that restrict COT-GAN. As such, the outcome, while noteworthy, is not surprising.

This result is strengthened by a comparison with the SinkGAN-based approaches: SinkGAN does not follow the same restrictions and, as we observe, is not improved as consistently by the SPATE-based embedding losses. In fact, in some cases the naive SinkGAN performs better than its derivatives using SPATE or Moran's I based embedding losses.
\begin{figure}[h!]
    \centering
    \includegraphics[width=0.49\textwidth]{figures/extreme_weather2.png}
    \caption{Selected samples for Extreme Weather dataset.}
    \label{fig:eweather}
\end{figure}
\begin{figure}[h]
    \centering
    \includegraphics[width=0.38\textwidth]{figures/tf2.png}
    \caption{Selected samples for Turbulent Flows dataset.}
    \label{fig:tf}
\end{figure}

We also observe that throughout all settings, models using Moran's I perform similarly to their naive counterparts. This confirms that in fact, simply using measures of spatial autocorrelation computed over a sequence is not sufficient for capturing complex spatio-temporal effects. On the contrary, the other two SPATE settings, $k$ and $kw$, both appear to have beneficial effects and improve performance.

In summary, our results highlight how COT-GAN combined with a non-anticipative measure of space-time association can improve the modeling of complex spatio-temporal patterns. This finding represents another step on the way towards deep learning methods specialized on the dynamics driving many systems on our planet.

Furthermore, we provide an investigation on the impact of the lengthscale parameter $l$ in the spatio-temporal expectations for $l \in \{1, 10, 20, 30, 50 \}$.  As shown in Figure \ref{fig:l},  $l=20$ leads to better EMD and KNN results whilst all MMD scores remain unchanged. For the results presented in this paper, we set $l=20$ in all our experiments.

\begin{figure}[h!]
    \centering

    \includegraphics[width=0.48\textwidth]{figures/fig6.pdf}
    \caption{Evaluations of $\text{SPATE-GAN}^{ksw}$ (left: EMD, middle: MMD, and right: KNN) on LGCP dataset given lengthscale $l \in \{1, 10, 20, 30, 50\}$. }
    \label{fig:l}
\end{figure}

\section{Introduction}
\input{intro}

\section{Related work}
\input{related_work}

\section{Method}
\input{method}

\section{Experiments}\label{sec:experiments}
\input{experiments}

\section{Conclusion}
\input{conclusion}

\subsection{SPATE: Spatio-temporal association}

\begin{figure*}[ht
\centering
\includegraphics[scale=0.99]{figures/fig1.pdf}
\caption{Illustrating the three proposed options to obtain \textit{spatio-temporal expectations} $\mu_{it}$ used in the computation of $\text{SPATE}$ for single-channel data.
\label{fig1}
\end{figure*}

For a static, discrete spatial pattern (e.g. a grid of pixels forming an image) consisting of continuous values $x \in \mathbb{R}^{n}$ where $n \in \mathbb{N}$ is the dimensionality of $x$, $x_i$ for $i \in 1,\dots,n$ represents the $i$'th pixel value on a regular grid.
The local Moran's I statistic $I_i$ is computed as follows:
\begin{equation}
\begin{split}
    I_{i} = (n_{i}-1) \frac{z_{i}}{\sum^{n_{i}}_{j=1} z_{j}^2} \sum^{n_{i}}_{j=1, j \neq i} w_{i,j} z_{j}
\end{split}
\end{equation}
where $z_i = x_{i} - \bar{x}$ is the deviance of observation $x_{i}$ from the global mean $\bar{x}$,
$n_{i}$ is the number of spatial neighbors of pixel $x_i$, $j$ indexes neighbors of $x_i$ for $j \in \{1, ..., n_i\}$ and $j \neq i$, and $w_{i,j}$ is a binary spatial weight matrix, indicating spatial neighborhood of observations $i$ and $j$. $I_{i}$ can be interpreted as a measure of similarity to neighboring
pixels: positive values imply homogeneous clusters, while negative values suggest outliers, change patterns or edges.

Now, let us assume that we observe a sequence of spatial patterns over time $t$: $x=(x_1,...,x_T) \in \mathbb{R}^{n\times T}$ where $n$ is the dimensionality of $x_t$ at each time $t$ and $T$ is the total time steps of the sequence.
Of course, a naive adoption of the approach above is simply to ignore the time component of a sequence and compute the local Moran's I values  $I_{it}$ around pixel $i$ using mean values $\bar{x}_{t}$ at each time $t$.
Unfortunately, this approach would strictly separate spatial and temporal effects. In fact, the much more realistic assumption is that space and time are not separable, but do in fact interact and form joint patterns. For this reason, we expand the concept of Moran's I for spatio-temporal expectations.
First, we follow the intuition outlined in \citet{Kulldorff2005} and define expected values of $\mu_{it}(x)$.
We refer to this approach as Kulldorff spatio-temporal expectation ("\textit{k}"):

\begin{equation} \label{eq:u_k}
\mu_{it}^{(k)} = \frac{\sum_{j} x_{jt} \sum_{t'} x_{it'}}{\sum_{j} \sum_{t'} x_{jt'}}.
\end{equation}

$\mu_{it}^{(k)}$ in \eqref{eq:u_k} involves utilizing all spatial units (pixels) available at time step $t$ and across all time steps at a single spatial unit (pixel position) $i$. This computation of the \textit{spatio-temporal expectations}
assumes independence of space and time, and thus the residual
$z_{it} = x_{it} - \mu_{it}^{(k)}$ can be thought of as a local measure of space-time interaction at pixel $i$ and time $t$.  Moreover, this formulation of $\mu_{it}$ makes two critical assumptions: (1) Different time steps are equally important, irrespective of how distant they are from the current time step. (2) At each time step, we assume availability of the whole time series (i.e. looking into the future is possible). We can modify the computation of $\mu_{it}$ by imposing alternative assumptions.

First, assuming that distant time steps have less significant impact on the current time step, we can integrate temporal weights into the computation, and apply decreasingly lower weights to more distant time steps. For example, one can consider an exponential kernel:

\begin{equation}
\mu_{it}^{(kw)} = \frac{\sum_j x_{jt} \sum_{t'} b_{tt'} x_{it'}}{\sum_j \sum_{t'} b_{tt'} x_{jt'}},
\end{equation}
where $b_{tt'} = \exp(-|t-t'| / l)$ and $l$ is the lengthscale of the exponential kernel. We refer to this approach as Kulldorff-weighted spatio-temporal expectation ("\textit{kw}").

Second, we can restrict the computation of $\mu_{it}$ at time step $t$ to only account for time steps $< t$, so that the expectation at each time step is independent of future observations.
Thus, the computation respects the generating logic of sequential data. We refer to this last approach as Kulldorff-sequential-weighted spatio-temporal expectation ("\textit{ksw}"):

\begin{equation}
\mu_{it}^{(ksw)} = \frac{\sum_j x_{jt} \sum_{t' < t} b_{tt'} x_{it'}}{\sum_j \sum_{t' < t} b_{tt'} x_{jt'}}.
\end{equation}

Note that for the first time step $t=0$, we cannot access past time steps to calculate spatio-temporal expectations $\mu_{i0}^{(ksw)}$.

We can now simply plug in our new spatio-temporal expectations into the Moran's I metric at time $t$ by replacing
spatial only expectations with a spatio-temporal expectation of our choice. As such, we define our novel measure of \textbf{spa}tio-\textbf{te}mporal association (SPATE),

\begin{equation} \label{eq:spate}
\begin{split}
    \begin{aligned}
    S_{it} (x, w) = (n_{i}-1) \frac{z_{it}}{\sum^{n_{x}}_{j=1} z_{jt}^2}  \sum^{n_x}_{j=1, j \neq i} w_{i,j} z_{jt}
    \end{aligned}
\end{split}
\end{equation}
where $z_{it} = x_{it} - \mu_{it}$ and $\mu_{it}$ can be any option from $\mu_{it}^{(k)}$, $\mu_{it}^{(kw)}$ and $\mu_{it}^{(ksw)}$. When using $\mu_{it}^{(ksw)}$, SPATE does not return values for $t=0$, as no previous time steps are available to calculate spatio-temporal expectations. See Fig \ref{fig1} for an illustration of the three proposed options.

SPATE measures spatio-temporal autocorrelation at the input resolution. Its behavior can be closely related to that of the Moran's I metric. While Moran's I evaluates the deviance $z_{i}$ between each pixel and the spatial expectation, SPATE does so by using the spatio-temporal expectation $z_{it}$.
Like Moran's I, SPATE acts as a detector of spatio-temporal clusters and change patterns. Like Moran's I, SPATE identifies positive and negative space-time autocorrelation, i.e. homogeneous areas of similar behavior and outliers that behave differently from their immediate neighborhood. The difference between Moran's I and SPATE is that the later explicitly captures space-time interactions. For example, if pixel $x_i$ and all other data points (not just its neighbors) are increased at a given time step $t$, Moran's I at time $t$ (for all points) will be high, but SPATE will not be. SPATE of $x_{it}$ will be high if \textit{(a)} pixel $x_{it}$ is high compared to its expectation at the same time $t$, \textit{(b)} its neighbors are high compared to their expectations, while \textit{(c)} its non-neighbors are not particularly high compared to their expectations.

In the $kw$ and $ksw$ settings, the lengthscale parameter $l$ governs whether the metric captures longer or shorter term temporal patterns. For example, if pixel $x_{it}$ and its neighbors increase slowly over time, that change will only cause SPATE to be high for larger lengthscales $l$, while smaller $l$ values imply that current values are compared to those that are close in time. As such, the lengthscale determines what changes are considered "slow" (incorporated into the mean, not detected as space-time interaction) and "fast" (current values are different from the mean, detected as space-time interaction). The $ksw$ setting further allows for scenarios where we might wish to compute the metric based on previous time-steps alone, i.e. to preserve sequential logic.

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.481\textwidth]{figures/embs1.png}
    \caption{The SPATE metric in its different forms computed for an example from the LGCP datasets.  While $\text{SPATE}^{k}$ and $\text{SPATE}^{kw}$ are visually undistinguishable, $\text{SPATE}^{ksw}$ changes as increasingly more past time steps become available, converging to $\text{SPATE}^{kw}$ at time $T$. We can also observe how the Moran's I metric remains static in time, while all versions of SPATE behave dynamically in space and time.}
    \label{fig:lgcp}
\end{figure}

\subsection{COT-GAN}
\label{subseq:cot-gan}

Built upon the theory of Causal Optimal Transport (COT), COT-GAN was introduced by \citet{Xu2020} as an adversarial algorithm to train implicit generative models for sequential learning.  COT can be considered as a maximization over the classical (Kantorovich) optimal transport (OT) with a temporal causality constraint, which restricts the transporting of mass on the arrival sequence at any time $t$ to depend on the starting sequence only up to time $t$. This motivated us to design the spatio-temporal expectation $\mu_{it}^{ksw}$ in order to respect the nature of sequential data that are generated in an autoregressive manner.

In sequential generation, we are interested in learning a model that produces $y=(y_1,...,y_T) \in \mathbb{R}^{n\times T}$ to mimic $x=(x_1,...,x_T) \in \mathbb{R}^{n\times T}$.
Given two probability measures
$\mu,\nu$ defined on $\mathbb{R}^{n\times T}$, and a cost function $c:\mathbb{R}^{n\times T}\times\mathbb{R}^{n\times T}\to\mathbb{R}$,
the causal optimal transport of $\mu$ into $\nu$ is formulated as
\begin{equation}\label{eq:COT}
\mathcal{W}^\mathcal{K}_c(\mu, \nu) := \inf_{\pi \in \Pi^{\mathcal{K}}(\mu, \nu)} \mathbb{E}^{\pi}[c(x,y)],
\end{equation}
where $\Pi^{\mathcal{K}}(\mu,\nu)$ is the set of probability measures $\pi$ on $\mathbb{R}^{n\times T}\times\mathbb{R}^{n\times T}$ with marginals $\mu,\nu$, which also satisfy the constraint
\begin{equation*} \label{eq:causal_pi}
\pi(dy_t|dx_{1:T})=\pi(dy_t|dx_{1:t})\quad \text{for all\, $t=1,\cdots,T-1$}.
\end{equation*} Such plans in $\Pi^{\mathcal{K}}(\mu,\nu)$ are called \emph{causal transport plans}. Here $c(x,y)$ is a cost function that measures the loss incurred by transporting a unit of mass from $x$ to $y$. $\mathcal{W}^\mathcal{K}_c(\mu,\nu)$ is thus the minimal total cost for moving the mass $\mu$ to $\nu$ in a causal way.

In comparison, the (classic) OT is defined on $\mathbb{R}^{n}$
and differs from COT by searching an optimal plan that leads to the least cost in a less restricted space $\Pi(\mu,\nu)$ which contains all transport plans,
\begin{equation}\label{eq:OT}
\mathcal{W}_c(\mu, \nu) := \inf_{\pi \in \Pi(\mu, \nu)} \mathbb{E}^{\pi}[c(x,y)].
\end{equation}

COT-GAN constructed an adversarial objective by reformulating the COT \eqref{eq:COT} as below,
\begin{equation}\label{eq:sup_OT}
\mathcal{W}^\mathcal{K}_{c}(\mu, \nu) = \sup_{c^\mathcal{K}\in\mathcal{C}^\mathcal{K}(\mu,c)} \inf_{\pi \in \Pi(\mu, \nu)} \{ \mathbb{E}^{\pi}[c^\mathcal{K}(x,y)]\},
\end{equation}
where $\mathcal{C}^\mathcal{K}(\mu,c)$ is a special family of costs that encode the causality constraint, see Appendix A for details. Note that the inner optimization problem is equivalent to OT in \eqref{eq:OT} with a specific class of cost function, i.e., $\mathcal{W}_{c^{\mathcal{K}}}(\mu, \nu) = \inf_{\pi \in \Pi(\mu, \nu)} \{ \mathbb{E}^{\pi}[c^\mathcal{K}(x,y)]\}$.

In the implementation of COT-GAN, learning is done via stochastic gradient descent (SGD) on mini-batches. Given a distribution $\zeta$ on some latent space $\mathcal{Z}$, the generator $g_\theta$ is a function that maps a latent variable $z \sim \zeta$ to the generated sequence $y$ in the path space.  Given a mini-batch of size $m$ from training data $\{x^d_{1:T}\}_{i=1}^m$ and that from generated samples $\{y^d_{1:T}\}_{i=1}^m$, we define the empirical measures for the mini-batches as
\begin{align*}
& \hat{\mu} :=\frac{1}{m}\sum_{d=1}^m\delta_{{x}^d_{1:T}},\quad \hat{\nu}_\theta:=\frac{1}{m}\sum_{d=1}^m\delta_{{y}^d_{1:T}},
\end{align*}
where $\hat{\nu}_\theta$ incorporates the parameterization of $g_\theta$.
Thus, the duality of COT given by \eqref{eq:sup_OT} between the empirical measures of two mini-batches can be written as
\begin{equation}\label{eq:sup_OT_mini}
\mathcal{W}^\mathcal{K}_{c}(\hat{\mu}, \hat{\nu}_\theta) = \sup_{c^{\mathcal{K}} \in \mathcal{C} ^{\mathcal{K}}(\hat{\mu}, c) } \mathcal{W}_{c^\mathcal{K}}(\hat{\mu}, \hat{\nu}_\theta).
\end{equation}

Computing \eqref{eq:sup_OT_mini} involves solving the optimization problem of classic OT.  COT-GAN opted for the Sinkhorn algorithm, see \cite{cuturi2013sinkhorn, GPC}, and a modified version of Sinkhorn divergence for an approximation.
Here we have the entropic-regularized OT defined as
\begin{align*}
\mathcal{W}_{c,\varepsilon}(\mu, \nu) := \inf_{\pi \in \Pi(\mu, \nu)} \{ \mathbb{E}^{\pi}[c(x,y)] - \varepsilon H(\pi)\},\quad \varepsilon>0,
\end{align*}
where $H(\pi)$ is the Shannon entropy of $\pi$.
Whilst the Sinkhorn divergence was also computed between two mini-batches, the authors of COT-GAN noticed the bias was better reduced by
the mixed Sinkhorn divergence,
\begin{align*}\label{mixSink}
\widehat{\mathcal{W}}_{c,\varepsilon}^{\text{mix}}(\hat{\mu}, \hat{\nu}, \hat{\mu}', \hat{\nu}') &:=
\mathcal{W}_{c,\varepsilon}(\hat{\mu}, \hat{\nu}) + \mathcal{W}_{c,\varepsilon}(\hat{\mu}', \hat{\nu}') \\
& \quad -\mathcal{W}_{c,\varepsilon}(\hat{\mu}, \hat{\mu}')-\mathcal{W}_{c,\varepsilon}(\hat{\nu}, \hat{\nu}'),
\end{align*}
where $\hat{\mu}$ and $\hat{\mu}'$ represent the empirical measures corresponding to two different mini-batches of the training data, and $\hat{\nu}$ and $\hat{\nu}'$ are the ones to the generated samples.

Finally,  we have the following objective function for COT-GAN:
\begin{equation*}\label{eq:objective}
\inf_{\theta} \sup_{\varphi} \bigg\{ \widehat{\mathcal{W}}_{c^\mathcal{K}_\varphi,\varepsilon}^{\text{mix}}(\hat{\mu}, \hat{\nu}_\theta, \hat{\mu}', \hat{\nu}'_\theta)
- \lambda \big[ p_{{\bf M}_{\varphi_2}} (\hat{\mu}) + p_{{\bf M}_{\varphi_2}} (\hat{\mu}') \big] \bigg\},
\end{equation*}
where the role of discriminator is played by the cost function $c^\mathcal{K}$ parameterized by $\varphi$, and
$p_{{\bf M}_{\varphi_2}}$ is the martingale penalization required for the formulation of the new class of costs $\mathcal{C}^\mathcal{K}_{\varphi}$, and $\varphi_2$ is a subset of the discriminator parameters $\varphi$, see details in Appendix A. Similar to common GAN training scheme, the discriminator learns the worst-case cost $c^\mathcal{K}_{\varphi}$ by maximizing the objective over $\varphi$, and the generator is updated by minimizing it over $\theta$.

\subsection{SPATE-GAN}

In SPATE-GAN, we integrate our newly devised spatio-temporal metric into the COT-GAN objective function.
We compute the embedding for each $x^d_{it}$ and $y^d_{it}$ in minibatches $\{x^d_{1:T}\}_{d=1}^m$ and $\{y^d_{1:T}\}_{d=1}^m$ by
\begin{align*}
\hat{x}^d_{it} = S_{it}(x^d_{1:T}, w) \quad \text{and} \quad
\hat{y}^d_{it} = S_{it}(y^d_{1:T}, w),
\end{align*}
where the binary spatial weight matrix $w$ is pre-defined.

The corresponding embeddings are then concatenated with the training data and generated samples on the channel dimension. We define the empirical measures for the concatenated sequences as
\begin{align*}
& \widehat\mu^{e}:=\frac{1}{m}\sum_{d=1}^m\delta_{\text{concat}({x}^d_{1:T},\hat{x}_{1:T}^d))},\\
& \widehat\nu^{e}_\theta:=\frac{1}{m}\sum_{d=1}^m\delta_{ \text{concat}({y}^d_{1:T},\hat{y}_{1:T}^d)},
\end{align*}
where $\text{concat}(., .)$ is an operator that concatenates inputs along the channel dimension.

We thus arrive at the objective function for SPATE-GAN:
\begin{align*}\label{eq:objective}
\inf_{\theta} \sup_{\varphi} & \bigg\{ \widehat{\mathcal{W}}_{c^\mathcal{K}_\varphi,\varepsilon}^{\text{mix}}(\widehat\mu^{e}, \widehat\nu^{e}_\theta, \widehat\mu^{e'}, \widehat\nu^{e'}_\theta) \\
& \quad - \lambda \big[ p_{{\bf M}_{\varphi_2}} (\widehat\mu^{e}) + p_{{\bf M}_{\varphi_2}} (\widehat\mu^{e'}) \big] \bigg\}.
\end{align*}
We maximize the objective function over $\varphi$ to search for a worst-case distance between the two empirical measures, and minimize it over $\theta$ to learn a distribution that is as close as possible to the real distribution.
The algorithm is summarized in Algorithm \ref{alg}. Its time complexity scales as $\mathcal{O}((J+2n)LTm^2)$ in each iteration where $J$ is the output dimension of the discriminator (see Appendix A for details), and $L$ is the number of Sinkhorn iterations (see \citet{GPC, cuturi2013sinkhorn} for details).

In the experiment section, we will compare SPATE-GAN with three different expectations $\mu_{it}^{(k)}$, $\mu_{it}^{(kw)}$ and $\mu_{it}^{(ksw)}$ in the computation of SPATE. Hence, we denote the corresponding models as $\text{SPATE-GAN}^k$, $\text{SPATE-GAN}^{kw}$, and $\text{SPATE-GAN}^{ksw}$, respectively.

Last, we would like to emphasize that, although all three embeddings consider the space-time interactions in a certain way, the non-anticipative assumption of $\mu_{it}^{(ksw)}$ is consistent with the generating process of the type of data we are investigating.  As the causality constraint in COT-GAN also restricts the search of transport plans to those that satisfy non-anticipative transporting of mass, $\text{SPATE-GAN}^{ksw}$ is a model that fully respects temporal causality in learning, whilst $\text{SPATE-GAN}^{k}$ and $\text{SPATE-GAN}^{kw}$ also combine information from the future.
\begin{algorithm}[t]
\SetAlgoLined
\KwData{$\{{x}^d_{1:T}\}_{d=1}^{n}$ (input data), $\zeta$ (latent distribution)}
\KwParam{$\theta_0$, $\varphi_0$ (parameter initializations), $m$ (batch size), $\varepsilon$ (regularization parameter),
$\alpha$ (learning rate), $\lambda$ (martingale penalty coefficient)}
 Initialize: $\theta \leftarrow \theta_0$, $\varphi \leftarrow \varphi_{0}$\\
 \For{$b = 1, 2,\dots$}{
    ~Sample $\{{x}^d_{1:T}\}_{d=1}^{m}$ from real data;\\[0.1cm]

    ~Sample $\{{z}^d_{1:T}\}_{d=1}^{m}$ from
    ~$\zeta$;\\[0.1cm]
    ~Generate sequences from latent: $(y_{1:T}^d) \leftarrow g_{\theta}({z}^d_{1:T})$;\\[0.1cm]
    ~Compute the embeddings: $\hat{x}^d_{it} = S_{it}(x^d_{1:T}, w), \quad \hat{y}^i_{it} = S_{it}(y^i_{1:T}, w)$;\\[0.1cm]
    ~Concatenated the data with embeddings: $\text{concat}({x}^d_{1:T},\hat{x}_{1:T}^d), \text{concat}({y}^d_{1:T},\hat{y}_{1:T}^d)$ ;\\[0.1cm]
    ~Update discriminator parameter: \\ [0.1cm]
    ${\varphi} \leftarrow {\varphi} + \alpha \nabla_\varphi \big( \widehat{\mathcal{W}}_{c^\mathcal{K}_\varphi,\varepsilon}^{\text{mix}}(\widehat\mu^{e}, \widehat\nu^{e}_\theta, \widehat\mu^{e'}, \widehat\nu^{e'}_\theta) - \lambda \big[ p_{{\bf M}_{\varphi_2}} (\widehat\mu^{e}) + p_{{\bf M}_{\varphi_2}} (\widehat\mu^{e'}) \big] \big)$; \\[0.1cm]
    ~Sample $\{{z}^d_{1:T}\}_{d=1}^{m}$ from  $\zeta$;\\[0.1cm]
    ~Generate sequences from latent: $(y_{1:T}^d) \leftarrow g_{\theta}({z}^d_{1:T})$;\\[0.1cm]
    ~Compute the embeddings: $\hat{x}^d_{it} = S_{it}(x^d_{1:T}, w), \quad \hat{y}^d_{it} = S_{it}(y^d_{1:T}, w)$;\\[0.1cm]
    ~Concatenated the data with embeddings: $\text{concat}({x}^d_{1:T},\hat{x}_{1:T}^d), \text{concat}({y}^d_{1:T},\hat{y}_{1:T}^d)$ ;\\[0.1cm]
    ~Update generator parameter: $\theta \leftarrow \theta - \alpha \nabla_\theta\left( \widehat{\mathcal{W}}_{c^\mathcal{K}_\varphi,\varepsilon}^{\text{mix}}(\widehat\mu^{e}, \widehat\nu^{e}_\theta, \widehat\mu^{e'}, \widehat\nu^{e'}_\theta) \right)$;
}
 \caption{training SPATE-GAN by SGD}
 \label{alg}
\end{algorithm}
\subsection{Autocorrelation metrics for spatio-temporal phenomena}

Analysing autoregressive patterns in spatial and spatio-temporal data has a long tradition in different academic domains (e.g. GIS, ecology) which over time developed diverse measures to describe these phenomena. The most commonly known of these metrics is the Moran's I index of global and local spatial autocorrelation. Originally proposed by \citet{Anselin1995}, Moran's I identifies both homogeneous spatial clusters and outliers. Applications of the metric range from identifying rare earth contamination \cite{Yuan2018a} to analysing land cover change patterns \cite{Das2017}. Throughout the years, Moran's I has also motivated several methodological expansions, analysing for example spatial heteroskedasticity \cite{Ord2012} and local spatial dispersion \cite{Westerholt2018}. Moran's I has also seen some expansions into the spatio-temporal domain.
\citet{Matthews2019} use the metric iteratively to model disease spread over time. \citet{Lee2017} and \citet{Gao2019} propose novel spatio-temporal expansions of the Moran's I metric, returning static outputs at a purely spatial resolution. \citet{Siino2018} design an extended Moran's I for spatio-temporal point processes. However, to the best of our knowledge, neither the Moran's I nor its spatio-temporal extensions have been applied to discrete spatio-temporal video data. It is evident that metrics of spatio-temporal autocorrelation can provide meaningful embeddings of complex data, capturing underlying patterns throughout a range of different application domains.

\subsection{Deep learning \& GANs for spatial and spatio-temporal data}

Deep learning describes a powerful family of methods capable of dealing with the highly complex and non-linear nature of many real world spatial and spatio-temporal patterns \cite{Yan2017,Aodha2019,Chu2019,Bao2020,Geng2019}.  Paradigms like physics-informed deep learning aim to devise methods which integrate (geo)physical constraints explicitly into neural network models \cite{Wang2020}. There is also an increasing number of studies tackling specific challenges associated with geographic data: \citet{Mai2020} and \citet{Yin2019} propose context-aware vector embeddings for geographic coordinates. \citet{Zammit-Mangion2019} propose to learn deep neural networks for spatial covariance patterns using injective warping functions. Intuitions for spatial autocorrelation, including the Moran's I metric, have been integrated into machine learning frameworks tackling model selection for ensemble learners \cite{Klemmer2019a}, spatial representation learning \cite{Zhang2019}, auxiliary task learning \cite{Klemmer2021a} or residual correlation graph neural networks \cite{Jia2020}. All these studies highlight the benefits of explicitly encoding spatial context into neural networks to improve performance.

Narrowing down on the GAN context specifically, we find that spatio-temporal applications have mostly focused on video data \cite{Xu2020,Tulyakov2018,Kim2020}. Beyond this, GANs have been used for conditional density estimation of traffic \cite{Zhang2020a}, trajectory prediction \cite{Huang2021} or extreme weather event simulation \cite{Klemmer2021b}. Nevertheless, to the best of our knowledge, metrics capturing spatio-temporal autocorrelation have never been integrated into GANs. As previous studies highlight the value on encoding spatial context, this work seeks to provide a first-principle approach of integrating metrics of spatio-temporal autocorrelation into GANs for modeling of complex spatio-temporal patterns.

\subsection{Embedding loss functions}
As SPATE-GAN integrates spatio-temporal metrics into COT-GAN as embeddings, here we briefly revisit previous work and insights into embedding losses. Embedding losses describe approaches where the loss function of a model is computed on an embedding of the data. This can have desirable outcomes, such as training stability or focusing on specific patterns in the data. Embedding losses have become popular in computer vision over the last years: \citet{Ghafoorian2019} use embedding losses to improve GAN-based lane detection. \citet{Filntisis2020} use visual-semantic embedding losses to improve predictions of bodily expressed emotions. \citet{Wang2020a} introduce CLIFFNet, utilizing hierarchical embeddings of depth maps for molecular depth estimation. \citet{Bailer2017} introduce a threshold loss to improve optical flow estimation. It is clear that embedding losses have shown great potential for particularly challenging visual problems, especially those involving complex spatio-temporal dynamics.


\section{Introduction}
The recent development of deep learning has promoted a series of applications in videos~\cite{wang2016temporal, tran2018closer, feichtenhofer2019slowfast}.
Meanwhile, practitioners have developed various large-scale benchmarks~\cite{abu2016youtube,carreira2017quo, goyal2017something} accessible to the traditional fully-supervised learning methods and have greatly facilitated video-related research. Nevertheless, the high cost of manual annotation involved in fully-supervised methods excludes the potential utilization of millions of uncurated videos on the Internet.
Therefore, video representation learning in an unsupervised manner is of great significance and emerges as a general trend.

Recently, unsupervised learning in images~\cite{oord2018representation,wu2018unsupervised,tian2019contrastive} has achieved competitive performances compared to their supervised counterparts, especially with the contrastive self-supervised learning formulation~\cite{he2020momentum, chen2020simple}. Inspired by these successes, various attempts have also been made in self-supervised video representation learning~\cite{qian2021spatiotemporal, han2020self}. However, videos are distinct from images due to the extra dimension of temporal dynamics.
\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{Figure/moco.png}
    \caption{The contrastive learning framework with the FAME method. We first randomly sample two clips from a video and use FAME to generate new clips by composing the original foreground onto various backgrounds from other videos. We then feed the augmented clips into the existing contrastive learning framework and perform self-supervised pretraining.}
    \label{fig:moco}
\end{figure}
Unfortunately, the existing video benchmarks present severe static bias~\cite{he2016human, li2018resound,choi2019sdn}, which induces the model to focus on the static and straightforward cues like background context instead of the motion patterns that are intrinsically more informative.
Therefore, when generalizing to novel benchmarks, the scene-biased model struggles to recognize some unseen action classes in the seen background.
For instance, the action of ``yoga'' in the pretrain dataset often happens in the gym, so the pretrained model inclines to bind the yoga action to the gym scene blindly but fails to understand the essence of the ``yoga'' action. On the other hand, when the unseen action class of ``push-up'' also occurs in the gym in the downstream dataset, the pretrained model has difficulty adapting to a new action that takes place in the same scene.
In short, previous models tend to focus on background context but neglect moving foreground. This phenomenon is more severe in self-supervised learning since the model is trained without semantic supervision, i.e., action labels.

Driven by the motivation of mitigating the background bias in the self-supervised learning framework, BE~\cite{wang2021removing} proposes a simple method of adding a specific static frame on all other frames in the video.
By doing so, their method can reduce the learning emphasis on the background to some extent and make the model generalize better.
However, this simple operation is relatively coarse and does not take good care of the foreground region. Although adding one static frame distracts the background pixels as expected, it damages the appearance and the motion patterns of the foreground objects at the same time, leading to impaired temporal modeling and motion understanding. To ameliorate the aforementioned limitation and help the model better grasp the foreground action, we present a new augmentation technique named \textbf{F}oreground-b\textbf{a}ckground \textbf{Me}rging (FAME).
Particularly, we separate each video's foreground and background regions, and then merge foreground areas with random backgrounds. In the separation step, we first circle out the edge region of the moving object as the seed region via frame difference. Then, we use color statistics to extrapolate the entire foreground area from the seed region. This efficient foreground discovery method extracts most areas of dynamic on which we expect the model to put the emphasis. Next, in the merging step, we fuse the extracted foreground regions of each video with random backgrounds from other videos to form action samples. This merging step aims to reduce the influence of the original background by introducing diverse backgrounds.
After that, we force the model to learn the consistent representation between original clips and distracting clips.
The training pipeline is shown in Figure~\ref{fig:moco}.
In this way, the model is asked to prioritize the motion patterns and alleviate the background bias in the self-supervised learning framework.

We evaluate FAME on two action recognition benchmarks. The superior experimental performance verifies that FAME enables self-supervised video representation learning to generalize better and distill the motion-aware representations. In short, we summarize our contributions as follows:
\begin{itemize}
    \item We propose a simple yet effective augmentation method for self-supervised video representation learning. Our approach helps the model mitigate background bias in video benchmarks and learn the motion-aware representations.

    \item Our method enhances the MoCo framework remarkably and achieves the start-of-the-art performance on two downstream tasks, action recognition and video retrieval, on two mainstream datasets, UCF101 and HMDB51.

\end{itemize}

\section{Related Work}
\noindent{\textbf{Contrastive Visual Representation Learning.}}
Recently, contrastive learning has greatly facilitated self-supervised visual representation learning~\cite{wu2018unsupervised,oord2018representation,tian2019contrastive,chen2020simple,he2020momentum}. It performs instance discrimination in a fully self-supervised manner to pull the representations of the same instance close and push those of different instances far away. Following this idea, \citet{wu2018unsupervised} proposes to formulates the instance discrimination as a non-parametric classification problem.
\citet{oord2018representation} mathematically proves that we could estimate mutual information with InfoNCE loss~\cite{gutmann2010noise}, which can be easily used for optimization. Later, \citet{he2020momentum} proposes MoCo
to make use of key representations calculated in previous iterations as negative samples to facilitate contrastive learning. Also, SimCLR~\cite{chen2020simple} proposes to employ a large batch size instead of the memory bank to expand the negative pool for more robust visual representation. Considering that SimCLR requires tremendous computational resources, we adopt the MoCo framework as a strong baseline for self-supervised pretraining in our work.

\noindent{\textbf{Self-supervised Video Representation Learning.}}
In video representation learning, there has been a line of works that employ diverse pretext tasks for self-supervised representation learning~\cite{misra2016shuffle,lee2017unsupervised,xu2019self}. The most prevalent approaches include temporal order prediction~\cite{misra2016shuffle}, video colorization~\cite{vondrick2018tracking}, spatio-temporal puzzling~\cite{kim2019self} and speed prediction~\cite{benaim2020speednet}. These methods generally employ manually designed tasks to seek the spatio-temporal cues in video data, but the performance is limited. Then, for further improvement, some works apply contrastive learning formulation into video representation learning~\cite{gordon2020watching,qian2021spatiotemporal}. Han et al. use InfoNCE loss to guide dense predictive coding in videos~\cite{han2019video,han2020memory}. \citet{asano2019self,han2020self} propose to leverage the consistency between different modalities to enhance video representation.
However, the video representations learned from these methods are mostly dominated by the background instead of the dynamic motions~\cite{wang2021removing}, which introduces strong background bias and impairs generalization ability in downstream applications. Therefore, we now propose FAME to construct positive samples with the same motions but different backgrounds for self-supervised pretraining in this work.

\noindent{\textbf{Video Background Bias Mitigation.}}
How to mitigate the background bias has been a long-standing topic for action recognition.
In the supervised scenario,
\citet{choi2019sdn} use an off-the-shelf human detector to mask out the human regions and train the model in an adversarial manner. Later, to make the self-supervised video representations more robust to the background bias, a line of works employ other natural supervision~\cite{huang2021self, xiao2021modist} to guide the model to capture motion information explicitly. However, these methods require more than one backbone to pretrain multi-modality data, resulting in an undesired computational cost. To better leverage the implicit motion information in videos, DSM~\cite{wang2021enhancing} aims to decouple the motion and context by deliberately constructing the positive/negative samples through spatial and temporal disturbance. BE~\cite{wang2021removing} proposes to add a static frame as background noise for static bias mitigation. But these two methods would erode the foreground moving objects, while our FAME meticulously extracts foreground regions and preserves high-quality motion patterns.

\begin{figure*}[!ht]
    \centering
    \includegraphics[width=0.9\linewidth]{Figure/FA.png}
    \caption{Separation stage of Foreground-background Merging. Given a video clip $X$, we can obtain seed region $S$ by the equation~\eqref{seed}. Then, we expand the seed region into the foreground region using color statistic assumptions.}
    \label{fig:FA}
\end{figure*}
\section{Approach}
In this section, we introduce our Foreground-background Merging (FAME) method. In section 3.1, we first revisit the vanilla contrastive learning framework
and then illustrate how our method is applied in this framework. In section 3.2, we elaborate on how to separate foreground regions using our method.
To clarify the notation, we denote the video clips as $X \in R^{C\times T \times H  \times W}$, where $C, T, H, W$ are respectively the dimension of the channel, timespan, height, width.

\subsection{Vanilla Contrastive Learning}

The vanilla contrastive learning approach employs instance discrimination to learn the feature representation in a fully self-supervised manner~\citep{chen2020simple,he2020momentum,grill2020bootstrap}. Generally, it aims to maximize the similarity between the query sample $q$ and its positive keys $k^+$, and minimize the similarity between $q$ and negative keys $k^-$. We empirically use InfoNCE loss~\cite{gutmann2010noise} for optimization:
\begin{align}
    \mathcal{L}_{nce} = -\log\frac{\sum_{k\in\{k^+\}}\exp(\text{sim}(q,k)/\tau)}{\sum_{k\in\{k^+,k^-\}}\exp(\text{sim}(q,k)/\tau)},
    \label{eq:infonce}
\end{align}
where $\tau$ is the temperature hyper-parameter controlling the concentration level of the distribution, and $\text{sim}(q,k)$ measures the cosine similarity between the latent embeddings, i.e., $\text{sim}(q,k)=q^Tk/(\norm{q}_2\norm{k}_2)$. In most existing works~\citep{feichtenhofer2021large}, $k^+$ is the set of clip embeddings extracted from the same video as $q$, and $k^-$ is the set of clip embeddings extracted from other videos.

However, this vanilla contrastive learning formulation in the video domain cannot fully utilize the dynamic motion information and tends to discriminate different instances according to the background cues~\citep{wang2021removing}. If the model depends excessively on the background but ignores the foreground object, such a misleading focus can risk the model's generalization ability.
Thus, we carefully design FAME as an augmentation technique to circumvent the negative impact of the background. We show the contrastive learning framework with FAME in Figure~\ref{fig:moco}. In detail, we randomly sample two clips from different timestamps. Before applying the basic augmentation, we use our proposed FAME method to compound the foreground of one clip with the background from other videos in the same mini-batch. After that, the two clips only resemble each other in the moving foreground objects but differ in the background context. Then, we feed these two clips into the 3D encoder and treat them as the positive keys while the rest of the clips serve as negative keys. Finally, we minimize the InfoNCE loss to pretrain the 3D encoder. By constructing the positive pair with the same foreground but diverse backgrounds, we guide the model to focus on temporal dynamics and suppress the impact of the background.

\subsection{Foreground-background Merging}

Motivated by mitigating background bias in self-supervised video representation learning, we intend to retain the foreground regions in original videos and shuffle the background areas among various videos. To achieve this goal, we propose the Foreground-background Merging method to augment the clips with minimal computation overhead.
Concretely, FAME consists of two stages, one for separation and the other for merging.

In the separation stage, we show the pipeline in Figure~\ref{fig:FA}. We first differentiate adjacent frames iteratively and then sum up the magnitude of the difference along channel and timespan dimensions to generate the seed region $S$. We formulate $S \in \mathbb{R}^{H  \times W}$ as
\begin{equation}
    S = \frac{1}{T-1}\sum_{c=1}^{C}\sum_{t=1}^{T-1} \norm{X_{c,t+1}-X_{c,t}}_1.
    \label{seed}
\end{equation}
The intuition is that moving foreground objects tend to possess a great magnitude in terms of frame difference, while the static backgrounds are minor in this metric. In practice, we find that the large values of the seed region $S$ usually correspond to the moving objects' edge region.
To expand the edge of the foreground objects into the whole foreground, we take inspiration from the unsupervised foreground discovery~\cite{stretcu2015multiple} for seed propagation. Specifically, we leverage the color
distributions to estimate the entire foreground. Denoting $N^{(F)}$ as the total number of pixels in the foreground region and $N_{x}^{(F)}$ as the number of the given color $x$ appearing in the foreground region, the probability of a given color $x$ appearing in the foreground region can be estimated as $P(x\mid F)=N_{x}^{(F)}/N^{(F)}$. Similarly, the probability of $x$ belonging to the background region is $P(x\mid B)=N_{x}^{(B)}/N^{(B)}$. In practice, we sample the foreground color distribution in the top $50\%$ of seed region $S$ and the background color distribution in the last $10\%$ of seed region $S$. Namely, in our setting, $N^{(F)}=\round{0.5\times H \times W}$ and $N^{(B)}=\round{0.1 \times H \times W}$. Given the above two distributions for the color $x$ and the assumption that all pixels with the same color have the same probability of being the foreground and background, we
approximate the foreground likelihood for a given color $x$ as
\begin{equation}
    \displaystyle P(F \mid x)= \frac{P(x \mid F)}{P(x \mid F)+P(x \mid B)}.
\end{equation}
Therefore, the soft segmentation mask $\displaystyle M \in \mathbb{R}^{H  \times W}$ can be calculated based on the color of each pixel. We formulate it as $\round{M}_{ij} = \displaystyle P(F \mid x_{ij})$, where $x_{ij}$ is the color at pixel $(i,j)$. To better filter out the background region, we binarize the mask
\begin{equation}
    \displaystyle \round{\widetilde{M}}_{ij} =
    \left\{
    \begin{aligned}
    1 & \text{,  if} \round{M}_{ij} \text{is among Top-$\round{\beta H W}$ of $M$,}   \\
    0 & \text{, otherwise,}
    \end{aligned}
    \right.
\end{equation}
where $\beta \in [0,1]$ is a hyper-parameter to describe the portion of the foreground.
Note that the mask we generate is constant with respect of timespan $T$ for the sake of computational efficiency. To do so, we view video clips as ``image'' when counting the color statistics.
In other words, we reduce $X$ over the timespan dimension, i.e., $\widetilde{X}=\sum_{t=1}^{T} X_{t}/T$. We have tried three variants to obtain the foreground mask $\widetilde{M}$. Experiment results in Table~\ref{tab:variants} verify that FAME works best among them. Moreover, all variants of FAME consistently enhance the representation ability greatly. It conforms to our intuition that retaining the original foregrounds and shuffling the backgrounds stimulate motion understanding.
Having foreground mask $\widetilde{M}$, we then conduct the stage of merging. Denoting $X,Y$ as foreground and background source clips, the synthetic clip $X_{\text{merge}} = X\otimes\widetilde{M}+Y\otimes(1-\widetilde{M})$, where $\otimes$ is the element-wise multiplication. As a result, we merge the moving foreground objects with random backgrounds.

\section{Experiments}
In this section, we first introduce the dataset and the implementation details for our experiments. Then, we report our evaluation results on downstream tasks: action recognition, video retrieval. Next, we conduct a set of ablation studies to analyze and validate our FAME method quantitatively. Finally, we investigate and make sense of what the model learns with FAME qualitatively.

\subsection{Datasets}
\subsubsection{Kinetics-400.} Kinetics-400~\cite{carreira2017quo} is a large-scale and high-quality dataset for action recognition, which consists of around 240K video clips with 400 human action classes. Each clip lasts about 10 seconds and is annotated with a single action class. We use the training set of Kinetics-400 to pretrain our model in a self-supervised manner.
\subsubsection{UCF101.}
UCF101~\cite{soomro2012ucf101} is a human action dataset. It contains over 13k clips covering 101 action classes. In our experiment, split 1 of UCF101 is used for pretrain and downstream tasks.
\subsubsection{HMDB51.}
HMDB51~\cite{kuehne2011hmdb} is a human action dataset with 51 action categories and around 7,000 manually annotated clips. We also use split 1 of HMDB51 in our experiments.

\subsection{Implementation Details}
\subsubsection{Self-supervised Pretraining.}
In the stage of self-supervised training,
we apply our FAME method on MoCo framework ~\cite{he2020momentum, chen2020improved}. We select two common backbone choices, R(2+1)D-18~\cite{tran2018closer} and I3D-22~\cite{carreira2017quo}, as the 3D encoder.

First, we randomly sample two different temporal clips in the same video as positive pair. Each clip consists of 16 frames with a temporal stride of 2. We spatially crop a random portion of clips and resize it to the size of $224 \times 224$ or $112 \times 112$. We then use FAME to distract one out of the positive pairs. Notice that the background videos are from the clips in the same mini-batch. Next, following the prior work~\cite{feichtenhofer2021large}, we perform the basic augmentation containing random grayscale, color jittering, random horizontal flip, and random Gaussian blur. All these augmentations are temporally consistent.
We pretrain the model for 200 epochs with a batch size of 64 on 8 Tesla V100 GPUs during the training phase. The SGD optimizer is adopted with the initial learning rate of $10^{-2}$ and weight decay of $10^{-4}$. For the implementation of MoCo, the number of the negative queue is set to 65536 for Kinetics-400, and 2048 for UCF101, respectively. We also swap the key/queue samples so that every sample can generate the gradient for optimization. More details about implementation are in Appendix.

\begin{table*}[!h]
\small
\centering
    \begin{tabular}{llllllll}
        \toprule

        Method & Backbone & Pretrain Dataset & Frames & Res. & Freeze & UCF101 & HMDB51 \\\hline
        CBT~\cite{sun2019learning} & S3D & Kinetics-600 & 16 & 112 & \Checkmark & 54.0 & 29.5 \\
        CCL~\cite{kong2020cycle} & R3D-18 & Kinetics-400 & 16 & 112  & \Checkmark & 52.1 & 27.8 \\
        MemDPC~\cite{han2020memory} & R3D-34 & Kinetics-400 & 40 & 224 & \Checkmark & 54.1 & 30.5\\

        RSPNet~\cite{chen2021rspnet} & R3D-18 & Kinetics-400 & 16 & 112  & \Checkmark & 61.8 & \textbf{42.8} \\
        MLRep~\cite{qian2021enhancing} & R3D-18 & Kinetics-400 & 16 & 112 & \Checkmark & 63.2 & 33.4 \\ \hdashline
        MoCo~\cite{chen2020improved} & R(2+1)D &  Kinetics-400 & 16 & 112 & \Checkmark & 67.4 &	39.8 \\
        \textbf{MoCo+FAME (Ours)} & R(2+1)D &  Kinetics-400 & 16 & 112 & \Checkmark & \textbf{72.2}(4.8$\uparrow$) &	42.2(2.4$\uparrow$) \\
        \hline
        \hline
        Shuffle~\cite{misra2016shuffle} & AlexNet & UCF101 & 3 & 256 & \XSolidBrush & 50.2 & 18.1 \\
        OPN~\cite{lee2017unsupervised} & VGG &  UCF101 & 4 & 80 & \XSolidBrush & 59.8 & 23.8 \\
        VCP~\cite{luo2020video} & R(2+1)D & UCF101 & 16 & 112  & \XSolidBrush & 66.3 & 32.2 \\

        PRP~\cite{yao2020video} & R(2+1)D & UCF101 & 16 & 112 & \XSolidBrush & 72.1 & 35.0\\
        IIC~\cite{tao2020self} & C3D & UCF101 & 16 & 112 & \XSolidBrush & 72.7 & 36.8 \\
         TempTrans~\cite{jenni2020video} & R(2+1)D & UCF101 & 16 & 112  & \XSolidBrush & 81.6 & 46.4 \\
         3DRotNet~\cite{jing2018self} & R3D-18 & Kinetics-400 & 16 & 112  & \XSolidBrush &62.9 &33.7\\
         Spatio-Temp~\cite{wang2019self} & C3D & Kinetics-400 & 16 & 112 & \XSolidBrush & 61.2 & 33.4
        \\

        Pace Prediction~\cite{wang2020self} & R(2+1)D & Kinetics-400 & 16 & 112  & \XSolidBrush & 77.1 & 36.6
        \\
        MemDPC~\cite{han2020memory}& R3D-34 & Kinetics-400 & 40 & 224 & \XSolidBrush & 78.1 & 41.2 \\
        SpeedNet~\cite{benaim2020speednet} & S3D-G & Kinetics-400 & 64 & 224  & \XSolidBrush & 81.1 & 48.8\\
        VideoMoCo~\cite{pan2021videomoco} & R(2+1)D & Kinetics-400 & 32 & 112 & \XSolidBrush & 78.7 & 49.2
        \\
        RSPNet~\cite{chen2021rspnet} & R(2+1)D & Kinetics-400 & 16 & 112  & \XSolidBrush & 81.1 &44.6 \\

        MLRep~\cite{qian2021enhancing} & R3D-18 & Kinetics-400 & 16 & 112 & \XSolidBrush & 79.1 & 47.6 \\

        ASCNet~\cite{huang2021ascnet} & R3D-18 & Kinetics-400 & 16 & 112 & \XSolidBrush & 80.5 & 52.3 \\
        SRTC~\cite{zhang2021incomplete} & R(2+1)D &  Kinetics-400 & 16 & 112 & \XSolidBrush & 82.0 & 51.2 \\\hdashline
        MoCo~\cite{chen2020improved} & R(2+1)D & Kinetics-400 & 16 & 112 & \XSolidBrush & 82.9 &	50.7\\
        \textbf{MoCo+FAME (ours)} & R(2+1)D & Kinetics-400 & 16 & 112 & \XSolidBrush &	 \textbf{84.8}(1.9$\uparrow$)	& \textbf{53.5}(2.8$\uparrow$) \\
        \hline
        DSM~\cite{wang2021enhancing}& I3D & Kinetics-400 & 16 & 224 & \XSolidBrush & 74.8 & 52.5 \\

        MoCo+BE~\cite{wang2021removing}& I3D & Kinetics-400 & 16 & 224 & \XSolidBrush & 86.8 & 55.4 \\

        \textbf{MoCo+FAME (ours)} & I3D & Kinetics-400 & 16 & 224 & \XSolidBrush & \textbf{88.6} & \textbf{61.1} \\
        \bottomrule
    \end{tabular}
    \caption{Comparison with the existing self-supervised video representation learning methods for action recognition on UCF101 and HMDB51. To compare fairly, we list each work's setting, including backbone architecture used, pretrain dataset and spatial-temporal resolution. Freeze (tick) indicates linear probe, and no freeze (cross) means finetune.}
    \label{tab:recognition}
\end{table*}

\subsubsection{Action Recognition.} After pretraining, we initialize the backbone with the pretrained parameters except for the last fully connected layer. There are two protocols of action recognition to validate the self-supervised representations. One is \textit{linear probe}. The encoder is frozen, and we only train the last fully connected layer. The second one is \textit{finetune}, where we train the whole network in a supervised fashion.
During the inference phase, we take the standard evaluation protocol~\cite{xu2019self, wang2020self, pan2021videomoco}. We uniformly sample ten $16$-frame video clips with a temporal stride of 2 from each testing video, then center crop and resize them to $224 \times 224$ or $112 \times 112$. We average the prediction of each testing video clip and report Top-1 accuracy to measure the action recognition performance.
\subsubsection{Video Retrieval.}
Without further training, we directly use the representation from the pretrained encoder for evaluation. Following~\citet{xu2019self, luo2020video}, we take video clips in the test set to query $k$ nearest neighbors in the training set. In detail, we average the representation of ten uniformly sampled clips to obtain the global representation. If the category of the testing clip appears in the $k$ nearest neighbors, it counts as a hit. We report Top-$k$ recall R@k for evaluation.

\subsection{Evaluation on Downstream Tasks}

\subsubsection{Action Recognition.} We compare our method with the existing self-supervised video representation learning approaches on action recognition. In Table~\ref{tab:recognition}, we report Top-1 accuracy on UCF101 and HMDB51. We do not consider the existing methods with a deeper backbone or non-single modality, e.g., optical flow, audio, and text.

\begin{table*}
\centering
    \begin{tabular}{lllllll}
        \toprule
        \multirow{2}{*}{Method} & \multirow{2}{*}{Backbone}  & \multicolumn{5}{c}{R@k}  \\\cline{3-7}
        &&R@1 & R@5 & R@10 & R@20 & R@50\\\hline
        SpeedNet~\cite{benaim2020speednet} & S3D-G &  13.0 & 28.1 & 37.5 & 49.5 & 65.0\\
        MLRep~\cite{qian2021enhancing} & R3D-18 &  41.5 & 60.0 & 71.2 & 80.1  & - \\
        ASCNet~\cite{huang2021ascnet} & R3D-18 &  58.9 & 76.3 & 82.2 & 87.5 & 93.4\\
        \textbf{FAME (ours)} & R(2+1)D & 62.3 & 75.1 & 80.9 & 86.9 & 93.0\\
        \bottomrule
    \end{tabular}
    \caption{Comparison with the existing self-supervised video representation learning methods for video retrieval. All methods are pretrained on Kinetics-400. We report the Top-$k$ recall R@k when k=1, 5, 10, 20, 50 on UCF101.}
    \label{tab:retrieval}
\end{table*}
Our method obtains the best result on UCF101 and a comparable result on HMDB51 in the linear probe setting. Even though MoCo~\cite{chen2020improved} serves a strong baseline and outperforms most previous methods, our FAME could still improve MoCo baseline by 4.8\% and 2.4\% respectively on UCF101 and HMDB51. MoCo+FAME also beats MLRep~\cite{qian2021enhancing}, which carefully designs the multi-level feature optimization and temporal modeling, by a large margin, i.e., about 9.0\% gain on both UCF101 and HMDB51. The outstanding performance demonstrates that our method can capture the moving foreground patterns and represent the temporal information.

In the finetune protocol, FAME with R(2+1)D backbone also achieves the best result on UCF101 and HMDB51. It indicates that FAME helps MoCo learn the scene-debiased and motion-aware representations on the Kinetics-400 dataset, which generalize well to the downstream dataset. Remarkably, FAME brings 1.9\% and 2.8\% performance gain on UCF101 and HMDB51 against the MoCo baseline. In comparison to other state-of-the-art methods, although SRTC~\cite{zhang2021incomplete} introduces two additional sub-loss terms to regularize the self-supervised pretraining, our simple formulation outperforms SRTC by $2.8\%$ and $2.3\%$ with the same backbone R(2+1)D.
Notably, we share similar motivation with BE~\cite{wang2021removing}, both hoping to alleviate the background bias problem by disturbing the background. BE directly adds a static frame to every other frame and regards this distracting video as the positive pair to the original video. This coarse disturbance ruins the foreground region, and our experiments testify that it does hurt the temporal modeling and motion understanding.
Using the same backbone I3D, FAME outperforms BE by 1.8\% and 5.7\% on UCF101 and HMDB51, respectively. It proves that our model can better mitigate the static background bias by further separating foreground and background.

\subsubsection{Video Retrieval.}
We report the performance comparison for video retrieval in Table~\ref{tab:retrieval}. Our method achieves significant performance gain on R@1. Remarkably, though ASCNet~\cite{huang2021ascnet} devises two particular tasks to learn appearance and speed consistency, we still reach higher Top-1 retrieval accuracy. It is because our motion-aware representations can precisely retrieve the action with the same semantics. Ours is slightly lower than ASCNet in R@5 to R@50 since FAME almost abandons background cues, while ASCNet may take trace of background shortcuts to retrieve samples of the same category when $k$ is large.

\subsection{Ablation Study}
In this section, we conduct thorough ablation studies to analyze how FAME improves self-supervised video representation learning. We choose split 1 of UCF101 as the pretrain dataset and I3D as the backbone for computational efficiency. All the Top-1 accuracy in the ablation study is measured under the protocol of finetune.
\begin{table}
    \centering
    \begin{tabular}{lll}
         \bottomrule $\beta$ & UCF101 & HMDB51 \\\hline
         1.0 (baseline) & 75.8 & 45.5 \\
         0.7 & 80.3 & 49.6  \\
         0.5 & 81.2  & \textbf{52.6} \\
         0.3 & \textbf{82.0}  & 51.6  \\
        \bottomrule
    \end{tabular}
    \caption{Top-1 accuracy with respect to $\beta$ on UCF101 and HMDB51. $\beta=0.3$ is optimal for UCF101 and $\beta=0.5$ achieves best on HMDB51.}
    \label{tab:beta}
\end{table}
\subsubsection{Importance of the area ratio of the foreground region.}
In this section, we inspect how the area of the foreground region contributes to the representation quality. We ablate $\beta$, the portion of the foreground, in the range of $\{1, 0.7, 0.5, 0.3\}$. Note that $\beta=1$ reverts to the baseline method without applying FAME. We report the performance in Table~\ref{tab:beta}.
It can be observed that the results of $\beta=0.3$ and $0.5$ vastly outperform baseline by $\sim 6\%$ on both UCF101 and HMDB51. The improvement of $\beta=0.7$ is also considerable, though slightly inferior to the smaller value of $\beta$ due to insufficient background replacement. This phenomenon shows that our method is insensitive to the hyper-parameter $\beta$, and the harder contrastive task formulated by FAME is more conducive to the representation quality.

\subsubsection{Impact of background source.}
Besides the foreground ratio, we also wonder how the source of background affects the representation ability to capture the motion.
Specifically, we aim to explore whether the performance would change dramatically using the background in the same video instead of other videos.
We perform an additional experiment where we merge the foreground of one video with the background sampled at different timestamps of the video itself. As shown in Table~\ref{tab:intra-inter}, we find that using the background from intra-video boosts the baseline with 1.6\% and 2.1\% improvement on UCF101 and HMDB51 and the introduction of other videos' backgrounds brings further improvement, i.e., 5.4\% and 7.1\% gain on UCF101 and HMDB51. In general, the intra-video background is almost the same as the original one, while the inter-video background is quite distinct. Thus, it demonstrates that the modification from the intra-video is not adequate to mitigate background bias while replacing the background with diverse scenes better strengthens motion pattern learning.

\begin{table}
    \centering
    \begin{tabular}{lll}
         \bottomrule Background  & UCF101 & HMDB51 \\\hline
         none & 75.8 & 45.5 \\
         intra-video & 77.4(1.6$\uparrow$) & 47.6(2.1$\uparrow$) \\
         inter-video & 81.2(5.4$\uparrow$) & 52.6(7.1$\uparrow$) \\
        \bottomrule
    \end{tabular}
    \caption{Top-1 accuracy on UCF101 and HMDB51 in terms of intra-/inter-video background. Using the inter-video background increases the accuracy more greatly than using the intra-video background.}
    \label{tab:intra-inter}
\end{table}
\subsubsection{Stronger background debiasing.}
To explore whether FAME is sufficiently strong to reduce the background bias in the contrastive learning, we design a stronger contrastive objective. That is, we apply FAME on both branches of MoCo and neither of the two processed video clips contains initial background information. We report the results in Table~\ref{tab:single}.
The slight difference in performance between those two settings proves that our default setting is strong enough for the model to learn the scene-debiased representations.
\begin{table}
    \centering
    \begin{tabular}{lllll}
         \bottomrule & \multicolumn{2}{c} {UCF101} & \multicolumn{2}{c} {HMDB51} \\
         $\beta$ & single & both & single & both \\\hline
         $0.7$ & 80.3 & 79.6 & 49.6 & 50.8 \\
         $0.5$ & 81.2 & 81.2 & 52.6 & 51.4 \\
         $0.3$ & 82.0 & 81.1 & 51.6 & 53.1 \\
        \bottomrule
    \end{tabular}
    \caption{Top-1 accuracy on UCF101 and HMDB51. We denote the operating FAME on single branch (default setting) as \textit{single} and the operating FAME on both branches as
    \textit{both}. Under various $\beta$, the corresponding value of single and both is approximately the same. }
    \label{tab:single}
\end{table}
\subsubsection{Variants of Foreground-background Separation.}
\begin{table}
    \centering
    \begin{tabular}{ccc}
    \bottomrule
      Method  &  UCF101  &  HMDB51  \\\hline
      baseline & 75.8 & 45.5 \\
      Gauss   & 77.9 & 46.4 \\
      Seed  & 80.4 & 51.3 \\
      Grid  & 81.5 & 51.5 \\
      FAME   & 81.2 & 52.6 \\\hdashline
      Grid$\dagger$ &86.5 &58.7 \\
      FAME$\dagger$ & 88.6 & 61.1 \\
    \bottomrule
    \end{tabular}
    \caption{Top-1 accuracy of various foreground-background separation methods on UCF101 and HMDB51. $\dagger$ indicates the pretrain dataset is Kinetics-400. All variants improve the baseline significantly and FAME performs best.}
    \label{tab:variants}
\end{table}

\begin{figure}[!h]
    \centering
    \includegraphics[width=\linewidth]{Figure/variants.png}
    \caption{The illustration about FAME and three variants of foreground-background separation.}
    \label{fig:variants}
\end{figure}
In order to verify that emphasizing moving foreground advances the motion understanding, we devise three variants of foreground mask: (i)  Gauss: we adopt a 2D Gaussian kernel matrix as the foreground mask. It derives from the assumption that videos are shot in the object-centric form.
(ii) Seed: we just take the seed region $S$ to characterize the foreground.
(iii) Grid: we separate foreground and background via the grid. In practice, the video is split into $4 \times 4$ grids spatially. We count the sum of $S$ in each grid and take the greatest eight grids as the foreground area.
A brief illustration is displayed in Figure~\ref{fig:variants}.

We compare FAME with these three variants in Table~\ref{tab:variants}. First, we note that all variants improve the baseline by a large margin, showing the effect of FAME. Furthermore, refining the foreground mask from Gauss, Seed, Grid to FAME continually increases the action recognition performance. Interestingly, we notice that Grid outperforms FAME slightly on UCF101.
We conjecture that it is because both the pretrain dataset and downstream dataset are UCF101. Thus, background bias can be leveraged as a shortcut for action recognition.
To delve into this phenomenon, we carry out an extra experiment on another pretrain dataset Kinetics-400. Top-1 accuracy of Grid variant is over 2\% lower than FAME on both UCF101 and HMDB51.
It indicates that a meticulous segmentation mask instead of a rough grid box is more effective in facilitating generalization ability.

\subsection{Visualization Analysis}
\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{Figure/cam.png}
    \caption{Class activation maps (CAM) visualization. Red areas indicate the important areas for the model to predict the action class. Comparing to MoCo, MoCo+FAME resists the impact of background and highlights the motion areas.}
    \label{fig:cam}
\end{figure}

To better demonstrate the effectiveness of FAME, we provide the CAM~\cite{zhou2016learning} visualization in Figure~\ref{fig:cam}. We train a linear classifier with the pretrained model similar to the linear probe and omit the last global average pooling layer to generate activation maps. With that, we can spot the contribution of each area and find crucial regions for discriminating the action class. We find that when integrated with FAME, MoCo can focus on moving foreground area rather than background context. For example, in the first row of Figure~\ref{fig:cam}, MoCo+FAME precisely captures the moving upper and lower body when the man is practicing TaiChi, while the MoCo baseline displays a dispersed highlight map and fails to attend to the motion area. In addition, we illustrate that the CAM activation map can almost overlap with the foreground mask generated by FAME. It testifies that our method enables the model to perceive the motion patterns and hinder the background bias.

\section{Conclusion}
In this work, we propose a new Foreground-background Merging (FAME) method to alleviate the background bias in self-supervised video representation learning.
Via Foreground-background Merging, we augment the original video by separating the foreground and background regions and fusing the original foreground with other videos' backgrounds. Then, the backbone model is trained to learn semantically consistent representation between the original video and the fused video. In this way, the model can learn the scene-debiased and motion-aware representations of videos. Experimental results on a bunch of downstream tasks manifest the effectiveness of our method.

\clearpage


\section{Introduction}
\input{sections/introduction}

\section{Method}
\input{sections/method}

\section{Experiments}
\input{sections/experiments}

\section{Conclusion}
\input{sections/conclusion}

\clearpage
\bibliographystyle{ACM-Reference-Format}

\subsection{Datasets}
\textbf{FEVER} \cite{thorne2018fever} is a crowdsourced fact verification dataset containing claim-evidence pairs based on Wikipedia articles. We only use the claims paired with a single evidence for training and evaluation in this work.

\noindent\textbf{Symmetric} \cite{schuster2019towards} is a test set based on the FEVER development dataset designed for unbiased evaluation. It eliminates the correlation of n-grams in the claim and labels by careful construction.

\noindent\textbf{Adversarial} \cite{Thorne2019adversarial} is an adversarially constructed dataset explicitly designed to induce errors in models trained on the FEVER dataset.

\noindent\textbf{Fool Me Twice (FM2)} \cite{eisenschlos2021fool} is a Wikipedia-based fact verification dataset composed of 13k claim-evidence pairs that are collected through the games among crowd-workers.

\subsection{Compared Methods}

\textbf{Data Augmentation Methods:}
We compare our method against two data augmentation techniques commonly used for various natural language processing tasks: Easy Data Augmentation (EDA) \cite{wei2019eda} and neural paraphrasing. EDA applies simple mutations, such as random swapping or synonym replacement, to the original sentence to generate new examples.
For neural paraphrasing, we use a GPT-2 model \cite{radford2019language} fine-tuned on back-translated data to paraphrase the original text \cite{krishna-etal-2020-reformulating}. For each original claim-evidence pair, we create a new pair that holds the same relation by transforming only the claim using these methods, leading to the augmentation ratio of augmented to original data 1:1.

\noindent\textbf{Regularization-based Debiasing Methods:}
We also compare with
two debiasing techniques that reduce the reliance on biases by regularizing the model on the biased samples.
The first one is an example re-weighting method that targets biases from the shortcut words \cite{schuster2019towards}. By re-weighting the importance of claims containing those words, it forces a model to focus on the hard examples in which relying on the bias results in incorrect predictions.
The other one is Product of Experts (PoE) \cite{mahabadi2020end}, which computes the training loss in an ensemble of the base model and the bias-only model. Similar to the first method, it controls the base model's loss depending on the prediction of the bias-only model for each example.

\subsection{Implementation Details}
For our experiments, we use the BERT-base-uncased model \cite{devlin2019bert}, which demonstrates competitive performance for fact verification tasks.
We fine-tune BERT with an additional layer on top of the [CLS] token embedding.
We concatenate the claim and the evidence, and insert [SEP] token in between to make the input sequence.
Following previous works, we set maximum sequence length to 128, batch size to 32, and optimize the model through a standard cross-entropy loss using the Adam optimizer~\cite{kingma2014adam} with a learning rate of 2e-5.
\minwoo{We train the model on the FEVER train set and evaluate the generalization performance using the development set for Symmetric, Adversarial and FM2 datasets.}
We train the model for 3 epochs with 5 different random seeds for all experiments and report the averaged result.
For our augmentation pipeline, we set the maximum span size for evidence modification $\tau$ to $3$, which produces an augmented dataset with an augmentation ratio of 1:0.58.

\subsection{Results on the Full Dataset}

First we augment the full FEVER train set with our approach and compare its performance in Table~\ref{tab:full_result}.
Our proposed method achieves 10.13\% improvement over the baseline and 3.6\% improvement over the previous state-of-the-art debiasing technique on the Symmetric dataset.
This result shows that our method is highly effective at preventing the model to predict from unnecessary biases.
Our approach also shows a 2.12\% improvement on the Adversarial dataset and 3.36\% improvement on the FM2 dataset compared to the baseline, indicating that our augmentation method benefits not just on the diagnostic dataset for lexical bias but for fact verification in general.
Finally, our method leads to the greatest overall improvement across the datasets out of all compared training methods,
This result empirically proves that the contrastive samples generated from our augmentation method enhance the factual reasoning capabilities by learning a more robust feature representation, achieving strong generalization.

Compared to our approach, the example re-weighting and PoE methods perform slightly worse on the Symmetric and Adversarial dataset and slightly better on the original FEVER development set and FM2 dataset.
On the other hand, EDA and paraphrasing augmentations show negligible performance improvement on the Symmetric dataset. These results suggest that simply training with more data does not necessarily help mitigate the bias in data.

\subsection{Ablation Studies}

We conduct ablation study to verify the effectiveness of the augmented samples generated from each step of our augmentation process.
The results shown in Table \ref{tab:full_result} reveal that a moderate performance improvement on the Symmetric, Adversarial and FM2 evaluation sets are attained even with only using the negative claims generated from the first step.
However, its performance on the Symmetric dataset is still significantly lower compared to the full augmentation method, implying that augmenting with negative claims alone is less effective for debiasing the model.

Training with the negative evidence augmented data exhibits a more competitive performance on the other hand, especially outperforming previous state-of-the-art technique on the Symmetric dataset.
The results imply that the key component of our debiasing approach is training the model with contrastive samples sharing the same claim, which enables the model to learn by comparing the claim to the input evidence instead of just relying on artifacts in the claim.
Nevertheless, the full augmentation method outperforms all ablations, indicating that contrastive data samples in general help the model learn more robust representations.

\subsection{Results on Low-resource Conditions}\label{SEC:LOW}

\minwoo{In the real world, the available training corpus size for fact verification is often lacking due to the expensive cost and difficulty of collecting and labelling the data.
Training on such a limited amount of data could lead to even more biased models due to the lack of samples to learn factual reasoning from.
Therefore, it is important to investigate whether the debiasing methods perform well in low-resource conditions.
}

Thus, we further evaluate our data augmentation method in low-resource conditions, simulating a data-scarce setting by sampling subsets of the original FEVER training data.
We perform a class-balanced sub-sample on the FEVER training data with 5 different random seeds, and train the model on each subset with another 5 different random seeds to account for statistical variations.
As in the previous experiments, we augment the subset with an augmentation ratio of 1:1 for EDA and back-translation methods, while our augmentation method results in an average augmentation ratio of 1:0.47.
We evaluate on the subsets of 0.1\% ($N=240$), 0.2\% ($N=483$), 0.5\% ($N=969$), and 1.0\% ($N=2,427$) of the original training data, and the results are presented in Figure \ref{fig:limited}. \\
\indent Our augmentation method shows a consistent improvement in the low-resource conditions over all evaluated datasets, and especially outperforms the baseline trained on the full dataset with just 1\% of the original training data for the Symmetric evaluation set. On the other hand, the PoE method shows little to no improvements on all of the datasets except FM2, and in some cases show a slight performance drop.
These results indicate that PoE, which relies on training a biased model to regularize learning from biased samples, do not generalize well in data-scarce settings.\\
EDA and paraphrasing augmentation show a moderate improvement over the baselines, verifying their effectiveness in low-resource conditions.
However, our augmentation method is still able to achieve a marked improvement over EDA and paraphrasing on Adversarial and FM2 datasets, implying that our augmentation approach is more robust and effective at generalizing to out-of-distribution samples.
In summary, the results with varying training dataset sizes show that our augmentation method is effective for low-resource domains.

\begin{figure}[t]
    \begin{center}
    \begin{tabular}{c}
      \includegraphics[width=0.8\columnwidth]{rsc/fig_limited_data_one_column.png}
    \end{tabular}
    \vspace{-2mm}
    \caption{Experimental results on varying training set size with different training methods. Train set sampling is done with five different seeds and the mean accuracy is reported.

    }
    \vspace{-4mm}
    \label{fig:limited}
    \end{center}
\end{figure}

\subsection{Task Formulation}
Given a textual claim $c$ and evidence $e$, the objective of the fact verification task is to predict whether the claim is supported by the evidence, refuted by the evidence, or the evidence has not enough information for verification.
We denote a sample in the dataset $\mathcal{D}$ of size $N$ as a triplet $(c,e,y) \in \mathcal{D}$, where $y \in \{SUP, REF, NEI\}$ is the data label.

\begin{table*}
  \caption{Experimental results on fact verification datasets. The mean and the standard deviation of the classification accuracy over five different runs are reported for each method.}
  \vspace{-2mm}
  \label{tab:full_result}
  \begin{tabular}{l|cc|cc|cc}
    \toprule
    \textbf{Train method} & \textbf{FEVER dev} & \textbf{Symmetric} & \textbf{Adversarial} & \textbf{FM2 dev} & \textbf{$\Delta$ sym.} & \textbf{$\Delta$ avg.} \\
    \midrule
    No augmentation (baseline) & 86.15 $\pm$ 0.42 & 58.77 $\pm$ 1.29 & 49.66 $\pm$ 0.37 & 40.81 $\pm$ 0.43 & - & - \\
    EDA & 85.09 $\pm$ 0.25 & 58.55 $\pm$ 1.63 & 51.41  $\pm$ 1.14 & 41.21 $\pm$ 1.11 & $-$0.22\% & +0.22\% \\
    Paraphrasing & 84.33 $\pm$ 0.34 & 59.02 $\pm$ 1.38 &  \textbf{52.53} $\pm$ 1.20 & 40.60 $\pm$ 0.71 & +0.25\% & +0.27\% \\
    \midrule
    Re-weighting & 85.56 $\pm$ 0.32 & 61.87 $\pm$ 1.16 & 49.92 $\pm$ 0.80 & 43.80 $\pm$ 0.46 & +3.10\% & +1.44\% \\

    Product of Experts (PoE) & \textbf{86.50} $\pm$ 0.35 & 65.30 $\pm$ 1.73 & 51.07 $\pm$ 1.20 & \textbf{46.69 $\pm$ 1.11} & +6.53\% & +3.54\% \\
    \midrule
    CrossAug (ours) & 85.34 $\pm$ 0.68 & \textbf{68.90 $\pm$ 1.68} &51.78 $\pm$ 1.02 & 44.17 $\pm$ 1.27 & \textbf{+10.13\%} & \textbf{+3.70\%} \\
     - Negative claim only augmentation & 85.70 $\pm$ 0.28 & 61.00 $\pm$ 0.71 & 51.96 $\pm$ 0.90 & 43.06 $\pm$ 0.40 & +2.23\% & +1.58\% \\
     - Negative evidence only augmentation & 85.87 $\pm$ 0.16 & 67.06 $\pm$ 0.99 & 51.46 $\pm$ 0.43 & 43.70 $\pm$ 0.97 & +8.29\% & +3.18\% \\
    \bottomrule
  \end{tabular}
  \vspace{-1mm}
\end{table*}

\subsection{Data augmentation pipeline}
In our proposed method, we generate three additional synthetic samples for each original claim-evidence pair through two stages of augmentation. Note that CrossAug utilizes only positive claims (SUP claims), which are verifiable by specific evidence, in the FEVER dataset. The whole process of our data augmentation pipeline is shown in Figure \ref{fig_overall}.

\noindent\textbf{(1) Negative Claim Generation:}
The first stage is to generate a \textit{negative claim} $c'$ based on a positive claim $c$ by adopting a neural sequence-to-sequence model. This generative process involves transforming of a positive claim, such as inserting a negation or replacing a word with antonyms.
\tony{In turn, the generated claim has a different meaning so that it is refuted by the evidence $e$ supporting a positive claim, which is why we call this negative claim.}
Through this process, we form a new data sample $(c', e, REF)$.

To this end, we fine-tune BART \cite{lewis2020bart} on WikiFactCheck-English dataset, which provides pairs of positive claims and their corresponding negative claims \cite{sathe2020automated}. In fine-tuning, positive claims are used as the source text and negative claims are taken as the target text of the model. To provide a richer context, we also fine-tune the model with the source and target text reversed since a positive claim can be seen as a refuted version of a negative claim.

\noindent\textbf{(2) Evidence Modification:}
The negative claim generated in the first stage often only differs from the positive claim by a few words, and can thus be seen as a span replacement. For example, ``\textit{over 30 days}" is simply substituted with ``\textit{less than 10 days}", as shown in Figure \ref{fig_overall}. This phenomenon is due to the characteristics of the data used for fine-tuning the model: the negative claim was manually written by annotators under the constraints of both the sentence length and the subject to be similar to the positive claim \cite{sathe2020automated}. We also observe that the words replaced in the positive claim are often found verbatim in the evidence. This is because the changed part of the claim usually matches the factual information taken from the evidence.

As the second stage of our data augmentation process, we build upon these observations to perform a lexical search-based evidence modification. First, we compare the positive claim $c$ with the negative claim $c'$ to identify the changed part in the first stage. Once the words replaced in the positive claim are recognized (``over 30 days" in Figure \ref{fig_overall}), we search for the same words in the evidence $e$ and replace them with the substituted words in the negative claim (``less than 10 days" in Figure \ref{fig_overall}). Since this substitution induces the same factual modification on the evidence as that applied to the negative claim, it can be logically concluded that the resulting modified evidence $e'$ supports the negative claim $c'$ and refutes the positive claim $c$.
Consequently, we form two additional contrastive samples $(c, e', REF)$ and $(c', e', SUP)$ in the second stage.

\smallskip
\noindent\textbf{Exceptional Cases:}
In the first stage, the generated negative claim $c'$ is occasionally the exact same as the positive claim $c$. For such samples, we skip performing augmentation.
Also, in the second stage, we carry out the evidence modification only when the number of replaced words in the claims are less than or equal to $\tau$, where $\tau$ is a threshold value. This is necessary to prevent invalid evidence modification. When the replaced part is large, it frequently contains inappropriate terms for reconstructing the evidence, such as non-factual words, producing an illogical sentence. However, we still keep the sample $(c',e, REF)$ from the first stage even when the evidence modification stage is skipped.