\section{Introduction}\label{sec1}
The optimal fingering printing approach proposed in a series of papers starting from
\cite{AT99} (herein AT99) followed by \cite{AS03} and \cite{SA03/2}), has been the corner stone in the detection and assessment of climate change due to human activities in the last 20 years, and has been the adopted methods by numerous Intergovernmental Panel on Climate Change (IPCC)
 reports on the climate change.
Recently, \cite{M21} (M21) offered a very critical assessment on
\cite{AT99},  suggesting there were serious problems with the approach.

The criticism of \cite{M21} is well summarized in its abstract "AT99 stated the GM Theorem incorrectly, omitting a critical condition altogether, their GLS method cannot satisfy the GM conditions, and their variance estimator is inconsistent by construction. Additionally, they did not formally state the null hypothesis of the RCT (residual consistency test) nor identify which of the GM conditions it tests, nor did they prove its distribution and critical values, rendering it uninformative as a specification test.”

This paper provides a statistical review on AT99  in light of the critics of \cite{M21}, and makes remarks on the key aspects of the AT99 formulation.

We provide conditions under which the optimal fingerprint approach would stand, which are largely related to the null simulation of the climate models and the accuracy of the covariance  offered by the climate models as approximation to the residual covariance of the observations.

\section{Review on AT99's Formulation}\label{sec2}

The main idea of AT99 is to utilize the Gauss-Markov (GM) theorem for the homogeneous linear regression model to make the detected fingerprints “optimal”, which means in the statistical terms,  to obtain estimates of the regression coefficients with the smallest variation so as to  attain the largest signal (regression coefficient estimates) to noise (standard deviation of the estimates) ratio. AT99 tried to
utilize the GM theorem's ability to generate the Best linear unbiased estimator (BLUE) for the regression coefficients. The BLUE property would ensure the best signal to noise ratio in the detected human fingerprints on the climate change.

The road to BLUE is not a direct one, as the homogeneous variation conditions for the GM theorem  were not readily  satisfied.
AT99 suggested an approach that is well connected to the feasible generalized regression as pointed out by \cite{M21} with an approach that is uncommon to statisticians and econometricians in  estimating the residual covariance matrix  base on separate simulation of climate models.

\subsection{Model and Key Aspect of Estimation} \label{sec2.1}

The Linear regression model considered in AT99 is
\begin{equation}
\mathbf{y}=\mathbf{X} \boldsymbol{\beta}+\mathbf{u}\label{AT1}\tag{AT1}
\end{equation}
where $\mathbf{y}$ is the $\ell$-dimensional  vector of observations on certain climate variables, $\mathbf{X}$ is the design matrix of $\ell \times m$ corresponding to $m$ (climate) model-simulated response-patterns, which are commonly called covariates in statistics, and are obtained from ensembles of climate models.
Throughout this review, the original equation numbers of AT99 are labeled as (AT.numeric number), while other equations are labelled separately.

The $\mathbf{u}$ represents  the “climate noise” which was assumed to be multivariate normally distributed with the covariance matrix
\begin{equation}
\mathbf{C}_{N} \equiv \mathcal{E} (\mathbf{uu}^{T}). \label{AT2}\tag{AT2}
\end{equation}
Here $\mathcal{E}$ denotes  the mathematical expectation on random variables in the physical world of observations.

\bigskip
\noindent{\bf Remark 1.} AT99 did not present the regression identification condition
\begin{equation}
E(\mathbf{u}\mid\mathbf{X}) =0 \label{eq:iden1}
\end{equation}
which has drawn heavy criticism from \cite{M21}.  Our reading is that AT99 implicitly assumed such a condition.
However, as will be shown in {\bf Remark 8} below, (\ref{eq:iden1}) is not enough for the suggested approach.

The major idea of AT99 is  to utilize the GM theorem for attaining the best  linear unbiased estimator(BLUE) of $\boldsymbol{\beta}$, which can attain the smallest mean square error  in the estimated $\boldsymbol{\beta}$-coefficients and hence the highest signal to noise ratio (SNR)  in the fingerprint detection.

As $\mathbf{\mathbf{C}}_N \ne \sigma^2 \mathbf{I}_{\ell}$ due to the spatial and temporal heterogeneity in the residuals of the observations, to satisfy the variance condition that
$$E(\mathbf{uu}^{T})= \sigma^2 \mathbf{I}_{\ell}$$
of the GM theorem,  AT99 suggested a pre-whitening operator via a $\kappa' \times \ell $ matrix $\mathbf{P}$
 such that
\begin{equation}
\mathcal{E} \left(\mathbf{P u u}^{T} \mathbf{P}^{T}\right)=\mathbf{P C_{N}} \mathbf{P}^{T}=\mathbf{I}\label{AT3}\tag{AT3}
\end{equation}

\bigskip
\noindent{\bf Remark 2.} AT99 was vague about the dimension of $\mathbf{P}$ and $\mathbf{I}$ in \eqref{AT3}, which should be $\mathbf{I}_{\kappa'}$ where $\kappa'$ is the rank of $\mathbf{P}$ and   $ \kappa' \le \ell$.  In the original GM theorem, $\kappa'=\ell$.
There should be another restriction that $\kappa' > m$, the dimension of $\boldsymbol{\beta}$,
in order to make  $\mathbf{X}^T \mathbf{P}^ T \mathbf{P} \mathbf{X}$ used in the generalized least square (GLS) full rank, which was only implicitly assumed in AT99 to ensure positive degree of freedom in the $\chi^2$-distribution in  (AT18).

Ignoring the randomness of $\mathbf{P}$ for the moment, if $\mathbf{P}$ is known,
Gauss-Markov theorem would imply that by left multiplying  $\mathbf{P}$ on all both sides of (\ref{ATeq:cond1}),  the ordinary least square (OLS) estimator $\boldsymbol{\beta}$  on the rotated data is
\begin{equation}
\begin{aligned}
\tilde{\boldsymbol{\beta}}&=\left(\mathbf{X}^{T} \mathbf{P}^{T} \mathbf{P} \mathbf{X}\right)^{-1} \mathbf{X}^{T} \mathbf{P}^{T} \mathbf{P y}\\
&=\left(\mathbf{X}^{T} \mathbf{C}_{N}^{-1} \mathbf{X}\right)^{-1} \mathbf{X}^{T} \mathbf{C}_{N}^{-1} \mathbf{y} \equiv \mathbf{F}^{T} \mathbf{y}
\end{aligned}
\label{AT4}\tag{AT4}
\end{equation}
where $\mathbf{F} = \left(\mathbf{X}^{T} \mathbf{P}^{T} \mathbf{P} \mathbf{X}\right)^{-1} \mathbf{X}^{T} \mathbf{P}^{T} \mathbf{P}$, and
the covariance of $\tilde{\boldsymbol{\beta}}$  is
\begin{equation}
Var(\tilde{\boldsymbol{\beta}})=\left(\mathbf{X}^{T} \mathbf{C}_{N}^{-1} \mathbf{X}\right)^{-1}.
\label{AT6}\tag{AT6}
\end{equation}

\bigskip
\noindent {\bf Remark 3.} The $\mathbf{\mathbf{C}}_N^{-1} $ is understood as the Moore–Penrose generalized inverse of $\mathbf{\mathbf{C}}_N$, which is $ \mathbf{P}^T \mathbf{P}$, in the case of $\kappa' < \ell$.  A more general expression for the variance is
\begin{equation}
Var(\tilde{\boldsymbol{\beta}})=\left(\mathbf{X}^{T}  \mathbf{P}^{T} \mathbf{P} \mathbf{X}\right)^{-1}
\label{eq:AT6a}
\end{equation}

{A key formulation of AT99 is on estimation of $\mathbf{\mathbf{C}}_N$ via separate simulations of certain climate model that would be consistent to the null setting of no human forcing on the climate, namely the so-called null setting in the climate models.
Specifically,  }
$\mathbf{C}_{N}$ is estimated by
\begin{equation}
\hat{\mathbf{C}}_{N}=\frac{1}{n} \mathbf{Y}_{N} \mathbf{Y}_{N}^{T}\label{AT12}\tag{AT12}
\end{equation}
where the columns of $\mathbf{Y}_{N}$ ($\ell\times n$ matrix)  represent $n$ response vectors of size $\ell$ from $n$ ensembles of the simulation runs of the null setting.

\bigskip
\noindent{\bf Remark 4.}  This external way of estimating the covariance matrix $\mathbf{\mathbf{C}}_N$  was quite different from the internal way commonly practised in statistics and econometrics.  {\bf The $n$ ensemble runs should be mutually independent of each other, and should be independent to the design matrix $\mathbf{X}$ in \eqref{AT1}. }
A merit of this external estimation of $\mathbf{\mathbf{C}}_N$ is to make the estimation of the error covariance separate from the potential regression model misspecification (for instance omitted variables and  non-linearity).
Let $\tilde{\mathbf{C}}_N$ be the the underlying covariance matrix of the ensembled null simulations to which $\hat{\mathbf{C}}_N$ converges to in probability.
A drawback of this external approach is that it may be  challenging to make
$\tilde{\mathbf{C}}_N$ match $\mathbf{\mathbf{C}}_N$ of the physical world. This is actually the most important factor that determines the estimated fingerprint $\tilde{\boldsymbol{\beta}}$ being BLUE or not.

\bigskip
\noindent{\bf Remark 5. }  A key requirement needed here is that the null simulation experiments should be statistically independent of the observed physical world
governing Model \eqref{AT1} in order to ensure the key identification condition (\ref{ATeq:cond1})
 in {\bf Remark 8}. As will be shown later, this independence would be the key to ensure $\tilde{\boldsymbol{\beta}}$ being unbiased.

Due to the limitation of the ensemble runs of the climate models,
$\hat{\mathbf{C}}_{N}$'s rank is only a $\kappa < \ell$, making $\hat{\mathbf{C}}_{N}$ non-invertible.  By  resorting to the generalized inverse  of $\hat{\mathbf{C}}_{N}$, $\mathbf{P}$ can be  approximated by a $\mathbf{P}^{(\kappa)}$.
{Specifically, support  $\hat{\mathbf{C}}_N$
 admits the spectral decomposition that $\hat{\mathbf{C}}_N=\sum_{i=1}^r \lambda_i \nu_i \nu_i^T$ where ${\lambda}_{i=1}^{\kappa}$ are the positive eigen-values of $\hat{\mathbf{C}}_N$ ranked in descending order
and ${\nu}_{i=1}^{\kappa}$ are the corresponding eigen-vectors.  One way to attain the $\kappa \times \ell$ matrix $\mathbf{P}^{(\kappa)}$
 is to define }
\[
\mathbf{P}^{(\kappa)} = (\lambda_1^{-1/2} \nu_1, \dots, \lambda_k^{-1/2} \nu_k)^T
\]
Then, it can be verified that
\begin{equation}
\mathbf{P}^{(\kappa)} \hat{\mathbf{C}}_N {\mathbf{P}^{(\kappa)}}^T = \mathbf{I}_{\kappa}
\end{equation}

\bigskip
\noindent{\bf Remark 6.} {The reduced rank of
$\hat{\mathbf{C}}_{N}$ implies that $\hat{\mathbf{C}}_{N}$ may not be a consistent estimator of  $\mathbf{C}_{N}$.
This is an aspect that receives the bulk of the criticism in \cite{M21}.
AT99 seems to suggest that  $\hat{\mathbf{C}}_{N}$ was designed to consistently estimate $\sum_{i=1}^k \lambda_i \nu_i \nu_i^T$, the first $\kappa$-terms in the spectral decomposition of $\mathbf{\mathbf{C}}_N$.  How to achieve this statistically is unclear to us.  }

\bigskip

\noindent{\bf Remark 7.}  Our understanding is that  $\hat{\mathbf{C}}_{N}$ is a consistent estimator of $\tilde{\mathbf{C}}_N$,
the underlying covariance matrix of the null simulation of the climate model, which may be different from $\mathbf{\mathbf{C}}_N$ due to the limitation of the climate model.   If $\tilde{\mathbf{C}}_N \ne \mathbf{\mathbf{C}}_N$ as suspected by \cite{M21}, the estimator $\tilde{\boldsymbol{\beta}}$ is still consistent although will not have the smallest variance, hence giving up the BLUE property but still unbiased and consistent under the independence condition as we will show shortly.

\bigskip

\noindent {\bf Remark 8.}  In order to employ the GM theorem,
the null climate experiments used to attain $\hat{\mathbf{C}}_N$ should be independent of
the observed physical "experiments" that governs  Model \eqref{AT1}. If the independence can be ensured,
which should be satisfied as the simulation runs should be statistically independent of the physical world.  Then,  $\mathbf{P}^{(\kappa)}$ matrix
would be independent of $\mathbf{u}$ and $\mathbf{X}$ in  \eqref{AT1}, and then
\begin{equation}
 E(\mathbf{u} \mid\mathbf{X}, \mathbf{P}^{(\kappa)}) = E(\mathbf{u}\mid\mathbf{X}) =0. \label{ATeq:cond1}
\end{equation}
The statistical meaning of (\ref{ATeq:cond1}) is actually the conditional independence between $\mathbf{u}$ and $\mathbf{P}^{(\kappa)}$ given $\mathbf{X}$, slightly weaker than the full independence as mentioned above.
{AT99 did not explicitly present this most primitive regression identification condition
$E(\mathbf{u}\mid\mathbf{X}) =0$, neither mention (\ref{ATeq:cond1}) above, which had caused severe criticism from \cite{M21}.}

It is readily shown that under (\ref{ATeq:cond1}), as $E(\mathbf{y} \mid \mathbf{X}, \mathbf{P}^{(\kappa)}) = \mathbf{X}\boldsymbol{\beta}$,
\begin{eqnarray}
&& E(\tilde{\boldsymbol{\beta}}) = E(E( \tilde{\boldsymbol{\beta}} \mid \mathbf{X}, \mathbf{P}^{(\kappa)})) \nonumber \\
&=&  E\left(\mathbf{X}^{T} \mathbf{P}^{(\kappa)^T} \mathbf{P}^{(\kappa)} \mathbf{X}\right)^{-1} \mathbf{X}^{T} \mathbf{P}^{(\kappa)^T} \mathbf{P} E(\mathbf{y} \mid\mathbf{X}, \mathbf{P}^{(\kappa)}) \nonumber \\
&=& \boldsymbol{\beta} \label{eq:unbiased}
\end{eqnarray}
indicating $\tilde{\boldsymbol{\beta}}$ being unbiased and having addressed the unbiased issue concerned by \cite{M21}.

\subsection{BLUE Property} \label{sec2.2}

We now consider the BLUE property offered by rotating the data by $\mathbf{P}^{(\kappa)}$, a key aspect of AT99 approach.  We assume  (\ref{ATeq:cond1}) and $\tilde{\mathbf{C}}_N=\mathbf{C}_N$, and show that  $\tilde{\boldsymbol{\beta}}$ is a restricted BLUE and is a full BLUE if $\kappa = \ell$.

Let  us  consider another arbitrary linear unbiased estimator $\hat{\boldsymbol{\beta}}$ of $\boldsymbol{\beta}$ such that  $\hat{\boldsymbol{\beta}}=\mathbf{A} \mathbf{P}^{(\kappa)}\mathbf{y}$ for an arbitrary $m \times \kappa$ constant matrix $\mathbf{A}$,
and
$$\mathbf{D} = \mathbf{A} - (\mathbf{X}^T \mathbf{P}^{(\kappa) T} \mathbf{P}^{(\kappa)} \mathbf{X})^{-1}\mathbf{X}^T \mathbf{P}^{(\kappa) T}.$$
The unbiasedness of $\hat{\boldsymbol{\beta}}$ suggests that
\begin{equation}
\begin{aligned}
E(\hat{\boldsymbol{\beta}}\mid\mathbf{X}, \mathbf{P}^{(\kappa)})&=E(\mathbf{A} \mathbf{P}^{(\kappa)} y\mid\mathbf{X}, \mathbf{P}^{(\kappa)})\\
&=\boldsymbol{\beta} +\mathbf{D} \mathbf{P}^{(\kappa)} \mathbf{X} \boldsymbol{\beta}= \boldsymbol{\beta}
\end{aligned}
\end{equation}
implying that $\mathbf{D} \mathbf{P}^{(\kappa)} \mathbf{X}\boldsymbol{\beta} =0$.

It can be shown that
\begin{eqnarray}
&& Var(\hat{\boldsymbol{\beta}}\mid\mathbf{X},\mathbf{P}^{(\kappa)}) \nonumber \\
&=&E(\mathbf{A}\mathbf{P}^{(\kappa)}\mathbf{uu}^T{\mathbf{P}^{(\kappa)}}^T\mathbf{A}^T\mid\mathbf{X},\mathbf{P}^{(\kappa)}) \nonumber \\
&=& \mathbf{D} \mathbf{D}^T+ Var(\tilde{\boldsymbol{\beta}}\mid\mathbf{X},\mathbf{P}^{(\kappa)}).  \label{eq:varBLUE}
\end{eqnarray}
As $\mathbf{D} \mathbf{D}^T\geq0$, $Var(\hat{\boldsymbol{\beta}}\mid\mathbf{X},\mathbf{P}^{(\kappa)})\geq Var(\tilde{\boldsymbol{\beta}}\mid\mathbf{X},\mathbf{P}^{(\kappa)})$.

\bigskip

\noindent{\bf Remark 9a.}  Note here the class of the linear estimators is restricted by $\mathbf{P}^{(\kappa)}$ when $\kappa < \ell$.   The demonstration above shows that the AT99 formulation with the enforcement (\ref{ATeq:cond1}) would make $\tilde{\boldsymbol{\beta}}$ is {\bf a restricted BLUE}, which would be a full unrestricted BLUE if $\kappa =\ell$.

\noindent{\bf Remark 9b.} The largest threat to the BLUE property of $\tilde{\boldsymbol{\beta}}$ is $\tilde{\mathbf{C}}_N \ne \mathbf{\mathbf{C}}_N$, which would render the minimum variance aspect of $\tilde{\boldsymbol{\beta}}$ although it is still unbiased and consistent under the routine conditions from the perspective of feasible GLS (\cite{W02}).

\subsection{Model Inconsistency Check} \label{sec2.3}

AT99 was aware of the importance of $\tilde{\mathbf{C}}_N$ being a good approximation of $\mathbf{C}_N$ or not.
Section 4 of AT99 is devoted to checking for this by examining  the residual inconsistency via a $\chi^2$ or a F-test.
The null hypothesis $\cal{H}_0$ was worded as follows:

"Our null-hypothesis, $\mathcal{H}_{0}$, is that the control simulation of climate variability is an adequate representation of variability in the real world in the truncated statespace which we are using for the analysis, i.e. the subspace defined by the first $\kappa$ EOFs of the control run does not include patterns which contain unrealistically low (or high) variance in the control simulation of climate variability. Because the effects of errors in observations are not represented in the climate model, $\mathcal{H}_{0}$ also encompasses the statement that observational error is negligible in the truncated state-space (on the spatio-temporal scales) used for detection. A test of $\mathcal{H}_{0}$, therefore, is also a test of the validity of this assumption."

\cite{M21} complained that "they did not formally state the null hypothesis of the RCT nor identify which of the GM conditions it tests, nor did they prove its distribution and critical values, rendering it uninformative as a specification test".

\bigskip

\noindent {\bf Remark 10.} The null hypothesis $\mathcal{H}_{0}$ in AT99 is actually $\mathcal{H}_0: \tilde{\mathbf{C}}_N = \mathbf{\mathbf{C}}_N= Var(\mathbf{u})$ where $\tilde{\mathbf{C}}_N$ is the underlying covariance matrix of the null simulation via the climate model representing "the control simulation of climate variability", and $\mathbf{\mathbf{C}}_N= Var(\mathbf{u})$ is "the  variability in the real world".   As having stated in {\bf Remark 7},
 $\hat{\tilde{\mathbf{C}}}_N$ would be a consistent estimator of $\tilde{\mathbf{C}}_N$ such that $\hat{\tilde{\mathbf{C}}}_N^{-1} = \tilde{\mathbf{C}}_N^{-1} + o_p(1)$ where $o_p(1)$ represents a term that converges to 0 in probability, and $\tilde{\mathbf{C}}_N^{-1}=\tilde{\mathbf{P}}^T \tilde{\mathbf{P}}$ is the generalized inverse. The latter is implicitly assumed by AT99 as shown in the following derivation for the $\chi^2_{\kappa-m}$ distribution.

Under $\mathcal{H}_{0}:\tilde{\mathbf{C}}_N = \mathbf{\mathbf{C}}_N$,  the test statistic
\begin{eqnarray}
&& r^2 = \tilde{\mathbf{u}}^T \hat{\mathbf{C}}_N^{-1} \tilde{\mathbf{u}} \nonumber \\
&= & (\mathbf{Y}-\mathbf{X}\tilde{\boldsymbol{\beta}})^T\hat{\tilde{C}}_N^{-1}(\mathbf{Y}-\mathbf{X}\tilde{\boldsymbol{\beta}})\nonumber \\
&= & \mathbf{Y}^T(\mathbf{I}-\mathbf{F} \mathbf{X}^T) \tilde{\mathbf{C}}_N^{-1}(\mathbf{I}-\mathbf{X}\mathbf{F}^T)\mathbf{Y} + o_p(1) \nonumber \\
&= & {\mathbf{u}^*}^T A {\mathbf{u}^*}+o_p(1) \nonumber
\end{eqnarray}
where $\mathbf{u}^*=\tilde{\mathbf{P}}^{(\kappa)} \mathbf{u} \sim N(0,\mathbf{I}_\kappa)$  and  under $\mathcal{H}_0$
$$\mathbf{A}= \mathbf{\mathbf{C}}_N^{\frac{1}{2}}(\mathbf{I}_\kappa-\mathbf{F}\mathbf{X}^T)\mathbf{\mathbf{C}}_N^{-1}(\mathbf{I}_{\kappa}-\mathbf{X}\mathbf{F}^T)\mathbf{\mathbf{C}}_N^{\frac{1}{2}}.$$

It can be shown that $\mathbf{A}^2=\mathbf{A}$ and $rank(\mathbf{A})= trace(\mathbf{A}) =\kappa - m$.  Thus, $r^{2} \sim \chi_{\kappa-m}^{2}$ as indicated in (AT18).

The key that  leads to $A$ being idempotent (the key for the $\chi^2$ distribution) is the assertion of $\tilde{\mathbf{C}}_N = \mathbf{\mathbf{C}}_N$ under $\mathcal{H}_{0}$.

\bigskip
\noindent {\bf Remark 11.} AT99 discussed on the power of the $\chi^2$-test via the spectral decomposition (EOF) of $\hat{\mathbf{C}}_N$. A formal power analysis can be conducted as follows. Under the alternative hypothesis $\mathcal{H}_1: \tilde{\mathbf{C}}_N \ne \mathbf{\mathbf{C}}_N$,
\begin{eqnarray}
r^2 &= & {\mathbf{u}^*}^T \tilde{\mathbf{A}} {\mathbf{u}^*}+o_p(1) \label{eq:RCTH1}
\end{eqnarray}
where $\tilde{\mathbf{A}} = \mathbf{\mathbf{C}}_N^{\frac{1}{2}}(\mathbf{I}-\tilde{\mathbf{F}}\mathbf{X}^T)\tilde{\mathbf{C}}_N^{-1}(\mathbf{I}-\mathbf{X}\tilde{\mathbf{F}}^T)\mathbf{\mathbf{C}}_N^{\frac{1}{2}}$
and $\tilde{\mathbf{F}}$ is the same as $F$ defined earlier except that it uses $\tilde{\mathbf{P}}$ to replace $\mathbf{P}$.
As $\tilde{\mathbf{A}}$ is no longer idempotent under $\mathcal{H}_1$, $r^2$ is no longer $\chi^2$ distributed. Let $\{\lambda_j\}_{j=1}^{\kappa}$ be the eigenvalues of $\tilde{\mathbf{C}}_N$ and $\{\chi_{1 j}^2\}_{j=1}^{\kappa}$ be the independent $\chi_1^2$ random variables, then it can be shown that
$r^2 \sim \sum_{j=1}^{\kappa} \lambda_j \chi_{1 j}^2$,
which is a weighted independent $\chi_1^2$ distribution and can be used to evaluate the power of the $\chi^2$-test.  If $\hat{\mathbf{C}}_N$ can be consistently estimated,  the eigen-values of $\tilde{\mathbf{A}}$ can be numerically computed and then the power of the RCT can be calculated under the (\ref{eq:RCTH1}).

\bigskip
\noindent{\bf Remark 12.} It is unclear on the powerfulness of the $\chi^2$ test in detecting differences in the underlying variance of the null simulation and the variance of the residuals in the observed physical world.
In particular, its relative performance to the likelihood ratio test
(\cite{AndersonBook}, p.412) should be investigated as $\mathcal{H}_0$ is vital
for the BLUE and the optimal fingerprinting detection approach.

\section{Connection to Feasible Generalized Least Square}

\cite{M21} pointed out that the approach of AT99 is connected to the  Feasible Generalized Least Square (FGLS) in econometrics and statistics (\cite{Carroll82}; \cite{W02}).  We would  spell out the link first and then make some comments.

Like AT99, the FGLS is a two-stage procedure with the first stage obtaining
an initial and yet consistent estimator $\hat{\boldsymbol{\beta}}_1$ of $\boldsymbol{\beta}$ (for instance the OLS without data rotation) based on Model \eqref{AT1}. Let
\begin{equation}
\hat{u}_i = y_i - x_i \hat{\boldsymbol{\beta}}_1 \quad \mbox{for $i=1, \cdots, \ell$}.  \label{eq:estu}
\end{equation}
Then, if Model \eqref{AT1} is valid, a consistent estimator of $\mathbf{\mathbf{C}}_N$ is
\begin{equation}
\hat{\mathbf{C}}_{N 1} = \ell^{-1} \sum_{i=1}^{\ell} \hat{u}_i \hat{u}_i^T. \label{eq：hatCN1}
\end{equation}
The second stage of FGLS is to rotate the data with the square root matrix $\mathbf{P}_1$ of $\hat{\mathbf{C}}_{N 1}^{-1}$,  as did for $\hat{\mathbf{C}}_N^{-1}$ with $\mathbf{P}^{(\kappa)}$, and attain the FGLS estimator
\begin{equation}
\tilde{\boldsymbol{\beta}}_{FGLS} = (\mathbf{X}^T \hat{\mathbf{C}}_{N 1}^{-1} X)^{-1} \mathbf{X}^T \hat{\mathbf{C}}_{N 1}^{-1} \mathbf{Y}  \label{eq:fgls}
\end{equation}
or in the notation of AT99
\begin{equation}
\tilde{\boldsymbol{\beta}}_{FGLS} = (\mathbf{X}^T \mathbf{P}_{N 1}^T \mathbf{P}_{N 1} \mathbf{X})^{-1} \mathbf{X}^T  \mathbf{P}_{N 1}^T \mathbf{P}_{N 1} \mathbf{Y}.  \label{eq:fgls1}
\end{equation}

\bigskip
\noindent  {\bf Remark 13.}  Comparing the above FGLS with that of AT99,
 the latter  may be viewed as a FGLS. The difference between the two is rested in the way of finding the estimates of  $\mathbf{\mathbf{C}}_N$. The FGLS does it internally with the observed data under Model \eqref{AT1}, while AT99 does it externally via separate simulation of climate models.  An drawback of the internal approach is that it would be more adversely impacted by the mis-specification of Model \eqref{AT1}, namely the estimated residual $\tilde{\mathbf{u}}$ would be impacted by the estimation in the regression part.  In contrast, the challenge of the AT99's external approach is rested in if $\tilde{\mathbf{C}}_N$ is indeed $\mathbf{\mathbf{C}}_N$, that is the very purpose of  the $\chi^2$ test. We believe the likelihood ratio test as mentioned in {\bf Remark 11} would be more powerful in detecting the differences and should be investigated.

\section{Conclusion}

Our review finds that the "optimal fingerprinting" approach is very much dependent on two key conditions, which would make the approach survive \cite{M21}'s criticism. One is the independent condition (\ref{eq:iden1}) and another is $\tilde{\mathbf{C}}_N =\mathbf{C}_N$. Both are related to how the null simulation of the climate model is conducted and how accurate the climate model is in approximating the real covariance $\mathbf{C}_N$ of the physical world.
We would think the first condition would be easier to argue than the second one, and would believe that  as the enduring effort in building better and more accurate climate models continues, the difference between $\tilde{\mathbf{C}}_N$ and $\mathbf{C}_N$ would be decreased, and the fingerprint approach  would be  closer to be "optimal".
The independence condition would provide the key justification to the  regression identification condition (\ref{ATeq:cond1}) in {\bf Remark 8}, which in turn ensures $\tilde{\boldsymbol{\beta}}$ is unbiased removing a key concern of \cite{M21}.

The reduced rank of $\hat{\mathbf{C}}_N$ would not make $\tilde{\boldsymbol{\beta}}$ losing the BLUE property all together, but only make it a restricted BLUE as indicated in {\bf Remark 9},  provided $\tilde{\mathbf{C}}_N = \mathbf{\mathbf{C}}_N$. In the case of $\tilde{\mathbf{C}}_N \ne \mathbf{\mathbf{C}}_N$, $\tilde{\boldsymbol{\beta}}$  would still be a consistent and unbiased estimator under standard conditions of the linear regression if the independence between "the two worlds" is maintained by the simulations of the  climate models.  Our remarks made in \ref{sec2.1}-\ref{sec2.2} should address most of “The five assumptions” raised by \cite{M21}.

The residual consistency test (RCT) proposed by AT99 is a valid test for checking on the agreement between the covariances ($\tilde{\mathbf{C}}_N$ vs $\mathbf{\mathbf{C}}_N$) of the two worlds as shown in \ref{sec2.3}, which means that a proper control of the probability of the type I error (rejecting when the underlying covariance matrix of the null simulation does match $\mathbf{\mathbf{C}}_N$ of  the physical world)
can be realized.  As mentioned in {\bf Remarks 11} and {\bf 12}, its control on the probability of the type II error (not rejecting when the covariances of the two worlds are actually different) need further study.

Moreover, we outline the connection between the "optimal fingerprinting" approach and the Feasible Generalized Least Square, which we hope would connect the field of climate change studies and the statistical approach for conducting generalized least square regression.   And more vigorous analysis on the asymptotic normality  of $\tilde{\boldsymbol{\beta}}$ can be carried out using the standard methods in asymptotic statistics with specific conditions, which would not to get into in this review.

\backmatter

\bmhead{Acknowledgments}
We thanks Hongbin Lin and Shanshan Luo for assistance.

\section*{Declarations}
\begin{itemize}
\item Funding
\\This research was supported by the National Natural Science Foundation of China (Grant No.92046021).
\item Conflicts of interest
\\ There is no conflict of interest.

\item Availability of data and material
\\Not applicable.

\item Code availability
\\Not applicable.
\end{itemize}

\input{output.bbl}

\end{document}


\section{Introduction}
 With the gradual development of Internet technology, information resources are growing at a high speed and the problem of "information overload" has emerged. It is difficult for users to get the information they need directly from the huge amount of information. Therefore, recommendation algorithms are usually used to solve the problem of "information overload", and the research of various recommendation methods~\cite{sedhain2015autorec, cheng2016wide, salah2018probabilistic} has become one of the main focuses in the field of computer science.

The traditional recommendation system based on collaborative filtering~\cite{he2016fast} has been the main direction of recommendation algorithm research because it can take the advantage of big data to find users with the most similar interests and recommend the micro-videos they are interested in. However, there are two problems that are difficult to solve in collaborative filtering, namely, the cold start problem caused by too little initial data of new users and the sparse interaction data problem caused by too many micro-videos in total and too few micro-videos interacting with users. To solve these two problems, researchers have tried to introduce various kinds of auxiliary information to complement the data, such as point-of-interest letters~\cite{cao2010mining}, comment information~\cite{zhang2013generating}, social networks ~\cite{jamali2010matrix}, and contextual information~\cite{adomavicius2011context}, which have been useful in specific scenarios, and researchers have started to explore how to design better models to fully utilize the auxiliary information and explore the applicability of The researchers started to explore how to design better models to fully utilize the auxiliary information and explore more general auxiliary information with wider applicability~\cite{wang2019neural}. Recently proposed approaches prefer to design end-to-end models to simultaneously learn the potential feature attributes and structural relationships of entities in the knowledge graph and use them for recommendation, which are implemented mainly by graph neural network~\cite{scarselli2008graph} (GNN) approaches to model the knowledge graph.

Compared with traditional graph learning, graph neural networks can not only learn the topology of graph networks but also aggregate feature information of neighbors, thus enabling effective learning of various structures in graph networks and playing a key role in subsequent recommendations~\cite{defferrard2016convolutional}. The original recommendation learning algorithm~\cite{mnih2007probabilistic} is mainly based on user-micro-video interaction for learning and modeling the user-micro-video connectivity, which makes it difficult to capture more complex relationships between nodes. While traditional graph learning generally works for graph topology and also takes less into account various feature information among nodes or nodes, compared with the original recommendation methods, graph neural network recommendation methods can alleviate the problems of cold start~\cite{lee2016enhancing,wei2021contrastive} and data sparsity~\cite{zhou2011functional} in traditional collaborative filtering methods. It can not only learn the topology of the graph network but also aggregate the various connections of neighboring nodes. Thus it can learn the information in the graph network more effectively and play a key role in the subsequent recommendation work.

Compared with traditional methods, graph neural networks can improve the accuracy of recommendation results. The application scenarios of recommendation systems often contain huge amounts of data, and traditional collaborative filtering recommendations often fail to make recommendations for users because they have too little interaction data, or a large number of users have high similarity to them due to a lot of interaction data of some users, which leads to less personalized recommendation results. For example, if user A likes movies 1, 2 and 3, and user B likes movie 1, the recommendation system will consider that user A and user B are similar and obtain the relationship that both 2 and 1 are the works of director c from the knowledge graph, and then give priority to The graph neural network can improve the diversity ~\cite{smith2019towards} and novelty~\cite{zhou2010solving} of recommendation results.

Traditional recommendation algorithms tend to have a large number of repetitive recommendations as well as high similarity recommendations, which result in a lack of novelty in the recommended results due to the single index for calculating similarity~\cite{Yu}. The inferable implicit information contained in the knowledge graph enhances the ability of the knowledge graph-based recommendation system to explore the potential interests of users and provide them with richer recommendation results. For example, if user A likes movies 4 and 5, the recommendation system can recommend 6 for user A based on the relationship that both 4 and 6 are the works of director d, and recommend 7 for user A based on the relationship that both 7 and 1 are starred by e. The graph neural network has good interpretability. Since the nodes of the knowledge graph have realistic meaning, they can be linked with the user's history to provide a basis for the user's recommendation. Graph neural networks have good scalability and wider applicability~\cite{NMCL}. The knowledge graph-based recommendation algorithm relies on the objective knowledge contained in the knowledge graph as well as the implicit derivable knowledge; in other words, the more knowledge the knowledge graph contains, the more powerful the knowledge graph-based recommendation algorithm is. Monti et al.~\cite{monti2017geometric} used graph neural networks to extract network representations of users and micro-videos, and then combined with recurrent neural networks for the message passing process. Berg et al.~\cite{berg2017graph} proposed a graph auto-coding framework to generate latent features of users and micro-videos by passing message aggregation on the user-micro-video graph.Hartford et al.~\cite{hartford2018deep} considered the problem of predicting relationships between two and more different sets of objects and introduced a weight binding scheme to the deep model.

The original recommendation learning algorithm is mainly based on user-micro-video interaction for learning, modeling the user-micro-video connection relationship, which is difficult to capture the more complex relationships between nodes. And traditional graph learning generally works for graph topology and also takes less account of various feature information among nodes and nodes~\cite{GRCN}. To address the above problems, this paper proposes a graph neural network-based model. We propose a personalized recommendation model based on graph neural network, which utilizes the feature that graph neural network can tap deep information of graph data more effectively, and transforms the input user rating information and micro-video side information into graph structure, ( including user-micro-video graph, micro-video-information graph, etc.) for effective feature extraction, based on the importance sampling strategy. The importance-based sampling strategy measures the importance of neighbor nodes to the central node by calculating the relationship tightness between the neighbor nodes and the central node, and selects the neighbor nodes for recommendation tasks based on the importance level, which can be more targeted to select the sampling neighbors with more influence on the target micro-video nodes. The pooling aggregation strategy, on the other hand, trains the aggregation weights by inputting the neighborhood node features into the fully connected layer before aggregating the neighborhood features, and then introduces the pooling layer for feature aggregation, and finally aggregates the obtained neighborhood aggregation features with the target node itself, which directly introduces a symmetric trainable function to fuse the neighborhood weight training into the model to better capture the different neighborhood nodes' differential features in a learnable manner to allow for a more accurate representation of the current node features. The model uses graph neural networks to efficiently model user preference information through information transfer and information aggregation on nodes, while introducing attention networks to finally obtain rating predictions.

\begin{figure}
	\centering
	  \includegraphics[width=0.5\textwidth]{Charts/graph1.pdf}
	  \vspace{-1mm}
	  \caption{Schematic diagram of multi-layer convolutional graph neural network.}
	\label{fig_1}
	\vspace{-5mm}
\end{figure}

\section{Related Work}
Graph neural networks were first proposed by Gori et al. in 2005~\cite{scarselli2005graph} and then further elucidated by Scarselli et al. in 2009~\cite{scarselli2008graph}. In early studies based on graph data, for the target node representation, researchers conducted learning by iteratively propagating neighbor information through recurrent neural networks. The computational effort in this process is very large, and many researchers hope to alleviate this problem, and subsequently this has become a research hotspot in recent years. The relationship between graph neural networks and network embeddings is close, and the relationship between them is briefly described here. Network embedding is the projection of nodes in a graph into a low-dimensional vector space while preserving the network topology and node information. thus producing a low-dimensional vector representation of the nodes, which is then used for graph data analysis (e.g., classification, recommendation, etc.). The general network embedding algorithms can be broadly classified into: matrix decomposition~\cite{koren2009matrix,xue2017deep,mnih2007probabilistic}, random wandering ~\cite{perozzi2014deepwalk} and deep learning methods~\cite{wang2016structural,hamilton2017inductive}. Among them, deep learning methods for network embedding also belong to graph neural networks, including algorithms based on graph self-encoders (e.g., SDNE~\cite{wang2016structural}) and graph convolutional neural networks with unsupervised learning (GraphSage~\cite{hamilton2017inductive}),and LightGCN~\cite{he2020lightgcn}.

GC-MC~\cite{berg2017graph} is a graph-based auto-encoder framework for matrix completion. The auto-encoder produces latent features of user and item nodes through a form of message passing on the bipartite interaction graph. These latent user and item representations are used to reconstruct the rating links through a bilinear decoder. The benefit of formulating matrix completion as a link prediction task on a bipartite graph becomes especially apparent when recommender graphs are accompanied with structured external information such as social networks. Combining such external information with interaction data can alleviate performance bottlenecks related to the cold start problem.

Spectral graph theory~\cite{shuman2013emerging} studies connections between combinatorial properties of a graph and the eigenvalues of matrices associated to the graph. L. Zheng et al. ~\cite{zheng2018spectral} proposed a spectral graph theory based method to leverage the broad information existing in the spectral domain. Specifically, to conquer the difficulties of directly learning from the spectral domain for recommendations, they first presented a new spectral convolution operation, which is approximated by a polynomial to dynamically amplify or attenuate each frequency domain. Then, they introduced a deep recommendation model, named Spectral Collaborative Filtering (SpectralCF), built by multiple proposed spectral convolution layers. SpectralCF directly performs collaborative filtering in the spectral domain.

  J. Sun et al.~\cite{sun2020neighbor} propose a novel graph convolutional neural network-based recommender system framework, Neighbor Interaction Aware Graph Convolutional Neural Networks, NIA-GCN. In NIA-GCN, they proposed a Pairwise Neighborhood Aggregation (PNA) layer to capture relationships between pairs of neighbors at each GCN layer. PNA applies element-wise multiplication between every two neighbor embeddings to learn user-user and item-item relationships, which are important signals in recommendation. Instead of recursively updating neighborhood embeddings, they introduced parallel graph convolution networks (Parallel-GCNs), which can independently aggregate information from node entities at different depths with respect to the central node. Such an approach is a much better match to the heterogeneous nature of the user-item bipartite graph. They proposed a novel Cross-Depth Ensemble (CDE) layer to capture the user-user, item-item and user-item relationships in neighborhoods in the graph. This layer allows predictions to take into account more complicated relationships across different depths in the graph.

  S. Ji et al.~\cite{ji2020dual} proposed a dual channel hypergraph collaborative filtering (DHCF). It is a dual channel learning strategy, which holistically leverages the divide-and-conquer strategy, is introduced to learn the representation of users and items so that these two types of data can be elegantly interconnected while still maintaining their specific properties. The hypergraph structure is employed for modeling users and items with explicit hybrid high-order correlations. The jump hypergraph convolution (JHConv) method is proposed to support the explicit and efficient embedding propagation of high-order correlations. Wang et al.~\cite{GRCN} constructed the user-item graph, whose edges corresponded to implicit feedback. With their proposed neural graph collaborative filtering (NGCF) method, the collaborative signal conveyed by the edges and high-order connectivity explicit modelled and injected into each user and item embedding. More recently, the GCN-based model has been introduced into multimedia recommendation in implicit feedback settings. Wei et al.~\cite{MMGCN} constructed the modal-specific bipartite graph with implicit data to model the user preference in multiple modalities. They developed a multimedia recommendation framework, dubbed multimodal graph convolutional network (MMGCN), which represented the user preference in each modality with her/his directly and indirectly connected neighbors.

\section{Methodology}
\subsection{Embedding Layer}

\begin{figure*}
	\centering
	  \includegraphics[width=1\textwidth]{Charts/graph5.pdf}
	  \vspace{-1mm}
	  \caption{Schematic diagram of our neural network recommendation model.}
	\label{fig_1}
	\vspace{-5mm}
\end{figure*}

We use an embedding vector $e_{ua} \in \mathbb{R}^d$ to describe a user $u_a$, where $d$ denotes the size of the embedding. This can be viewed as constructing the parameter matrix as an embedding lookup table, as the following equation shown:
\begin{equation}
E=[e_{ul}, \dots, e_{um}, e_{il}, \dots, e_{im}, e_{1}, \dots, e_{n}],
\end{equation}
where $e_{ul}, \dots, e_{um}, e_{il}$ represents user embedding, $e_{il}, \dots, e_{im}, e_{1}$ represents micro-video embedding. $e_{il}, \dots, e_{im}, e_{1}$ represents rates. In the GCFA framework, we jointly consider user embedding, micro-video embedding and rate embedding, feed the end-user embedding and micro-video embedding into a multilayer perceptron, and then output the predicted of the scores.

Create the user embedding matrix $U$ and the project embedding matrix $V$. Initialize using the common initializer Xavier, which keeps the gradient size of each layer approximately the same. Then input the users in turn, and the user adjacency matrix $N$ to aggregate the user neighbor feature adjacencies as follows:
\begin{equation}
u^n=\sum_{n\in N(u)} g(r_{u,n})n,
\end{equation}
where $N(u)$ is the set of users' neighbors and $g(r_{u,n})$ is the normalization of the relationship weights between users and their neighbors, as shown in follows:
\begin{equation}
g(r_{u,n})=\frac{exp(r_{u,n})}{\sum_{n\in N(u)} exp(r_{u,n})},
\end{equation}
Then the aggregated neighborhood features are nonlinearly aggregated with the user's own features, as shown below:
\begin{equation}
agg_u=\delta(w\cdot (u+u^n)+b),
\end{equation}
where $\delta$ is the nonlinear activation function. We use Relu as the activation function, $w$ is the transformation weight, and $b$ is the bias term.

\subsection{Propagation Layer}

For a connected user-micro-video pair, we define the message propagate from $i_s$ to $u_a$ as:
\begin{equation}
h_a=\theta(w\cdot Aggre_{i}(\{x_{sa}, \forall{s}\in C(a)\}+b)),
\end{equation}
where $C(a)$ denotes the set of micro-videos that user $u_a$ has interaction with,  $x_{sa}$ denotes the interaction between user $u_a$ and micro-video $i_s$. Based on the embedding representation, $Aggre_{i}$ is the micro-video aggregation function. In addition $\theta$ denotes the nonlinear activation function and $w$, $b$ are the weights and biases of the neural network, respectively.

For the interaction between user $u_a$ with rating level $r$ and micro-video $i_s$, a two-layer neural network is used to combine the micro-video embedding $e_i$ with the viewpoint embedding $e_r$, thus modeling the viewpoint-based representation $x_{as}$ of the interaction. This can be expressed as g with the tandem of micro-video embedding and viewpoint embedding as input and the output as The fusion network for the interaction viewpoint-based representation. $x_{as}$ is represented as follows:
\begin{equation}
x_{as}=g_v ([e_i \oplus e_r]),
\end{equation}
where $\oplus$ denotes the concatenation of two vectors.

The traditional aggregation operation $Aggre_{i}$ is based on the mean operator, which takes the mean value of the elements of the vector in $\{x_{as}, \forall{s}\in C(a)\}$. The mean value-based aggregation function is as follows:
\begin{equation}
h_a=\theta(w\cdot \sum_{s\in C(a)} \beta_a x_{as} +b),
\end{equation}
In the mean-based aggregation, $\beta_a$ is fixed to $\frac{1}{C(a)}$ for all micro-vide0s. The approach assumes that all interactions contribute equally to understanding the user. In fact, not every interaction behavior has the same impact on user based viewpoint embedding, so we allow various interactions to contribute differently to the user's latent factor. Assign unique weights to each $(u_a, i_s)$ pair:
\begin{equation}
h_a=\theta(w\cdot \sum_{s\in C(a)} \beta_{as} x_{as} +b),
\end{equation}

When understanding user preferences from historical interactions $C(a)$, $ \beta_{as}$ denotes the attentional weight of the contribution of the interaction with $i_s$ to the potential factor of user $u_a$ when understanding the potential factor of user $u_a$. In particular, we parameterize micro-video attention $ \beta_{as}$ using an attention network consisting of two layers of networks. The inputs to the attention network are the interaction-based viewpoint representation $x_{as}$ and the embedding $e_{ua}$ of the target user $u_a$. The attention network is defined as follows:
\begin{equation}
\beta_{as}^*=w_2^T \cdot \theta ([x_{as} \oplus e_{ua}]+b_1)+b_2,
\end{equation}

The above attention scores are normalized using the Softmax function to obtain the target attention weights such that the contribution to the user potential factor of user $u_a$ is:
\begin{equation}
\beta_{as}=\frac{exp(\beta_{as}^*)}{\sum_{s\in C(a)} exp(\beta_{as}^*)},
\end{equation}
Substitute equation(10) into equation(8) to calculate $h_a$.

Improving the representation of $u_a$ by aggregating information from user's neighbors. Concatenate $h_a$ and $e_{ua}$. The results are fed into a two-layer neural network, and the first layer embedding is obtained through the attention mechanism and the embedding propagation layer to optimize the optimize the initial embedding.
\begin{equation}
e_{ua}^{(l)}=LeakyReLU(m_{u\gets u}+\sum_{s\in C(a)} m_{u\gets i}),
\end{equation}
The activation function of LeakyReLU allows messages to encode small signals, both positive and negative. Similarly, we can obtain a representation of the micro-video $e_{ib}^{(l)}$ by propagating information about its associated users $i_b$.
\begin{equation}
e_{ib}^{(l)}=LeakyReLU(m_{i\gets i}+\sum_{i\in B(b)} m_{i\gets u}),
\end{equation}
In summary, the advantage of embedding propagation layers is the explicit use of single-layer connection information to associate users and micro-video representations.

\subsection{Loss Function}
The loss function of the model is shown below:
\begin{equation}
\begin{split}
& L=\sum_{(u,v)\in M} -log\theta(u_a^T i_s)+ log(1-\theta(u_a^T i_s)) \\
& +\frac{\lambda}{2} (\|U\|^2_2 + \|V\|^2_2 +\|W\|^2_2,
\end{split}
\end{equation}
The former term is the cross-entropy loss between the true and predicted scores, and the latter term is a regularization term used to avoid overfitting due to large parameter differences.
\section{EXPERIMENTS}
\subsection{Dataset}
 \begin{figure*}
    \centering
    \subfigure[AUC on Movielens]{
      \includegraphics[width=0.45\textwidth]{Charts/graph3.pdf}
      \label{fig_visualize_1_1}
    }
    \subfigure[AUC on Amazon]{
      \includegraphics[width=0.45\textwidth]{Charts/graph4.pdf}
      \label{fig_visualize_2_3}
    }

    \vspace{-10pt}
    \caption{AUC Comparison Histogram.}
    \label{fig_3}
    \vspace{-5pt}
 \end{figure*}

\begin{table}
  \centering
  \caption{ecommendation Accuracy Performance Comparison.}

  \label{table_3}
  \setlength{\tabcolsep}{5.0mm}
  \begin{tabular}{|c|c|c|c|}
    \hline
    \textbf{Model}&\textbf{AUC}&\textbf{NDCG@1}&\textbf{NDCG@2}\\
    \hline
    \textbf{BPR}&$0.7345$&$0.7536$&$0.9213$\\
    \hline
    \textbf{NCF}&$0.7436$&$0.7569$&$0.9311$\\
    \hline
    \textbf{FCMC}&$0.7497$&$0.7641$&$0.9422$\\
    \hline
    \textbf{GraphRec}&$0.7536$&$0.7753$&$0.9495$\\
    \hline
    \textbf{Ours}&$0.7721$&$0.7938$&$0.9567$\\
    \hline

  \end{tabular}
  \vspace{-2mm}
\end{table}

The datasets used in this experiment are MovieLens-1M and Amazon.

MovieLens-1M~\cite{harper2015movielens}: This movie rating dataset has been widely used to evaluate collaborative filtering algorithms. We used the version containing 1,000,209 ratings from 6,040 users for 3,900 movies. While it is a dataset with explicit feedbacks, we follow the convention~\cite{he2017neural} that transforms it into implicit data, where each entry is marked as 0 or 1 indicating whether the user has rated the micro-video. After transforming, we retain a dataset of $1.0\%$ density.

Amazon Instant Video~\cite{mcauley2015image}: The dataset consists of 426,922 users, 23,965 videos and 583,933 ratings from Amazon.com. Similarly, we transformed it into implicit data and removed users with less than 5 interactions. As a result, a dataset of $0.12\%$ density is obtained.

\subsection{Baselines}

BPR~\cite{rendle2012bpr}: We use Bayesian Personalized Ranking based Matrix Factorization. BPR introduces a pair-wise loss into the Matrix Factorization to be optimized for ranking.

NCF~\cite{he2017neural}: Neural Collaborative Filtering fuses matrix factorization and Multi-Layer Perceptron (MLP) to learn from user-item interactions. The MLP endows NCF with the ability of modelling non-linearities between users and items.

GCMC~\cite{berg2017graph}: Graph Convolutional Matrix Completion utilizes a graph auto-encoder to learn the connectivity information of a bipartite interaction graph for latent factors of users and items.

GraphRec~\cite{fan2019graph}: this model uses GNNs to integrate node information and topology for consistent modeling of user-project graphs and user-user social graphs.

\subsection{Evaluation Metrics}

The most commonly used metrics for video recommendation and even general-purpose recommendation systems are ranking metrics, of which the most representative metrics are the area under the feature curve (AUC) and the normalized discounted cumulative gain (NDCG)~\cite{schutze2008introduction}. The AUC measures the ability of the model to discriminate between all positive and negative samples relative to each other, and the NDCG measures the distance of the model ranking results from the ideal ranking results.

\subsection{Result Analysis}
We first compared the overall recommendation results We compare the overall recommendation results, as shown in Table 1.
From Table 1, we can see that, compared with the existing models, our proposed model is more effective in terms of AUC, NDCG@1, NDCG@2. The average relative improvement value is about $1.7\%$. For the recommendation system model The improvement value is significant for the recommended system model.

NCF obtains better performance than BPR. Both methods utilize only scoring information. However, NCF is based on a neural network architecture, which shows the power of neural network models in recommender systems. graphRec outperforms GCMC. both methods utilize only graph neural networks. However, GraphRec is based on the embedding propagation architecture, which illustrates the power of the embedding propagation model. The method in this paper outperforms all baseline methods in most cases. In contrast to GraphRec, the model in this paper utilizes attention mechanisms in the user micro-video graph to fuse both interaction behaviors of user micro-videos and user perspectives. This illustrates the importance of simultaneously fusing the interaction behavior of user micro-videos and user viewpoints. In the Book-Crossing dataset, where the data is sparse, the proposed model achieves more obvious advantages compared with all other models, which proves that the improved strategy proposed in this paper is useful in capturing important features and compensating for the sparsity of data. As can be seen from Figure 3, our model achieves a smaller training training loss.
\begin{figure}
	\centering
	  \includegraphics[width=0.5\textwidth]{Charts/graph2.pdf}
	  \vspace{-1mm}
	  \caption{Comparison of model loss curves.}
	\label{fig_1}
	\vspace{-5mm}
\end{figure}

\section{Conclusion and Future Work}
The original recommendation learning algorithm is mainly based on user-micro-video interaction for learning, modeling the user-micro-video connection relationship, which is difficult to capture the more complex relationships between nodes. And traditional graph learning generally works for graph topology and also takes less account of various feature information among nodes and nodes. To address the above problems, this paper proposes a graph neural network-based model. We propose a personalized recommendation model based on graph neural network, which utilizes the feature that graph neural network can tap deep information of graph data more effectively, and transforms the input user rating information and micro-video side information into graph structure, ( including user-micro-video graph, micro-video-information graph, etc.) for effective feature extraction, based on the importance sampling strategy. The importance-based sampling strategy measures the importance of neighbor nodes to the central node by calculating the relationship tightness between the neighbor nodes and the central node, and selects the neighbor nodes for recommendation tasks based on the importance level, which can be more targeted to select the sampling neighbors with more influence on the target micro-video nodes. The pooling aggregation strategy, on the other hand, trains the aggregation weights by inputting the neighborhood node features into the fully connected layer before aggregating the neighborhood features, and then introduces the pooling layer for feature aggregation, and finally aggregates the obtained neighborhood aggregation features with the target node itself, which directly introduces a symmetric trainable function to fuse the neighborhood weight training into the model to better capture the different neighborhood nodes' differential features in a learnable manner to allow for a more accurate representation of the current node features. The model uses graph neural networks to efficiently model user preference information through information transfer and information aggregation on nodes, while introducing attention networks to finally obtain rating predictions.


\section{Introduction}
Text retrieval is a task which aims to search in a large corpus for texts that are most relevant given a query. The final retrieval result is informative and can benefit a wide range of down streaming applications, such as open domain question answering~\cite{karpukhin2020dense,Li2021EncoderAO}, machine reading comprehension~\cite{Rajpurkar2016SQuAD1Q,Nishida2018RetrieveandReadML}, and web search systems~\cite{huang2020embedding,liu2021pre}.

Since the corpus is massive in general, a practical text retrieval system usually leverages a retrieve-then-reranking architecture due to the efficiency and effectiveness trade-off. Specifically, given a search query, a retriever is involved to efficiently recall a relatively small size of relevant texts from the large corpus. Then the query and the initial retrieval results will be passed through more delicate reranking stages to produce the final ranked results. In recent years, the emergence of deep pre-trained language models (PTM)~\cite{kenton2019bert,liu2019roberta} has brought a storm in natural language processing (NLP) and information retrieval fields. Text retrieval systems armed with pre-trained language models have become a dominant paradigm to improve the overall performance compared to traditional statistical methods (e,g. BM25). Considering the large computational cost of transformer-based PTMs, the retrieval stage and reranking stage usually use different model architectures: representation-focused model and interaction-focused model respectively~\cite{fan2021pre}, as illustrated in Figure \ref{fig:two-type-model}.

\begin{figure}[t]
    \centering
    \includegraphics[width=1.0\columnwidth]{two_stage.pdf}
    \caption{Model architectures of representation-focused model (left) and interaction based model (right).}
    \label{fig:two-type-model}
\end{figure}

Based on the model structures presented in Figure \ref{fig:two-type-model}, previous approaches to multi-stage text retrieval roughly fall into two groups. The first group tend to separately optimize the retrieval or reranking stage by designing task-oriented PTMs~\cite{gao2021condenser,gao2021unsupervised,ma2021b} or better model training strategies~\cite{xiong2021approximate,gao2021rethink}. Studies in the second group attend to jointly optimize the two stages by adversarial training~\cite{zhang2021adversarial} or knowledge distillation~\cite{ren2021rocketqav2,lu2022erniesearch}. However, this kind of joint learning method mainly focused on improving the retrieval stage performance by absorbing merits from the reranking stage.

In a global view, the retrieval stage and reranking stage are highly correlated. Intuitively, better retrieval results will definitely provide a more enlightening signal to the reranking stage~\cite{gao2021rethink} for both the training and inference phase. Furthermore, since the two individual stages adopt different model architectures (representation-focused vs interaction-focused), we infer that the focus of the two stages is also different despite the sharing target to estimate the relevance of query and document. Empirically, we find that simply a weighted combination of the relevance scores produced by the two stages can further improve the overall performance, which also supports our hypothesis above. These observations inspire us whether it is possible to make an additional module that couples both coarse retrieval and fine reranking features to facilitate the final retrieval performance.

In this paper, we propose to add an additional reranking stage with a lightweight reranking model, which we noted as Hybrid List Aware Transformer Ranker (HLATR). As we tend to incorporate both the two-stage features and the reranking training objective is list aware, we adopt the transformer model structure~\cite{vaswani2017attention} for HLATR. As introduced in~\cite{vaswani2017attention}, the content for each position of the sequential inputs includes the embedding itself and the position embedding corresponded. Here, we carry out the feature coupling process by taking the logit produced by the PTM based reranking model as the embedding of each document, and we compute the position embedding based on the retrieval ranking order of each document in the retrieval stage. A stack of transformer layers will be used as the encoder part. In HLATR, we can effectively fuse the retrieval and reranking features and model the list aware text ranking process.

To verify the effectiveness and robustness of HLATR model, we conduct experiments on two commonly used large text retrieval benchmarks, namely the MS MARCO passage and document ranking datasets~\cite{Campos2016MSMA}. Empirically experiment results demonstrate that HLATR can remarkable improve the final text retrieval performance with different combinations of retrieval and reranking architectures. Further we design sufficient subsidiary experiments to prove that HLATR is lightweight and easy to practical.

\section{Related Work}
In order to balance efficiency and effectiveness, existing text retrieval systems usually leverage a retrieval-then-reranking framework.

For the retrieval stage, traditional sparse retrieval methods adopt the exact term matching signal for text relevance estimation (e,g. TF-IDF~\cite{luhn1957statistical} and BM25~\cite{robertson2009probabilistic} method). Recently, with the rapid development of the deep text representation method, many works shifted to training dense retrievers by taking BERT or other deep neural networks as text encoders~\cite{karpukhin2020dense,nogueira2019passage}. These models utilize dual encoders to encode queries and documents separately, which allows pre-encoding and indexing of corpus. Under this paradigm, various optimization strategies have been proposed to improve the retrieval performance including designing task-specific pre-trained model~\cite{gao2021unsupervised,gao2021condenser}, hard negatives mining strategy~\cite{xiong2021approximate} and multi-view text representation~\cite{hu2021multi}.

For the reranking stage, previous studies use an interaction-focused model which takes the pair of query and document as input. By considering the token level similarity, these models always have a better ranking power~\cite{guo2016deep}. By such a full interaction computation strategy, the PTM based reranking model usually reranks a small number of documents produced by the retrieval stage. ~\cite{ma2021b} propose to enhance the pre-trained language model by constructing scale pseudo relevant query-document pairs.  ~\cite{gao2021rethink} propose to replace the traditional binary cross-entropy loss~\cite{nogueira2019passage} with a contrastive loss for better list aware ranking. ~\cite{pu2021yes} attend to dynamically select hard negative samples to construct high-quality semantic space the trained reranking model.

Apart from the separate optimization methods presented above, there are also studies seeking to jointly optimize the retrieval and reranking stage. In ~\cite{ren2021rocketqav2}, they simultaneously optimize the retrieval model and the ranking model with a hybrid data augmentation technology. AR2~\cite{zhang2021adversarial} uses a Generative adversarial network (GAN)~\cite{creswell2018generative} method which regards the retrieval model as the ``generative mode'' and the ranker model as the ``discriminative model''. All of these jointly learning methods need a large extra training cost than the traditional two-stage training method. In contrast, HLTAR requires only a very small additional consumption cost.

\begin{figure*}
    \centering
    \includegraphics[width= 2.0\columnwidth]{intro.pdf}
    \caption{An illusion of the conventional retrieve-then-reranking framework (solid line part) and our HLTAR ranking model (solid line and dashed line part).}
    \label{fig:frame}
\end{figure*}

\section{Methodology}

\subsection{Problem Overview}
The retrieve-then-reranking framework for text retrieval task is presented in Fig~\ref{fig:frame}. Given a query $q$, a text retrieval system aims to return a small size list of relevant documents from a large document set $\mathcal{D}=\{d_i\}_{i=1}^N$. Existing retrieval systems usually consist of a retrieval stage and a reranking stage with different focuses on efficiency and effectiveness. Here, for simplicity, we only introduce the dense retrieval model and reranking model based on pre-trained language model (as illustrated in Figure~\ref{fig:two-type-model}).

The retrieval stage takes $q$ and $\mathcal{D}$ as input and aims to return all potentially relevant documents $\mathcal{D}_r \in \mathcal{D}$ refer to the relevance between query and document, where $|\mathcal{D}_r| \ll N$. Dense retrieval model focuses on leaning to map text into quality continuous vector representation. By taking the pre-trained language model as the encoder, the retrieval model can be formulated as:
\begin{equation}
\label{eq:retrieval}
    score(q,d) = f(E^Q(q), E^D(d)),
\end{equation}
where $E^Q$ and $E^D$ are query and document encoders, and $f$ is the relevance estimation function.

Following the retrieval stage, a reranking stage takes $q$ and $\mathcal{D}_r$ as input to perform a more refined relevance evaluation for each query-document pair $(q, d)$, where $d \in \mathcal{D}_r$. Commonly, the reranking model adopts the interaction-based model. Without loss of generality, the reranking method could be abstracted by the following formulation:
\begin{equation}
\label{eq:reranking}
    score(q,d) = f(E^R(q,d)),
\end{equation}
where $E^R$ notes the PTM based encoder and $f$ is the evaluation function that computes the relevance score.

In the retrieval stage, query and document are encoded independently. Therefore, the embedding representation of $\mathcal{D}$ can be pre-encoded. While in the reranking stage, the final representation of each query-document pair can not be pre-encoded due to the interaction-based architecture. As the result, we can only rerank a small set of documents as a result of the large computational cost of PTM.

Despite adopting different model architectures, the retrieval and reranking stage share the same target for evaluating the relevance of query and document. Certainly, the training of retrieval and reranking models relies on large-scale labeled data. Previous studies~\cite{karpukhin2020dense,gao2021rethink} proposed to optimize the PTM based retrieval and reranking model with a contrastive learning objective. Concretely, for each query $q$, we form a group $G_d$ with a single relevant positive document $d^+$ and multiple negatives. By taking the scoring function defined in equations (\ref{eq:retrieval})(\ref{eq:reranking}), the contrastive loss for each query $q$ can be denoted as:
\begin{equation}
    \mathcal{L}_q := -log\frac{exp(score(q, d^{+}))}{\sum_{p\in G_d} exp(score(q, d))}.
\end{equation}

\subsection{HLATR}
We target on building a lightweight reranking module that couples feature from both the retrieval and reranking stage. The overall architecture of our proposed model is illustrated in Figure \ref{fig:hltar}. Specifically, our model consists of a feature combination layer along with a transformer encoder.

\subsubsection{Multi-stage Feature Fusion}
As the HLATR model is pipelined with the previous reranking model, HLATR takes the query $q$ and $D_r^{'}$ as input, where $D_r^{'}$ is the ranking results produced by the PTM based reranking module. Obviously, $|D_r| = |D_r^{'}| = Z$, where $Z$ denotes the reranking size. Since we adopt the transformer model architecture as the encoder, we carry out the multi-stage features fusion via coupling the retrieval ranking order in the retrieval stage and the final logit representation generated by the reranking model of each document.

For document $d\in D_r^{'}$ with retrieval order $i$ (where $0 \leq i \leq Z$), we map the $i$-th order into a learnable position embedding $\mathbf{pe_{i}} \in {\mathbf{R}^{d}}$ with dimension $d$, and which will be jointly optimized during the learning process. The position embedding is randomly initialized. The strategy of introducing retrieval order as a feature makes our model not only practicable for systems that use dense retrieval models, but also for others based on traditional statistical retrieval methods (e.g, BM25).

For the reranking stage, we hire the final encoded representation $E^R(q,d)$, which is converted to an embedding representation sharing the same dimension as the position embedding:
\begin{equation}
    v_i = W_{v}^TE^R(q,d_i),
\end{equation}
where $W_{v}$ is the trainable projection matrix.

Then, for each $d_i$ in $D_r^{'}$, we can directly add $pe_{i}$ and $v_i$ to build the input matrix $H^0 \in R^{d \times Z}$:
\begin{align*}
    H^0 = \begin{pmatrix} LN(pe_{1} + v_1) \\  LN(pe_{2} + v_2) \\  ... \\  LN(pe_{Z}+v_Z)
    \end{pmatrix},
\end{align*}
where LN represents the LayerNorm~\cite{ba2016layer} operation.

\subsubsection{Transformer Encoder}
The learning process of the reranking model is list aware. The key point of a reranking module is to model and learn the partial order relationship between different documents within the entire candidates pool. Thus, we adopt the transformer architecture~\cite{vaswani2017attention} as the encoder owing to its long-distance modeling ability and full attention mechanism. The self-attention mechanism in the transformer can directly model the mutual influences for any different query-documents pair sample.

The entire encoder module consists of $L$ transformer layers. The hidden size of each Transformer layer is set equal to the input dimension $d$. For the first Transformer layer, it takes $H^0$ as input. While for the $l$-th Transformer layer $T^{l}$, it takes the output of the previous layer as input:
\begin{align*}
    H^{1} = T^1(H^0), \\
    H^{l} = T^l(H^{l-1}).
\end{align*}

\subsubsection{Learning Objective}
\label{sec:loss}
Finally, we use a linear projection layer to predict the relevance score between the input query $q$ and $D_r^{'}$. For each $d_i \in D_r^{'}$, the final relevance score can be represented as:
\begin{equation}
    score(q, d_i) = W^TH^l_i.
\end{equation}
Similar to the previous stage,  we optimize HLATR with a list-wise contrastive loss~\cite{gao2021rethink}:
\begin{equation}
   \mathcal{L}_{\{q, D_r^{'}\}} = -log\frac{exp(score(q, d^{+}))}{\sum_{d\in D_r^{'}} exp(score(q, d))} .
\end{equation}

\begin{figure}[t]
    \centering
    \includegraphics[width= 1.0\columnwidth]{model.pdf}
    \caption{Thee detailed network structure of our HLATR (Hybrid List Aware Transformer Reranking) model.}
    \label{fig:hltar}
\end{figure}

From the perspective of the learning objective, HLATR can also be classified as a reranking model, which takes a query $q$ and a document list as input and then produces the relevance score of each query-document pair among all candidate documents. HLATR is different from the previous reranking model based on the pre-trained language model. We sort out the difference between the two modules from three aspects:

\noindent 1) Superficially, the input of HLATR and the PTM based reranking model is the query $q$ and its corresponding candidate document list. Indeed, the PTM based reranking model uses the raw content of query and document as input, but HLATR uses the combination of features generated by the previous two stages as input.

\noindent 2) Moreover, the calculation paradigm of the final ranking score is also different. In the PTM based reranking model, the relevance score of each query-doc pair is calculated independently. In the HLATR model, the relevance score of the entire input document list is calculated at one time through the serialization computing mechanism of the transformer architecture. Such a list aware calculation method allows us to better model the mutual relationship between the different documents in the entire input document list.

\noindent 3) During the training process, for PTM based reranking model, we have to control the data size of each mini-batch via a random sampling strategy when constructing $G_q$ for each query $q$ due to the high computational cost of PTM. The lightweight nature of HLATR allows us to use the entire retrieval results to construct the set of negatives, thereby greatly expanding the number of negative samples in each mini-batch. A previous study has proven that the larger number of negative samples can effectively improve the reranking performance under the contrastive learning optimization objective~\cite{gao2021rethink}.

\section{Experiment}
\subsection{Setup}

\noindent \textbf{Dataset} In this paper, we conduct experiments on two widely-used large-scale text retrieval datasets: MS MARCO Passage (called ``Passage'' below for simplicity) dataset and MS MARCO Document (called ``Document'' below) dataset. The MS MARCO dataset is constructed from Bing\'s search query logs and web documents retrieved by Bing. The statistic information of MS MARCO dataset is presented in Table~\ref{tab:static}. Since the number of parameters of HLATR is much smaller compared with the PTM reranking model, we only need to use a small amount of data to train the HLATR model. For the passage dataset, there exists an extra dev dataset which contains $34,000$ queries which can be used as training dataset. For the document dataset, we split $50,000$ queries from the orinal training dataset as the final training dataset for HLATR.

\vspace{0.15cm}
\noindent \textbf{Evaluation Metric} Following the evaluation methodology used in previous work, we report MRR@10 and MRR@100 results for the MS MARCO dev dataset due to the label of test data is not available.

\begin{table}[]
\caption{Statistics of MA MARCO dataset. ``\#Length'' represents the average text length and ``\#Relvant'' represents the number of the labeled relevant documents per query. We only count the query which at least has one relevant document.}
\label{tab:static}
\begin{tabular}{@{}c|c|c@{}}
\toprule
Dataset             & Document & Passage \\ \midrule
\#Query             & 367013/5193               & 502939/6980  \\
\#Document          & 3213835                   & 8841823  \\
\#Length            & 56.8                        &  1127.8     \\
\#Relevant  & 1.04                        & 1.06       \\ \bottomrule
\end{tabular}
\end{table}

\begin{table*}[t]
\caption{MRR@10 and MRR@100 metrics for MS MARCO Passage and Document Dataset with different kinds of retrieval and reranking models. We bold the best performances of each combination of retrieval and reraking model. The results of HLATR are statistically significant difference ($p$ < $0.01$) compared to other models.}
\label{tab:res}
\resizebox{2.0\columnwidth}{!}{
\begin{tabular}{@{}cccccccccc@{}}
\toprule
\multicolumn{2}{c}{Model/Dataset} &
  \multicolumn{4}{c}{Passage} &
  \multicolumn{4}{c}{Document} \\ \midrule
\multicolumn{1}{c|}{\multirow{2}{*}{PTM}} &
  \multicolumn{1}{c|}{\multirow{2}{*}{}} &
  \multicolumn{2}{c|}{Sparse} &
  \multicolumn{2}{c|}{Dense} &
  \multicolumn{2}{c|}{Sparse} &
  \multicolumn{2}{c}{Dense} \\ \cmidrule(l){3-10}
\multicolumn{1}{c|}{} &
  \multicolumn{1}{c|}{} &
  MRR@10 &
  \multicolumn{1}{c|}{MRR@100} &
  MRR@10 &
  \multicolumn{1}{c|}{MRR@100} &
  MRR@10 &
  \multicolumn{1}{c|}{MRR@100} &
  MRR@10 &
  MRR@100 \\ \midrule
\multicolumn{2}{c|}{Retrieval Only}  &
  18.7 &
  \multicolumn{1}{c|}{19.8} &
  38.3 &
  \multicolumn{1}{c|}{39.4} &
  20.9 &
  \multicolumn{1}{c|}{22.2} &
  36.8 &
  37.9 \\ \midrule
\multicolumn{1}{c|}{\multirow{3}{*}{BERT$_{\text base}$}} &
  \multicolumn{1}{c|}{Reranking} &
  34.4 &
  \multicolumn{1}{c|}{35.5} &
  40.1 &
  \multicolumn{1}{c|}{41.1} &
  40.4 &
  \multicolumn{1}{c|}{40.9} &
  43.4 &
  44.2 \\
\multicolumn{1}{c|}{} &
  \multicolumn{1}{c|}{WCR} &
  34.5 &
  \multicolumn{1}{c|}{35.6} &
  41.5 &
  \multicolumn{1}{c|}{42.4} &
  40.6 &
  \multicolumn{1}{c|}{41.1} &
  44.3 &
  45.1 \\
\multicolumn{1}{c|}{} &
  \multicolumn{1}{c|}{HLATR} &
  \textbf{35.0} &
  \multicolumn{1}{c|}{\textbf{36.0}} &
  \textbf{42.0} &
  \multicolumn{1}{c|}{\textbf{42.9}} &
  \textbf{42.0} &
  \multicolumn{1}{c|}{\textbf{42.5}} &
  \textbf{44.5} &
  \textbf{45.2} \\ \midrule
\multicolumn{1}{c|}{\multirow{3}{*}{BERT$_{\text large}$}} &
  \multicolumn{1}{c|}{Reranking} &
  35.4 &
  \multicolumn{1}{c|}{36.3} &
  41.0 &
  \multicolumn{1}{c|}{42.0} &
  40.6 &
  \multicolumn{1}{c|}{41.1} &
  44.2 &
  45.0 \\
\multicolumn{1}{c|}{} &
  \multicolumn{1}{c|}{WCR} &
  35.6 &
  \multicolumn{1}{c|}{36.6} &
  42.0 &
  \multicolumn{1}{c|}{42.9} &
  40.8 &
  \multicolumn{1}{c|}{41.3} &
  45.2 &
  45.9 \\
\multicolumn{1}{c|}{} &
  \multicolumn{1}{c|}{HLATR} &
  \textbf{35.9} &
  \multicolumn{1}{c|}{\textbf{36.7}} &
  \textbf{42.4} &
  \multicolumn{1}{c|}{\textbf{43.4}} &
  \textbf{42.2} &
  \multicolumn{1}{c|}{\textbf{42.8}} &
  \textbf{45.6} &
  \textbf{46.3} \\ \midrule
\multicolumn{1}{c|}{\multirow{3}{*}{RoBERTa$_{\text base}$}} &
  \multicolumn{1}{c|}{Reranking} &
  34.9 &
  \multicolumn{1}{c|}{36.1} &
  40.8 &
  \multicolumn{1}{c|}{41.7} &
  40.6 &
  \multicolumn{1}{c|}{41.0} &
  44.3 &
  45.1 \\
\multicolumn{1}{c|}{} &
  \multicolumn{1}{c|}{WCR} &
  35.2 &
  \multicolumn{1}{c|}{36.3} &
  42.0 &
  \multicolumn{1}{c|}{42.9} &
  41.0 &
  \multicolumn{1}{c|}{41.6} &
  45.1 &
  45.8 \\
\multicolumn{1}{c|}{} &
  \multicolumn{1}{c|}{HLATR} &
  \textbf{35.5} &
  \multicolumn{1}{c|}{\textbf{36.5}} &
  \textbf{42.5} &
  \multicolumn{1}{c|}{\textbf{43.5}} &
  \textbf{42.0} &
  \multicolumn{1}{c|}{\textbf{42.4}} &
  \textbf{45.4} &
  \textbf{46.1} \\ \midrule
\multicolumn{1}{c|}{\multirow{3}{*}{RoBERTa$_{\text large}$}} &
  \multicolumn{1}{c|}{Reranking} &
  36.2 &
  \multicolumn{1}{c|}{37.2} &
  41.6 &
  \multicolumn{1}{c|}{42.6} &
  40.8 &
  \multicolumn{1}{c|}{41.2} &
  45.2 &
  45.9 \\
\multicolumn{1}{c|}{} &
  \multicolumn{1}{c|}{WCR} &
  36.4 &
  \multicolumn{1}{c|}{37.4} &
  42.9 &
  \multicolumn{1}{c|}{43.9} &
  41.2 &
  \multicolumn{1}{c|}{41.6} &
  45.7 &
  46.4 \\
\multicolumn{1}{c|}{} &
  \multicolumn{1}{c|}{HLATR} &
  \textbf{36.8} &
  \multicolumn{1}{c|}{\textbf{37.8}} &
  \textbf{43.7} &
  \multicolumn{1}{c|}{\textbf{44.5}} &
  \textbf{42.3} &
  \multicolumn{1}{c|}{\textbf{42.7}} &
  \textbf{45.9} &
  \textbf{46.6} \\ \bottomrule
\end{tabular}
}
\end{table*}

\begin{table}[]
\centering
\caption{Hyperameters for HLATR, Ranker for Passage Dataset (Ranker(P)) and Document (Ranker(D)), lr represents learning rate, $k$ is the number of hard negatives, $n$ is the number of transformer layers, head is the attention head for each transformer layer, $d$ is the dimension of $v$.}
\label{tab:hyper}
\begin{tabular}{c|c|c|c}
\toprule
 & HLATR & Rerenk(P) & Ranker(D)\\ \midrule
lr             & 1e-3 & 1e-5 & 1e-5\\
batch size     & 1024 & 256 & 128\\
epoch          & 40   & 3 & 3\\
k              & 100  & 8 & 4 \\
n              & 4    & 12/24 & 12/24\\
head           & 2    & 12 & 12\\
d              & 128  & 768/1024 & 768/1024 \\ \bottomrule
\end{tabular}
\end{table}

\noindent \textbf{Implementation Details} To verify the efficiency and robustness of the HLATR module, we implement the experiments with different retrieval and reranking models. For the retrieval stage, we report results based on both sparse and dense retrieval methods. Specifically, we choose the BM25 method as the sparse retrieval model, and two state-of-the-art dense retrieval models coCondenser~\cite{gao2021unsupervised} and ANCE~\cite{xiong2021approximate} for passage and document retrieval respectively. For the reranking stage, we leverage the commonly used reranking method based on the pre-trained language model as presented in~\cite{nogueira2019passage,gao2021rethink}. However, we conduct experiments based on different pre-trained language models (BERT$_{\text base}$, BERT$_{\text large}$, RoBERTa$_{\text base}$, RoBERTa$_{\text large}$) to verify the robustness of HLATR.

We implement the BM25 retrieval process by the pyserini toolkit~\cite{lin2021pyserini}. For ANCE\footnote{https://github.com/microsoft/ANCE} and coCondenser\footnote{https://github.com/luyug/Condenser} methods, we use the publicly released checkpoint by authors. Following previous studies, for the passage dataset, we use the body field as the document text. For the document dataset, we combine the url, title and body field as the final document text. The maximum token length is set to $128$ and $512$ respectively. Detailed hype-parameters used in our experiments can be found in Table \ref{tab:hyper}.

\vspace{0.15cm}
\noindent \textbf{Baselines} We compare HLATR with the conventional retrieve-then-reranking models and the simple Weighted Combination of two-stage Ranking models (WCR). For each query-document pair, the WCR method directly weights combined the relevance scores produced by the retrieval stage reranking stage as the final ranking score:
\begin{equation}
\alpha f(E^Q(q), E^D(d)) + (1-\alpha)f(E^R(q,d)),
\end{equation}
where $\alpha \in (0,1)$.

\subsection{Results}
Table~\ref{tab:res} presents the final results of our method for both passage and document datasets. We highlight the best performances of each metric in bold. We find that our approach achieves the best performance among all the baselines.

Our model improves the text retrieval performances by a large margin compared to the conventional retrieve-then-reranking two-stage architecture. Specifically, the HLATR enhanced text retrieval systems improve the MRR@10 metric on passage and document datasets from $40.1$ to $42.0$ and $43.4$ to $44.5$ respectively under the dense retrieval and BERT-base reranking setting. This observation illustrates the effectiveness of combining features from the previous retrieval and reranking stage to improve the final ranking performance.

We can also observe that HLATR achieves consistent improvement whether using sparse or dense retrieval methods. We train the reranking models based on different pre-trained language models. Firstly, whether for the BERT or RoBERTa model, the performance of the large model is stably better than that of the base model, which is consistent with the results of previous studies~\cite{gao2021rethink,ma2021b}. Moreover, based on different reranking models, the improvement of the HLATR model is consistent. The above experimental result strongly depicts the robustness of our HLATR model.

For each experimental setting, we also report the results of the WCR method. We can see that the WCR method can effectively improve the retrieval performance, which once again proves the effectiveness of the motivation in coupling retrieval and reranking features. However, we can also observe that the effect of the WCR model still lags behind the HLATR model, which further illustrates that HLATR is a model advisable architecture for large-scale text retrieval task from the perspective of coupling features.

\subsection{Discussion and Analysis}
\label{sec:discussion}

\noindent \textbf{Multi-stage Feature Coupling} We further evaluate the influence of the feature coupling mechanism in HLATR. The experimental results are presented in Table \ref{tab:ablation}. ``w/o retrieval'' denotes the setting that removes the retrieval feature and only uses the reranking stage feature as input for HLATR. Similarly, ``w/o reranking'' notes the setting that we remove the reranking stage feature for HLATR. Unlike the ``w/o retrieval'' setting, only adopting ``position embedding'' as input for the transformer encoder is extra-conventional. Therefore, we replace the reranking stage feature vector with the query embedding and document embedding generated by the dense retrieval model. We tried several methods to concatenate these two embeddings and we find that the element-wise multiply performs best. So we only report the result based on the element-wise multiply concatenation method.

From Table \ref{tab:ablation}, we can observe that when the input feature of HLATR only contains one of the retrieval or reranking features alone, the overall ranking performance is still slightly improved. For example, the ``w/o reranking'' setting improves the MRR@10 value from $38.3$ to $39.0$ compared to the simple retrieval performance. Notably, the coupled feature input will lead to an improvement with a larger margin. This experimental result shows that coupling multi-stage features can indeed improve the final reranking performance.

\begin{table}[]
\centering
\caption{Ablation experiments result on MS MARCO Passage dataset. Dense model for retrieval stage and BERT$_{base}$ model for reranking stage.}
\label{tab:ablation}
\begin{tabular}{@{}c|cc@{}}
\toprule
\multirow{2}{*}{Model} & \multicolumn{2}{c}{Passage} \\ \cmidrule(l){2-3}
                       & MRR@10       & MRR@100      \\ \midrule
Retrieval              & 38.3         & 39.4         \\
Reranking                 & 40.8         & 41.8         \\
HLATR                   & {\bf 42.6}         & {\bf 43.5}         \\
w/o retrieval      & 41.2         & 42.2         \\
w/o ranker         & 39.0         & 39.9         \\
 \bottomrule
\end{tabular}
\end{table}

\begin{table}[]
\centering
\caption{Experiment result on MS MARCO Passage dataset of HLATR and its two variants.}
\label{tab:exp-variants}
\begin{tabular}{@{}c|cc@{}}
\toprule
\multirow{2}{*}{Model} & \multicolumn{2}{c}{Passage} \\ \cmidrule(l){2-3}
                       & MRR@10       & MRR@100      \\ \midrule
HLATR                   & {\bf 42.6}         & {\bf 43.5}         \\
WCR         & 41.8         & 42.8         \\
HLATR$_{linear}$            & 42.4         & 43.3         \\
HLATR$_{bce}$               & 41.7         & 42.6         \\ \bottomrule
\end{tabular}
\end{table}

\vspace{0.15cm}
\noindent \textbf{Architecture analysis of HLATR} To better analyze the advantages of our model in tackling the list aware text reranking problem, we conduct experiments with two variants:

\noindent 1) HLATR$_{linear}$: In this model, we replace the multi-layer transformer with a two-layer MLP model with ReLU as the activation function.

\noindent 2) HLATR$_{bce}$: In this model, we replace the original contrastive loss with binary cross-entropy loss, which can be formulated as:
\begin{align*}
    & \mathcal{L}_{bce}(q,D_r^{'}) = \frac{1}{Z}\sum_i^Z(y_i\log(\sigma(score(q,d_i))) \\
    &+(1-y_i)\log(1-\sigma(score(q,d_i))),
\end{align*}
where $y_i$ is the $\{0,1\}$ label indicates whether this document is relevant to the query and $\sigma$ denotes the sigmoid activation function.

The experiment result is presented in table \ref{tab:exp-variants}, from which we can see that with just a $2$-layer MLP encoder, HLATR can still outperform the WCR method significantly. However, compared with the HLATR model that uses the transformer structure as encoder, the performance of HLATR$_{linear}$ is degraded. We can infer that for such a list aware reranking task, the transformer model has a better ability in modeling the mutual associations between different documents in the input sequence.

Further, by comparing the performance of the HLATR$_{bce}$ model and the original model, we can conclude that the contrastive learning objective is more proper for the text ranking learning process. The essential reason is that the text reranking task is an optimization problem with imbalanced data. In the input sequence, the number of negative samples is much larger than the number of positive samples, and the loss of contrastive learning is more advantageous in dealing with such a data imbalance scenario.

\begin{table}[]
\centering
\caption{Time cost for different reranking models and HLATR model for per 1000 queries. The time cost for RoBERTa$_{base}$ and RoBERTa$_{large}$ is similar with the BERT$_{base/large}$ model. So we only report the result of BERT models.}
\label{tab:time}
\begin{tabular}{c|c|c}
\toprule
Model      & Passage   & Document   \\ \midrule
BERT$_{\text base}$        & 600ms         & 800ms          \\ \midrule
BERT$_{\text large}$      & 900ms          & 1200ms         \\ \midrule
HLATR        & 2ms         & 2ms          \\ \bottomrule
\end{tabular}
\end{table}

\vspace{0.15cm}
\noindent \textbf{Computational Cost of HLATR} To test the efficiency of HLATR, here we report the inference time cost for different kinds of ranking models and the HLATR model. All experiments are conducted on NVIDIA Tesla 32G V100.

In Table~\ref{tab:time}, we can see that the time cost of HLATR is significantly smaller than the PTM based reranking models. On the passage reranking dataset, HLATR is $300$/$450$ times more computationally efficient. Therefore, although we add an extra reranking stage, it only adds a little time cost compared with the whole retrieve-then-reranking process due to its small size of parameters and effective way to calculate the relevance score.

\noindent \textbf{Hyper-parameters of Transformer} In this part, we wish to explore how the architecture of the transformer-based encoder affects the ranking performance. In Fig~\ref{fig:analysis}, we compare the MRR@10 metric with different settings of number of transformer layers and its hidden dimension $d$ on the Passage Ranking dataset. We can see that the performance increases first and drop later when the number of layer increases. The larger $d$ is, the inflection point comes earlier. A small dimension size like $128$ leads to a better performance than a large dimension size like $512$. This can be attributed that the aim of HLATR is only designed to make use of the first two-stages features produced by the large-scale pre-trained model. So that it does not need a big number of parameters. Consequently, we need to balance the ``width'' and the ``depth'' of HLATR to achieve the best performance.

\begin{figure}[h]
    \centering
    \resizebox{0.9\columnwidth}{!}{
    \includegraphics{analysis.pdf}
    }
    \caption{The impact of Transformer architecture with different dimensions of hidden states $d$ and number of layers. The horizontal axis represents the number of transformer layers and the vertical axis represents the MRR@10 metric.}
    \label{fig:analysis}
\end{figure}

\section{Conclusion}
In this paper, we intuitively propose a unified hybrid list aware transformer reranking (HLATR) model via coupling retrieval and reranking features for multi-stage text retrieval task. The HLATR model is lightweight and can be easily applied to existing systems. Experiments on two large scale text retrieval datasets show our framework significantly outperforms previous baselines.