\section{Introduction}
\setcounter{equation}{0}

The amplitude for a $\Delta S = +1$
neutral-current transition $K(k) \to \pi(p) \ell^+(p_+) \ell^-(p_-)$,
$\ell = e , \mu$, where $(K,\pi) \in \{(K^+ , \pi^+) , (K^0 , \pi^0)\}$,
takes the form \cite{EPdR87,DAmbrosio:1998gur,DAmbrosio:2018ytt}
\begin{eqnarray}
\hspace{-0.6cm}
{\cal A}^{K \to \pi \ell^+ \ell^-} \! (s) \!\!&=& \!\!
- e^2 \times {\bar{\rm u}} (p_-) \gamma_\rho {\rm v} (p_+) \times \frac{1}{s}
\nonumber\\
&& \hspace{-0.6cm}
\times
\bigg\{
i \! \int \! d^4 x \,
\langle\pi(p)  \vert T \{  j^\rho  (0) {\cal L}^{\Delta S = 1}_{\mbox{\scriptsize non-lept}} (x)  \} \vert K(k) \rangle
\nonumber\\
&& \hspace{-0.6cm}
+ \, \left( - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\times \frac{C_{7V} (\nu)}{4 \pi \alpha} \times s
\nonumber\\
&& \hspace{-0.6cm}
\times \, \langle\pi(p)  \vert ({\bar s} \gamma^\rho d) (0) \vert K(k) \rangle
\bigg\}
\lbl{amplitude}
\\
\hspace{-0.6cm}
&=& \!\!
- e^2 \times {\bar{\rm u}} (p_-) ({\not\! k} + {\not\!\! p}) {\rm v} (p_+)
\times \frac{W_{K\pi}(z)}{16 \pi^2 M_K^2}
.
\nonumber
\end{eqnarray}
Here $s= (k-p)^2 = (p_+ + p_-)^2 = z M_K^2$ is the square of the di-lepton invariant mass, $M_K$ denotes
the kaon mass [for our present purposes, it is not necessary to distinguish between the masses of charged
and neutral kaons], $e$ is the electric charge, $G_{\rm F}$ the Fermi constant, and $V_{ud}$, $V_{us}$ are elements of the CKM matrix.
Each matrix element occurring in the first expression has to be evaluated in QCD
with three active flavours. In particular, the electromagnetic current $j^\rho$ is made up from the
contributions of the $u$, $d$ and $s$ quarks,
\begin{equation}
\hspace{-0.005cm}
j^\rho (x) = \!\sum_{q = u,d,s} \! e_q ({\bar q} \gamma^\rho q) (x),~
e_u = +\frac{2}{3} , \, e_d = e_s = - \frac{1}{3}
.
\end{equation}
Furthermore, ${\cal L}^{\Delta S = 1}_{\mbox{\scriptsize non-lept}} (x)$
represents the effective La\-gran\-gian for $\Delta S = +1$ transitions
\cite{Gaillard:1974nj,Altarelli:1974exa,Witten:1976kx,Shifman:1975tn,Wise:1979at,Gilman:1979bc},
and involves the two current-current
four-quark operators, as well as the QCD penguin operators, modulated by the appropriate Wilson coefficients
\begin{equation}
{\cal L}^{\Delta S = 1}_{\mbox{\scriptsize non-lept}} (x) =
\left(- \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud}\right) \times \sum_{I=1}^6 C_I (\nu) Q_I ( x ; \nu)
.
\end{equation}
The above representation of ${\cal A}^{K \to \pi \ell^+ \ell^-}$
holds at order ${\cal O}(\alpha G_{\rm F})$, $\alpha = e^2/4\pi$ denoting the fine-structure constant,
and with a corresponding form factor for each $(K, \pi)$ channel,
that is for the CP conserving transitions $K^+ \to \pi^+ \ell^+ \ell^-$ and $K_S \to \pi^0 \ell^+ \ell^-$,
but also for the direct-CP violating part of the amplitude for the decay $K_L \to \pi^0 \ell^+ \ell^-$.
The purpose of the perhaps somewhat unfamiliar first expression of ${\cal A}^{K \to \pi \ell^+ \ell^-}$
given in Eq. \rf{amplitude} above is to explicitly display the two main components of the weak transition form factor defined
by the second expression in Eq. \rf{amplitude},
\begin{equation}
W_{K\pi}(z) = W_{K\pi}^{\rm LD}(z ; \nu) + W_{K\pi}^{\rm SD}(z ; \nu)
.
\lbl{SD_LD}
\end{equation}
The first part is dominated by long-distance contributions, the second one is generated
at short distances, at the electroweak scale or beyond in case new physics sets in at even
higher energies.
Notice the appearance of an ultraviolet subtraction scale $\nu$ in each of the two terms
in Eq. \rf{SD_LD}.
Their sum $W_{K\pi}(z)$ appears in the amplitude ${\cal A}^{K \to \pi \ell^+ \ell^-}$
entering the physical decay rate and should of course not depend on $\nu$ anymore.
Explicitly, one has
\begin{eqnarray}
&&
\hspace{-1.5cm}
\left[ s (k + p)_\rho - (M_K^2 - M_\pi^2) (k - p)_\rho \right]
\times \frac{W_{K\pi}^{\rm LD}(z ; \nu)}{16 \pi^2 M_K^2}
\nonumber\\
&&
= i \! \int \! d^4 x \,
\langle\pi(p)  \vert T \{  j_\rho  (0) {\cal L}^{\Delta S = 1}_{\mbox{\scriptsize non-lept}} (x)  \} \vert K(k) \rangle
\lbl{W_LD}
\end{eqnarray}
for the long-distance dominated part, whereas the contribution from short distances, which
arises from a local term \cite{Witten:1976kx,Gilman:1979ud,Dib:1988md,Dib:1988js}
\begin{equation}
{\cal L}^{\Delta S = 1}_{\rm lept} (x ; \nu) =
- \frac{G_{\rm F}}{\sqrt{2}} V_{us}^* V_{ud} C_{7V} (\nu)  Q_{7V} (x)
,
\lbl{eff_lag_lept}
\end{equation}
with $Q_{7V} = ( {\bar s}^i d_i )_{V-A} ( {\bar \ell} \ell )_V$, reads
\begin{equation}
\frac{W_{K\pi}^{\rm SD}(z ; \nu)}{16 \pi^2 M_K^2} = - \left( - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\times \frac{C_{7V} (\nu)}{4 \pi \alpha} \times C_{K\pi} f_+ (s)
\lbl{W_SD}
\end{equation}
in terms one of the form factor $f_+(s)$ describing the matrix element of the $\Delta S = +1$
neutral current
\begin{eqnarray}
&&
\hspace{-1.25cm}
\langle \pi (p) \vert \left( {\bar s} \, \gamma_\rho  d \right) \! (0)  \vert K (k) \rangle
\nonumber\\
&&
=
C_{K\pi} \left[ (k + p )_\rho f_+^{K\pi} (s) + (k-p)_\rho f_-^{K\pi} (s) \right]
.
\lbl{Kl3_ff}
\end{eqnarray}
In these last formulas, $C_{K\pi}$ denotes a Clebsch-Gordan coefficient, chosen such that
$f_+(0)=1$ in the flavour-$SU(3)$ limit.
In both Eqs. \rf{W_LD} and \rf{Kl3_ff} the terms proportional to $(k-p)_\rho$ do not contribute
to the amplitude ${\cal A}^{K \to \pi \ell^+ \ell^-}$ due to the conservation of the leptonic
current, $(k-p)_\rho \, {\bar{\rm u}} (p_-) \gamma^\rho {\rm v} (p_+) = 0$. For the sake of
completeness, let us recall that in the standard model there is also a contribution to the short-distance
part of the amplitude ${\cal A}^{K \to \pi \ell^+ \ell^-}$ coming from a term proportional to
$C_{7A}  Q_{7A} (x)$, with $Q_{7A} = ( {\bar s}^i d_i )_{V-A} ( {\bar \ell} \ell )_A$.
This term does not play any role in the present discussion, as it does not involve the short-distance
scale $\nu$ and is anyway CKM suppressed. We will therefore not mention it any further.

In $W_{K\pi}^{\rm SD}(z ; \nu)$ the scale dependence is entirely carried by the Wilson coefficient
$C_{7V} (\nu)$,
\begin{equation}
\nu \frac{d C_{7V} (\nu)}{d \nu} = \frac{\alpha}{\alpha_s (\nu)} \sum_{J=1}^6 \gamma_{J,7V} (\alpha_s)\,  C_J (\nu)
.
\lbl{scale_C7V}
\end{equation}
The anomalous dimensions $\gamma_{J,7V} (\alpha_s)$ are known to leading
\cite{Gilman:1979ud,Dib:1988md,Dib:1988js,Flynn:1988ve} and to next-to-leading
\cite{Buras:1994qa} orders,
\begin{equation}
\gamma_{J,7V} (\alpha_s) = \gamma_{J,7V}^{(0)} \, \frac{\alpha_s}{4\pi}
+ \gamma_{J,7V}^{(1)} \left(\frac{\alpha_s}{4\pi}\right)^2 + \cdots
.
\end{equation}
In $W_{K\pi}^{\rm LD}(z ; \nu)$, the scale dependence arises from the singular
structure, at short distances, of the time-ordered product of the electromagnetic
current with the $\Delta S = +1$ effective Lagrangian [both $j^\mu$ and
${\cal L}_{\mbox{\scriptsize non-lept}}^{\Delta S = +1}$ are finite operators].
This short-distance singularity can be studied perturbatively within the
operator-product expansion (OPE) \cite{Wilson:1969zs,Wilson:1972ee}.
After renormalization [we use dimensional regularization in the
${\overline{\rm MS}}$ subtraction scheme, and below $q^\mu$ denotes a {Euclidean} momentum,
with $q^2=s<0$, whose components become simultaneously large] one obtains the general structure,
cf. Ref. \cite{DAmbrosio:2018ytt},
\begin{eqnarray}
&&
\hspace{-1.35cm}
\lim_{q \to \infty} i
\!\int d^4 x \, e^{i q \cdot x}
T \{ j^\mu (x)
{\cal L}_{\mbox{\scriptsize non-lept}}^{\Delta S = +1} (0) \}
\nonumber\\
&& \hspace{-1.35cm}
=
 [ q^\mu q^\rho - q^2 \eta^{\mu\rho} ] \!\times \! {\bar s} \gamma_\rho (1 - \gamma_5) d
\lbl{OPE}
\\
&& \hspace{-1.35cm}
\times \! \left(\! - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\!\times \! \frac{1}{4 \pi} \sum_{I=1}^6 C_I (\nu) \xi_I (\alpha_s \, ; \nu^2/q^2)
+ \, {\cal O} (q)
,
\nonumber
\end{eqnarray}
with
\begin{equation}
\xi_I (\alpha_s \, ;  \nu^2/s) =
\sum_{p\ge 0} \sum_{r =0}^{p+1}
\xi^I_{pr}
\alpha_s^p (\nu) \ln^r (-\nu^2/s)
.
\lbl{xi_I}
\end{equation}
The subleading terms in the OPE have finite coefficients, and do therefore not
depend on the renormalization scale $\nu$. Consequently [recall that $z\equiv s/M_K^2$],
\begin{eqnarray}
\hspace{-0.65cm}
\nu \frac{d W_{K\pi}^{\rm LD}(z ; \nu)}{d\nu}
\!\!&=&\!\! - 16 \pi^2 M_K^2 C_{K\pi} f_+ (s) \times \left( - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\nonumber\\
&& \!\!
\times \frac{1}{4 \pi} \nu \frac{d}{d\nu} \sum_{I=1}^6 C_I (\nu) \xi_I (\alpha_s \, ; \nu^2/s)
.~~~
\end{eqnarray}
From Eqs. \rf{OPE} and \rf{xi_I} one infers that in the {Euclidean} region $z\to - \infty$
the form factor $W_{K\pi}^{\rm LD}(z ; \nu)$ behaves asymptotically like
powers of $\ln (-z)$ times powers of the strong coupling $\alpha_s$.

In the present Letter, we wish to discuss how the above short-distance behaviour can be reproduced,
up to the order ${\cal O} (\alpha_s)$, by a model involving an infinite number of equally-spaced
[in mass squared] zero-width resonances. Models of this kind were considered in various contexts
in the past, see for instance the articles \cite{Shifman:2000jv,Golterman:2001nk,Golterman:2001pj} and references therein.
Such Regge-type models find their justification in the properties of the QCD spectrum in the limit
of a large number of {colors} \cite{tHooft74,tHooft:1974pnl,Witten79}. More recently quite efficient methods,
based on the Converse Mapping Theorem \cite{FGD95} and the notion of harmonic sums, were developed in order
to handle such models \cite{Friot:2005cu,Rafael:2012sy,Greynat:2013cta}.
In the case at hand, the general idea put forward in Ref. \cite{DAmbrosio:2018ytt}
consists in decomposing the long-distance dominated part of the form factor as a sum
\begin{equation}
W_{K\pi}^{\rm LD} (z) = W_{K\pi}^{\pi\pi} (z) + W_{K\pi}^{\rm res} (z ; \nu)
.
\lbl{model}
\end{equation}
The first term describes the contribution from the resonant $P$-wave
two-pion intermediate state to $W_{K\pi}(z)$. It is constructed upon assuming
that it is given by an unsubtracted dispersion integral. The absorptive part consists
of the two-pion spectral density $\rho_{K\pi}^{\pi\pi} (s)$,
and is obtained upon inserting a two-pion intermediate state in the
representation of the form factor given in Eq. \rf{amplitude}. This contribution
is not relevant for the discussion of the short-distance properties, and we
refer the interested reader to Ref. \cite{DAmbrosio:2018ytt} for details.
The second term describes the contributions from the intermediate
states with higher thresholds as a sum of zero-width resonances.
The weight with which each resonance contributes must be chosen such as to reproduce
the short-distance behaviour given in Eqs. \rf{OPE} and \rf{xi_I}. In Ref. \cite{DAmbrosio:2018ytt},
this matching has been achieved at order ${\cal O} (\alpha_s^0)$, which involves only
a constant term and a term linear in $\ln (-z)$. Here we wish to extend this matching to
the order ${\cal O} (\alpha_s)$, where we also encounter a term quadratic in $\ln (-z)$.

The remainder of this Letter is thus organized as follows. In section \ref{SD_coeffs}
we determine the required coefficients $\xi_{01}^I$, $\xi_{11}^I$ and $\xi_{12}^I$, relying
on a renormalization-group argument given in Ref. \cite{DAmbrosio:2018ytt}. For
{pedagogical} reasons, we then review (Section \ref{matching_LO}) the matching at order ${\cal O} (\alpha_s^0)$,
making the discussion of Ref. \cite{DAmbrosio:2018ytt} simpler and, hopefully, also
more intuitive. The matching at order ${\cal O} (\alpha_s)$ is then presented in Section \ref{matching_NLO}.
Finally, we discuss some consequences and features of our results (Section \ref{discussion}).

\section{Determination of the short-distance coefficients}\label{SD_coeffs}
\setcounter{equation}{0}

From the perspective of perturbative QCD, the matrix element that defines the long-distance
dominated part $W_{K\pi}^{\rm LD}(z ; \nu)$ is described by the {Feynman} diagrams of the
type shown {\rm in} Fig. \ref{OPE_QCD_fig}. This is obviously not a realistic description
of $W_{K\pi}^{\rm LD}(z ; \nu)$, except at short distances, when the momentum transfered
to the electromagnetic current becomes large in the space-like or Euclidean domain, $s\to -\infty$.
The leading contribution to the OPE of the electromagnetic current and the four-quark operators
describing $\Delta S = +1$ transitions involves the neutral-current operator ${\bar s} \gamma_\rho (1 - \gamma_5) d$,
and corresponds to the diagram $(a)$ of Fig. \ref{OPE_QCD_fig}. The calculation with a bare
four-quark operator $Q_I$ is straightforward, see Ref. \cite{DAmbrosio:2018ytt}, and gives
\begin{eqnarray}
&&
\hspace{-1.35cm}
\lim_{q \to \infty} i
\!\int d^4 x \, e^{i q \cdot x}
T \{ j^\mu (x)
{\cal L}_{\mbox{\scriptsize non-lept}}^{\Delta S = +1} (0) \}
\nonumber\\
&& \hspace{-1.35cm}
= [ q^\mu q^\rho - q^2 \eta^{\mu\rho} ] \times [{\bar s} \gamma_\rho (1 - \gamma_5) d](0)
\times \left( - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\nonumber\\
&& \hspace{-1.35cm}
\times \frac{1}{4 \pi} \sum_{I=1}^6 C_I \left\{ \xi_{00}^I
- \xi_{01}^I \left[ \frac{2}{D-4} + \ln\left(\frac{- q^2}{\nu^2}\right) \right]
+ {\cal O}(\alpha_s) \right\}
\nonumber\\
&& \hspace{-1.35cm}
+ \ {\cal O} (q)
.
\end{eqnarray}
with
\begin{eqnarray}
\xi_{01}^I &=& \frac{1}{4\pi} \, \frac{8}{9}
\times
\left(
N_c \, , \, 1 \, , \, -1 \, , \, -N_c \, , \, 0 \, , \, 0
\right),
\nonumber\\
\xi_{00}^I &=& \frac{\xi_{01}^I}{3} \times
\left\{
\begin{array}{l}
 2~~ {\rm NDR} \\
 \\
 5~~ {\rm HV}
\end{array}
\right.
,
\lbl{xi_LO}
\end{eqnarray}
\begin{figure*}[t]
\center\epsfig{figure=Kpill_SD_NLO.eps,height=5.5cm}
\caption{Diagrams contributing to the leading
short-distance behaviour of the time-ordered product of the
electromagnetic current (materialized by the wiggly line on the left)
with one of the four-quark operators, whose insertion is
shown by the crossed-circle ($\oplus$) vertex on the right of diagram $(a)$,
of order ${\cal O}(\alpha_s^0)$, and of the diagrams
$(b)$ to $(e)$. The latter also form a representative subset of
the diagrams contributing to the gluonic corrections at order ${\cal O}(\alpha_s)$.
The vertex represented by a circular black blob ({\large$\bullet$}) on the right of diagram $(f)$ corresponds
to the insertion of an order ${\cal O}(\alpha_s)$
{counterterm} into the lowest-order diagram $(a)$. The other external lines
represent an incoming $d$ quark and an outgoing $s$ quark, while the
loop consists of a  ${\bar q}$-$q$ pair, where $q=u,d,s$.} \label{OPE_QCD_fig}
\end{figure*}

\noindent
The result for $\xi_{00}^I$ depends on the scheme used to handle the Dirac matrices
in $D$ dimensions, here either naive dimensional regularization (NDR)
\cite{Chanowitz:1979zu} or the 't Hooft-Veltman (HV) scheme \cite{tHooft:1972tcz,Breitenlohner:1977hr}.
The divergent part is removed through the renormalization of the (bare) coupling $C_{7V}$.
Notice also that at this order the Wilson coefficients are not yet running.
In the presence of QCD corrections, the four-quark
operators and the corresponding Wilson coefficients are renormalized. But this
does not take care of all the divergences, since two external lines of the $Q_I$ operators
are closed into a loop with the insertion of the electromagnetic current, see Fig. \ref{OPE_QCD_fig}.
These remaining divergences are again absorbed through the higher-order renormalization
of $C_{7V}$. To all orders in the powers of $\alpha_s$, and after renormalization
through minimal subtraction in the ${\overline{\rm MS}}$ scheme, the leading term in the OPE
then takes the form given in Eqs. \rf{OPE} and \rf{xi_I}.
From the scale dependence of the Wilson coefficients $C_{7V}(\nu)$, given in Eq. \rf{scale_C7V},
and $C_I(\nu)$,
\begin{equation}
\nu \frac{d C_I (\nu)}{d \nu} = \sum_{J=1}^6 \gamma_{J,\, I} (\alpha_s) C_J (\nu)
,
\lbl{scale_CI}
\end{equation}
one infers that the total form factor in Eq. \rf{SD_LD} will be scale independent provided the equation
\begin{equation}
\nu \frac{d \xi_I (\alpha_s \, ; \nu^2/s)}{d\nu} + \sum_{J=1}^6 \gamma_{I,J} (\alpha_s) \xi_J (\alpha_s \, ; \nu^2/s)
= - \frac{\gamma_{I,7V} (\alpha_s)}{\alpha_s (\nu)}
\lbl{RG_equ}
\end{equation}
holds. At order ${\cal O}(\alpha_s^0)$, this allows to recover
the values of the coefficients $\xi_{01}^I$ from the known one-loop anomalous
dimension matrices.
Including first-order QCD corrections, one obtains, after renormalization,
\begin{eqnarray}
&&
\hspace{-1.25cm}
\lim_{q \to \infty} i
\!\int d^4 x \, e^{i q \cdot x}
T \{ j^\mu (x)
{\cal L}_{\mbox{\scriptsize non-lept}}^{\Delta S = +1} (0) \}
\lbl{OPE_NLO}
\\
&& \hspace{-1.0cm}
= [ q^\mu q^\rho - q^2 \eta^{\mu\rho} ] \times {\bar s} \gamma_\rho (1 - \gamma_5) d
\times \left( - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\nonumber\\
&& \hspace{-1.0cm}
\times \frac{1}{4 \pi} \sum_{I=1}^6 C_I (\nu) \left\{ \xi_{00}^I - \xi_{01}^I \ln\left(\frac{- q^2}{\nu^2}\right)
+ \alpha_s (\nu) \left[ \xi_{10}^I
\right.\right.\nonumber\\
&& \hspace{-1.0cm}
\left.\left.
- \xi_{11}^I \ln\left(\frac{- q^2}{\nu^2}\right)
+ \xi_{12}^I \ln^2\left(\frac{- q^2}{\nu^2}\right) \right] + {\cal O}(\alpha_s^2) \right\}
+ {\cal O} (q)
.
\nonumber
\end{eqnarray}
From the above renormalization-group argument and the known two-loop
anomalous dimension matrices \cite{Altarelli:1980fi,Buras:1989xd,Buras:1991jm,Buras:1992tc,Ciuchini:1993vr},
one infers the relations
\begin{equation}
\xi_{12}^I = \frac{1}{(4\pi)^2} \, \frac{4}{27}  \left(N_c - \frac{1}{N_c} \right)
\times
\left(
0 \, , \, - 8 \, , \, + 11 \, , \, N_f \, , \, 0 \, , \, N_f
\right) \!,
\end{equation}
and
\begin{eqnarray}
&& \hspace{-1.25cm}
\xi^I_{11} = \frac{1}{(4\pi)^2} \, \frac{8}{3}  \left(N_c - \frac{1}{N_c} \right)
\nonumber\\
&& \hspace{-0.75cm}
\times
\left\{
\begin{array}{l}
\!\!\left( \frac{N_c}{2} \, , \, -\frac{19}{18} \, , \, \frac{17}{9} \, , \, \frac{7}{6} - \frac{N_c}{2} \, , \, 0 \, , \, \frac{7}{6} \right)~~ {\rm NDR} \\
\\
\!\!\left( \frac{N_c}{2} \, , \, -\frac{5}{18} \, , \, \frac{13}{9} \, , \, \frac{7}{6} - \frac{N_c}{2} \, , \, 0 \, , \, \frac{7}{6} \right)~~ {\rm HV}
\end{array}
\right.
\!.
\end{eqnarray}
The coefficients $\xi_{10}^I$ cannot be obtained this way, and their
determination would require a full two-loop calculation, which we will
however not need to attempt here.

\section{The matching at order ${\cal O} (\alpha_s^0)$}\label{matching_LO}
\setcounter{equation}{0}

 It is convenient to represent the resonance contribution to the form factor in
 Eq. \rf{model} as a dispersive integral
\begin{equation}
W^{\rm res}_{K\pi} (z ; D) = \frac{C_{K\pi} f_+^{K\pi} (z M_K^2)}{4\pi} \int dx \, \frac{\rho^{\rm res}_{K\pi}(x ; D)}{x - z M_K^2 - i0}
,
\end{equation}
where the spectral density is constructed order by order in the expansion in powers
of the strong coupling $\alpha_s$,
\begin{equation}
\rho^{\rm res}_{K\pi}(x ; D) = \rho^{\rm res;0}_{K\pi}(x ; D) + \rho^{\rm res;1}_{K\pi}(x ; D)
+ \cdots ,
\end{equation}
and is defined in $D$ dimensions. This last point requires some explanation.
Indeed, by power counting the naive superficial degree of divergence of the diagrams in Fig. \ref{OPE_QCD_fig}
is two, and becomes actually zero once the Ward identity following from the conservation of the electromagnetic current
is implemented. Therefore, in four dimensions the form factor satisfies a once-subtracted dispersion relation, which
would thus constitute an appropriate representation to start with. However, the information from
short distances at our disposal, and summarized in the preceding section, comes from calculations
done within a dimensional renormalization scheme with minimal subtraction and not
within a momentum subtraction one. Since we want to make direct use of this information without transforming it
first into a different scheme, we choose instead to start from an unsubtracted dispersion relation in $D$ dimensions.
As will hopefully become clear in the sequel, far from being an unnecessary complication, this choice even presents some advantages
in actually guiding our intuition in the process of constructing an appropriate ansatz for the spectral density
$\rho^{\rm res;0}_{K\pi}(x ; D)$ or $\rho^{\rm res;1}_{K\pi}(x ; D)$.

At the one-loop level, i.e. order ${\cal O} (\alpha_s^0)$, the resonance representation we are looking for
should match, at large negative values of $s$, the behaviour in the same limit
of the unrenormalized QCD diagram $(a)$ of Fig. \ref{OPE_QCD_fig}. In order to reproduce a logarithmic
behaviour at short distances, an infinite set of resonances is required \cite{Witten79}. We will consider a simple
Regge-type description of the resonance spectrum in terms of equally spaced [in mass squared] zero-width states.
Accordingly, the general structure of $\rho^{\rm res;0}_{K\pi}(x ; D)$ is given by
\begin{eqnarray}
\hspace{-0.5cm}
\rho^{\rm res;0}_{K\pi}(x ; D) \!\!&=&\!\! A_0 (D) (4\pi)^{2 - \frac{D}{2}} \left( \frac{M^2}{\nu^2_{\rm MS}} \right)^{\frac{D}{2} - 2}
\Gamma \left(2 - \frac{D}{2} \right)
\nonumber\\
&& \!\! \times\sum_{n\ge 1} M^2 \mu_n^{(0)} (D) \delta (x - nM^2)
,
\end{eqnarray}
with so-far unspecified functions $A_0(D)$ and $\mu_n^{(0)} (D)$. The additional prefactors simply
account for the structure of diagram $(a)$ of Fig. \ref{OPE_QCD_fig} in $D$ dimensions. Since the one-loop
divergence is already contained in the factor $\Gamma (2 - D/2)$, $A_0(D)$ has to be regular at $D=4$.
It accounts for the scheme dependence, for instance in handling Dirac matrices
in $D$ dimensions, see Section \ref{SD_coeffs}. $M$ denotes
the mass of the lowest-lying resonance and $\nu_{\rm MS}$ denotes the renormalization scale in the
minimal subtraction scheme.
Then, with $w\equiv -s/M^2$,
\begin{eqnarray}
\hspace{-0.5cm}
\int dx \, \frac{\rho^{\rm res;0}_{K\pi}(x ; D)}{x+wM^2} \!\!&=& \!\!
A_0 (D) (4\pi)^{2 - \frac{D}{2}} \left( \frac{M^2}{\nu^2_{\rm MS}} \right)^{\frac{d}{2} - 2}
\nonumber\\
&&\!\! \times \,
\Gamma \left(2 - \frac{D}{2} \right)
\sum_{n\ge 1} \frac{\mu_n^{(0)} (D)}{n+w}
.
\end{eqnarray}
The value of the above dispersive integral at $w=0$ reproduces the divergent part of
the diagram $(a)$ of Fig. \ref{OPE_QCD_fig}
\begin{eqnarray}
\hspace{-0.5cm}
\int \frac{dx}{x} \, \rho^{\rm res;0}_{K\pi}(x ; D) \!\!\!&=&\!\!\!
A_0 (D) (4\pi)^{2 - \frac{D}{2}} \left( \frac{M^2}{\nu^2_{\rm MS}} \right)^{\frac{d}{2} - 2}
\nonumber\\
&&
\!\!\! \times \Gamma \left(2 - \frac{D}{2} \right)
\sum_{n\ge 1} \frac{\mu_n^{(0)} (D)}{n}
.
\end{eqnarray}
$A_0(4)$ being so far a free parameter, one may, without loss of generality, require that
\begin{equation}
\sum_{n\ge 1} \frac{\mu_n^{(0)} (D)}{n} = 1 + (D-4) \mu^{(0)} + {\cal O} ((D-4)^2)
,
\lbl{cond_1}
\end{equation}
with $\mu^{(0)}$ a constant that remains unspecified for the time being.
Then
\begin{eqnarray}
\hspace{-0.55cm}
\int \frac{dx}{x} \, \rho^{\rm res;0}_{K\pi}(x ; D) \!\!&=& \!\!
A_0(4) \bigg[
- \frac{2}{D-4} + \ln \frac{\nu^2}{M^2}
\\
&& \!\!
- 2 \frac{A_0^\prime(4)}{A_0(4)}
- 2 \mu^{(0)}
\bigg] + {\cal O} (D-4),
\nonumber
\end{eqnarray}
with $\nu \equiv \nu_{\rm MS} e^{-\gamma_E/2} \sqrt{4\pi}$
the subtraction scale in the ${\overline{\rm MS}}$ scheme.
The constant $\mu^{(0)}$ can actually be absorbed without loss
of generality into $A_0^\prime(4)$, which has also not been specified so far.
The remaining, subtracted, dispersive integral
\begin{eqnarray}
\hspace{-0.65cm}
w M^2 \! \int \frac{dx}{x} \, \frac{\rho^{\rm res;0}_{K\pi}(x ; D)}{x+wM^2} \!\!&=& \!\!
A_0 (D) (4\pi)^{2 - \frac{D}{2}} \left( \frac{M^2}{\nu^2_{\rm MS}} \right)^{\frac{d}{2} - 2}
\\
&& \times \,
\Gamma \left(2 - \frac{D}{2} \right) \sum_{n\ge 1} \frac{w \mu_n^{(0)} (D)}{n(n+w)}
\nonumber
\end{eqnarray}
should then be finite as $D\to 4$. This in turn will be the case if we require
\begin{equation}
\mu_n^{(0)} (D) = (D-4) {\bar\mu}^{(0)}_n + {\cal O} ((D-4)^2)
\lbl{cond_2}
\end{equation}
and
\begin{equation}
\xi(w) = \sum_{n\ge 1} \frac{{\bar\mu}_n^{(0)}}{n(n+w)} ~~ {\rm converges}
.
\lbl{cond_3}
\end{equation}
Consequently
\begin{equation}
w M^2 \int \frac{dx}{x} \, \frac{\rho^{\rm res;0}_{K\pi}(x ; D)}{x+wM^2} =
A_0 (4) (- 2 w) \xi (w) + {\cal O} (D-4)
.
\end{equation}
Finally, in order to reproduce the correct matching with the short-distance behaviour,
one must also require
\begin{equation}
- 2 w \xi (w) = \ln w + {\rm Cst} + \cdots
\end{equation}
as $w \to +\infty$. At this stage, one may observe that ${\mu}_n^{(0)} \propto (D-4) {\bar\mu}^{(0)}$,
with ${\bar\mu}^{(0)}$ a constant,
would provide a convergent series $\xi (w)$, endowed with an asymptotic logarithmic behaviour.
But it would fail to satisfy the condition \rf{cond_1}. As we now show, this defect can be easily
repaired. Intuitively, the task will consist in providing a convergence factor for the sum \rf{cond_1} when $D<4$,
but which is no longer operative for $D=4$, otherwise the sum $\xi(w)$ would converge too quickly,
and would no longer exhibit a logarithmic behaviour for large values of $w$. Instead, it would rather behave
as a constant or an inverse power of $w$.
Consider therefore the ansatz
\begin{equation}
\mu_n^{(0)} (D) = {\rm a}^{(0)} (D) n^{\frac{D}{2} - 2}
.
\end{equation}
Then one has
\begin{equation}
\sum_{n\ge 1} \frac{\mu_n^{(0)} (D)}{n} = {\rm a}^{(0)} (D) \zeta \left( 3 - \frac{D}{2}  \right)
,
\end{equation}
where $\zeta(s)$ denotes Riemann's zeta-function, see \cite[section 25.2]{DLMF}.
Since, as $D\to 4$, $\zeta(3-D/2) \sim -2/(D-4) + \gamma_E$, where $\gamma_E$ is
the Euler constant, any choice of the form
\begin{equation}
{\rm a}^{(0)} (D) = \frac{1}{\zeta \left( 3 - \frac{D}{2}  \right) + f(D)}
,
\end{equation}
where $f(D)$ is an arbitrary function regular at $D=4$, will lead to
\begin{equation}
\sum_{n\ge 1} \frac{\mu_n^{(0)} (D)}{n} = 1 + \frac{f(4)}{2} (D-4) + {\cal O}((D-4)^2)
\end{equation}
and thus satisfy the condition \rf{cond_1} with $\mu^{(0)} = f(4)/2$.
The condition \rf{cond_2} is then also satisfied, with ${\bar\mu}_n = - 1/2$, so that
\begin{equation}
\Gamma \left(2 - \frac{D}{2} \right) \mu_n (D) = 1 + {\cal O} (D-4)
.
\end{equation}
The condition \rf{cond_3} is met as well, with
\begin{equation}
- 2 w \xi (w) = \sum_{n\ge 1} \frac{w}{n(n+w)} = \gamma_E + \psi(1+w)
.
\end{equation}
Putting everything together, one ends up with
\begin{eqnarray}
&&
\hspace{-1.25cm}
\int dx \, \frac{\rho^{\rm res;0}_{K\pi}(x ; D)}{x+wM^2} =
A_0(4) \bigg\{
- \frac{2}{D-4} + \ln \frac{\nu^2}{M^2} - 2 \frac{A_0^\prime(4)}{A_0(4)}
\nonumber\\
&&
-2 \mu^{(0)} - \left[ \gamma_E + \psi (1 + w) \right]
\bigg\}  + {\cal O}(D-4)
.
\end{eqnarray}
In this last formula, the di-gamma function $\psi$ resums the dispersive integral
\begin{equation}
\psi(1+w) = - \gamma_E \!+\! M^2 w \int \! \frac{dx}{x} \, \frac{1}{x + M^2 w} \sum_{n\ge 1} M^2 \delta (x - nM^2)
.
\end{equation}
Finally, for large positive $w$ the di-gamma function behaves as $\psi(1+w) \sim \ln w = \ln(-s/M^2)$, so that the correct
short-distance behaviour is also recovered, provided one takes
\begin{equation}
A_0 (4) = - 16 \pi^2 M_K^2 \left( - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\times \sum_{I=1}^6 C_I (\nu) \xi_{01}^I
\end{equation}
and
\begin{eqnarray}
\hspace{-0.67cm}
A_0^\prime(4) + \left( \mu^{(0)} + \frac{\gamma_E}{2} \right) A_0(4) \!\!\!&=&\!\!\!
+ 16 \pi^2 M_K^2 \! \left( - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\nonumber\\
&& \!\!\!
\times \sum_{I=1}^6 C_I (\nu) \frac{\xi_{00}^I}{2}
.
\end{eqnarray}
Minimal subtraction of the divergence leads to the renormalized dispersion relation
\begin{eqnarray}
&&
\hspace{-1.37cm}
\int dx \, \frac{\rho^{\rm res;0}_{K\pi}(x ; D)}{x+wM^2} =
- 16 \pi^2 M_K^2 \! \left( \! - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\!\times \! \sum_{I=1}^6 C_I (\nu)
\nonumber\\
&&
\hspace{+1.5cm}
\times\left[
\xi_{01}^I \left(\ln \frac{\nu^2}{M^2} - \psi(1+w) \right) + \xi_{00}^I
\right]
\nonumber\\
&&
\hspace{+1.5cm}
+ \, {\cal O} (D-4)
,
\lbl{ren_disp_rho_0}
\end{eqnarray}
with
\begin{eqnarray}
\hspace{-0.65cm}
\rho^{\rm res;0}_{K\pi}(x ; D) \!\!&=& \!\! - 16 \pi^2 M_K^2 \left( - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\times \sum_{I=1}^6 C_I (\nu)
\nonumber\\
&& \!\!
\times
\bigg\{
\xi_{01}^I - \frac{D-4}{2} \left[\xi_{00}^I + (f(4) + 2 \gamma_E) \xi_{01}^I \right] \! \bigg\}
\nonumber\\
&& \!\!
\times
\left( \frac{M^2}{\nu^2} \right)^{\frac{D}{2} - 2}
\Gamma \left(2 - \frac{D}{2} \right)
\lbl{rho_0}
\\
&& \!\!
\times
\sum_{n\ge 1} \frac{M^2 n^{\frac{D}{2} - 2}}{\zeta \left( 3 - \frac{D}{2} \right) + f(D)} \, \delta (x - nM^2)
.
\nonumber
\end{eqnarray}

We may now, in some sense, reverse-engineer the whole construction, starting directly from the representation
of $\rho^{\rm res;0}_{K\pi}(x ; D)$ in $D$ dimensions given in Eq. \rf{rho_0} above and showing that it satisfies
the required properties. This will be useful in Section \ref{matching_NLO}, where we will only sketch the
construction of the spectral density $\rho^{\rm res;0}_{K\pi}(x ; D)$, give the result, and show that it indeed
exhibits the appropriate features.
Using the Mellin representation
\begin{equation}
\frac{1}{1+\frac{w}{n}} = \int_{c_1-i\infty}^{c_1+i\infty} \frac{du}{2 \pi i} \left(\frac{w}{n}\right)^{-u} \Gamma(u) \Gamma(1-u)
,
\lbl{mellin}
\end{equation}
where $0< c_1 <1$, one has
\begin{eqnarray}
&&
\hspace{-1.25cm}
\int dx \, \frac{\rho^{\rm res;0}_{K\pi}(x ; D)}{x+wM^2}
\nonumber\\
&&
\hspace{-1.25cm}
= - 16 \pi^2 M_K^2 \left( - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\times \sum_{I=1}^6 C_I (\nu)  \bigg\{
\xi_{01}^I - \frac{D-4}{2}
\nonumber\\
&&
\hspace{-0.9cm}
\times \big[\xi_{00}^I + (f(4) + 2 \gamma_E) \xi_{01}^I \big] \bigg\}
\times
\left( \frac{M^2}{\nu^2} \right)^{\frac{D}{2} - 2}
\nonumber\\
&&
\hspace{-0.9cm}
\times
\frac{\Gamma \left(2 - \frac{D}{2} \right)}{\zeta \left( 3 - \frac{D}{2} \right) + f(D)} \times
\int_{c_1-i\infty}^{c_1+i\infty} \frac{du}{2 \pi i} w^{-u} \Gamma(u) \Gamma(1-u)
\nonumber\\
&&
\hspace{-0.9cm}
\times
\zeta \left( 3 - \frac{D}{2} -u \right)
.
\end{eqnarray}
The first singularity of the integrand lying on the left of the fundamental band $c_1={\rm Re} \, u \in ]0,2-D/2[$
is a simple pole at $u=0$, coming from the factor $\Gamma(u)$. According to the Converse Mapping Theorem
\cite{FGD95} one therefore has
\begin{eqnarray}
&&
\hspace{-1.25cm}
\int dx \, \frac{\rho^{\rm res;0}_{K\pi}(x ; D)}{x+wM^2}
\nonumber\\
&&
\hspace{-1.25cm}
\underset{w \rightarrow 0}{=}
- 16 \pi^2 M_K^2 \!\left( \! - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\times \sum_{I=1}^6 C_I (\nu)  \bigg\{
\xi_{01}^I - \frac{D-4}{2}
\nonumber\\
&&
\hspace{-0.6cm}
\times \big[\xi_{00}^I + (f(4) + 2 \gamma_E) \xi_{01}^I \big] \bigg\}
\times
\left( \frac{M^2}{\nu^2} \right)^{\frac{D}{2} - 2}
\nonumber\\
&&
\hspace{-0.6cm}
\times
\frac{\Gamma \left(2 - \frac{D}{2} \right) \zeta \left( 3 - \frac{D}{2} \right)}{\zeta \left( 3 - \frac{D}{2} \right) + f(D)}
\nonumber\\
&&
\hspace{-1.25cm}
\underset{D \rightarrow 4}{=}
- 16 \pi^2 M_K^2 \!\! \left( - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\times \sum_{I=1}^6 C_I (\nu)  \bigg\{
\frac{-2}{D-4} \xi_{01}^I
\nonumber\\
&&
\hspace{-0.6cm}
+ \, \xi_{00}^I + \xi_{01}^I \gamma_E + \xi_{01}^I \ln\frac{M^2}{\nu^2} \bigg\}
+ {\cal O} (D-4)
.
\end{eqnarray}
The first singularity of the integrand lying on the right of the fundamental band $c_1={\rm Re} \, u \in ]0,2-D/2[$
is a {simple} pole occurring at $u=2-D/2$, with $\zeta \left( 3 - \frac{D}{2} -u \right) \sim -1/(u-2+D/2)$,
so that, according to the Converse Mapping Theorem,
\begin{eqnarray}
&&
\hspace{-1.25cm}
\int dx \, \frac{\rho^{\rm res;0}_{K\pi}(x ; D)}{x+wM^2}
\nonumber\\
&&
\hspace{-1.25cm}
\underset{w \rightarrow +\infty}{\sim}
- 16 \pi^2 M_K^2 \left( - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\times \sum_{I=1}^6 C_I (\nu)  \bigg\{
\xi_{01}^I
\nonumber\\
&&
\hspace{-0.35cm}
- \, \frac{D-4}{2} \left[\xi_{00}^I + (f(4) + 2 \gamma_E) \xi_{01}^I \right] \bigg\}
\times
\left( \frac{M^2}{\nu^2} \right)^{\frac{D}{2} - 2}
\nonumber\\
&&
\hspace{-0.35cm}
\times \,
\frac{\Gamma \left(2 - \frac{D}{2} \right)}{\zeta \left( 3 - \frac{D}{2} \right) + f(D)}
\times \Gamma \left(2 - \frac{D}{2} \right) \Gamma \left(\frac{D}{2} - 1 \right)
\nonumber\\
&&
\hspace{-0.35cm}
\times \,
\left[ w^{\frac{D}{2} - 2} + {\cal O} \left( w^{-1} \right) \right]
.\qquad~
\end{eqnarray}
Considering next the limit $D\to 4$, one indeed recovers the expected result:
\begin{eqnarray}
&&
\hspace{-1.25cm}
\int dx \, \frac{\rho^{\rm res;0}_{K\pi}(x ; D)}{x+wM^2}
\nonumber\\
&&
\hspace{-1.25cm}
\underset{w \rightarrow +\infty}{\sim}
- 16 \pi^2 M_K^2 \left( - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\times \sum_{I=1}^6 C_I (\nu)
\\
&&
\hspace{-0.35cm}
\times \,  \bigg\{
\frac{-2}{D-4} \xi_{01}^I + \xi_{00}^I + \xi_{01}^I \ln\frac{\nu^2}{-s} \bigg\}
+ {\cal O}(D-4)
.
\nonumber
\end{eqnarray}
Summarizing the preceding analysis, we find {indeed} that the spectral density \rf{rho_0}
reproduces the minimally subtracted dispersive integral in Eq. \rf{ren_disp_rho_0}.
Let us notice, at this stage, that the result \rf{ren_disp_rho_0} for $D=4$ does no
longer depend at all on $f(D)$. We will return to this point in Section \ref{discussion} below.

\section{The matching at order ${\cal O} (\alpha_s)$}\label{matching_NLO}
\setcounter{equation}{0}

In order to include the ${\cal O}(\alpha_s)$ corrections that arise at two loops, it is necessary to start with
a somewhat more involved expression of the spectral density. We again let ourselves be guided by the perturbative
structure of this contribution. On the basis of the order ${\cal O}(\alpha_s)$ diagrams shown in Fig. \ref{OPE_QCD_fig},
we are led to consider as a starting point the sum of two terms,
\begin{equation}
\rho^{\rm res;1}_{K\pi}(x ; D) = \rho^{\rm res;1a}_{K\pi}(x ; D) + \rho^{\rm res;1b}_{K\pi}(x ; D)
\end{equation}
with
\begin{eqnarray}
&& \hspace{-1.3cm}
\rho^{\rm res;1a}_{K\pi}(x ; D) = A_{1a} (D) (4\pi)^{4 - D} \!\left( \frac{M^2}{\nu^2_{\rm MS}} \right)^{D-4}
\! \left[\Gamma \left(2 - \frac{D}{2} \right)\right]^2
\nonumber\\
&& \hspace{0.7cm}
\times \sum_{n\ge 1} M^2 \mu_n^{(1a)} (D) \, \delta (x - nM^2)
,
\nonumber\\
&& \hspace{-1.3cm}
\rho^{\rm res;1b}_{K\pi}(x ; D) =
\frac{A_{1b} (D)}{D-4} \, (4\pi)^{2 - \frac{D}{2}} \left( \frac{M^2}{\nu^2_{\rm MS}} \right)^{\frac{D}{2} - 2}
\Gamma \left(2 - \frac{D}{2} \right)
\nonumber\\
&& \hspace{0.7cm}
\times \sum_{n\ge 1} M^2 \mu_n^{(1b)} (D) \, \delta (x - nM^2)
,
\end{eqnarray}
where both functions $A_{1a} (D)$ and $A_{1b} (D)$ are regular at $D=4$.
The first term, $\rho^{\rm res;1a}_{K\pi}(x ; D)$, corresponds to the genuine two-loop diagrams,
like the graphs $(b)$ to $(e)$ of Fig. \ref{OPE_QCD_fig}. The second term, $\rho^{\rm res;1b}_{K\pi}(x ; D)$,
stands for the diagram $(f)$, i.e. the one-loop graph $(a)$ with the insertion of a one-loop counterterm
proportional to $1/(D-4)$. Without loss of generality, one may impose a normalization
condition like \rf{cond_1} for each set of coefficients separately, and from there proceed as in
Section \ref{matching_LO}. We will not go through the details of this straightforward exercise, but rather
state the final result and show that it satisfies the required properties. Before that, let us make a few
useful remarks.

With the normalizations set as in Eq. \rf{cond_1} for both $\rho^{\rm res;1a}_{K\pi}(x ; D)$
and $\rho^{\rm res;1b}_{K\pi}(x ; D)$, the corresponding dispersive integrals at $w=0$ display double and
{ simple} poles at $D=4$. The { simple} poles contain a contribution proportional to $\ln (M^2/\nu^2)$, which,
on general grounds \cite{Collins74}, is not allowed, and hence has to cancel in the sum of the two dispersive integrals.
For this to happen, we need to impose the condition $A_{1b} (4) = 4 A_{1a} (4)$.

Next, $\rho^{\rm res;1b}_{K\pi}(x ; D)$ has to describe the same structure {as} $\rho^{\rm res;0}_{K\pi}(x ; D)$
up to an additional factor $\propto 1/(D-4)$ coming from the inserted counterterm. It is thus natural to consider
for it the ansatz
\begin{equation}
\mu_n^{(1b)} (D) = \frac{n^{\frac{D}{2} - 2}}{\zeta\left( 3 - \frac{D}{2} \right) + {\tilde f} (D)}
,
\end{equation}
where ${\tilde f}(D)$ is an arbitrary function regular at $D=4$.
Then the resulting subtracted dispersive integral
\begin{equation}
w M^2 \int \frac{dx}{x} \, \frac{\rho^{\rm res;1b}_{K\pi}(x ; D)}{x+wM^2}
\end{equation}
still contains a term proportional to $\psi (1+w)/(D-4)$, which has eventually
to be canceled by a similar contribution from the dispersive integral involving
$\rho^{\rm res;1a}_{K\pi}(x ; D)$.

Turning to the latter, we consider the ansatz
\begin{equation}
\mu_n^{(1a)} (D) = \frac{n^{y(D-4)}}{\zeta\left( 1+4y-yD \right) + g (D)}
,
\end{equation}
where $g(D)$ is an arbitrary function regular at $D=4$, and $y$ is a so-far free parameter.
Considering the dispersive integral
\begin{equation}
w M^2 \int \frac{dx}{x} \, \frac{\rho^{\rm res;1a}_{K\pi}(x ; D)}{x+wM^2}
,
\end{equation}
we find that the required cancellation of the unwanted pole terms takes place for the choice $y=1$.

Adjusting the remaining free parameters such as to reproduce the asymptotic
behaviour of Eq. \rf{OPE_NLO}, we finally arrive at the result\footnote{Up to a factor $(-1)^n$,
the Stieltjes constants $\gamma_n$ give the coefficients
of the Taylor expansion of the regular part of $\zeta (s)$ at $s=1$, with $\gamma_0 = \gamma_E = 0.577216\cdots$,
$\gamma_1=-0.0728\cdots$, $\gamma_2=-0.00969\cdots$; see \cite[section 25.2]{DLMF}.}
\begin{eqnarray}
&&
\hspace{-1.25cm}
\rho^{\rm res;1}_{K\pi}(x ; D)
\nonumber\\
&&
\hspace{-1.25cm}
= - 16 \pi^2 M_K^2 \left( - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right) \alpha_s (\nu)
\sum_{I=1}^6 C_I (\nu)
\nonumber\\
&&
\hspace{-1.25cm}
\times
\left\{
\left( \frac{M^2}{\nu^2} \right)^{D-4}
\left[ \Gamma \left(2 - \frac{D}{2} \right) \right]^2 e^{\gamma_E(4-D)} \xi_{12}^I
\right.
\nonumber\\
&&
\times \sum_{n\ge 1} M^2
\frac{n^{D-4}}{\zeta\left( 5-D \right)}
\,\delta(x-nM^2)
\nonumber\\
&&
\hspace{-1.25cm}
+ \left( \frac{M^2}{\nu^2} \right)^{\frac{D}{2}-2}
\Gamma \left(2 - \frac{D}{2} \right)  \frac{e^{\gamma_E(4-D)/2}}{D-4}
\nonumber\\
&& \times
\bigg\{ 4 \xi_{12}^I
+ (D-4)\big[\xi_{11}^I + 2 \gamma_E \xi_{12}^I\big]
\nonumber\\
&&
- \frac{(D-4)^2}{2} \bigg[  \xi_{10}^I + \xi_{11}^I \gamma_E
- 2 \xi_{12}^I \Big( \frac{\pi^2}{6} + \gamma_1  \Big) \bigg] \bigg\}
\nonumber\\
&& \left. \times
\sum_{n\ge 1} M^2 \frac{n^{\frac{D}{2} - 2}}{\zeta\left( 3 - \frac{D}{2} \right) }
\,\delta(x-nM^2)
\right\}
.
\end{eqnarray}
It is a straightforward exercise, making use of the techniques described in the
second part of Section \ref{matching_LO}, to show that this spectral density leads
to the desired properties. For instance, after minimal subtraction of the simple
and double poles at $D=4$, the renormalized dispersive integral reads
\begin{eqnarray}
&&
\hspace{-1.25cm}
\int dx \, \frac{\rho^{\rm res;1}_{K\pi}(x ; D)}{x+wM^2}
\nonumber\\
&&
\hspace{-1.25cm}
= - \, 16 \pi^2 M_K^2 \left( - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\alpha_s (\nu) \sum_{I=1}^6 C_I (\nu)  \Bigg\{ \xi_{12}^I  {\widetilde\psi} (w)
\nonumber\\
&&
\hspace{-1.25cm}
+ \left[ 2  \xi_{12}^I \ln \frac{M^2}{\nu^2} - \xi_{11}^I
\right]   \psi(1+w)
+ \xi^I_{10}
- 2 \left( \frac{\pi^2}{6} + \gamma_1  \right) \xi_{12}^I
\nonumber\\
&&
\hspace{-1.25cm}
+
\left[
 \xi^I_{12} \ln \frac{M^2}{\nu^2}   - \xi^I_{11} \right]
\ln \frac{M^2}{\nu^2} \Bigg\}
+
{\cal O}(D-4)
,
\end{eqnarray}
with
\begin{eqnarray}
\hspace{-0.5cm}
{\widetilde\psi} (w) \!\!\!&=&\!\!\! 2 M^2 w \int \frac{dx}{x} \, \frac{1}{x + M^2 w} \sum_{n\ge 1} M^2 (\ln n) \delta (x - nM^2)
\nonumber\\
\!\!\!&=&\!\!\! 2 w \sum_{n\ge 1} \frac{\ln n}{n(n+w)}
.
\end{eqnarray}
We now need to establish some properties of the function ${\widetilde\psi} (w)$,
which is defined by an absolutely convergent sum as long as $w$ is not equal to
a negative integer. Thus, ${\widetilde\psi} (0)=0$ and
\begin{equation}
{\widetilde\psi}^\prime (0) = 2 \sum_{n\ge 1} \frac{\ln n}{n^2} = - 2 \zeta^\prime (2)
.
\end{equation}
Next, using the relation \rf{mellin}
and
\begin{equation}
\sum_{n\ge 1} n^{u-2} \ln n = - \zeta^\prime (2-u)
,
\end{equation}
one obtains the following Mellin representation of the function ${\widetilde\psi} (w)$
\begin{equation}
{\widetilde\psi} (w) = -2 w \! \int_{c_1-i\infty}^{c_1+i\infty} \!\frac{du}{2 \pi i} w^{-u} \Gamma(u) \Gamma(1-u)
\times \zeta^\prime (2-u)  .
\lbl{Mellin_psi-tilde}
\end{equation}
The first singularity on the left of the fundamental strip $0<c_1<1$ lies at $u=0$. It consists of a simple pole coming from
the factor $\Gamma(u)$ of the integrand. Therefore, the Converse Mapping Theorem allows to state that
${\widetilde\psi} (w) = -2w\zeta^\prime (2) + \cdots$ as $w\to 0$, a property already established before.
The first singularity one encounters on the right of the fundamental strip consists of a triple pole and a
simple pole, both located at $u=+1$,
\begin{equation}
\Gamma(u) \Gamma(1-u) \zeta^\prime (2-u) \underset{u \rightarrow +1}{\sim}
\frac{1}{(u-1)^3} + \frac{\frac{\pi^2}{6} + \gamma_1}{u-1} +  \gamma_2 + {\cal O}(u-1)
.
\end{equation}
The Converse Mapping Theorem then allows to conclude that
\begin{equation}
{\widetilde\psi} (w) \underset{w \rightarrow +\infty}{\sim}
\ln^2 w + 2  \left( \frac{\pi^2}{6} + \gamma_1  \right) + \cdots
,
\end{equation}
which is precisely what is required. The function ${\widetilde\psi} (w)$
does not seem to be related in an obvious way to the standard sets of functions that have been studied in
the mathematical literature, see e.g. \cite{DLMF} and references therein. It is thus
interesting to notice that the Mellin representation \rf{Mellin_psi-tilde} of ${\widetilde\psi} (w)$,
which holds for ${\rm Re} w >0$, can be recast into [the integral is understood as its Cauchy principal
value for $w$ real and positive]
\begin{eqnarray}
&& \hspace{-1.25cm}
{\widetilde\psi} (w) =
2 \int_0^\infty \frac{dx}{x} \frac{w}{x-w} \, [\psi(1+x) + \gamma_E]
\nonumber\\
&& \hspace{-0.2cm}
+ 2  [\psi(1+w) + \gamma_E]\ln w
.
\end{eqnarray}
This representation allows to extend the function ${\widetilde\psi} (w)$ to negative
values of ${\rm Re} w$. The integral is then regular, while the second term
reproduces the poles of ${\widetilde\psi} (w)$ when $w$ equals a strictly negative integer.
The determination of $\ln w$ is given by $w\to w-i\epsilon$, with $\epsilon>0$ and infinitesimal,
in agreement with the prescription $s \to s+i\epsilon$.
This representation also proves quite useful for the numerical evaluation of ${\widetilde\psi} (w)$.

\begin{figure}[t]
~\includegraphics[width=8.0cm]{calWresNLO_HV_NDR_pm1.eps}
\caption{The function ${\cal W}_{K\pi}^{\rm res} (z ; \nu)$ defined in Eq. \rf{Wres_final}
as a function of $z$ for $\nu=2~{\rm GeV}$ and $M=1~{\rm Gev}$. The light-red (purple)
band corresponds to the NDR (HV) scheme. The red (blue) line {corresponds} to the truncation
of ${\cal W}_{K\pi}^{\rm res} (z ; \nu)$ to lowest order in the NDR (HV) scheme.
For the ease of visualization, the location of the poles at $z=nM^2/M_K^2$ has been slightly
shifted off the real-$z$ axis.} \label{plot_calW}
\end{figure}

\section{Discussion}\label{discussion}
\setcounter{equation}{0}

\begin{figure*}[t]
\includegraphics[width=8.26cm]{aplus_NLO.eps} ~~~ \includegraphics[width=8.2cm]{bplus_NLO.eps}
\caption{The variation of $a_+$ (left plot) and of $b_+$ (right plot) as a
function of the short-distance renormalization scale $\nu$. The scale dependence at lowest order
corresponds to the red (blue) lines in the NDR (HV) scheme. The scale dependence after
inclusion of ${\cal O}(\alpha_s)$ corrections corresponds to the light-red (purple)
bands in the NDR (HV) scheme.} \label{plots_scale}
\end{figure*}

We have shown that it is possible to construct a function $W_{K\pi}^{\rm res} (z ; \nu)$
through an infinite sum of zero-width resonances with a Regge-type spectrum, such that its short-distance
behaviour matches, at order ${\cal O}(\alpha_s)$, the one obtained from the leading term in the OPE. After renormalization
in the ${\overline{\rm MS}}$ scheme of dimensional regularisation, and up to the order ${\cal O} (\alpha_s)$,
this function reads [$w \equiv -z M_K^2/M^2$]
\begin{eqnarray}
&& \hspace{-1.35cm}
W_{K\pi}^{\rm res} (z ; \nu) =
- 16 \pi^2 M_K^2 \! \left( \! - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\!\frac{C_{K\pi} f_+^{K\pi} (z M_K^2)}{4\pi}
\nonumber\\
&& \hspace{0.1cm}
\times  \sum_{I=1}^6 C_I (\nu)\Bigg\{
\xi_{00}^I
+ \alpha_s (\nu) \xi^I_{10}
\nonumber\\
&& \hspace{0.1cm}
+
\ln \frac{\nu^2}{M^2} \left[ \xi_{01}^I + \alpha_s (\nu)
\left( \xi^I_{11}
+ \xi^I_{12} \ln \frac{\nu^2}{M^2} \right) \right]
\nonumber\\
&& \hspace{0.1cm}
- \psi(1+w) \left[ \xi_{01}^I + \alpha_s (\nu) \left( \xi_{11}^I + 2 \xi_{12}^I \ln \frac{\nu^2}{M^2} \right) \right]
\nonumber\\
&& \hspace{0.1cm}
+ \alpha_s (\nu) \, \xi_{12}^I \left( {\widetilde\psi} (w) - \frac{\pi^2}{3} - 2 \gamma_1 \right)
\!\Bigg\}
\nonumber\\
&&
\equiv
- 16 \pi^2 M_K^2 \! \left( \! - \frac{G_{\rm F}}{\sqrt{2}} V_{us} V_{ud} \right)
\!\frac{C_{K\pi} f_+^{K\pi} (z M_K^2)}{4\pi}
\nonumber\\
&& \hspace{0.1cm}
\times
{\cal W}_{K\pi}^{\rm res} (z ; \nu)
.
\lbl{Wres_final}
\end{eqnarray}
This expression involves two scales. The first one, $M$, is the scale of the
lowest resonance in the spectrum besides the $\rho(770)$, which is already taken
into account by the contribution $W_{K\pi}^{\pi\pi} (z)$ in Eq. \rf{model}, see
Ref. \cite{DAmbrosio:2018ytt}. The first resonance appearing in $W_{K\pi}^{\rm res} (z ; \nu)$
is thus the $K^*(892)$ or the $\phi(1020)$, hence $M\sim 1~{\rm GeV}$. The second
scale in Eq. \rf{Wres_final} is the renormalization scale $\nu$. It represents
the onset of the perturbative regime of QCD. In practice, the description of the spectrum
in terms of well-identified resonances extends to a few radial excitations of the resonances
mentioned just above, for instance $\rho(1450)$, $\rho(1570)$, $\phi(1680)$..., before
they merge into the continuum. This means $\nu\sim 2~{\rm GeV}$.

The function ${\cal W}_{K\pi}^{\rm res} (z ; \nu)$ is shown in Fig. \ref{plot_calW}.
For positive values of $z$ it displays the expected infinite series of equally-spaced poles,
whereas for negative values of $z$ the asymptotic regime sets in rapidly.
In order to draw these plots, some knowledge of the coefficients $\xi_{10}^I$,
which are not fixed by the renormalization-group constraint \rf{RG_equ}, is needed.
For the sake of illustration, we have estimated these coefficients to vary in the range
$-\xi_{11}^I \le \xi_{10}^I \le + \xi_{11}^I$, taking the relation \rf{xi_LO} between
the lowest-order coefficients $\xi_{00}^I$ and $\xi_{01}^I$ as a guide. The same choice
also applies to Fig. \ref{plots_scale}.

In the remainder of the Letter, we wish to address in turn two issues that we think
deserve to be given some consideration, namely: i) the size of the residual dependence
with respect to the short-distance scale $\nu$ and ii) some features and properties of
the resonance model that we have constructed, as well as possible improvements.

\subsection{Residual scale dependence}

By construction, the expression of $W_{K\pi}^{\rm res} (z ; \nu)$ displayed
in Eq. \rf{Wres_final} provides a form factor that is independent of the
renormalization scale $\nu$ at order ${\cal O}(\alpha_s)$,
\begin{equation}
\nu \frac{d}{d \nu} \left[ W_{K\pi}^{\rm res} (z ; \nu) + W_{K\pi}^{\rm SD} (z ; \nu) \right]
= {\cal O}(\alpha_s^2)
.
\end{equation}
We expect the residual scale dependence to be weaker { than} the one that
results from the matching at lowest order only. We illustrate these
changes in the case of the two constants $a_+$ and $b_+$ corresponding
to the values of the form factor $W_{K\pi} (z)$, for $(K,\pi) = (K^+ , \pi^+)$,
and of its derivative at $z=0$, respectively.
For this purpose, we add to the sum of $W_{K\pi}^{\rm res}$ and $W_{K\pi}^{\rm SD}$
the contribution $W_{K\pi}^{\pi\pi}$ of the two-pion intermediate
state evaluated in Ref. \cite{DAmbrosio:2018ytt}, see Eq. \rf{model} above.
This means $a_+\vert_{\pi\pi}=-1.58$ and $b_+\vert_{\pi\pi}=-0.76$.
The improvement when going from lowest order
to next-to-leading order can be appreciated from the plots shown in Fig. \ref{plots_scale}
[details on the numerical aspects can be found in the appendix]. We notice that
indeed both the scale dependence and the scheme dependence become less pronounced when order
${\cal O}(\alpha_s)$ corrections are included. Our ignorance of the coefficients $\xi_{10}^I$
induces, under the conditions stated above, an uncertainty that amounts, at $\nu=2~{\rm GeV}$,
to about $5\%$ in both $a_+$ and $b_+$. We also observe that the corrections
coming from the resonance and short-distance parts are quite small as compared to the
contribution from the two-pion state.

This improvement in the control over the scheme and scale {dependences} allows
us to refine somewhat our evaluation of the coefficients $a_+$ and $b_+$ made in
Ref. \cite{DAmbrosio:2018ytt}. We obtain
\begin{equation}
a_+ = - 1.59(8) , \quad b_+ = -0.82(6).
\end{equation}
These values are still at variance with the experimental determinations
$a_+^{\rm exp} =-0.593(11)$ and $b_+^{\rm exp}=-0.675(43)$, see Ref. \cite{DAmbrosio:2018ytt}
for a detailed discussion, especially as far as $a_+$ is concerned. On the other
hand, as discussed in Ref. \cite{DAmbrosio:2018ytt}, in order to reach a
definite conclusion, the contributions
$a_+\vert_{\pi\pi}$ and $b_+\vert_{\pi\pi}$ need to be evaluated within
a tighter framework than the one adopted there,
and other exclusive contributions, in particular from two-kaon states,
should eventually be accounted for explicitly, see also the discussion below.
Work in this direction is on its way \cite{in_progress}.

\subsection{Properties and features of the resonance model}

The resonance model that we have constructed in Sections \ref{matching_LO} and \ref{matching_NLO}
is by far not unique. Discussing the arbitrariness of the construction in full generality represents
a formidable, if not impossible, task, given the fact that the corresponding dispersive integrals
are only constrained to reproduce the divergences of perturbative QCD at $z=0$ and the leading
asymptotic behaviour from the OPE at $z\to -\infty$. The determination of sub-leading terms in the OPE
might be a way to constrain the resonance model further. We will, however, not address this possibility
here, but rather state a few remarks concerning the model that we have constructed above.

Some of the arbitrariness of the construction is embodied in the functions $A_0(D)$ and $f(D)$ in Section \ref{matching_LO},
or $A_{1a}(D)$, $A_{1b}(D)$ ${\tilde f}(D)$ and $g(D)$ in Section \ref{matching_NLO}. These functions do
not appear anymore in the final expression
\rf{Wres_final}, partly because they are absorbed in the matching to the coefficients $\xi_{rp}^I$, partly
because they contribute only to the terms of order ${\cal O}(D-4)$ in the renormalized dispersive integral.

These features certainly do not exhaust all the arbitrariness of the model. While the contribution of the
resonances with higher masses will be constrained by the short-distance behaviour, one might expect that
the {description} of the lower-lying resonances like $K^*(892)$ or $\phi(1020)$ [recall that the contribution
from $\rho(770)$ is already taken care of by $W_{K\pi}^{\pi\pi}$] provided by this model may be less realistic
from a phenomenological point of view.
One way to circumvent this possible drawback would be to consider additional intermediate states in a more
explicit way, i.e. by extending the decomposition in Eq. \rf{model} to, for instance \cite{DAmbrosio:2018ytt}
\begin{equation}
W_{K\pi}^{\rm LD} (z) = W_{K\pi}^{\pi\pi} (z) + W_{K\pi}^{K\pi} (z) + W_{K\pi}^{K{\bar K}} (z) + W_{K\pi}^{\rm res} (z ; \nu)
.
\end{equation}
We will, however, not pursue this matter in the present Letter, and leave the discussion of such an extension and of the corresponding
multi-channel analysis it requires for future work.


\section{Introduction}
\label{Introduction}

Relation extraction aims to identify the relationship between two nominals, typically in the context of a sentence, making it essential to natural language understanding. Consequently, it is a key component of many natural language processing applications, such as information extraction~\cite{fader_openie_2011}, knowledge base population~\cite{tac_kbp_2010}, and question answering~\cite{yu_improvednr_2017}. Table~\ref{tab:re_examples} lists exemplary relation mentions.

\begin{table}[ht!]
    \centering
    \begin{tabular}{p{6cm} l l l}
        \hline
        Sentence & Subject & Object & Relation \\
        \hline
        Mr. \underline{Scheider} played the \underline{police chief} of a resort town menaced by a shark. & Scheider & police chief & per:title \\[20pt]
        The measure included \underline{Aerolineas}'s domestic subsidiary, \underline{Austral}. & Aerolineas & Austral & org:subsidiaries \\[20pt]
        \underline{Yolanda King} died last May of an apparent \underline{heart attack}. & Yolanda King & heart attack & per:cause\_of\_death \\[20pt]
        The \underline{key} was in a \underline{chest}. & key & chest & Content-Container \\[5pt]
        The \underline{car} left the \underline{plant}. & car & plant & Entity-Origin \\[5pt]
        Branches overhang the \underline{roof} of this \underline{house}. & roof & house & Component-Whole\\
        \hline
    \end{tabular}
    \caption{Relation extraction examples, taken from TACRED (1-3) and SemEval 2010 Task 8 (4-6). TACRED relation types mostly focus on named entities, whereas SemEval contains semantic relations between concepts.}
    \label{tab:re_examples}
\end{table}

State-of-the-art relation extraction models, traditional feature-based and current neural methods, typically rely on explicit linguistic features: prefix and morphological features~\cite{mintz_distantsf_2009}, syntactic features such as part-of-speech tags~\cite{zeng_relationcv_2014}, and semantic features like named entity tags and WordNet hypernyms~\cite{xu_improvedrc_2016}. Most recently, \citet{zhang_graphco_2018} combined dependency parse features with graph convolutional neural networks to considerably increase the performance of relation extraction systems.

However, relying on explicit linguistic features severely restricts the applicability and portability of relation extraction to novel languages. Explicitly computing such features requires large amounts of annotated, language-specific resources for training; many unavailable in non-English languages. Moreover, each feature extraction step introduces an additional source of error, possibly cascading over multiple steps.
Deep language representations, on the other hand, have shown to be a very effective form of unsupervised pre-training, yielding contextualized features that capture linguistic properties and benefit downstream natural language understanding tasks, such as semantic role labeling, coreference resolution, and sentiment analysis~\cite{peters_deepcw_2018}. Similarly, fine-tuning pre-trained language representations on a target task has shown to yield state-of-the-art performance on a variety of tasks, such as semantic textual similarity, textual entailment, and question answering~\cite{radford_improvinglu_2018}.

In addition, classifying complex relations requires a considerable amount of annotated training examples, which are time-consuming and costly to acquire. \citet{ruder_universallm_2018} showed language model fine-tuning to be a sample efficient method that requires fewer labeled examples.

Besides recurrent (RNN) and convolutional neural networks (CNN), the Transformer ~\cite{vaswani_attention_2017} is becoming a popular approach to learn deep language representations. Its self-attentive structure allows it to capture long-range dependencies efficiently; demonstrated by the recent success in machine translation~\cite{vaswani_attention_2017}, text generation~\cite{liu_generatingwb_2018}, and question answering~\cite{radford_improvinglu_2018}.

In this paper, we propose TRE: a \textbf{T}ransformer based \textbf{R}elation \textbf{E}xtraction model. Unlike previous methods, TRE uses deep language representations instead of explicit linguistic features to inform the relation classifier. Since language representations are learned by unsupervised language modeling, pre-training TRE only requires a plain text corpus instead of annotated language-specific resources. Fine-tuning TRE, and its representations, directly to the task minimizes explicit feature extraction, reducing the risk of error accumulation. Furthermore, an increased sample efficiency reduces the need for distant supervision methods~\cite{mintz_distantsf_2009, riedel_modeling_2010}, allowing for simpler model architectures without task-specific modifications.

The contributions of our paper are as follows:
\begin{itemize}
    \item We describe TRE, a Transformer based relation extraction model that, unlike previous methods, relies on deep language representations instead of explicit linguistic features.

    \item We are the first to demonstrate the importance of pre-trained language representations in relation extraction, by outperforming state-of-the-art methods on two supervised datasets, TACRED and SemEval 2010 Task 8.

    \item We report detailed ablations, demonstrating that pre-trained language representations prevent overfitting and achieve better generalization in the presence of complex entity mentions. Similarly, we demonstrate a considerable increase in sample efficiency over baseline methods.

    \item We make our trained models, experiments, and source code available to facilitate wider adoption and further research.
\end{itemize}

\section{TRE}
This section introduces TRE and its implementation. First, we cover the model architecture (Section~\ref{sec:model_architecture}) and input representation (Section~\ref{sec:input_representation}), followed by the introduction of unsupervised pre-training of deep language model representations (Section~\ref{sec:unsuperv_pretraining}). Finally, we present supervised fine-tuning on the relation extraction task (Section~\ref{sec:superv_re}).

\subsection{Model Architecture}
\label{sec:model_architecture}
TRE is a multi-layer Transformer-Decoder~\cite{liu_generatingwb_2018}, a decoder-only variant of the original Transformer~\cite{vaswani_attention_2017}. As shown in Figure~\ref{fig:model_architecture}, the model repeatedly encodes the given input representations over multiple layers (i.e., Transformer blocks), consisting of masked multi-headed self-attention followed by a position-wise feedforward operation:
\begin{equation}
    \begin{tabular}{l}
        $h_0 = T W_e + W_p$ \\
        $h_l = transformer\_block(h_{l-1})\ \forall\ l\ \in\ [1, L]$ \\
    \end{tabular}
\end{equation}
Where $T = (t_{1}, \ldots, t_{k})$ is a sequence of token indices in a sentence\footnote{In this work a ``sentence'' denotes an arbitrary span of contiguous text, rather than an actual linguistic sentence. For pre-training the input consists of multiple linguistic sentences, whereas relation extraction is applied to a single one. A ``sequence'' refers to the input token sequence.}. $W_e$ is the token embedding matrix, $W_p$ is the positional embedding matrix, $L$ is the number of Transformer blocks, and $h_l$ is the state at layer $l$. Since the Transformer has no implicit notion of token positions, the first layer adds a learned positional embedding $e_p \in \mathbb{R}^d$ to each token embedding $e^p_t \in \mathbb{R}^d$ at position $p$ in the input sequence. The self-attentive architecture allows an output state $h^p_l$ of a block to be informed by all input states $h_{l-1}$, which is key to efficiently model long-range dependencies. For language modeling, however, self-attention must be constrained (masked) not to attend to positions ahead of the current token.
For a more exhaustive description of the architecture, we refer readers to~\citet{vaswani_attention_2017} and the excellent guide ``The Annotated Transformer''\footnote{\url{http://nlp.seas.harvard.edu/2018/04/03/attention.html}}.

\begin{figure}[ht!]
\centering
\includegraphics[width=\textwidth]{TRE_architecture.pdf}
\caption{\textbf{(left)} Transformer-Block architecture and training objectives. A Transformer-Block is applied at each of the $L$ layers to produce states $h_1$ to $h_L$. \textbf{(right)}~Relation extraction requires a structured input for fine-tuning, with special delimiters to assign different meanings to parts of the input.}\label{fig:model_architecture}
\end{figure}

\subsection{Input Representation}
\label{sec:input_representation}
Our input representation (see also Figure~\ref{fig:model_architecture}) encodes each sentence as a sequence of tokens. To make use of sub-word information, we tokenize the input text using byte pair encoding (BPE)~\cite{sennrich_subword_2016}. The BPE algorithm creates a vocabulary of sub-word tokens, starting with single characters. Then, the algorithm iteratively merges the most frequently co-occurring tokens into a new token until a predefined vocabulary size is reached. For each token, we obtain its input representation by summing over the corresponding token embedding and positional embedding.

While the model is pre-trained on plain text sentences, the relation extraction task requires a structured input, namely a sentence and relation arguments.
To avoid task-specific changes to the architecture, we adopt a traversal-style approach similar to~\citet{radford_improvinglu_2018}. The structured, task-specific input is converted to an ordered sequence to be directly fed into the model without architectural changes. Figure~\ref{fig:model_architecture} provides a visual illustration of the input format. It starts with the tokens of both relation arguments $a^1$ and $a^2$, separated by delimiters, followed by the token sequence of the sentence containing the relation mention, and ends with a special classification token. The classification token signals the model to generate a sentence representation for relation classification. Since our model processes the input left-to-right, we add the relation arguments to the beginning, to bias the attention mechanism towards their token representation while processing the sentence's token sequence.

\subsection{Unsupervised Pre-training of Language Representations}
\label{sec:unsuperv_pretraining}
Relation extraction models benefit from efficient representations of long-term dependencies \cite{zhang_graphco_2018} and hierarchical relation types \cite{re_relation_att}. Generative pre-training via a language model objective can be seen as an ideal task to learn deep language representations that capture important lexical, syntactic, and semantic features without supervision~\cite{linzen_assessingta_2016, radford_improvinglu_2018, ruder_universallm_2018}, before fine-tuning to the supervised task -- in our case relation extraction.

Given a corpus $\mathcal{C} = \{c_1, \ldots, c_n\}$ of tokens $c_i$, the language modeling objective maximizes the likelihood
\begin{equation}
    \label{eq:lm_objective}
    L_1(\mathcal{C}) = \sum_i \log P(c_i | c_{i-1}, \ldots, c_{i-k}; \theta)
\end{equation}
where k is the context window considered for predicting the next token $c_i$ via the conditional probability $P$. TRE models the conditional probability by an output distribution over target tokens:
\begin{equation}
    P(c) = softmax(h_L W^T_e),
\end{equation}
where $h_L$ is the sequence of states after the final layer $L$, $W_e$ is the embedding matrix, and $\theta$ are the model parameters that are optimized by stochastic gradient descent.

\subsection{Supervised Fine-tuning on Relation Extraction}
\label{sec:superv_re}
After pre-training with the objective in Eq.~\ref{eq:lm_objective}, the language model is fine-tuned on the relation extraction task. We assume a labeled dataset $\mathcal{D} = ([x_i^1, \ldots, x_i^m], a^1_i, a^2_i, r_i)$, where each instance consists of an input sequence of tokens $x^1, \ldots, x^m$, the positions of both relation arguments $a^1$ and $a^2$ in the sequence of tokens, and a corresponding relation label $r$. The input sequence is fed to the pre-trained model to obtain the final state representation $h_L$. To compute the output distribution $P(r)$ over relation labels, a linear layer followed by a softmax is applied to the last state $h^m_L$, which represents a summary of the input sequence:
\begin{equation}
    P(r | x^1, \ldots, x^m) = softmax(h^m_L W_r)
\end{equation}
During fine-tuning we optimize the following objective:
\begin{equation}
    L_2(\mathcal{D}) = \sum^{|\mathcal{D}|}_{i=1} \log P(r_i | x_i^1, \ldots, x_i^m)
\end{equation}
According to \citet{radford_improvinglu_2018}, introducing language modeling as an auxiliary objective during fine-tuning improves generalization and leads to faster convergence. Therefore, we adopt a similar objective:
\begin{equation}
    L(\mathcal{D}) = \lambda * L_1(\mathcal{D}) + L_2(\mathcal{D}),
\end{equation}
where $\lambda$ is the language model weight, a scalar, weighting the contribution of the language model objective during fine-tuning.

\section{Experiment Setup}
\label{experiment_setup}
We run experiments on two supervised relation extraction datasets: The recently published TACRED dataset~\cite{zhang_position_aware_2017} and the SemEval dataset~\cite{hendrickx_semeval2010t8_2010}. We evaluate the PCNN implementation of \citet{zeng_distant_2015}\footnote{\url{https://github.com/thunlp/OpenNRE}} on the two datasets and use it as a state-of-the-art baseline in our analysis section. In the following we describe our experimental setup.

\subsection{Datasets}
\label{subsec:exp_datasets}

Table \ref{tab:dataset_stats} shows key statistics of the TACRED and SemEval datasets. While TACRED is approximately 10x the size of SemEval 2010 Task 8, it contains a much higher fraction of negative training examples, which makes classification more challenging.
\begin{table}[ht!]
    \centering
    \begin{tabular}{l c c c}
        \hline
        Dataset & relation types & examples & negative examples \\
        \hline
        TACRED & 42 & 106,264 & 79.5\% \\
        SemEval 2010 Task 8 & 19 & 10,717 & 17.4\% \\
        \hline
    \end{tabular}
    \caption{Comparison of the datasets used for evaluation}
    \label{tab:dataset_stats}
\end{table}

\noindent \textbf{TACRED}: The \textit{TAC} \textit{R}elation \textit{E}xtraction \textit{D}ataset~\cite{zhang_position_aware_2017} contains 106k sentences with entity mention pairs collected from the TAC KBP evaluations\footnote{\url{https://tac.nist.gov/2017/KBP/index.html}} 2009--2014, with the years 2009 to 2012 used for training, 2013 for evaluation, and 2014 for testing. Sentences are annotated with person- and organization-oriented relation types, e.g.\ \emph{per:title}, \emph{org:founded}, and \emph{no\_relation} for negative examples. In contrast to the SemEval dataset the entity mentions are typed, with subjects classified into person and organization, and objects categorized into 16 fine-grained classes (e.g., date, location, title). As per convention, we report our results as micro-averaged F1 scores. Following the evaluation strategy of \citet{zhang_position_aware_2017}, we select our best model based on the median validation F1 score over 5 independent runs and report its performance on the test set.

\noindent \textbf{SemEval 2010 Task 8}: The SemEval 2010 Task 8 dataset~\cite{hendrickx_semeval2010t8_2010} is a standard benchmark for binary relation classification, and contains 8,000 sentences for training, and 2,717 for testing. Sentences are annotated with a pair of untyped nominals and one of 9 directed semantic relation types, such as\ \emph{Cause-Effect}, \emph{Entity-Origin}, as well as the undirected \emph{Other} type to indicate no relation, resulting in 19 distinct types in total.  We follow the official convention and report macro-averaged F1 scores with directionality taken into account by averaging over 5 independent runs. The dataset is publicly available\footnote{\scriptsize\url{https://drive.google.com/file/d/0B_jQiLugGTAkMDQ5ZjZiMTUtMzQ1Yy00YWNmLWJlZDYtOWY1ZDMwY2U4YjFk}} and we use the official scorer to compute our results.

\subsection{Pre-training}
\label{subsec:exp_pretraining}
Since pre-training is computationally expensive, and our main goal is to show its effectiveness by fine-tuning on the relation extraction task, we reuse the language model\footnote{\url{https://github.com/openai/finetune-transformer-lm}} published by~\citet{radford_improvinglu_2018} for our experiments. The model was trained on the BooksCorpus~\cite{zhu_aligningba_2015}, which contains around 7,000 unpublished books with a total of more than 800M words of different genres. It consists of $L=12$ layers (blocks) with 12 attention heads and 768 dimensional states, and a feed-forward layer of 3072 dimensional states. We reuse the model's byte pair encoding vocabulary, containing 40,000 tokens, but extend it with task-specific ones (i.e., start, end, and delimiter tokens). Also, we use the learned positional embeddings with supported sequence lengths of up to 512 tokens.

\subsection{Entity Masking}
\label{subsec:exp_masking}
We employ four different entity masking strategies. Entity masking allows us to investigate the model performance while providing limited information about entities, in order to prevent overfitting and allow for better generalization to unseen entities. It also enables us to analyze the impact of entity type and role features on the model's performance. For the simplest masking strategy \textit{UNK}, we replace all entity mentions with a special unknown token. For the \textit{NE} strategy, we replace each entity mention with its named entity type. Similarly, \textit{GR} substitutes a mention with its grammatical role (subject or object). \textit{NE + GR} combines both strategies.

\subsection{Hyperparameter Settings and Optimization}
\label{subsec:exp_hyperparams}
During our experiments we found the hyperparameters for fine-tuning, reported in~\cite{radford_improvinglu_2018}, to be very effective. Therefore, unless specified otherwise, we used the Adam optimization scheme~\cite{adam_2015} with $\beta_1=0.9$, $\beta_2=0.999$, a batch size of 8, and a linear learning rate decay schedule with warm-up over 0.2\% of training updates. We apply residual, and classifier dropout with a rate of 0.1. Also, we experimented with token dropout, but did not find that it improved performance. Table~\ref{tab:best_hyperparameters} shows the best performing hyperparameter configuration for each dataset. On SemEval 2010 Task 8, we first split 800 examples of the training set for hyperparameter selection and retrained on the entire training set with the best parameter configuration.

\begin{table}[ht!]
    \centering
    \begin{tabular}{l c c c c c} \\\hline
        & Epochs & Learning Rate & Warmup Learning Rate & $\lambda$ & Attn. Dropout  \\\hline
        TACRED & 3 & 5.25e-5 & 2e-3 & 0.5 & 0.1 \\
        SemEval & 3 & 6.25e-5 & 1e-3 & 0.7 & 0.15 \\
        \hline
    \end{tabular}
    \caption{Best hyperparameter configuration for TACRED and SemEval}
    \label{tab:best_hyperparameters}
\end{table}

\section{Results}
\label{sec:results}
This section presents our experimental results. We compare our TRE model to other works on the two benchmark datasets, demonstrating that it achieves state-of-the-art performance even without sophisticated linguistic features. We also provide results on model ablations and the effect of the proposed entity masking schemes.

\subsection{TACRED}
On the TACRED dataset, TRE outperforms state-of-the-art single-model systems and achieves an F1 score of 67.4 (Table~\ref{tab:results_tacred}). Compared to SemEval, we observe methods to perform better that are able to model complex syntactic and long-range dependencies, such as PA-LSTM~\cite{zhang_position_aware_2017} and C-GCN~\cite{zhang_graphco_2018}. Outperforming these methods highlights our model's ability to implicitly capture patterns similar to complex syntactic features, and also capture long-range dependencies.

We would like to point out that the result was produced by the same ``entity masking'' strategy used in previous work~\cite{zhang_position_aware_2017, zhang_graphco_2018}. Similar to our \textit{NE + GR} masking strategy, described in Section~\ref{subsec:exp_masking}, we replace each entity mention by a special token; a combination of its named entity type and grammatical role. While we achieve state-of-the-art results by providing only named entity information, unmasked entity mentions decrease the score to 62.8, indicating overfitting and, consequently, difficulties to generalize to specific entity types. In Section \ref{subsec:effect_entity_masking}, we analyze the effect of entity masking on task performance in more detail.

\begin{table}[ht!]
    \centering
    \begin{tabular}{l c c c} \hline
        System & P & R & F1 \\\hline
        LR$^\mathrm{\dagger}$ \citet{zhang_position_aware_2017} & 72.0 & 47.8 & 57.5 \\

        CNN$^\mathrm{\dagger}$ \citet{zhang_position_aware_2017} & \textbf{72.1} & 50.3 & 59.2 \\
        Tree-LSTM$^\mathrm{\dagger}$ \citet{zhang_graphco_2018} & 66.0 & 59.2 & 62.4 \\

        PA-LSTM$^\mathrm{\dagger}$ \citet{zhang_graphco_2018} & 65.7 & 64.5 & 65.1 \\
        C-GCN$^\mathrm{\dagger}$ \citet{zhang_graphco_2018} & 69.9 & 63.3 & 66.4 \\
        TRE (ours) & 70.1 & \textbf{65.0} & \textbf{67.4} \\\hline
    \end{tabular}
    \caption{TACRED single-model test set performance. We selected the hyperparameters using the validation set, and report the test score of the run with the median validation score among 5 randomly initialized runs. $\mathrm{\dagger}$ marks results reported in the corresponding papers.}
    \label{tab:results_tacred}
\end{table}

\subsection{SemEval}
\label{subsec:res_semeval}
On the SemEval 2010 Task 8 dataset, the TRE model outperforms the best previously reported models, establishing a new state-of-the-art score of 87.1 F1 (Table~\ref{tab:results_semeval2}).
The result indicates that pre-training via a language modeling objective allows the model to implicitly capture useful linguistic properties for relation extraction, outperforming methods that rely on explicit lexical features (SVM~\cite{rink_semrel_2010}, RNN~\cite{zhang_relationcv_2015}). Similarly, our model outperforms approaches that rely on explicit syntactic features such as the shortest dependency path and learned distributed representations of part-of-speech and named entity tags (e.g., BCRNN~\cite{cai_bidirectionalrc_2016}, DRNN~\cite{xu_improvedrc_2016}, CGCN~\cite{zhang_graphco_2018}).
\begin{table}[ht!]
\centering
\begin{tabular}{l l l l}
\hline
System & P & R & F1 \\\hline
SVM{$^\mathrm{\dagger}$}~\citet{rink_semrel_2010} & -- & -- & 82.2\\
PA-LSTM{$^\mathrm{\dagger}$}~\citet{zhang_graphco_2018} & -- & -- & 82.7 \\
C-GCN{$^\mathrm{\dagger}$}~\citet{zhang_graphco_2018} & -- & -- & 84.8 \\
DRNN{$^\mathrm{\dagger}$}~\citet{xu_improvedrc_2016} & -- & -- & 86.1 \\
BRCNN{$^\mathrm{\dagger}$}~\citet{cai_bidirectionalrc_2016} & -- & -- & 86.3 \\
PCNN~\citet{zeng_distant_2015} & 86.7 & 86.7 & 86.6\\
TRE (ours) & 88.0 & 86.2 & $\mathbf{87.1}$ ($\pm 0.16$)\\\hline
\end{tabular}
\caption{SemEval single-model test set performance. $\mathrm{\dagger}$ marks results reported in the corresponding papers. We report the mean and standard deviation across 5 randomly initialized runs.}
    \label{tab:results_semeval2}
\end{table}

Similar to~\citet{zhang_graphco_2018}, we observe a high correlation between entity mentions and relation labels. According to the authors, simplifying SemEval sentences in the training and validation set to just \textit{``subject and object''}, where ``subject'' and ``object'' are the actual entities, already achieves an F1 score of 65.1. To better evaluate our model's ability to generalize beyond entity mentions, we substitute the entity mentions in the training set with a special unknown \textit{(UNK)} token. The token simulates the presence of unseen entities and prevents overfitting to entity mentions that strongly correlate with specific relations.
\begin{table}[ht!]
    \centering
    \begin{tabular}{l c c l}
        \hline
        System & P & R & F1 \\\hline
        PA-LSTM{$^\mathrm{\dagger}$} \citet{zhang_graphco_2018} & -- & -- & 75.3 \\
        C-GCN{$^\mathrm{\dagger}$} \citet{zhang_graphco_2018} & -- & -- & 76.5 \\
        TRE (ours) & 80.3 & 78.0 & \textbf{79.1} ( $\pm$ 0.37) \\\hline
    \end{tabular}
    \caption{SemEval single-model test set performance with all entity mentions masked by an unknown \textit{(UNK)} token. $\mathrm{\dagger}$ marks results reported in the corresponding papers. Due to the small test set size, we report the mean and standard deviation across 5 randomly initialized runs.}
    \label{tab:results_semeval_masked}
\end{table}
Our model achieves an F1 score of 79.1 (Table \ref{tab:results_semeval_masked}), an improvement of 2.6 points F1 score over the previous state-of-the-art. The result suggests that pre-trained language representations improve our model's ability to generalize beyond the mention level when predicting the relation between two previously unseen entities.

\section{Analysis \& Ablation Studies}
\label{analysis_discussion}
Although we demonstrated strong empirical results, we have not yet isolated the contributions of specific parts of TRE. In this section, we perform ablation experiments to understand the relative importance of each model component, followed by experiments to validate our claim that pre-trained language representations capture linguistic properties useful to relation extraction and also improve sample efficiency. We report our results on the predefined TACRED validation set and randomly select 800 examples of the SemEval training set as a validation set.

\subsection{Effect of Pre-training}
Pre-training affects two major parts of our model: language representations and byte pair embeddings. In Table \ref{tab:ablation_masking}, we first compare a model that was fine-tuned using pre-trained representations to one that used randomly initialized language representations. On both datasets we observe fine-tuning to considerably benefit from pre-trained language representations. For the SemEval dataset, the validation F1 score increases to $85.6$ when using a pre-trained language model and no entity masking, compared to $75.6$ without pre-training. We observe even more pronounced performance gains for the TACRED dataset, where using a pre-trained language model increases the validation F1 score by $20$ to $63.3$. With entity masking, performance gains are slightly lower, at $+8$ on the SemEval dataset and $+9.4$ (UNK) respectively $+3.8$ (NE$+$GR) on the TACRED dataset. The larger effect of pre-training when entity mentions are not masked suggests that pre-training has a regularizing effect, preventing overfitting to specific mentions. In addition, the contextualized features allow the model to better adapt to complex entities. Our observations are consistent with the results of~\citet{ruder_universallm_2018}, who observed that language model pre-training considerably improves text classification performance on small and medium-sized datasets, similar to ours.

\begin{table}[ht!]
    \begin{center}
        \begin{tabular}{l@{\hskip .5in} c c c@{\hskip .3in} c c c}
            \hline
             & \multicolumn{2}{c}{\textbf{SemEval}} & & \multicolumn{3}{c}{\textbf{TACRED}} \\

            & None & UNK & & None & UNK & NE + GR \\
            \hline
            Best model & \textbf{85.6} & 76.9 & & 63.3 & 51.0 & \textbf{68.0} \\
            -- w/o pre-trained LM & \textbf{75.6} & 68.2 & & 43.3 & 41.6 & \textbf{64.2} \\
            -- w/o pre-trained LM and BPE & 55.3 & \textbf{60.9} & & 38.5 & 38.4 & \textbf{60.8} \\
            \hline
        \end{tabular}
        \caption{Ablation with and without masked entities for SemEval \textbf{(left)} and TACRED validation set \textbf{(right)}. We report F1 scores over 5 independent runs.}
        \label{tab:ablation_masking}
    \end{center}
\end{table}

In addition, we train a model from scratch without pre-trained byte pair embeddings. We keep the vocabulary of sub-word tokens fixed and randomly initialize the embeddings. Again, we observe both datasets to benefit from pre-trained byte-pair embeddings. Because of its small size, SemEval benefits more from pre-trained embeddings, as these can not be learned reliably from the small corpus. This increases the risk of overfitting to entity mentions, which can be seen in the lower performance compared to UNK masking, where entity mentions are not available. For the TACRED dataset, model performance drops by approximately $3-5\%$ with and without entity masking when not using pre-trained byte pair embeddings.

\subsection{Which Information is captured by Language Representations?}
Undoubtedly, entity type information is crucial to relation extraction. This is confirmed by the superior performance on TACRED (Table~\ref{tab:ablation_masking}) when entity and grammatical role information is provided (NE$+$GR). The model achieves a validation F1 score of $68.0$, compared to $63.3$ without entity masking. Without pre-trained language representations, the model with NE$+$GR masking still manages to achieve a F1 score of $64.2$. This suggests that pre-trained language representations capture features that are as informative as providing entity type and grammatical role information.
This is also suggested by the work of~\citet{peters_deepcw_2018}, who show that a language model captures syntactic and semantic information useful for a variety of natural language processing tasks such as part-of-speech tagging and word sense disambiguation.

\subsection{Effect of Entity Masking}
\label{subsec:effect_entity_masking}
Entity masking, as described in Section~\ref{subsec:exp_masking}, can be used to limit the information about entity mentions available to our model and it is valuable in multiple ways. It can be used to simulate different scenarios, such as the presence of unseen entities, to prevent overfitting to specific entity mentions, and to focus more on context. Table~\ref{tab:robustness} shows F1 scores on the TACRED validation dataset for different entity masking strategies. As we saw previously, masking with entity and grammatical role information yields the best overall performance, yielding a F1 score of $68.0$. We find that using different masking strategies mostly impacts the recall, while precision tends to remain high, with the exception of the UNK masking strategy.

When applying the UNK masking strategy, which does not provide any information about the entity mention, the F1 score drops to $51.0$. Using grammatical role information considerably increases performance to an F1 score of $56.1$. This suggests that either the semantic role type is a very helpful feature, or its importance lies in the fact that it provides robust information on where each argument entity is positioned in the input sentence. When using NE masking, we observe a significant increase in recall, which intuitively suggests a better generalization ability of the model. Combining NE masking with grammatical role information yields only a minor gain in recall, which increases from $65.3\%$ to $67.2\%$, while precision stays at $68.8\%$.

\begin{table}[ht!]
    \begin{center}
        \begin{tabular}{p{3cm} c c c}
            \hline
            Entity Masking & Precision & Recall & F1 \\
            \hline
            None & \textbf{69.5} & 58.1 & 63.3 \\
            UNK & 56.9 & 46.3 & 51.0 \\
            GR & 63.8 & 50.1 & 56.1 \\
            NE & 68.8 & 65.3 & 67.0 \\
            NE + GR & 68.8 & \textbf{67.2} & \textbf{68.0} \\
            \hline
        \end{tabular}
        \caption{TACRED validation F1 scores for TACRED with different entity masking strategies.}
        \label{tab:robustness}
    \end{center}
\end{table}

\subsection{Sample Efficiency}
We expect a pre-trained language model to allow for a more sample efficient fine-tuning on the relation extraction task.
To assess our model's sample efficiency, we used stratified subsampling splits of the TACRED training set with sampling ratios from 10\% to 100\%. We trained the previously presented model variants on each split, and evaluated them on the complete validation set using micro-averaged F1 scores, averaging the scores over 5 runs.

\begin{figure}[ht!]
\centering
\includegraphics[width=\textwidth]{sample_efficiency.png}
\caption{Micro-averaged F1 score on the validation set over increasing sampling ratios of the training set}\label{fig:sample_efficiency}
\end{figure}

The results are shown in Figure~\ref{fig:sample_efficiency}. The best performing model uses a pre-trained language model combined with NE$+$GR masking, and performs consistently better than the other models. There is a steep performance increase in the first part of the curve, when only a small subset of the training examples is used. The model reaches an F1 score of more than $60$ with only $20\%$ of the training data, and continues to improve with more training data.

The next best models are the TRE model without a pre-trained language model, and the TRE model without NE$+$GR masking. They perform very similar, which aligns well with our previous observations. The PCNN baseline performs well when masking is applied, but slightly drops in performance compared to the TRE models after $30\%$ of training data, slowly approaching a performance plateau of around $61$ F1 score. The PCNN baseline without masking performs worse, but improves steadily due to its low base score. The TRE model without a language model seems to overfit early and diminishes in performance with more than $70\%$ training data. Interestingly, the performance of several models drops or stagnates after about 80\% of the training data, which might indicate that these examples do not increase the models' regularization capabilities.

\section{Related Work}
\label{related_work}
\noindent  \textbf{Relation Extraction} \quad Providing explicit linguistic features to inform the relation classification is a common approach in relation extraction. Initial work used statistical classifiers or kernel based methods in combination with discrete syntactic features~\cite{zelenko_kernel_2003, mintz_distantsf_2009, hendrickx_semeval2010t8_2010}, such as part-of-speech and named entities tags, morphological features, and WordNet hypernyms. More recently, these methods have been superseded by neural networks, applied on a sequence of input tokens to classify the relation between two entities. Sequence based methods include recurrent~\cite{socher_semanticct_2012, zhang_relationcv_2015} and convolutional~\cite{zeng_relationcv_2014, zeng_distant_2015} neural networks. With neural models, discrete features have been superseded by distributed representations of words and syntactic features~\cite{turian_wordra_2010, pennington_glove_2014}. \citet{xu_semanticrc_2015, xu_classifyingrv_2015} integrated shortest dependency path (SDP) information into a LSTM-based relation classification model. Considering the SDP is useful for relation classification, because it focuses on the action and agents in a sentence~\cite{bunescu_shortest_2005,socher_groundedcs_2014}. \citet{zhang_graphco_2018} established a new state-of-the-art for relation extraction on the TACRED dataset by applying a combination of pruning and graph convolutions to the dependency tree.
Recently, \citet{verga_transformer_2018} extended the Transformer architecture~\cite{vaswani_attention_2017} by a custom architecture  for biomedical named entity and relation extraction. In comparison, our approach uses pre-trained language representations and no architectural modifications between pre-training and task fine-tuning.

\noindent \textbf{Language Representations and Transfer Learning} \quad
Deep language representations have shown to be a very effective form of unsupervised pre-training. \cite{peters_deepcw_2018} introduced embeddings from language models (ELMo), an approach to learn contextualized word representations by training a bidirectional LSTM to optimize a language model objective. \citet{peters_deepcw_2018} results show that replacing static pre-trained word vectors~\cite{mikolov_efficienteo_2013,pennington_glove_2014} with contextualized word representations significantly improves performance on various natural language processing tasks, such as  semantic similarity, coreference resolution, and semantic role labeling. \citet{ruder_universallm_2018} showed language representations learned by unsupervised language modeling to significantly improve text classification performance, to prevent overfitting, and to also increase sample efficiency. \cite{radford_improvinglu_2018} demonstrated that general-domain pre-training and task-specific fine-tuning, which our model is based on, achieves state-of-the-art results on several question answering, text classification, textual entailment, and semantic similarity tasks.

\section{Conclusion}
We proposed TRE, a Transformer based relation extraction method that replaces explicit linguistic features, required by previous methods, with implicit features captured in pre-trained language representations. We showed that our model outperformes the state-of-the-art on two popular relation extraction datasets, TACRED and SemEval 2010 Task 8. We also found that pre-trained language representations drastically improve the sample efficiency of our approach. In our experiments we observed language representations to capture features very informative to the relation extraction task.

While our results are strong, important future work is to further investigate the linguistic features that are captured by TRE. One question of interest is the extent of syntactic structure that is captured in language representations, compared to information provided by dependency parsing. Furthermore, our generic architecture enables us to integrate additional contextual information and background knowledge about entities, which could be used to further improve performance.

\acks{This research was partially supported by the German Federal Ministry of Education and Research through the projects DEEPLEE (01IW17001) and BBDC2 (01IS18025E), and by the German Federal Ministry of Transport and Digital Infrastructure through the project DAYSTREAM (19F2031A).
}