\section{Benchmarks}

A common complaint about C++ abstractions in HPC is that they hinder
compiler optimizations. While that was largely true in the past, a
number of developments have improved the situation. More recent C++
standards introduce capabilities and constraints which help the compiler
optimize code. Furthermore, with the widespread adoption of C++
abstraction layers in industry, significant work has gone into
optimizing commonly used compilers. To demonstrate that \texttt{mdspan}
does not introduce overheads compared to using raw pointers with manual
indexing, we will show benchmark results both for the version using
\texttt{mdspan} and an equivalent implementation using raw pointers.
Since the difference in most benchmarks is very small, most figures in
this section show overhead of the \texttt{mdspan} version over the raw
pointer variant. Negative overhead indicates cases where the
\texttt{mdspan} version was faster.

\begin{figure*}[!ht]
\centering
\includegraphics[width=0.95\textwidth]{figures/raw_vs_mdspan_normalized.pdf}
\caption{An overview of selected benchmark comparisons of mdspan and raw pointer performance.  Each benchmark is normalized to the average execution time of the raw pointer case.  Details of each of these benchmarks are described in the text.}
\label{raw-vs-mdspan-overview}
\end{figure*}

Figure \ref{raw-vs-mdspan-overview} shows a normalized comparison of
\texttt{mdspan} versions of several selected benchmarks with the same
benchmark expressed with raw pointers. A more thorough elaboration
follows. Most of the benchmarks showed overheads within the measurement
noise, and no benchmarks showed overhead greater than 10\%. Examination
of generated assembly (and, at least in the case of the Intel compiler,
optimization reports) indicates similar---usually
identical---vectorization of the \texttt{mdspan} and raw pointer
versions of our benchmarks.

\subsection{Methodology}

All benchmarks were prepared and executed using the Google Benchmark
microbenchmarking support library.\cite{googlebenchmark} Table
\ref{machines} lists the test systems and compilers used for
benchmarking. Unless otherwise stated, CPU benchmarks were run on
Mutrino, and GPU benchmarks were run on Apollo. CPU benchmarks are
serial unless labeled ``OpenMP'', in which case they were parallelized
with the OpenMP \texttt{parallel\ for} directive on the outermost loop
(with the intent of measuring typical basic usage of OpenMP). CPU
benchmarks were compiled with GCC 8.2.0, Intel ICC 18.0.5, and the
latest Clang development branch (GitHub hash \texttt{1fcdcd0}, which is
LLVM SVN revision 370135; labeled as ``Clang 9 (develop)'' herein). GPU
benchmarks were compiled with NVIDIA's NVCC version 10.1, using GCC
5.3.0 as the host compiler. The source code of all benchmarks is
available on the mdspan implementation repository that accompanies this
paper (see Implementation section above). A brief description of each
benchmark is also included here for completeness. These benchmarks tend
to focus on the three-dimensional use case (which we view as the
smallest ``relatively non-trivial'' number of dimensions), but spot
checks with larger numbers of dimensions---up to 10---yielded similar
results and led to similar conclusions.

\begin{table}[htbp]
\caption{Test Systems and Software}
\begin{center}
\input{machine_table}
\label{machines}
\end{center}
\end{table}

\subsubsection{\texorpdfstring{\texttt{Sum3D}
Benchmark}{Sum3D Benchmark}}

Intended as a ``simplest possible'' benchmark, this benchmark simply
sums over all of the entries in a 3D \texttt{mdspan}. The raw pointer
version (as with all of the benchmarks) does the same thing, but uses
hard-coded index arithmetic. Both right-most fast-running and left-most
fast-running loop structures and layouts were tested (and yielded
similar results), and only the right layout, right loop structure
results are discussed in this paper, for brevity. The relevant portion
of the source code for this benchmark, for an input \texttt{mdspan}
named \texttt{s} and an output named \texttt{sum}, looks like:

\begin{Shaded}
\begin{Highlighting}[]
\ControlFlowTok{for}\NormalTok{(}\DataTypeTok{ptrdiff\_t}\NormalTok{ i = }\DecValTok{0}\NormalTok{; i \textless{} s.extent(}\DecValTok{0}\NormalTok{); ++i) \{}
  \ControlFlowTok{for}\NormalTok{ (}\DataTypeTok{ptrdiff\_t}\NormalTok{ j = }\DecValTok{0}\NormalTok{; j \textless{} s.extent(}\DecValTok{1}\NormalTok{); ++j) \{}
    \ControlFlowTok{for}\NormalTok{ (}\DataTypeTok{ptrdiff\_t}\NormalTok{ k = }\DecValTok{0}\NormalTok{; k \textless{} s.extent(}\DecValTok{2}\NormalTok{); ++k) \{}
\NormalTok{      sum += s(i, j, k);}
\NormalTok{    \}}
\NormalTok{  \}}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\subsubsection{\texorpdfstring{\texttt{Stencil3D}
Benchmark}{Stencil3D Benchmark}}

This benchmark takes the sum of all of the neighboring points in
three-dimensional space from an input \texttt{mdspan} and stores it in
the corresponding entry of the output \texttt{mdspan}. In terms of
structured grid computations, it has a ``stencil size'' of one (which is
\texttt{d}, a \texttt{constexpr\ ptrdiff\_t} variable known to the
optimizer, in the code excerpt below). The relevant portion of the
source code for this benchmark, for an input \texttt{mdspan} named
\texttt{s} and an output \texttt{mdspan} named \texttt{o}, looks like
this:

\begin{Shaded}
\begin{Highlighting}[]
\ControlFlowTok{for}\NormalTok{(}\DataTypeTok{ptrdiff\_t}\NormalTok{ i = d; i \textless{} s.extent(}\DecValTok{0}\NormalTok{){-}d; ++i) \{}
  \ControlFlowTok{for}\NormalTok{(}\DataTypeTok{ptrdiff\_t}\NormalTok{ j = d; j \textless{} s.extent(}\DecValTok{1}\NormalTok{){-}d; ++j) \{}
    \ControlFlowTok{for}\NormalTok{(}\DataTypeTok{ptrdiff\_t}\NormalTok{ k = d; k \textless{} s.extent(}\DecValTok{2}\NormalTok{){-}d; ++k) \{}
      \DataTypeTok{value\_type}\NormalTok{ sum\_local = }\DecValTok{0}\NormalTok{;}
      \ControlFlowTok{for}\NormalTok{(}\DataTypeTok{ptrdiff\_t}\NormalTok{ di = i{-}d; di \textless{} i+d+}\DecValTok{1}\NormalTok{; ++di) \{}
        \ControlFlowTok{for}\NormalTok{(}\DataTypeTok{ptrdiff\_t}\NormalTok{ dj = j{-}d; dj \textless{} j+d+}\DecValTok{1}\NormalTok{; ++dj) \{}
          \ControlFlowTok{for}\NormalTok{(}\DataTypeTok{ptrdiff\_t}\NormalTok{ dk = k{-}d; dk \textless{} k+d+}\DecValTok{1}\NormalTok{; ++dk) \{}
\NormalTok{            sum\_local += s(di, dj, dk);}
\NormalTok{          \}}
\NormalTok{        \}}
\NormalTok{      \}}
\NormalTok{      o(i,j,k) = sum\_local;}
\NormalTok{    \}}
\NormalTok{  \}}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\subsubsection{\texorpdfstring{\texttt{TinyMatrixSum}
benchmark}{TinyMatrixSum benchmark}}

This benchmark applies a batched sum operation to large number of small
(in this paper, 3x3) matrices, accumulating from the input Nx3x3
\texttt{mdspan} into an Nx3x3 \texttt{mdspan}. The relevant portion of
the source code for this benchmark, for an input \texttt{mdspan} named
\texttt{s} and an output \texttt{mdspan} named \texttt{o}, looks like
this:

\begin{Shaded}
\begin{Highlighting}[]
\ControlFlowTok{for}\NormalTok{(}\DataTypeTok{ptrdiff\_t}\NormalTok{ i = }\DecValTok{0}\NormalTok{; i \textless{} s.extent(}\DecValTok{0}\NormalTok{); ++i) \{}
  \ControlFlowTok{for}\NormalTok{(}\DataTypeTok{ptrdiff\_t}\NormalTok{ j = }\DecValTok{0}\NormalTok{; j \textless{} s.extent(}\DecValTok{1}\NormalTok{); ++j) \{}
    \ControlFlowTok{for}\NormalTok{(}\DataTypeTok{ptrdiff\_t}\NormalTok{ k = }\DecValTok{0}\NormalTok{; k \textless{} s.extent(}\DecValTok{2}\NormalTok{); ++k) \{}
\NormalTok{      o(i,j,k) += s(i,j,k);}
\NormalTok{    \}}
\NormalTok{  \}}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\subsubsection{\texorpdfstring{\texttt{Subspan3D}
benchmark}{Subspan3D benchmark}}

This benchmark performs the same operations as the \texttt{Sum3D}
benchmark, but uses two calls to \texttt{subspan}, instead of accessing
the entries of the \texttt{mdspan} in the ``normal'' way
(\texttt{operator()} with three integer indices). It is intended to
stress the abstraction overhead (or lack thereof) in the implementation,
since \texttt{subspan} is the most complex part of the \texttt{mdspan}
implementation from a C++ perspective. Note that this is not the
intended use case of the \texttt{subspan} function, though it serves as
a reasonable worst-case proxy. The relevant portion of the source code
for this benchmark, for an input \texttt{mdspan} named \texttt{s} and an
output named \texttt{sum}, looks like this:

\begin{Shaded}
\begin{Highlighting}[]
\ControlFlowTok{for}\NormalTok{(}\DataTypeTok{ptrdiff\_t}\NormalTok{ i = }\DecValTok{0}\NormalTok{; i \textless{} s.extent(}\DecValTok{0}\NormalTok{); ++i) \{}
  \KeywordTok{auto}\NormalTok{ sub\_i = subspan(s, i, all, all);}
  \ControlFlowTok{for}\NormalTok{ (}\DataTypeTok{ptrdiff\_t}\NormalTok{ j = }\DecValTok{0}\NormalTok{; j \textless{} s.extent(}\DecValTok{1}\NormalTok{); ++j) \{}
    \KeywordTok{auto}\NormalTok{ sub\_i\_j = subspan(sub\_i, j, all);}
    \ControlFlowTok{for}\NormalTok{ (}\DataTypeTok{ptrdiff\_t}\NormalTok{ k = }\DecValTok{0}\NormalTok{; k \textless{} s.extent(}\DecValTok{2}\NormalTok{); ++k) \{}
\NormalTok{      sum += sub\_i\_j(k);}
\NormalTok{    \}}
\NormalTok{  \}}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

\subsubsection{\texorpdfstring{\texttt{MatVec}
benchmark}{MatVec benchmark}}

The \texttt{MatVec} benchmark performs a simple dense matrix-vector
multiply operation. It is aimed at demonstrating the impact of layout
choice on performance, particularly in the context of performance
portability of parallelization across diverse hardware platforms.
Consider this serial implementation:

\begin{Shaded}
\begin{Highlighting}[]
\ControlFlowTok{for}\NormalTok{(}\DataTypeTok{ptrdiff\_t}\NormalTok{ i = }\DecValTok{0}\NormalTok{; i \textless{} A.extent(}\DecValTok{0}\NormalTok{); ++i) \{}
  \ControlFlowTok{for}\NormalTok{(}\DataTypeTok{ptrdiff\_t}\NormalTok{ j = }\DecValTok{0}\NormalTok{; j \textless{} A.extent(}\DecValTok{1}\NormalTok{); ++j) \{}
\NormalTok{    y(i) += A(i,j) * x(j);}
\NormalTok{  \}}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

When parallelizing the outer loop via OpenMP, C++17 standard parallel
algorithms, or CUDA, the optimal layout depends on the hardware. On
CPUs, the compiler will vectorize the inner loop over \texttt{j}; thus,
unit-stride access on the second dimension of \texttt{A} is optimal. On
GPUs, no implicit auto-parallelization happens, so unit-stride access on
the first dimension is optimal. Being able to make this layout change in
the type of \texttt{A}---without actually changing the algorithm---means
that the algorithm can be generic over different architectures.

\subsection{Results: Compiler Comparison}

Figure \ref{compiler-comparison} shows a comparison of \texttt{mdspan}
overheads relative to the raw pointer analog for serial versions of
several benchmarks. With the exception of the \texttt{TinyMatrixSum}
benchmark using dynamic extents, overheads on all of the benchmarks were
either completely or very nearly within the experimental noise. The
outlier in this regard, \texttt{TinyMatrixSum} with dynamic extents, is
an interesting case study in the brittleness of modern loop optimizers,
whether or not C++ abstraction is involved. To a first approximation,
the authors believe the explanation for this is as follows: if the
compiler heuristic guesses that the inner loop sizes are too large, the
resulting optimization decisions (such as the amount of unrolling) are
inefficient for a 3x3 matrix. How the use of \texttt{mdspan} interacts
with the compiler's heuristic for generating this guess varies from
compiler to compiler. For instance, with the latest version of Clang,
the optimizer actually happens to make a \emph{better} guess, leading to
a ``negative overhead.''

\begin{figure}[htbp]
\centering
\includegraphics[width=0.45\textwidth]{figures/compiler_comparison.pdf}
\caption{Comparison of overheads, relative to raw pointer implementations, of the serial versions of various benchmarks across different compilers.}
\label{compiler-comparison}
\end{figure}

In many ways, the optimizer brittleness in this single outlier presents
a strong argument for the sort of genericness that \texttt{mdspan}
provides. As C++ continues to evolve, more compiler-specific extensions
that let programmers give hints to guide compiler optimization are
likely to trickle in. Maintaining such hints inside the logic of
application code is often impractical or impossible, but incorporating
that information into the \texttt{mdspan} accessor (particularly if such
accessors can be vendor-provided), over which most algorithms can be
generic, is a completely reasonable proposition in many cases.

\subsection{Results: Effect of Static Extents}

\begin{figure}[htbp]
\centering
\includegraphics[width=0.45\textwidth]{figures/static_extents_tinymatrixsum_compilers.pdf}
\caption{Comparison of speedups, relative to the fully dynamic version, of the TinyMatrixSum benchmark.  "D" indicates a dynamic expression of the particular extent, while "S" indicates a static expression (for instance, "DxDxS" indicates that the first extent, 1 million, was expressed dynamically, the second extent, 3, was expressed dynamically, and the third extent, 3, was expressed statically).}
\label{static-extents}
\end{figure}

Figure \ref{static-extents} shows the speedup achieved when using static
extents for the two inner dimensions as opposed to dynamic extents. When
programmers provide them as static extents, the compiler is able to
unroll the inner loops fully, resulting in nearly two times better
performance on the test system Mutrino. The effect of static extents on
the compiler's ability to optimize can vary significantly from compiler
to compiler based on design decisions internal to the compiler's
implementation.

\subsection{Results: Effect of Layout Abstraction}

The benchmark in Figure \ref{layout-matvec} was run on the ARM ThunderX2
(test system Astra), Intel SkyLake (test system Blake), and NVIDIA
TitanV (test system Apollo) platforms using OpenMP parallelization for
the CPUs and CUDA for the GPU. On the CPU systems the use of
\texttt{layout\_right} (for the matrix) provides the better performance,
with \texttt{layout\_left} being 3x-7x slower. On the GPU, however, the
\texttt{layout\_left} version achieves a 10x higher throughput. The
results shown represent performance measured in terms of algorithmic
memory throughput (that is, the count of memory accesses in the
algorithm divided by run time.)

\begin{figure}[htbp]
\centering
\includegraphics[width=0.45\textwidth]{figures/matvec_figure.pdf}
\caption{Comparison of absolute memory bandwidths for the MatVec benchmark with different memory layouts.}
\label{layout-matvec}
\end{figure}

\subsection{\texorpdfstring{Results: Overhead of
\texttt{subspan}}{Results: Overhead of subspan}}

For recent versions of GCC and Clang, the results are essentially
identical to the raw pointer implementation of \texttt{Sum3D}, as shown
in Figure \ref{subspan-gcc-and-clang}. (There is no raw pointer
implementation of \texttt{Subspan3D}, since the whole point is that it
would be identical to \texttt{Sum3D}.) For ICC 18.0.5, the results
showed significant overhead, rendering the GCC and Clang results
invisible---as much as 400\%. (The absolute magnitudes of the raw
pointer timings were similar across all three compilers, so this is a
genuine measurement of overhead introduced by the ICC frontend). Using
the more recent ICC 19.0.3.199, we were able to obtain much more
reasonable results in C++17 mode. Interestingly, though, the C++14
results \emph{with the same compiler version} were much more similar to
the ICC 18.0.5 results, indicating that the difference arises, at least
in part, from more modern C++ abstractions being easier for modern
compilers to understand. These results are shown in Figure
\ref{subspan-intel}.

\begin{figure}[htbp]
\centering
\includegraphics[width=0.45\textwidth]{figures/subspan_overhead_gcc_and_clang.pdf}
\caption{Comparison of overheads, relative to raw pointer implementations, of the Subspan3D benchmark for GCC and Clang.}
\label{subspan-gcc-and-clang}
\end{figure}

\begin{figure}[htbp]
\centering
\includegraphics[width=0.5\textwidth]{figures/subspan_overhead_intel.pdf}
\caption{Comparison of overheads, relative to raw pointer implementations, of the Subspan3D benchmark for ICC 19.0.3.199.  Note that this compiler was not available on our primary testing machine, so the test system Blake was used for this benchmark.}
\label{subspan-intel}
\end{figure}

\section{Conclusions}

Based on the \texttt{View} data structure in the Kokkos C++ Performance
Portable Programming Model, \texttt{mdspan} introduces a
multi-dimensional array view abstraction into the C++ Standard. We have
presented both the ISO C++ design and a production-oriented
implementation of \texttt{mdspan}. The design includes layout and
accessor abstractions that address performance portability concerns.
Besides controlling memory access patterns and data access semantics,
the abstractions also open the door for incorporating heterogeneous
memory paradigms via strong typing. Using a number of microbenchmarks,
we have demonstrated that our implementation of \texttt{mdspan} has (in
most cases) negligible overhead, compared to using raw pointers to
represent multi-dimensional arrays. The implementation can be used with
a C++11 standard-compliant compiler, and thus can be used with currently
available toolchains on typical supercomputing systems. The
standardization of \texttt{mdspan} lays the foundation for further
efforts, such as standardized linear algebra,\cite{wg21_p1673} which can
help to address future performance portability needs of HPC and
heterogeneous computing use cases.

\section{Acknowledgments}

This work was carried out in part at Sandia National Laboratories.
Sandia National Laboratories is a multimission laboratory managed and
operated by National Technology \& Engineering Solutions of Sandia, LLC,
a wholly owned subsidary of Honeywell International Inc., for the U. S.
Department of Energy's National Nuclear Security Administration under
contract DE-NA0003525.

\section{Design}

\texttt{mdspan} provides a class template for creating types of objects
that represent, but do not own, a contiguous piece (or ``span'') of
memory that is to be treated as a multi-dimensional entity with one or
more dimensional constraints. Together, these dimensional constraints
form a \emph{multi-index domain}. In the simple case of a
two-dimensional entity, for instance, this multi-index domain
encompasses the row and column indices of what is typically called a
matrix. For instance,

\begin{Shaded}
\begin{Highlighting}[]
\DataTypeTok{void}\NormalTok{ some\_function(}\DataTypeTok{float}\NormalTok{* data) \{}
  \KeywordTok{auto}\NormalTok{ my\_matrix =}
\NormalTok{    mdspan\textless{}}\DataTypeTok{float}\NormalTok{, dynamic\_extent, dynamic\_extent\textgreater{}(}
\NormalTok{      data, }\DecValTok{20}\NormalTok{, }\DecValTok{40}
\NormalTok{    );}
  \CommentTok{/* ... */}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

says to create an object that interprets memory starting at the pointer
\texttt{data} as a matrix with the shape 20 rows by 40 columns. Extents
can be provided either statically (i.e., at compile time) or dynamically
(as shown above), and static extents can be mixed with dynamic extents:

\begin{Shaded}
\begin{Highlighting}[]
\DataTypeTok{void}\NormalTok{ another\_function(}\DataTypeTok{float}\NormalTok{* data) \{}
  \KeywordTok{auto}\NormalTok{ another\_matrix =}
\NormalTok{    mdspan\textless{}}\DataTypeTok{float}\NormalTok{, }\DecValTok{20}\NormalTok{, dynamic\_extent\textgreater{}(}
\NormalTok{      data, }\DecValTok{40}
\NormalTok{    );}
  \CommentTok{/* ... */}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

This code snippet also treats \texttt{data} as a 20 by 40 matrix, but
the first of these dimensions is ``baked in'' to the type at compile
time---all instances of the type
\texttt{mdspan\textless{}double,\ 20,\ dynamic\_extent\textgreater{}}
will have 20 rows.

The design is greatly simplified by delegating the ownership and
lifetime management of the data to orthogonal constructs. Thus,
\texttt{mdspan} merely interprets existing memory as a multi-dimensional
entity, leaving management of the underlying memory to the user. This
follows a trend of similar constructs recently introduced to C++, such
as
\texttt{string\_view}\cite{wg21_is_14882:2017, cppreference_string_view}
and \texttt{span}\cite{wg21_is_14882:2020_cd, cppreference_span}. These
constructs allow ``API funnelling,'' which makes it easy for libraries
to support users' own types instead of forcing users to use a specific
type. Library interfaces can take \texttt{string\_view}s or
\texttt{mdspan}s, and library users can add interfaces to their own
types that return a suitable \texttt{string\_view} or \texttt{mdspan}.
This design pattern enables easy adoption by existing codebases which
have their own matrix types. Since \texttt{mdspan} is non-owning, users
can always create an \texttt{mdspan} that refers to a matrix owned by
another object. Older abstractions also take this approach. Iterators,
which have been central to C++ algorithm design for decades, are also
non-owning entities which delegate lifetime management as a separate
concern.\cite{stepanov2009}

References to entries in these matrices are obtained by giving a
multi-index (that is, an ordered set of indices) to the object's
\texttt{operator()}, which has been overloaded for this purpose:

\begin{Shaded}
\begin{Highlighting}[]
\CommentTok{// add 3.14 to the value on the row with index 10}
\CommentTok{// and the column with index 5}
\NormalTok{some\_matrix(}\DecValTok{10}\NormalTok{, }\DecValTok{5}\NormalTok{) += }\FloatTok{3.14}\NormalTok{;}
\CommentTok{// print the value of the entry in the row with}
\CommentTok{// index 0 and the column with index 38}
\NormalTok{printf(}\StringTok{"}\SpecialCharTok{\%f}\StringTok{"}\NormalTok{, some\_matrix(}\DecValTok{0}\NormalTok{, }\DecValTok{38}\NormalTok{));}
\end{Highlighting}
\end{Shaded}

The length of each dimension is accessed via the \texttt{extent} member
function. It takes an index to indicate the dimension. A loop to
multiply all entries of the matrix by a scalar could thus look like
this:

\begin{Shaded}
\begin{Highlighting}[]
\ControlFlowTok{for}\NormalTok{(}\DataTypeTok{int}\NormalTok{ row = }\DecValTok{0}\NormalTok{; row \textless{} my\_mat.extent(}\DecValTok{0}\NormalTok{); ++row)}
  \ControlFlowTok{for}\NormalTok{(}\DataTypeTok{int}\NormalTok{ col = }\DecValTok{0}\NormalTok{; col \textless{} my\_mat.extent(}\DecValTok{1}\NormalTok{); ++col)}
\NormalTok{    my\_mat(row, col) *= }\FloatTok{2.0}\NormalTok{;}
\end{Highlighting}
\end{Shaded}

Arbitrary slices of an \texttt{mdspan} can be taken using the
\texttt{subspan} function:

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{auto}\NormalTok{ my\_tens = mdspan\textless{}}\DataTypeTok{float}\NormalTok{, }\DecValTok{3}\NormalTok{, }\DecValTok{4}\NormalTok{, }\DecValTok{5}\NormalTok{, }\DecValTok{20}\NormalTok{\textgreater{}(data);}
\KeywordTok{auto}\NormalTok{ my\_matrix = subspan(my\_tens,}
  \DecValTok{2}\NormalTok{, all, pair\{}\DecValTok{2}\NormalTok{, }\DecValTok{4}\NormalTok{\}, }\DecValTok{0}
\NormalTok{);}
\end{Highlighting}
\end{Shaded}

The above snippet creates a 4 by 2 matrix sub-view of \texttt{my\_tens}
where the entries \texttt{i,\ j} correspond to index 2 in the first
dimension of \texttt{my\_tens}, index \texttt{i} in the second
dimension, \texttt{j+2} in the third dimension, and \texttt{0} in the
fourth dimension. This relatively verbose syntax for slicing was
preferred over other approaches, because slicing needs can vary
substantially across different domains. Domain-specific syntax can
easily be built on top of \texttt{subspan}.

Just as \texttt{std::string} is actually a C++ alias for
\texttt{std::basic\_string}\cite{wg21_is_14882:2017,cppreference_string_view},
\texttt{std::mdspan} is an alias for \texttt{std::basic\_mdspan}.
Whereas \texttt{std::mdspan} only provides control over the scalar type
and the extents, \texttt{std::basic\_mdspan} exposes more customization
points. It is templated on four parameters: the scalar type, the extents
object, the layout, and the accessor. In the following sections, we will
describe these parameters and their value for improving performance or
increasing portability.

\subsection{Extents Class Template}

In \texttt{basic\_mdspan} the extents are provided via an
\texttt{extents} class template. As with the \texttt{mdspan} alias
template, the parameters are either static sizes or the
\texttt{dynamic\_extent} tag.

\begin{Shaded}
\begin{Highlighting}[]
\DataTypeTok{void}\NormalTok{ some\_function(}\DataTypeTok{float}\NormalTok{* data) \{}
  \KeywordTok{auto}\NormalTok{ my\_matrix =}
\NormalTok{    basic\_mdspan\textless{}}\DataTypeTok{float}\NormalTok{, extents\textless{}}\DecValTok{20}\NormalTok{, dynamic\_extent\textgreater{}\textgreater{}(}
\NormalTok{      data, }\DecValTok{40}
\NormalTok{    );}
  \CommentTok{/* ... */}
\NormalTok{\}}
\end{Highlighting}
\end{Shaded}

The ability to provide extents statically can help significantly with
compiler optimizations. For example, a compiler may be able to unroll
small inner loops completely if the extents are known at compile time.
Knowing exact counts and sizes can also help with vectorization and the
optimizer's cost model. A typical example of this in HPC is operations
on a batch of small matrices or vectors, where the dimensions of each
item are dictated by a physics property or the way the system was
discretized, rather than by the problem size. When this sort of problem
interacts with generic code, such information would be lost unless
static extents can be part of the \texttt{mdspan} type itself. The
\texttt{TinyMatrixSum} benchmark (below) provides a proxy for problems
with this sort of behavior.

\subsection{Layout abstraction}

Modern C++ design requires library authors to orthogonalize certain
aspects of the design into customization points, over which algorithms
may be written generically. The most commonplace example of this is the
\texttt{Allocator}
abstraction\cite{wg21_is_14882:2017,cppreference_allocator_ntr}, which
controls memory allocation for standard containers like
\texttt{std::vector}\cite{wg21_is_14882:2017,cppreference_vector}. Most
algorithms on containers do not change regardless of how the underlying
data is allocated. The \texttt{Allocator} abstraction allows such
algorithms to be generic over the form of memory allocation used by the
container.

An example of one such aspect in the current context is the layout of
the underlying data with respect to the multi-index domain. While a
high-quality-of-implementation matrix multiply would definitely
specialize for different data layouts, the simplest possible
implementation would only need to know how to get and store data
associated with a given multi-index into the underlying memory. This
also describes the majority of use cases from the perspective of the
caller of such algorithms, where only the semantics of a mathematical
matrix multiply are needed regardless of data layout. The grouping of a
single set of mathematical semantics under a common algorithm name
(regardless of layout) serves as a conduit for performance portability,
and additionally reduces the cognitive load for the writer and
particularly the reader of the code.

The canonical example, again with reference to data layout, is the
portability of access patterns in code that may run on a
latency-optimizer processed (e.g., CPU) or on a bandwidth-optimized
processor (e.g., GPU). GPUs need to coalesce accesses (that is, stride
across execution agents) because of the vector nature of the underlying
hardware, whereas CPUs want to maximize locality (that is, assign
contiguous chunks to the same execution agent) in order to increase
cache reuse.

The abstraction for representing data layout generically is called the
\texttt{LayoutMapping}. The primary task of the \texttt{LayoutMapping}
is to represent the transformation of a multi-index into a single,
scalar memory offset. A large number of algorithms on multi-dimensional
arrays have semantics that depend only on the data as retrieved through
the multi-index domain, indicating that this transformation is a prime
aspect for orthogonalization into a customization point. (Note that many
algorithms have \emph{performance} characteristics that depend on this
transformation, but the separation of semantic aspects of an algorithm
from its performance characteristics is critical to modern programming
model design. The fact that the \texttt{LayoutMapping} abstraction
promotes this separation is further evidence of its utility as a
customization point.)

A brief survey of existing practice (such as the BLAS Technical Standard
\cite{BLAS}, Eigen\cite{eigenweb}, and MAGMA\cite{magma}) reveals an
initial set of layout mappings that such an abstraction must support.
These include, at minimum,

\begin{itemize}
\tightlist
\item
  row-major or column-major layouts (represented by the \texttt{TRANS}
  parameters in BLAS), that generalize to describe layouts where the
  fast-running index is left-most or right-most;
\item
  strided layouts (represented by the \texttt{LD} parameters in BLAS),
  that generalize to any in a class of layouts that can describe the
  distance in memory between two consecutive indices in a particular
  dimension with a constant (specific to that dimension); and
\item
  symmetric layouts (e.g., from the \texttt{xSYMM} algorithms in BLAS),
  which also include generalizations like whether the upper or lower
  triangle is stored (the \texttt{UPLO} parameter in BLAS) and whether
  the diagonal is stored explicitly, implicitly, or in some separate,
  contiguous storage.
\end{itemize}

In addition to similarities, it also helps to look at what differences
these layout mappings may introduce, over which some algorithms may not
be generic. In general, as many previous researchers have
noted,\cite{sutton2011design} the design of generic concepts for
customization typically begins with the algorithms, not the data
structures. Much of the design of \texttt{LayoutMapping} can be
motivated with some very simple algorithms. Consider an algorithm,
\texttt{scale}, that takes an \texttt{mdspan} and a scalar and
multiplies each entry, in place, by the scalar. For brevity, we will
only consider the two-dimensional case here (though much of this
motivation can be done even in the one-dimensional case). If such an
algorithm is to be implemented in the simplest possible way---iterating
over the rows and column indices and scaling each element---the
implementation would fail to meet the semantic requirements of the
algorithm for symmetric layouts, since non-diagonal entries reference
the same memory. Thus, it is necessary for certain algorithms to know
whether each multi-index in the domain maps to a unique offset in the
codomain (the space of all offsets that could be valid results of the
mapping). (An example of an algorithm for which this requirement is
\emph{not} needed is \texttt{dot\_product}.) The \texttt{LayoutMapping}
customization expresses this property through the requirement that it
provide an \texttt{is\_unique} method. Many algorithms are difficult or
impossible to implement on general non-unique layouts. However, in the
simple case of \texttt{scale}, the algorithm \emph{could} be implemented
for any layout that is simply \emph{contiguous}, by viewing the codomain
of the layout as a one-dimensional \texttt{mdspan} and scaling each item
that way. Contiguousness is expressed through the requirement of an
\texttt{is\_contiguous} method, and the size of the codomain is
expressed through the \texttt{required\_span\_size} required method.
Similarly, as previously observed, many existing implementations (such
as the BLAS) can specially handle any layout with regular strides.
Layout mappings can express whether they are strided using the
\texttt{is\_strided} method. Finally, all of these aspects need to be
expressible statically and dynamically, so for layout mappings where the
uniqueness, stridedness, and continguousness are consistently
\texttt{true} for all instances of the type, the
\texttt{is\_always\_unique}, \texttt{is\_always\_strided}, and
\texttt{is\_always\_contiguous} hooks are provided in the concept. These
requirements allow, for instance, algorithms that cannot support layouts
lacking certain properties to fail at compile time rather than run time.
The requirements on the \texttt{LayoutMapping} concept are summarized in
Table \ref{layoutreqs}.

\begin{table}[htbp]
\caption{Requirements on the \texttt{LayoutMapping} Concept}
\begin{center}
\input{layout_policy_table}
\label{layoutreqs}
\end{center}
\end{table}

\subsection{Accessor abstraction}

After several design iterations,\cite{wg21_p0009} the authors came to
the conclusion that many of the remaining customizations could be
encapsulated in the answer to one question: how should the
implementation turn an instance of some pointer type and an offset
(obtained from the \texttt{LayoutMapping} abstraction) into an instance
of some reference type? The \texttt{Accessor} customization point is
designed to provide all of the necessary flexibility in the answer to
this question. Our exploration in this space began with a couple of
specific use cases: a non-aliasing \texttt{Accessor}, similar to the
\texttt{restrict} keyword in C,\cite{c18standard} and an atomic
\texttt{Accessor}, where operations on the resulting reference use
atomic operations. The former needs to customize the pointer type to
include implementation-specific annotations (usually some variant of the
C-style \texttt{restrict} keyword) that indicate the pointer does not
alias pointers derived from other sources within the same context
(usually a function scope). The latter needs to customize the reference
type produced by the dereference operation to have it return a
\texttt{std::atomic\_ref\textless{}T\textgreater{}}.
(\texttt{std::atomic\_ref\textless{}T\textgreater{}} was merged into the
C++ Standard working draft during the C++20 cycle, and will likely be
officially approved as part of the C++20 balloting process when that
process completes sometime in 2020\cite{wg21_p0019}.) These requirements
immediately led us to include customizable \texttt{reference} and
\texttt{pointer} type names as part of the \texttt{Accessor} concept.
Marrying these two customizations could take several forms. One
possibility is to have a function that simply takes a \texttt{pointer}
and returns a \texttt{reference}. However, this requires the
\texttt{pointer} type to be arbitrarily offsettable---e.g., using
\texttt{operator+} or \texttt{std::advance}. A simpler approach that
removes this requirement is to have a customization point that takes the
\texttt{pointer} and an offset and returns the \texttt{reference}
directly. We chose the latter in order to simplify the requirements on
the \texttt{pointer} type, and named this required method
\texttt{access}.

The issue of offsetting a \texttt{pointer} to create another
\texttt{pointer}, while not necessarily separable from the creation of a
\texttt{reference}, is nonetheless also a concern that \texttt{Accessor}
needs to address for the implementation of the \texttt{subspan}
function. We named this customization with a required method
\texttt{offset}. The type of the \texttt{pointer} retrieved when
arbitrarily offsetting a \texttt{pointer} type may not necessarily match
the input pointer type. For instance, in the case of an overaligned
pointer type used for easy vectorization, a pointer derived from an
arbitrary (runtime) offset to this pointer cannot guarantee the
preservation of this alignment. Thus, the \texttt{Accessor} is allowed
to provide a different \texttt{Accessor}, named with the required type
name \texttt{offset\_policy}, that differs in type from itself (and
thus, for instance, may differ in its \texttt{pointer} type). Finally,
given an arbitrary \texttt{pointer} type, the current design requires
the ability to ``decay'' this type into an ``ordinary'' C++ pointer for
compatibility with \texttt{std::span}, which does not support
\texttt{pointer} type customization. The requirements on the
\texttt{Accessor} concept are summarized in Table \ref{accessreqs}.

\begin{table}[htbp]
\caption{Requirements on the \texttt{Accessor} Concept}
\begin{center}
\input{access_policy_table}
\label{accessreqs}
\end{center}
\end{table}

\subsubsection{Accessor Use Case: Non-Aliasing Semantics}

As a concrete example, the (trivial) \texttt{Accessor} required to
express non-aliasing semantics (similar to the \texttt{restrict} keyword
and supported in many C++ compilers as \texttt{\_\_restrict}) is shown
in Figure \ref{restrict-accessor}. This differs from the default
accessor (\texttt{std::accessor\_basic\textless{}T\textgreater{}}) only
in the definition of the nested type \texttt{pointer}. Interestingly,
because the design of \texttt{mdspan} requires the \texttt{pointer} to
be used as a parameter (in \texttt{access}) before it is ever turned
into a reference, \texttt{mdspan} is able to skirt the well-known issues
surrounding the meaning of the \texttt{restrict} qualifier on a data
member of a
struct.\cite{gcc_docs_restrict,msvc_docs_restrict,finkel2014restrict}

\begin{figure}[!h]

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{template}\NormalTok{ \textless{}}\KeywordTok{class}\NormalTok{ T\textgreater{}}
\KeywordTok{struct}\NormalTok{ RestrictAccessor \{}
  \KeywordTok{using} \DataTypeTok{element\_type}\NormalTok{ = T;}
  \KeywordTok{using}\NormalTok{ pointer = T* }\ExtensionTok{\_\_restrict}\NormalTok{;}
  \KeywordTok{using}\NormalTok{ reference = T\&;}
\NormalTok{  reference access(pointer p, }\DataTypeTok{ptrdiff\_t}\NormalTok{ i)}
    \AttributeTok{const} \KeywordTok{noexcept}
\NormalTok{  \{ }\ControlFlowTok{return}\NormalTok{ p[i]; \}}
\NormalTok{  pointer offset(pointer p, }\DataTypeTok{ptrdiff\_t}\NormalTok{ i)}
    \AttributeTok{const} \KeywordTok{noexcept}
\NormalTok{  \{ }\ControlFlowTok{return}\NormalTok{ p + i; \}}
\NormalTok{\};}
\end{Highlighting}
\end{Shaded}

\caption{An Accessor that provides an expression of non-aliasing semantics for mdspan.}
\label{restrict-accessor}
\end{figure}

\subsubsection{Accessor Use Case: Atomic Access}

Frequently in HPC applications, it is necessary to access a region of
memory atomically for only a small portion of its lifetime. Constructing
the entity to be atomic for the entire lifetime of the underlying
memory, as is done with \texttt{std::atomic}, may have unacceptable
overhead for many HPC use cases. As an entity that references a region
of memory for a subset of that memory's lifetime, \texttt{mdspan} is
ideally suited to be paired with a fancy reference type that expresses
atomic semantics (that is, all operations on the underlying memory are
to be performed atomically by the abstract machine). With the
introduction of \texttt{std::atomic\_ref} in C++20, all that is needed
is an accessor that customizes the reference type and provides an
\texttt{access} method that constructs such a reference. An
implementation of such an \texttt{Accessor} is shown in Figure
\ref{atomic-accessor}.

\begin{figure}[!h]

\begin{Shaded}
\begin{Highlighting}[]
\KeywordTok{template}\NormalTok{ \textless{}}\KeywordTok{class}\NormalTok{ T\textgreater{}}
\KeywordTok{struct}\NormalTok{ AtomicAccessor \{}
  \KeywordTok{using} \DataTypeTok{element\_type}\NormalTok{ = T;}
  \KeywordTok{using}\NormalTok{ pointer = T*;}
  \KeywordTok{using}\NormalTok{ reference = atomic\_ref\textless{}T\textgreater{};}
\NormalTok{  reference access(pointer p, }\DataTypeTok{ptrdiff\_t}\NormalTok{ i)}
    \AttributeTok{const} \KeywordTok{noexcept}
\NormalTok{  \{ }\ControlFlowTok{return}\NormalTok{ atomic\_ref\{ p[i] \}; \}}
\NormalTok{  pointer offset(pointer p, }\DataTypeTok{ptrdiff\_t}\NormalTok{ i)}
    \AttributeTok{const} \KeywordTok{noexcept}
\NormalTok{  \{ }\ControlFlowTok{return}\NormalTok{ p + i; \}}
\NormalTok{\};}
\end{Highlighting}
\end{Shaded}

\caption{An Accessor that provides an expression of atomic reference semantics for mdspan.}
\label{atomic-accessor}
\end{figure}

\subsubsection{Accessor Use Case: Bit-Packing}

Similar to the infamous
\texttt{std::vector\textless{}bool\textgreater{}}, the accessor
abstraction can be used to return a fancy reference type that references
individual bits packed into the bytes of underlying memory. (Unlike
\texttt{std::vector\textless{}bool\textgreater{}}, though,
\texttt{std::accessor\_basic\textless{}bool\textgreater{}} does not do
this by default.)

\subsubsection{Accessor Use Case: Strong Pointer Types for Heterogeneous
Memory Spaces}

Heterogeneity often requires a program to access multiple, potentially
disjoint memory spaces. Thus far, vendor-provided APIs for heterogeneity
have tended to represent this memory with plain-old raw pointers. An
important emerging paradigm in modern programming model design is
so-called ``strong types'' (also called ``opaque typedefs'' or ``phantom
types''),\cite{strongTypes,phantomOrigin} wherein meaning is opaquely
attached to the form of the type. For instance,
\texttt{distance\textless{}double\textgreater{}} and
\texttt{temperature\textless{}double\textgreater{}} might be different
concrete types that the compiler forbids mixing, even though they both
would use \texttt{double} for storage and arithmetic. Applied to
heterogeneity, the paradigm would suggest replacing raw pointers with
opaque typedefs indicating things like their compatibility or
accessibility. This not only introduces safety with respect to memory
access by an execution resource, but also allows generic software design
strategies where execution mechanisms can be deduced from the type of
the data structure. In \texttt{mdspan}, such strong typing can be
injected via the customization of the associated pointer type in the
\texttt{Accessor}. Initially, of course, such extensions will be outside
of the C++ Standard (e.g., OpenMP, HIP, SYCL, and older versions of
CUDA), but this design provides a means of forward compatibility if and
when the Standard addresses the concept of heterogeneous memory
resources natively in C++.

\section{Implementation}

This work is accompanied by a production-oriented implementation of the
proposed \texttt{std::mdspan}, available at
\href{https://github.com/kokkos/mdspan}{github.com/kokkos/mdspan}, the
details of which are discussed in this section. While the proposal it
implements targets C++23, the implementation includes compatibility
modes for C++17, C++14, and C++11. The implementation also includes a
couple of macros, \texttt{MDSPAN\_INLINE\_FUNCTION} and
\texttt{MDSPAN\_FORCE\_INLINE\_FUNCTION}, that can be used to add the
appropriate markings to functions and function templates, such as
\texttt{\_\_device\_\_} for CUDA compatibility. It has been tested on
various versions of numerous C++ compilers, including GCC, Clang,
Intel's ICC, Microsoft's MSVC, and NVIDIA's NVCC. The implementation
modestly extends the proposal in several places (mostly with respect to
typos in the latter), all of which are documented in the implementation
repository (link given above).

\section{Introduction}

One of the primary concerns of the high-performance computing (HPC)
community\cite{DOEPPP} is performance portability. \emph{Performance
portability} means that a single code base can perform well on many
different platforms. Over the last decade in particular, numerous
projects\cite{gridcpp2018,alpaka2016,occa2014,raja2014,kokkos2014} have
tried to address various challenges associated with it. The recent
announcement of the first exascale-class platforms, introducing
architectures which were previously not deployed in the HPC community,
has increased the urgency of finding solutions to performance
portability concerns. One of the projects which has found significant
success in adoption is Kokkos,\cite{kokkos2014,kokkosgithub} a C++
performance-portable programming model originally developed at Sandia
National Laboratories, but now maintained by a group spanning four
United States National Laboratories as well as the Swiss National
Supercomputing Centre.

Arguably the most significant innovation of the Kokkos project was its
\texttt{View} data structure, a multi-dimensional array abstraction
which addresses concerns of performance portability such as data layout
and data access customization. This array abstraction is now used at the
heart of many HPC software projects\cite{kokkosprojects}, and is proving
to be critical for meeting the challenges of preparing code bases for
the exascale era. While maintaining these capabilities in an
HPC-specific solution is workable for now, there are a number of reasons
why it would be beneficial to have the core capabilities become part of
international programming language standards. Doing so would enable
tighter integration into other language and library capabilities, such
as the proposed ISO C++ Linear Algebra library\cite{wg21_p1673}. It
would simplify interface compatibility between different HPC products,
and would further seamless integration with external products used in
applications not specific to HPC. For example, the proposed ISO C++
Audio library\cite{wg21_p1386} has expressed interest in using this
abstraction.

To that end the Kokkos team initiated a collaboration with other
stakeholders to design a multi-dimensional array for the ISO C++
Standard, that also addresses the concerns of performance portability
addressed by the Kokkos \texttt{View} abstraction. The result of this
over five-year process is \texttt{std::mdspan}, described herein and
proposed to the ISO C++ standard in the proposal P0009\cite{wg21_p0009}.
The design allows for a mix of static and dynamic array dimensions,
enables control of the data layout, and has customization points to
control how data are accessed. The latter includes use cases that
involve hardware-specific special load and store paths.

In this work we describe each of the design aspects of \texttt{mdspan},
with examples demonstrating their impact for performance and portability
concerns, as well as benchmarks of the production-quality reference
implementation developed by the authors.

\section{Ease of Use}

\bibliographystyle{IEEEtran}


\subsubsection{Multilingual Argument Assessment}
While \citet{gretz2020quality} mention
using a group of annotators with whom they have worked before for the assessment of arguments written in English, no such group was available to us for the non-English languages.
In addition, no test questions (TQs) were available, since they are typically formed from existing labeled data.
Initially, the first issue was addressed by relying on workers from appropriate countries, and the second by using machine-translated arguments from \argumentDatasetName, with a high-confidence label \emph{in English}, as TQs.
At first, since the quality label is sensitive to translation (as described in \sectionRef{subsec:translated_label_assessment}) such TQs were limited to stance.

A pilot on Spanish arguments showed a good \agreementName for stance ($0.71$), yet a low value for quality ($0.04$).
The results showed that many of the annotators labeled a vast majority ($>$\percentage{80}) of the arguments as high-quality,
even though they were instructed to consider only half as such.
Therefore, only those labeling $\leq80\%$ of arguments as high-quality were allowed further work. Others were excluded and their argument quality answers were ignored.

A second pilot extended this procedure to other languages.  However, the size of the workforce meeting the above criteria was small for \germanShort, \frenchShort and \dutchShort, preventing progress altogether for the last two.
This required integrating TQs for quality question despite the risk of the quality label changing due to the automatic translation.
To mitigate that risk, one of the authors carefully monitored each annotation task, reviewed TQs which many annotators answered incorrectly, and disabled those in which the translation introduced errors in the correct label or made the text unclear.

\section{Appendices}

\subsection{Additional Results}
\label{sec:appendix}

\subsubsection{Stance Classification on Machine-Translated Arguments}
\label{appendix:stance_mt_17l_results}

As described in \sectionRef{empirical_evaluation},
our evaluations were conducted on 6 European languages (EN, ES, FR, IT, DE, and NL).
Models were trained for several language groups: \emph{RM} -- for the Romance languages, \emph{WG} -- the West-Germanic, and \emph{6L} -- a model that covers all the TLs in our evaluation.

We further explored the translate-train approach by augmenting the training data of our models with machine-translated data of other language families.
First, we trained a model for the North-Germanic family (\emph{NG}) with three languages -- Danish (DA), Swedish (SV), and Norwegian (NB).
Next, we combined the Romance languages with  the two German families (\emph{RM, WG, and NG}), and created the \emph{9L} model with $9$ languages.
Finally, we trained a model with a relatively large number of languages ($17$) and a variety of language families. This model, denoted \emph{17L}, consists of all the languages in \emph{9L} and $8$ additional languages: Slavic languages -- Polish (PL), Slovak (SK), Russian (RU); Semitic -- Arabic (AR), Hebrew (HE); and Chinese/Japonic -- Simplified Chinese (ZH), Traditional Chinese (ZT), and Japanese (JA).

The stance classification results on
the \pseudoTestEviDatasetName pseudo-test for
all $17$ languages using all the aforementioned models are presented in \tableRef{tab_sbc_stance_mt}.
We see that expanding the training set beyond the six languages in \emph{6L} by adding more distant languages, as in the \emph{9L} and \emph{17L} models, does not significantly improve the performance on English.
On average,
training on the TL is better than training on the original English arguments (average performance over all $17$ languages
is $73.7\%$ with \emph{EN} and $86.5\%$ with \emph{TL}). Training on all $17$ languages tends to yield the best performance (with an average of $88.9\%$), though training on a subset of them is often nearly as good - and sometimes even better, especially on the \emph{6L} and \emph{9L} groups.

\input{tables/sbc_stance_mt}

\subsubsection{Per-Topic Analysis Information}
\tableRef{tab:sbc_test_motions} contains a list of $15$ topics that are included the test set of the \argumentDatasetName dataset, along with their IDs used during error analysis.

\input{tables/sbc_test_motions_ids}

\figureRef{fig_sbc_stance_hg_zs_per_topic} shows
per-language results of the zero-shot baseline on the $15$ topics of the \humanGeneratedArgDatasetName evaluation dataset.
As indicated by the overall results on the same data in \figureRef{fig_sbc_stance_hg}, we see high performance on the three Romance languages consistently across most topics, and low performance on DE and NL for about half of the topics.

\begin{figure}[th]
\includegraphics[width=\linewidth,
trim={1.0cm, 0.2cm, 1.4cm, 0.8cm},
clip]{figures/fig_sbc_stance_hg_zs_per_topic.pdf}
\caption{Per-topic stance classification
average macro-F1 results per language on \humanGeneratedArgDatasetName test set for the \emph{ZS} baseline. The dashed black line is the performance on English.}
\label{fig_sbc_stance_hg_zs_per_topic}
\end{figure}

\subsection{Annotation Details}
\label{appendix:annotation_details}
This section describes further annotation details, such as the adjustment of the argument assessment annotation task to multiple languages, and the guidelines used in each annotation task.
\input{Appendix/annotation_scaling}
\input{Appendix/argument_authoring_guidelines}
\input{Appendix/argument_assessment_guidelines}
\input{Appendix/evidence_assessment_guidelines}

\subsubsection{Argument Assessment Guidelines}
In this annotation, the guidelines for all languages were the same.
\\ \\
In the following task you should answer two questions concerning an argument suggested in the context of a debatable topic.

\begin{enumerate}
    \vspace{-0.2cm}\item What is the stance of the argument towards the topic? (supporting, contesting or neutral)
    \vspace{-0.2cm}\item For someone with this stance towards the topic, is this a high-quality argument to use? (yes or no)
\end{enumerate}
IMPORTANT! For the second question please answer "YES" only for high-quality arguments, and only for about half of the time.
\\ \\
Your answers will be monitored not only using test questions. If you are interested in participating in future similar tasks, please answer thoroughly.

\subsubsection{Argument Authoring Guidelines}
Below is an example of the argument authoring guidelines for German. The guidelines for the other languages were similar.
\\ \\
PLEASE READ: \\ \\
All your submitted arguments will be assessed for their quality. For each argument determined as a high-quality one, you will receive a bonus of up to 0.4\$.

\subsection*{Overview}
In the following task you are presented with a debatable topic, to which you should suggest high quality supporting/contesting arguments in \underline{German}.
\\ \\
A supporting/contesting argument will be considered as a high-quality one, if a person preparing a speech to support/contest the topic, respectively, will be likely to use this argument as is in her speech.
\\ \\
\textbf{Note:} Copying texts from the web or elsewhere is prohibited. The content you provide must be written by you in your own language.
\\ \\
\textbf{Requirements}
\begin{itemize}
\vspace{-0.2cm}\item The argument must be phrased in \underline{German}.
\vspace{-0.2cm}\item The argument must either clearly support or clearly contest the topic.
\vspace{-0.2cm}\item You should write a single argument in each text box.
\end{itemize}

\subsubsection{Evidence Assessment Guidelines}
Below is an example of the evidence assessment guidelines for German.
The guidelines for the other languages were similar.

\section*{General instructions}

In this task you are given a topic and evidence candidates for the topic. The candidates are in German. Consider each candidate independently. For each candidate please select Accept if and only if it satisfies ALL the following criteria:

\begin{itemize}
    \vspace{-0.2cm}\item The candidate clearly supports or clearly contests the given topic. A candidate that is neutral towards the topic should not be accepted.
    \vspace{-0.6cm}\item The candidate represents a coherent, stand-alone statement, that one can articulate (nearly) “as is” while discussing the topic, with no need to change/remove/add more than two words.
    \vspace{-0.2cm}\item The candidate represents valuable evidence to convince one to support or contest the topic. Namely, it is not merely a belief or merely a claim, rather it provides an indication whether a belief or a claim is true. A candidate which presents detailed information (typically quantitative) that clearly support or clearly contest the topic, should be accepted.
\end{itemize}
If you select Accept, you should further indicate whether the evidence supports the topic (Pro) or contests it (Con).
\\
\textbf{Note:} if you are unfamiliar with the topic, please briefly read about it in a relevant data source like Wikipedia.

\section*{Examples}

The following examples outline several candidates along with their suggested annotations; please read all these examples before performing the task.
\\ \\
\textbf{Topic:} We should ban the sale of violent video games to minors.
\\ \\
\underline{Example 1}
\\\vspace{-2mm}\\
\textit{The research clearly suggests that, among other risk factors, exposure to violent video games can lead to aggression and other potentially harmful effects.}
\\\vspace{-3mm}\\
\textbf{Annotation:} Accept – Pro
\\\vspace{-3mm}\\
\textbf{Note:} even though the text is not explicitly referring to the proposed ‘ban’ policy, it should still be accepted, since highlighting the negative aspects of violent video games can be used to support the suggested ban.
\\ \\
\underline{Example 2}
\\\vspace{-2mm}\\
\textit{A university of Oxford study negates the idea that violent video game content leads to violence.}
\\\vspace{-3mm}\\
\textbf{Annotation:} Accept - Con
\\\vspace{-3mm}\\
\textbf{Note:} here as well, even though the proposed ‘ban’ policy is not explicitly mentioned, the text should be accepted since clearly it can be used to contest the suggested ban.
\\ \\
\underline{Example 3}
\\\vspace{-2mm}\\
\textit{There is no reason to suppose that violent video games cause harm to children.}
\\\vspace{-3mm}\\
\textbf{Annotation:} Reject
\\\vspace{-3mm}\\
\textbf{Reason:} The candidate states a claim. It does not offer any additional information to convince the reader that this claim is true.
\\ \\
\underline{Example 4}
\\\vspace{-2mm}\\
\textit{The American Psychological Association argues that violent video-game play leads to increased moral sensitivity.}
\\\vspace{-3mm}\\
\textbf{Annotation:} Accept - Con
\\\vspace{-3mm}\\
\textbf{Reason:} The candidate states a claim, but the fact that it is raised by an authority figure (organization or human) turns it into a valuable evidence.
\\ \\
\underline{Example 5}
\\\vspace{-2mm}\\
\textit{Kennelly said there is no scientific evidence that violent video games cause “serious harm” in kids such as heightened aggression that would require protection of the law.}
\\\vspace{-3mm}\\
\textbf{Annotation:} Accept - Con
\\\vspace{-3mm}\\
\textbf{Note:}  If you are not certain whether the speaker is an authority figure or not, you should typically give him/her the benefit of the doubt and consider them as such (in this case the speaker is Matthew F. Kennelly, a United States District Judge). However,  if the candidate states a claim and the speaker is only mentioned by he/she/they you should reject it.
\\ \\
\underline{Example 6}
\\\vspace{-2mm}\\
\textit{The issue as “Psychological research confirms that violent video games can increase children's aggression.”}
\\\vspace{-3mm}\\
\textbf{Annotation:} Reject
\\\vspace{-3mm}\\
\textbf{Reason:} The candidate does not represent a coherent, stand-alone statement.
\\ \\
\underline{Example 7}
\\\vspace{-2mm}\\
\textit{Some studies have clearly demonstrated that video game violence is leading to serious aggressive behaviour in real life, although other studies have shown the opposite.}
\\\vspace{-3mm}\\
\textbf{Annotation:} Reject
\\\vspace{-3mm}\\
\textbf{Reason:} The pro/con stance of the candidate towards the topic is unclear, since the end of the text contradicts its beginning.
\\ \\
\underline{Example 8}
\\\vspace{-2mm}\\
\textit{The Entertainment Software Association reports that 17\% of violent video game players are boys under the age of eighteen.}
\\\vspace{-3mm}\\
\textbf{Annotation:} Reject
\\\vspace{-3mm}\\
\textbf{Reason:} The candidate states a fact with no clear pro/con stance towards the topic.
\\ \\
\underline{Example 9}
\\\vspace{-2mm}\\
\textit{Studies show that watching violent movies increases aggression amongst youth.}
\\\vspace{-3mm}\\
\textbf{Annotation:} Reject
\\\vspace{-3mm}\\
\textbf{Reason:} The candidate is not related to the topic as it discusses violent movies and not violent video games.
\\ \\
\underline{Example 10}
\\\vspace{-2mm}\\
\textit{Another 2001 meta-analyses and a more recent 2009 study focusing specifically on serious aggressive behavior concluded that video game violence is not related to serious aggressive behavior in real life.}
\\\vspace{-3mm}\\
\textbf{Annotation:} Accept - Con
\\\vspace{-3mm}\\
\textbf{Note:}  Even though the candidate’s first word better be omitted to make it a stand-alone statement, this is a minor change which is acceptable.
\\ \\
\underline{Example 11}
\\\vspace{-2mm}\\
\textit{Limiting the sale of violent video games will cause 15,000 people to lose their jobs.}
\\\vspace{-3mm}\\
\textbf{Annotation:} Accept - Con
\\\vspace{-3mm}\\
\textbf{Note:}  The candidate presents a specific numeric piece of information that clearly contest the topic. You are not expected to fact check the provided piece of information, don’t reject such a candidate just because you are not sure that the provided piece of information is true.

\section{Conclusions}
\label{sec:conclusions}

We have examined the translate-train paradigm for three multilingual argument mining tasks: stance classification, evidence detection, and argument quality, evaluating a wide range of multilingual models on machine-translated and human-authored data.
These tasks differ in their complexity, as reflected in the agreement of annotators on the correct label, the extent to which this label is preserved across translation, and, ultimately, in the accuracy of the models.

Accordingly, our results show that the translate-train approach is well suited for stance classification, as performance improves when
augmenting the English training data with automatic translations from other languages.
For evidence detection, adding data from the target language or related languages improves performance, yet adding more languages is not helpful.

For both tasks on the evidence data, adding more English
training data improves performance.
In these cases, augmenting the large English training set with data of other languages only leads to a marginal gain
for stance classification, and even degrades performance for evidence detection.

In contrast with the above two tasks, the results on argument quality show that training only on English is at least as good, if not better, than any translate-train model.
This is reflected by the clearly opposite trends observed in \figureRef{fig_sbc_arg_quality_hg} vs. those observed in \figureRef{fig_sbc_stance_hg}.

Taken together, our results confirm the validity of the common translate-train paradigm for argument mining tasks such as stance and evidence detection, for which the label is relatively well preserved under translation. However, for the more subtle argument-quality task, where the label -- as might be expected -- is far less preserved, a new approach might be needed.
Future work might wish to explore how translation can preserve not only the semantics of texts, but also finer aspects that contribute to its quality.

\section{Data Sets}
\label{sec:datasets}

\subsection{Translated Data}
\label{par:argmining_data}
\paragraph{English Datasets}
The sources for our translated training data and the ”pseudo test” sets are two existing argument mining datasets in English, collected by our colleagues as part of our work on Project Debater.\footnote{Project Debater is the first AI system that can debate humans on complex topics: \url{https://www.research.ibm.com/artificial-intelligence/project-debater/}}
One is a corpus of 30,497 arguments on $71$ controversial topics, annotated for their stance towards the topic and for their quality \cite{gretz2020quality}. This dataset (referred herein as \argumentDatasetName) is used for the stance classification and argument quality tasks.

The second dataset is a corpus of 35,211 sentences from Wikipedia on 321 controversial topics, annotated for their stance towards the topic and the extent to which they can serve as an evidence for the topic \cite{working-solution}.
This dataset (referred as \evidenceDatasetName) is used for the stance classification and evidence detection tasks.
Example \ref{ex:data} shows an argument and an evidence for one topic.

\input{data_examples}

A third dataset was used to augment the training data for evidence detection and stance classification -- the so-called VLD dataset of \citet{working-solution}, which includes around 200k sentences from newspaper articles,
pertaining to 337 topics.\footnote{The stance labels of the \evidenceDatasetName and VLD datasets are not part of their official releases. They are included in our release.}

\paragraph{Data Selection}
\label{subsec:data_selection}
The \argumentDatasetName dataset was filtered for stance classification by selecting arguments with a clear stance (confidence $> 0.75$) for training and evaluation.
For argument quality, arguments with a clear agreement on their quality were selected -- quality score above $0.9$ or below $0.4$.

A positive label for evidence detection was assigned for evidence from \evidenceDatasetName with a score above $0.7$, and a negative label to those with score below $0.3$ (those with in-between scores were not used).
For stance classification on the evidence data, all sentences with a non-neutral stance were selected, since the \evidenceDatasetName dataset does not provide a confidence score for the stance label.

The VLD corpus was also filtered, taking sentences with evidence score above $0.95$ or below $0.05$.
This yielded a total of 52,037 sentences for training, of which
19,406 have a positive label.

\paragraph{Translation}
\label{par:mt_data}
We used the Watson Language Translator\footnote{\url{www.ibm.com/watson/services/language-translator/}} to translate the selected English data into $5$ languages: German (DE), Dutch (NL), Spanish (ES), French (FR), and Italian (IT).
The translation
is a one-time process, which can be applied to any target language (TL) that the MT engine in use supports.
The labels for the MT data were projected from the English data.

Following the data splits provided in the official release of each English corpus (into a training, development, and test sets), the translations of the training data were used for fine-tuning mBERT, and the translations of the test data were used for evaluation.
The translations of the test data of \argumentDatasetName and \evidenceDatasetName\xspace into the $5$ non-English languages -- the pseudo test sets -- are herein referred to as \pseudoTestArgDatasetName and \pseudoTestEviDatasetName.
The statistics of the translated English data are summarized in \tableRef{tab:selected_english_data_stats}.

\subsection{Human-Generated Data}
\label{par:hg_data}

\input{tables/selected_english_data_stats}

Arguments written in the TL provide a more realistic evaluation set than translated texts, specifically for tasks where labels are not well-preserved across automatic translation.
Therefore, we created a new multilingual evaluation set by collecting arguments in all $5$ languages (ES, FR, IT, DE, and NL)
for all the $15$ topics of the  \argumentDatasetName test set, using the \textit{\figureEightName}\footnote{\url{www.appen.com}} crowdsourcing platform.
The human-authored evaluation dataset is herein referred to as \humanGeneratedArgDatasetName.

\paragraph{Annotation Setup}
Initially, crowd contributors wrote up to two pairs of arguments per topic, with one argument supporting the topic and another contesting it in each pair.
Next, the arguments were assessed for their stance and quality by $10$ annotators (per-language).
Given an argument, they were asked to determine the stance of the argument towards the topic and to assess whether it is of high quality.
The full argument annotation guidelines are included in the \supp, and \tableRef{tab:human_generated_labeling_stats} details the number of arguments collected and labeled per language.
To set a common standard, annotators were instructed to mark about half of the arguments they labeled as high quality.
Annotation quality was controlled by integrating test questions (TQs) with an a-priori known answer in between the regular questions, measuring the per-annotator accuracy on these questions, and excluding underperformers.

A per-annotator average agreement score was computed by considering all peers sharing at least $50$ common answers, calculating pairwise Cohen's Kappa \cite{cohenKappa} with each of them, and averaging.
Those not having at least $5$ peers meeting this criterion
were excluded and their answers were discarded.
Averaging the annotator agreements yields the average inter-annotator agreement (\agreementName) of each question.

To derive a label (or score) for each question we use the WA-score of \citet{gretz2020quality}.
Roughly, answers are aggregated with a weight proportional to the agreement score for the annotators who chose them.
At least $5$ answers were required for a question to be considered as labeled.

Scaling the annotation from English to new languages required some adjustments, such as restricting participation to countries in which the TL is commonly spoken, and the use of TQs for the argument quality question. Further details are provided in the \supp.

\input{tables/human_generated_corpus}

\paragraph{Results}
\tableRef{tab:human_generated_labeling_stats} presents the \agreementName for all TLs and each task for the human-generated dataset.
For stance, the agreement is comparable to previously reported values for English ($0.69$ by \citet{toledo-etal-2019-quality} and $0.83$ for \argumentDatasetName).
For quality, the agreement is significantly better than previously reported on \argumentDatasetName ($0.12$ by \citet{gretz2020quality}),  presumably due to the use of TQs in this task, which were not included before.
The annotation in each of the non-English languages involved a distinct group of annotators, producing varying annotation quality among languages which is reflected in their \agreementName values.

The results also include the percentage of arguments labeled as supporting arguments, computed separately for each annotator and averaged over all annotators.
All values are close to $0.5$, confirming that the collected arguments are balanced for stance, as instructed.
Similarly, the results show the  percentage of arguments labeled as high quality, averaged over all annotators, confirming that annotators mostly followed the instruction to label about half of the arguments as high quality.

The same confidence filtering thresholds described in \sectionRef{subsec:data_selection} were applied to
the data of \humanGeneratedArgDatasetName.
The statistics of the arguments selected for evaluation are shown in \tableRef{tab:human_generated_labeling_stats} (right).

\section{Experimental Setup}
\label{empirical_evaluation}

Our experiments are aimed at providing a comparative analysis of the translate-train approach when trained on different subsets of languages, and identifying when that approach is beneficial on the three argumentation tasks.
We begin by describing the setup used in all experiments.

\paragraph{Training Configuration}
We used the BERT-Base multilingual cased model configuration (12-layer, 768-hidden, 12-heads, total of 110M parameters) with a sentence-topic pair input.
Training was performed on one GPU.

The parameters configuration of the binary classification tasks, namely, stance classification and evidence detection, was:
maximum sequence length of $128$, batch size of $32$, dropout rate of $0.1$ and learning rate of 5e-5. Each model was fine-tuned over $10$ epochs, using a cross-entropy loss function.
The regression model for argument quality prediction, similar to the one used by \citet{gretz2020quality}, used
a maximum sequence length of $100$, a batch size of $32$, a dropout rate of $0.1$ and a learning rate of 2e-5.
Each model was fine-tuned over $3$ epochs, using a mean-squared-error loss function.
In all cases, the model from the last epoch was selected for evaluation.

\paragraph{Translate-Train Models}
\label{par:lang_groups}
For each task, mBERT was trained using data translated into one of the target languages (ES, FR, IT, DE and NL).
These per-language models, denoted herein as \emph{TL}, are the simplest application of the translate-train approach.
Two more models were trained for the language families that are represented in the above languages together with English:
\emph{RM} -- for the Romance languages (ES, FR, IT), and \emph{WG} -- for the West-Germanic languages (EN, DE, NL).
Each language family model was trained using the data of the languages in that family.
Lastly, a model was trained on data from all 6 languages (denoted \emph{6L}).
To summarize, our evaluation includes $4$ models based on the translate-train approach (\emph{TL}, \emph{6L}, \emph{RM} and \emph{WG}) for each task and TL.

\paragraph{Baselines}
Another mBERT model, denoted \emph{EN}, was trained on the source English data.
Using this model, the results of two baselines are reported:
(i) zero-shot (denoted \emph{ZS}) -- which passes an input text in a non-English language to the \emph{EN} model, utilizing the cross-lingual transfer capabilities of mBERT; (ii) translate-test (denoted \emph{TT}) -- in which an input text is machine translated into English, and that translation is provided as input to the \emph{EN} model.
The results on English data
are also reported as a performance benchmark for other languages.
Obviously, for English, the results of the \emph{TL} model and the \emph{ZS} and \emph{TT} baselines are identical.

\input{figures/arguments_stance_results}

\paragraph{Related and Distant Languages}
For each TL, we define two types of models. The \emph{RL} model is the one trained on \emph{related} languages from the same family as the TL, and  \emph{DL} is the model that is trained on languages that are more \emph{distant} from the TL.
In other words, given a TL, the \emph{RL} model refers to the language family that includes the TL, and the \emph{DL} model refers to the other family that does not include the TL.
For example, in the case of the TL being German, \emph{TL} denotes the model trained only on translated data in German;
\emph{RL} -- the \emph{WG} model trained on the 3 West-Germanic languages (including German);
\emph{DL} -- the \emph{RM} model trained on the 3 Romance languages.

\paragraph{Evaluation Metrics}
The reported metrics are macro-F1 for the classification tasks (stance classification and evidence detection), and Pearson correlation for the regression setting of argument quality.

\section{Introduction}
\label{sec:intro}

Argument mining has received much attention in recent years, with research mainly focused on English and, to some extent, German texts. Recent advancements in Natural Language Understanding suggest that in order to train appropriate models for argument mining tasks in other languages, we do not need to manually label text in these languages, but rather employ transfer learning from the English-based models \cite{eger-etal-2018-cross}.

In this work we examine three argument mining tasks: (1) \emph{stance classification}: given a topic and
an argument that supports or contests the topic, determine the argument's stance towards the topic; (2) \emph{evidence detection}: given a topic and a sentence, determine if the sentence is an evidence relevant to the topic; (3) \emph{argument quality}: given a topic and a relevant argument, rate the argument so that higher-quality arguments are assigned a higher score.

To facilitate transfer learning from English datasets for these tasks, we employ Multilingual BERT (mBERT) released by \citet{devlin2019bert},
a pre-trained language model that supports 104 languages, and use it mainly in a translate-train approach.
Namely, the English dataset is automatically translated into the desired language(s) using machine translation (MT); an augmented dataset composed of the original English text and all the translated copies is created; the mBERT model is fine-tuned on a subset of the dataset; and the resultant model is then used to solve the relevant downstream task in the desired language.
Previous works have suggested that translating the original dataset to as large a number of languages as possible is beneficial \cite{liang2020xglue}.
In this work, we show a more nuanced picture, where often selecting a subset of related languages is preferable.

In addition, we also examine the translate-test approach, in which one creates a classification model only with English data.
At prediction time, the non-English text is automatically translated into English, and then analyzed by the model.
This approach is less appealing since the initial translation step increases prediction run-time, and on our data also tends to perform worse.

We examine two text sources for performance evaluation on non-English texts. The first one is a "pseudo test" set -- an automatic translation of an English evaluation set for a task.
While such texts can be easily generated, it is not clear how well they represent "real" texts, authored by humans. Hence, we also examine human-authored texts, in several non-English languages, collected via crowdsourcing specifically for this work.
Both datasets are released as part of this work.\datasetURL

When translating the evaluation set, either automatically or by a human translator, one would like to assume that the initial label of the English text is maintained after translation.
While this is often the case, we show that this assumption becomes more dubious as the argument mining task becomes more complex and subjective, as well as when the original labels are not clearly agreed upon.

In summary, the main contributions of this paper are:
(1) a comparative analysis of the translate-train approach on three central argument mining tasks using different subsets of languages,
showing that training on more data helps, but that, in some cases, training on related languages is sufficient;
(2) multilingual benchmark datasets for the three tasks;
(3) an analysis of the three tasks, showing how well labels are preserved across translation, and the impact that has on the success of the translate-train approach.

\section*{Acknowledgments}
Special thanks go to Michal Jacovi, Christopher Sciacca,
and Mohamed Nasr for coordinating the manual translation effort.
We thank the teams who performed the translation, and Benjamin Sznajder for helping with the analysis.
We also thank the anonymous reviewers for their valuable comments.

\section{Related Work}
\label{sec:related_work}
Argument mining has been expanding from identifying argumentative passages to a variety of Natural Language Understanding (NLU) tasks (\citet{stede2018argumentation,lawrence2020argument}).
In this work, we explore three argumentation tasks in multilingual settings.

Stance detection (or classification) is often contrasted with sentiment analysis, in that the task is not simply to classify the sentiment of a text, but rather its stance w.r.t. some given target. Early work on this task includes \citet{thomas2006get, lin2006side}, while in the context of argument mining, the task has probably been introduced by \citet{sobhani2015argumentation}.
As with many other NLU tasks, earlier works developed classifiers based on various features (e.g., \citet{barhaim2017stance}), while more modern approaches rely on deep learning. See \citet{schiller2020stance} for a recent benchmarking report on such methods.

Research on stance detection in a multilingual setting is rather recent. \citet{zotova2020multilingual} explore stance detection in Twitter for Catalan and Spanish; \citet{lai2020multilingual} do this for political debates in social media in these two languages as well as French and Italian; \citet{vamvas2020x} analyze the stance of comments in the context of the Switzerland election in German, French and Italian.
Stance detection is reminiscent of the Natural Language Inference (NLI) problem, where one is given two sentences, and the objective is to determine whether one entails the other, contradicts it or is neutral. This task had been researched extensively, with \citet{conneau2018xnli} providing a 15-language benchmark for the multilingual setting. Another earlier work on a related task is addressing the support/attack relation prediction of two arguments in Italian \cite{basile-2016-arg-italian}.

Evidence detection is the task of determining, given some text and a topic, whether the text can serve as evidence in the context of the topic \cite{rinott-etal-2015-show}.
We follow \citet{working-solution}, in defining \emph{evidence} as \textit{a single sentence that clearly supports or contests the
topic, yet is not merely a belief or a claim.
Rather, it provides an indication for whether a relevant belief or a claim is true}, and, since we use their datasets, we restrict our analysis to evidence of type Study and Expert.
In a multilingual setting, a similar task, that of premise detection, was considered in \citet{eger-etal-2018-cross} for German, French, Spanish, and Chinese; in \citet{fishcheva2019cross} for Russian; in \citet{eger2018pd3} for French and German; and in \citet{aker2017projection} for Chinese.

Argument quality prediction is the task of evaluating the quality of an argument, either on an objective scale - the input is an argument and the output is a quality score; or in a relative manner - the input is a pair of arguments and the output is which of them is of higher quality.
While there are many, arguably independent, dimensions for quality \cite{wachsmuth2017computational}, it seems that people - and, consequently, algorithms - can usually perform this task in a consistent manner \cite{habernal2016argument, wachsmuth-etal-2017-argumentation, toledo-etal-2019-quality,gretz2020quality}.
To the best of our knowledge, this task was not previously considered in a multilingual setting.

In contrast with previous multilingual research on argument mining, in this work we address three different problems, of varying complexity, over a relatively large number of languages.
This allows us to draw more holistic conclusions on the efficacy -- and pitfalls -- of transfer learning in the argument mining domain.
It is interesting to compare these conclusions with other wide-scope multilingual NLU research, such as the XTREME \cite{hu2020xtreme} and XGLUE \cite{liang2020xglue}.

\section{Results}
\label{results}

The results below for arguments (for stance classification on that data and argument quality) are averages over $5$ evaluation runs of randomly initialized models that were trained in the same manner.
For evidence sentences (stance classification on that data and evidence detection), the results are from a single evaluation run.

\subsection{Stance Classification}
\label{subsec:stance_classification_results}

\paragraph{Arguments}

\figureRef{fig_sbc_stance_hg} shows the evaluation results on the human-generated arguments of \humanGeneratedArgDatasetName.
For the non-English languages, the performance over the \emph{ZS} baseline improves when adding translated data, even when that data is from distant languages (\emph{DL}).
The other baseline \emph{TT} is better, yet the best performance is attained by the \emph{6L}
models
-- significantly so for 3 of the 5 languages (ES, FR and DE).
Notably, ordering the translate-train models by their performance yields the same order for all languages: \emph{DL} is always the worst, followed by \emph{TL}, \emph{RL} and the best performing model \emph{6L}.

Repeating the same experiments on the pseudo-test data of \pseudoTestArgDatasetName resulted in similar trends, depicted in \figureRef{fig_sbc_stance_mt}.
Further augmentation of the training data with translations to more languages beyond the languages included in the training of the \emph{6L} models (e.g. with 9 or 17 languages) did not significantly improve performance on these languages.
These results are detailed in \tableRef{tab_sbc_stance_mt} within the \supp.

\input{figures/evidence_results}

\paragraph{Evidence} To explore whether the observed trends are data-specific, we repeated the evaluation of the stance classification task with
the \pseudoTestEviDatasetName dataset of evidence sentences from Wikipedia.
Training was performed on the training set of that corpus (called Wikipedia models).
The results on its pseudo-test set are depicted in \figureRef{fig:evidence_stance_wiki}.
For the
non-English
languages, the best performing models are \emph{6L} and \emph{RL}, consistent with our findings for arguments (\figureRef{fig:argument_stance_results}).

The VLD evidence curpus allows further exploration of the stance classification task within the evidence domain.
We trained models on a larger dataset of translated evidence combining the Wikipedia data and selected data from the VLD corpus (called Extended models).
\figureRef{fig:evidence_stance_extended} shows the results obtained using
these models.
Overall, the performance of the Extended models is significantly better than the performance of the Wikipedia models, in almost all cases, and the \emph{TL} models become competitive even with the \emph{6L} models.

\paragraph{Performance on English}
In comparison with the \emph{ZS} baseline (trained only on English), adding translated training data improves performance on English (leftmost bars in Figures \Ref{fig_sbc_stance_hg} and \ref{fig:evidence_stance_wiki}), for both domains.
For the evidence data, even translations to distant languages (\emph{DL}) help the Wikipedia model, yet when a lot of training data is available in English (leftmost bars in \ref{fig:evidence_stance_extended}), there is no significant gain from adding translations to the training set.

\paragraph{Summary}
Overall, the best performing models for the stance classification task are the \emph{RL} and \emph{6L} models, in both domains.
Our finding that the \emph{6L} models outperform the \emph{TL} models is consistent with previous results on the XNLI task \citep{hu2020xtreme}.
Interestingly, translated data can be used to improve performance on English as well.

The \emph{ZS} and \emph{TT} baselines are almost always outperformed by the best translate-train model.
However, when a large-scale English corpus is available (\figureRef{fig:evidence_stance_extended}), the \emph{TT} baseline becomes comparable to the best translate-train models.

\subsection{Evidence Detection}
\label{subsec:evidence_detection_results}

To examine whether the above observations are task-specific, we move on to the task of evidence detection.
The results for that task on the
\pseudoTestEviDatasetName pseudo-test are
depicted in \figureRef{fig:evidence_detection_wikipedia} (Wikipedia models) and in \figureRef{fig:evidence_detection_extended} (Extended models).

In contrast with the stance results, where in most cases the \emph{6L} models were best, for evidence detection performance may degrade when adding languages.
The best performing translate-train models are either the \emph{TL} or \emph{RL} models, in all cases.

As in the stance classification results for this corpus, the additional training data used in the Extended models improves performance.
In addition, the English benchmark results for the Wikipedia models (leftmost bars in  \figureRef{fig:evidence_detection_wikipedia}) can improve by adding languages, or by adding English data (\emph{ZS} bar for EN in \figureRef{fig:evidence_detection_extended}), but there is no significant gain from doing both
(leftmost bars in \figureRef{fig:evidence_detection_extended}).

\subsection{Argument Quality Prediction}
\label{subsec:argument_quality_results}

Moving to our last task, \figureRef{fig_sbc_arg_quality_hg} shows the Pearson correlation results on the human-generated arguments, between the predicted
quality score and the labeled argument quality score.
In contrast with the stance results, adding data from related languages (the \emph{RL} bars) does not help, and training on the English dataset (the \emph{ZS} bars) is sufficient to obtain a competitive model.\footnote{The performance of the \emph{ZS} model on English is $0.61$ -- comparable to its performance for FR and DE.}
We suspect that the reason for this is that this task is more complex and nuanced than the previous two.

\section{Analysis}

The performance of a translate-train model may be affected, among other factors, by the translation quality, the extent in which a task-specific label is preserved across that translation, and, for our data, the discussed topic.
These are analyzed below.

\subsection{Translation Quality Assessment}
We assessed our machine translation quality by computing the BLEU score \cite{papineni2002bleu} between the English arguments from the test set of  \argumentDatasetName
and the same arguments after translation to a TL and back to English.
For all languages, these scores are above $0.5$ (see \tableRef{tab:mt_bleu}), suggesting the translations are of high quality.

\input{figures/argument_quality}

\subsection{Translated Label Assessment}
\label{subsec:translated_label_assessment}
An important prerequisite for training and evaluating models on automatically translated texts is that the labels of the original texts are preserved under translation, which depends on the specific task at hand.
Example \ref{ex:back-translation} shows one argument and its translation to Spanish and back to English. The translation preserves the original stance,
but the argument quality is degraded.
Hence, we annotated a sample of the translated texts to assess how often this  happens in each task.
The annotation focuses on one Romance and one West-Germanic language -- Italian and German.

\input{tables/mt_bleu}

\input{quality_example}

\paragraph{Annotation Setup}
$14$ arguments were randomly sampled from each
topic of the \argumentDatasetName test set, yielding $210$ arguments per language.
Similarly, two sentences were sampled from each
topic in the \evidenceDatasetName test set, producing $200$ sentences per language.
All texts were machine translated and human translated by native speakers of each TL.
Both translations of each argument were labeled for their stance and quality, as in \sectionRef{par:hg_data}.
Similarly, the potential evidence sentences were annotated for whether they are valid evidence, and those which are so were also annotated with their stance towards the topic, as in \citet{working-solution}.
In this annotation, TQs were formed from translated texts, with the correct answer taken from the English labels.
The full evidence annotation guidelines are included in the \supp.

\paragraph{Results}
\tableRef{tab:translated_arguments_assessment} shows the assessment results for all tasks and the two languages.
The obtained \agreementName is on par with previously reported values for these tasks (as detailed in \sectionRef{par:hg_data}), though somewhat lower for evidence detection.
The table further shows Pearson correlation between the original English WA-scores, and the WA-scores of the translated texts.
For evidence detection and argument quality, this computation was performed on texts matching the criteria defined in \sectionRef{subsec:data_selection}.
The correlation for evidence stance classification was computed on sentences with at least $6$ stance labels on their translated version.

The results show that for both datasets,
stance is well preserved after translation.
For evidence detection, the correlation is lower, yet the difference between \machineTranslated and \humanTranslated is small,
suggesting the change in the labels is not due to the automatic translation.
Thus, the use of translated texts in
these tasks is acceptable, for both training and evaluation.

For argument quality, the correlation is considerably lower, and there is a significant difference between \machineTranslated  and \humanTranslated in \italianShort, as may be expected for such a nuanced task.
This could be the reason that the translate-train models do not improve performance for this task -- since the quality label is not maintained when an argument is translated, projecting these labels into translated texts introduces significant noise into such training data.

\input{tables/translated_arguments_assessment}

\subsection{Per-Topic Analysis}

In both of our data domains, arguments and evidence, the texts are relevant to a specific topic, and the obtained performance may depend on that topic in various ways.
Focusing on stance classification, we measured the per-topic performance on the human-generated arguments of \humanGeneratedArgDatasetName.
\figureRef{fig_sbc_stance_hg_avg5tls_per_topic} shows these results averaged over the $5$ non-English
languages, for the \emph{TL} and \emph{6L} translate-train models, and the \emph{ZS} and \emph{TT} baselines.
The topics are ordered by their performance on English.\footnote{\tableRef{tab:sbc_test_motions} in the \supp lists the topic for each topic ID.}

The results demonstrate the performance variability among the different topics.
For some, the average performance on the
non-English languages is close to their performance on English (e.g. topics 9 or 10), yet for others it is far from it (e.g. topic 5).
The performance of the \emph{ZS} baseline is low
for topics 2 through 8, from which $5$ are discussing imposing a ban.
This implies that the stance towards the discussed topic, or the  "action" within the topic (e.g. ban, legalize, etc.) may be an important factor.

We further manually analyzed the results on Topic 10, a low-performing
outlier in French for the \emph{ZS} baseline (see \figureRef{fig_sbc_stance_hg_zs_per_topic} in the \supp).
A native speaker examined $3$ batches of $20$ arguments each, containing:
1) prediction errors from that topic;
2) randomly sampled correct predictions from the same topic;
3) all 4 prediction errors and 16 randomly sampled correct predictions from
the topic with the highest performance (Topic 1).
Within the first batch, $40\%$ of the samples were incoherent or syntactically wrong arguments, compared to only $20\%$ in each of the other two batches.

\begin{figure}[t]
\includegraphics[width=\linewidth,
trim={1.0cm, 0.2cm, 1.4cm, 0.8cm},
clip]{figures/fig_sbc_stance_hg_per_topic.pdf}
\caption{Per-topic stance classification average macro-F1 results on
\humanGeneratedArgDatasetName, averaged over the $5$ non-English languages.
The topics are ordered by their performance on English with the \emph{EN} model (dashed black line).
}
\label{fig_sbc_stance_hg_avg5tls_per_topic}
\end{figure}