\section{Introduction}

One of the desirable properties of contextualized models, such as BERT \cite{BERT} and its derivatives, lies in their ability to associate dynamic representations to words, i.e., embeddings that can change depending on the context.
This provides the basis for the model to distinguish different meanings (senses) of words without the need to resort to an explicit sense disambiguation step.
The conventional evaluation framework for this property has been Word Sense Disambiguation \cite[WSD]{navigli:09}.
However, evaluation benchmarks for WSD are usually tied to  external sense inventories (often WordNet \cite{Fellbaum:98}), making it extremely difficult to evaluate systems that do not explicitly model sense distinctions in the inventory, effectively restricting the benchmark to inventory-based sense representation techniques and WSD systems.
This prevents a direct evaluation of lexical semantic capacity for a wide range of inventory-free models, such as the dominating language model-based contextualized representations.

\newcite{pilehvar2019-wic} addressed this dependence on sense inventories by reformulating the WSD task as a simple binary classification problem: given a target word $w$ in two different contexts, $c_1$ and $c_2$, the task is to identify if the same meaning (sense) of $w$ was intended in both $c_1$ and $c_2$, or not.
The task was framed as a dataset, called Word-in-Context (WiC), which is also a part of the widely-used SuperGLUE benchmark \cite{wang2019superglue}.
Despite allowing a significantly wider range of models for direct WSD evaluations, WiC is limited to the English language only, preventing the evaluation of models in other languages and in cross-lingual settings.

In this paper, we present a new evaluation benchmark, called XL-WiC
, that extends the WiC dataset to 12 new languages from different families and with different degrees of resource availability: Bulgarian (BG), Chinese (ZH), Croatian (HR), Danish (DA), Dutch (NL), Estonian (ET), Farsi (FA), French (FR), German (DE), Italian (IT), Japanese (JA) and Korean (KO)\xspace.
With over 80K instances, our benchmark can serve as a reliable evaluation framework for contextualized models in a wide range of heterogeneous languages.
XL-WiC can also serve as a suitable testbed for cross-lingual experimentation in settings such as zero-shot or few-shot transfer across languages.
As an additional contribution, we tested several pretrained multilingual models on XL-WiC, showing that they are generally effective in transferring sense distinction knowledge from English to other languages in the zero-shot setting.
However, with more training data at hand for target languages, monolingual approaches gain ground, outperforming their multilingual counterparts by a large margin.

\section{Related Work}

XL-WiC is a benchmark for inventory-independent evaluation of WSD models (Section \ref{rel:WSD}), while the multilingual nature of the dataset makes it an interesting resource for experimenting with cross-lingual transfer (Section \ref{rel:cross}).

\subsection{Word Sense Disambiguation}
\label{rel:WSD}

The ability to identify the intended sense of a polysemous word in a given context is one of the fundamental problems in lexical semantics. It is usually addressed with two different kinds of approaches relying on either sense-annotated corpora \cite{bevilacquaandnavigli-2020-ewiser,scarlinietal:20,terraluke-2020-bem} or knowledge bases \cite{Moroetal:14tacl,agirreetal:14,scozzafava-etal-2020-syntagrank}. Both are usually evaluated on dedicated benchmarks, including at least five WSD tasks in Senseval and SemEval series, from 2001 \cite{edmonds-cotton-2001-senseval} to 2015 \cite{moro-navigli-2015-semeval} that are included in the \citet{raganatoetal:17}'s test suite.
All these tasks are framed as classification problems, where disambiguation of a word is defined as selecting one of the predefined senses of the word listed by a sense inventory.
This brings about different limitations such as restricting senses only to those defined by the inventory,
or forcing the WSD system to explicitly model sense distinctions at the granularity level defined by the inventory

Stanford Contextual Word Similarity \cite{huang-etal-2012-improving} is one of the first datasets that focuses on ambiguity but outside the boundaries of sense inventories, and as a similarity measurement between two words in their contexts.
\newcite{pilehvar2019-wic} highlighted some of the limitations of the dataset that prevent a reliable evaluation, and proposed the Word-in-Context (WiC) dataset.
WiC is the closest dataset to ours, which provides around 10K instances (1400 instances for 1184 unique target nouns and verbs in the test set), but for the English language only.

\subsection{Cross-lingual NLP}
\label{rel:cross}

A prerequisite for research on a language is the availability of relevant evaluation benchmarks.
Given its importance, construction of multilingual datasets has always been considered as a key contribution in NLP research and numerous benchmarks exist for a wide range of tasks, such as semantic parsing \cite{hershcovich-etal-2019-semeval}, word similarity
\cite{camacho-collados-etal-2017-semeval,barzegar-etal-2018-semr}, sentence similarity \cite{cer-etal-2017-semeval},
or WSD \cite{navigli-etal-2013-semeval,moro2015semeval}.
A more recent example is XTREME \cite{hu2020xtreme}, a benchmark that covers around 40 languages in nine syntactic and semantic tasks.

On the other hand, pre-trained language models have recently proven very effective in transferring knowledge in cross-lingual NLP tasks \cite{BERT,XLMR}.
This has further magnified the requirement for rigorous multilingual benchmarks that can be used as basis for this direction of research \cite{artetxe2020call}.

\begin{table*}[ht]
    \centering
    \resizebox{\linewidth}{!}{
    \begin{tabular}{l lll l}
    \toprule
    \bf Lang. &
    \bf ~~Target Word & \multicolumn{1}{l}{\bf ~~Sentence 1} & \multicolumn{1}{l}{\bf ~~Sentence 2} & \bf Label\\

     \midrule
    EN &
    ~~Beat&
    ~~We beat the competition.  &
    ~~Agassi beat Becker in the tennis championship. & True\\
    \midrule
    DA &
    ~~Tro&
    ~~Jeg tror på det, min mor fortalte.&
    ~~Maria troede ikke sine egne øjne.&
    True
    \\
    ET &
    ~~Ruum &
    ~~Ühel hetkel olin väljaspool aega ja ruumi.&
    ~~Ümberringi oli lõputu tühi ruum. &
    True \\

    FR &
    ~~Causticité&
    ~~Sa causticité lui a fait bien des ennemis. &
    ~~La causticité des acides. &
    False \\
    KO &
    \begin{CJK}{UTF8}{}
    \CJKfamily{mj}
    틀림
    \end{CJK} &
    \begin{CJK}{UTF8}{}
    \CJKfamily{mj}
    틀림이 있는지 없는지 세어 보시오. \end{CJK} &
    \begin{CJK}{UTF8}{}
    \CJKfamily{mj}
     그 아이 하는 짓에 틀림이 있다면 모두 이 어미 죄이지요.
    \end{CJK} &
    False\\
    ZH &
    \begin{CJK}{UTF8}{}
    \CJKfamily{mj}
    發
    \end{CJK} &
    \begin{CJK}{UTF8}{}
    \CJKfamily{mj}
    建築師希望發大火燒掉城市的三分之一。
    \end{CJK} &
    \begin{CJK}{UTF8}{}
    \CJKfamily{mj}
    如果南美洲氣壓偏低，則印度可能發乾旱
    \end{CJK} &
    True\\
    \raisebox{0.6\height}{FA} &
    \raisebox{0.3\height}{~~\includegraphics[scale=0.4]{figures/fa.PNG}} &
    \includegraphics[scale=0.4]{figures/fa1.PNG} &
    \includegraphics[scale=0.4]{figures/fa2.PNG} &
    \raisebox{.6\height}{False}
    \\
    \bottomrule
    \end{tabular}
    }
    \caption{Sample instances from XL-WiC for different languages.
    }
    \label{tab:xlwic_examples}
\end{table*}

\section{XL-WiC: The Benchmark}

In this section, we describe the procedure we followed to construct the XL-WiC\xspace benchmark. Our framework is based on the original WiC dataset, which we extend to multiple languages

\subsection{English WiC}

Each instance of the original WiC dataset \cite{pilehvar2019-wic} is composed of a target word (e.g., \textit{justify}) and two sentences where the target word occurs (e.g., ``{\it Justify} the margins'' and ``The end {\it justifies} the means'').
The task is a binary classification: to decide whether the same sense of the target word (\textit{justify}) was intended in the two contexts or not.
The dataset was built using example sentences from resources such as Wiktionary, WordNet \cite{miller1995wordnet} and VerbNet \cite{VerbNet}.

\subsection{XL-WiC}
We followed \newcite{pilehvar2019-wic} and constructed XL-WiC based on example usages of words in sense inventories.
Example usages are curated in a way to be self contained and clearly distinguishable across different senses of a word; hence, they provide a reliable basis for the binary classification task.
Specifically, for a word $w$ and for all its senses $\{s_1^w,...,s_n^w\}$, we extract from the inventory all the example usages.
We then pair those examples that correspond to the same sense $s_i^w$ to form a positive instance (True label) while examples from different senses (i.e., $s_i^w$ and $s_j^w$ where $i \ne j$) are paired as a negative instance (False label).

We leveraged two main sense inventories for this extension: Multilingual WordNet (Section \ref{multilingual-wordnet}) and Wiktionary (Section \ref{wiktionary}).

\begin{table*}[!ht]
    \centering
    \setlength{\tabcolsep}{5.0pt}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{llrrrrrrrrrrrrrrr}
    \toprule
    \multirow{2}{*}{\bf Split} &
    \multirow{2}{*}{\bf Stat} &

    \multicolumn{1}{c}{\bf WiC} & \multicolumn{9}{c}{\bf Multilingual WordNet} &
    &

    \multicolumn{3}{c}{\bf Wiktionary} \\

    \cmidrule(lr){3-3}
    \cmidrule(lr){4-13}
    \cmidrule(lr){14-16}

    &
    &
    \multicolumn{1}{c}{\bf \lang{en}} &
    \multicolumn{1}{c}{\bf \lang{bg}} &
    \multicolumn{1}{c}{\bf \lang{da}} &
    \multicolumn{1}{c}{\bf \lang{et}} &
    \multicolumn{1}{c}{\bf \lang{fa}} &
    \multicolumn{1}{c}{\bf \lang{hr}} &

    \multicolumn{1}{c}{\bf \lang{ja}} &
    \multicolumn{1}{c}{\bf \lang{ko}} &
    \multicolumn{1}{c}{\bf \lang{nl}} &
    \multicolumn{1}{c}{\bf \lang{zh}} &
    &
    \multicolumn{1}{c}{\bf \lang{de}} &
    \multicolumn{1}{c}{\bf \lang{fr}} &
    \multicolumn{1}{c}{\bf \lang{it}}\\
    \midrule

    \multirow{3}{*}{\bf Train} & Instances
    &
    \num{5428} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &

    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    &
    \num{48042} &
    \num{39428} &
    \num{1144}\\

    &
    Unique Words
    &
    \num{1265} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &

    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    &
    \num{23213} &
    \num{20221} &
    \num{721}\\

    & Avg. Context Len   & 16.8 &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    \multicolumn{1}{c}{--} &
    & 32.7 & 32.3 & 23.2\\
    \midrule
    \multirow{3}{*}{\bf Dev} &
    Instances &
    \num{638} &
    \num{998} &
    \num{852} &
    \num{98} &
    \num{200} &
    \num{104} &

    \num{208} &
    \num{404} &
    \num{250} &
    \num{3046} &
    &
    \num{8870} &
    \num{8588} &
    \num{198}\\

    &
    Unique Words &
    \num{599} &
    \num{354} &
    \num{542} &
    \num{63} &
    \num{174} &
    \num{82} &

    \num{137} &
    \num{183} &
    \num{150} &
    \num{867} &
    &
    \num{4383} &
    \num{3517} &
    \num{136}\\
    &
    Avg. Context Len & 17.1& 8.4 & 32.6 & 19.8&24.8 &17.9 &20.8 &5.7 &20.1 &45.7& &32.5 & 34.3 &23.2\\
     \midrule
    \multirow{3}{*}{\bf Test} &
    Instances
    &
    \num{1400} &
    \num{1220} &
    \num{3406} &
    \num{390} &
    \num{800} &
    \num{408} &

    \num{824} &
    \num{1014} &
    \num{1004} &
    \num{5538} &
    &
    \num{24268} &
    \num{22232} &
    \num{592}\\
    &
    Unique Words
    &
    \num{1184} &
    \num{567} &
    \num{2088} &
    \num{276} &
    \num{533} &
    \num{305} &

    \num{476} &
    \num{475} &
    \num{600} &
    \num{1888} &
    &
    \num{11734} &
    \num{3517} &
    \num{394}\\
    & Avg. Context Len  & 17.2 & 8.5& 32.6 &19.4 & 23.5 & 18.1 & 20.6& 6.0&19.9 &46.0 & & 32.9& 36.4& 23.4\\
    \bottomrule
    \end{tabular}
    }
    \caption{Statistics for WordNet and Wiktionary datasets for different languages.}
    \label{tab:stats}
\end{table*}

\subsubsection{Multilingual WordNet}
\label{multilingual-wordnet}

WordNet \cite{miller1995wordnet} is the \textit{de facto} sense inventory for English WSD.
The resource was originally built as an English lexical database in 1995, but since then there have been many efforts to extend it to other languages \cite{bond2012survey}.
We took advantage of these extensions to construct XL-WiC. In particular, we processed the WordNet versions of
Bulgarian \cite{simovandpetya-2010-wordnet-bg},
Chinese \cite{huangetal-2014-wordnet-zh},
Croatian \cite{raffaelli-etal-2008-wordnet-hr},
Danish \cite{pedersen-et-al-2009-wordnet-da},
Dutch \cite{postma-et-al-2016-wordnet-nl},
Estonian \cite{vider-and-orav-2002-wordnet-et},
Japanese \cite{isahara-et-al-2008-wordnet-ja},
Korean \cite{yoon-et-al-2009-wordnet-ko} and
Farsi \cite{Shamsfard2009SemiAD}.\footnote{We tried other WordNet versions such as Albanian \cite{ruci:2008-albanet},
Basque \cite{pociello-etal-2008-wnterm}, Catalan \cite{benitez1998methods}, Galician \cite{guinovart2011galnet}, Hungarian \cite{mihaltz-et-al-2008-wordnet-hu}, Italian \cite{pianta-etal-2002-wordnet-it}, Slovenian \cite{fiser-etal-2012-wordnet-sl} and Spanish \cite{atserias-et-al-2004-wordnet-es}; but, they did not contain enough examples.}

\paragraph{Farsi: Semi-automatic extraction. }
\label{sec:farsi-dataset}
FarsNet v3.0 \cite{Shamsfard2009SemiAD} comprises 30K synsets with over 100K word entries.
Many of these synsets are mapped to the English database; however, each synset provides just one example usage for a target word.
This prevents us from applying the automatic extraction of positive examples.
Therefore, we utilized a semi-automatic procedure for the construction of the Farsi set. To this end, for each word, we extracted all example usages from FarsNet, and asked an annotator to group them into positive and negative pairs.
The emphasis was to make a challenging dataset with sense distinctions that are easily interpretable by humans.
This can also be viewed as a case study to understand the real gap between human and machine performance in settings where manual curation of instances is feasible.
\paragraph{Filtering.} WordNet is often considered to be a fine-grained resource, especially for verbs \cite{duffield2007criteria}. In some cases, the exact meaning of a word can be hard to assess, even for humans. For example, WordNet lists 29 distinct meanings for the noun \textit{line}, two of which correspond to the horizontally and the vertically organized line formations.
Therefore, to cope with this issue, we followed \citet{pilehvar2019-wic} and filtered out all pairs whose target senses were connected by an edge (including sister-sense relations) in WordNet's semantic network or if they belonged to the same supersense, i.e., one of the 44 lexicographer files\footnote{\url{wordnet.princeton.edu/documentation/lexnames5wn}} in WordNet which cluster concepts into semantic categories, e.g., \textit{Animal}, \textit{Cognition}, \textit{Food}, etc.
\begin{CJK}{UTF8}{}
    \CJKfamily{mj}
For example, the Japanese instance ``成長中の企業は大胆な指導者いなければならない'' (``Growing \textit{companies} must have bold leaders''), ``彼は安定した大きな企業に投資するだけだ'' (``He just invested in big stable \textit{companies}'')
for the target word ``企業'' (``company'') is discarded as its corresponding synsets, i.e., ``An organization created for business ventures'' and ``An institution created to conduct business'', are grouped under the same supersense in WordNet, i.e., \textit{Group}.
\end{CJK}

Finally, all datasets are split into development\footnote{
Development sets are intended to be used for different purposes, such as training or validation, as we will show in our experiments in Section \ref{results:wordnet}.}} and test.
At the end of this step, we ensure that both test and development sets have the same number of positive and negative instances. An excerpt of examples included in some of our datasets are shown in Table \ref{tab:xlwic_examples}

\subsubsection{Wiktionary}
\label{wiktionary}

Wiktionary is one of the richest free collaborative lexical databases, available for dozens of languages.
In this online resource, each word is provided with definitions for its various potential meanings, some of which are paired with example usages.
However, each language has a specific format, and therefore the compilation of these examples requires a careful language-specific parsing.
We extracted examples for three European languages for which we did not have WordNet-based dat
, namely French, German, and Italian.\footnote{French WordNet, WoNeF \cite{pradet2014wonef}, is built automatically; German WordNet, GermaNet \cite{hamp1997germanet}, has a very restrictive license; and the Italian WordNet \cite{pianta-etal-2002-wordnet-it} provides too few examples.} Once these examples were compiled, the process to build the final dataset was analogous to that for the WordNet-based datasets (see Section \ref{multilingual-wordnet}), except for the filtering step, which was not feasible as Wiktionary entries are not connected through paradigmatic relations as in WordNet.

For the case of Wiktionary, the number of examples was considerably higher; therefore, we also compiled language-specific training sets, which enabled a comparison between cross-lingual and monolingual models (see Section \ref{results:wiktionary}).
All Wiktionary datasets are split into balanced training, development and test splits, in each of which there are equal number of positive and negative instances.

\subsection{Statistics}

Table \ref{tab:stats} shows the statistics of all datasets, including the total number of instances, unique words, and the context length average.\footnote{We used the multilingual Stanford NLP toolkit, Stanza \cite{qi-et-al-2020-stanza}, with the available pre-trained neural models.}
Wiktionary-based datasets are substantially larger than the WordNet-based ones, and also provide training sets.
The Chinese datasets feature longer contexts on average and contain the largest number of development and testing instances among WordNet-based datasets.
Korean, on the other hand, is the one with the shortest contexts, which is expected given its agglutinative nature.
As for the training corpora, German and French datasets contain almost ten times the number of instances in the English training set.
This allows us to perform a large-scale comparison between cross-lingual and monolingual settings (see Section \ref{results:wiktionary}) as well as a few-shot analysis (Section \ref{sec:fewshot}).

\subsection{Validation and human performance}
\label{sec:validation}

To verify the reliability of the datasets, we carried out manual evaluation for those languages for which we had access to annotators.
To this end, we presented a set of 100 randomly sampled instances from each dataset to the corresponding annotator in the target language.\footnote{For Farsi, three checker annotators were involved in the validation, each annotating the 100-instance subset. In this case, we report the average accuracy.}
Annotators were all native speakers of the target language with high-level education.
They were provided with a minimal guideline: a brief explanation of their task and a few tagged examples.
We did not provide any lexical resource (or any other detailed instructions) to the annotators with the emphasis to make a challenging dataset with sense distinctions that are easily interpretable to the layman.
Given an instance, i.e., a pair of sentences containing the same target word, their task consisted of tagging it with a \textit{True} or \textit{False} label, depending on the the intended meanings of the word in the two contexts.

Table \ref{tab:validation} reports human performance for eight datasets in XL-WiC.
All accuracy figures are around 80\%, i.e., in the same ballpark as the original WiC English dataset, which attests the reliability of underlying resources and the construction procedure.
The only exception is for Farsi, for which the checker annotators agree with the gold labels in 97\% of the instances (by average).
This corroborates our emphasis on the annotation procedure for this manually-created dataset to have sense distinctions that are easily interpretable by humans.
As for Wiktionary, the human agreements are lower than those for the WordNet counterparts.\footnote{Even if not part of XL-WiC, we also compiled and validated a small Italian WordNet dataset to compare it with its Wiktionary counterpart. Table \ref{tab:italian} in Appendix includes an additional table comparing the nature of these two datasets (including zero-shot cross-lingual transfer results).}
This was partly expected given that the semantic network-based filtering step (see Section \ref{multilingual-wordnet}) was not feasible for the case of Wiktionary datasets due to the nature of the underlying resource.

\begin{table}[t]
    \centering
    \setlength{\tabcolsep}{4.6pt}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{ll lllllll ll}
    \toprule

   \multicolumn{1}{c}{\textbf{WiC}} & &
      \multicolumn{6}{c}{\textbf{WordNet}} & &
      \multicolumn{2}{c}{\textbf{Wiktionary}}  \\
      \cmidrule(lr){1-1} \cmidrule(lr){3-8} \cmidrule(lr){10-11}

      \multicolumn{1}{c}{\bf \lang{en}} & &
      \multicolumn{1}{c}{\bf \lang{da}} &
      \multicolumn{1}{c}{\bf \lang{fa}} &
      \multicolumn{1}{c}{\bf \lang{it}} &

      \multicolumn{1}{c}{\bf \lang{ja}} &
      \multicolumn{1}{c}{\bf \lang{ko}} &
      \multicolumn{1}{c}{\bf \lang{zh}} & &
      \multicolumn{1}{c}{\bf \lang{de}} &
      \multicolumn{1}{c}{\bf \lang{it}} \\
    \midrule

    80.0$^*$ & &
    87.0 &
    97.0 &
    82.0 &

    75.0 &
    76.0 &
    85.0 & &
     74.0 &
    78.0 \\
    \bottomrule
    \end{tabular}
    }
    \caption{Human performance (in terms of accuracy) for different languages in XL-WiC. $^*$From the original English WiC dataset.
    }
    \label{tab:validation}
\end{table}

\section{Experimental Setup}
\label{experimentalsetup}

For our experiments, we implemented a simple, yet effective, baseline based on a Transformer-based text encoder \cite{vaswanietal:17} and a logistic regression classifier, following \citet{wang2019superglue}.
The model takes as input the two contexts and first tokenizes them, splitting the input words into sub-tokens. The encoded representations of the target words are concatenated and fed to the logistic classifier.
For those cases where the target word was split by the tokenizer into multiple sub-tokens, we followed
\newcite{BERT} and considered the representation of its first sub-token.

As regards the text encoder, we carried out the experiments with three different multilingual models, i.e., the multilingual version of BERT \cite{BERT} ({\bf mBERT\xspace}) and the base and large versions of XLM-RoBERTa \cite{XLMR} ({\bf XLMR-base\xspace} and {\bf XLMR-large\xspace}, respectively).
In the monolingual setting, we used the following language-specific models: BERT-de\xspace\footnote{ \url{huggingface.co/dbmdz/bert-base-german-cased}}, CamemBERT-large\xspace  \cite{camembert}\footnote{\url{huggingface.co/camembert/camembert-large}}, BERT-it\xspace \footnote{\url{huggingface.co/dbmdz/bert-base-italian-xxl-cased}}, and ParsBERT\footnote{\url{github.com/hooshvare/parsbert}} \cite{farahani2020parsbert}, respectively, for German, French, Italian, and Farsi. As for all the other languages covered by the WordNet datasets, i.e., Bulgarian, Chinese, Croatian, Danish, Dutch, Estonian, Japanese and Korean, we used the pre-trained models made available by TurkuNLP.\footnote{\url{github.com/TurkuNLP/wikibert}} We refer to each language-specific model as L-BERT.

In all experiments we trained the baselines to minimize the binary cross-entropy loss between their prediction and the gold label with the Adam \cite{kingma-ba-2015-adam} optimizer. Training is carried out for 10 epochs with the learning rate  fixed to $1e^{-5}$ and weight decay set to $0$.
As for tuning, results are reported for the best training checkpoint (among the 10 epochs) according to the performance on the development set

\begin{table*}[!ht]
\centering
\setlength{\tabcolsep}{11.0pt}
\resizebox{\linewidth}{!}{
\begin{tabular}{l ccccccccc}
\toprule
\bf Model  &\bf \lang{bg}  &\bf \lang{da} &\bf  \lang{et} &\bf \lang{fa} &\bf \lang{hr}&\bf \lang{ja} &\bf \lang{ko} &\bf \lang{nl} & \bf \lang{zh} \\
\midrule
\multicolumn{10}{r}{\textit{Zero-shot cross-lingual setting} ~~ \textit{\textbf{Train}: EN ~~--~~ \textbf{Dev}: EN}}\\
\midrule
mBERT\xspace &   58.28 & 64.86 & 62.56 & 71.50 &  63.97 & 62.26 & 59.76 & 63.84 & 69.36 \\
XLMR-base\xspace &  60.73 &  64.79 & 62.82 & 69.88 & 62.01 & 60.44 & 66.96 & 65.73 & 65.78 \\
XLMR-large\xspace &  \textbf{66.48} &  \textbf{71.11} & \textbf{68.71} & \textbf{75.25} & \textbf{72.30} & 63.83 & \textbf{69.63} & \textbf{72.81} & \textbf{73.15} \\
\midrule
\midrule
\multicolumn{10}{r}{
\textit{Test instances translated to English} ~~
\textit{\textbf{Train}: EN ~~--~~ \textbf{Dev}: EN}}\\
\midrule
mBERT\xspace &  63.52 & 62.71 & 68.46 & -- & 60.54 & 63.95 & -- & 66.53 & --  \\
XLMR-base\xspace & 60.98 & 60.24 & 62.82 & --& 60.78 & 61.77 & --  & 64.64 & --  \\
XLMR-large\xspace & 64.43 & 66.64 & 63.84 & --& 69.85 & 64.44 & --  & 72.11 & -- \\
L-BERT\xspace & 64.02  & 65.38 & 64.62 & -- & 69.61 & \textbf{65.90} & -- & 68.43 & -- \\
\midrule
\multicolumn{10}{r}{
\textit{\textit{Train and Dev instances translated from English to target language} ~~ \textbf{Train}: T-EN ~~--~~ \textbf{Dev}: T-EN}}\\
\midrule
mBERT\xspace      & 56.97 & 60.25 & 59.48  & -- & 66.91 & 58.13 & -- & 60.06 & -- \\
XLMR-base\xspace   & 56.07 & 52.85 & 57.18 & -- & 64.22 & 56.19 & -- & 60.56 & -- \\
XLMR-large\xspace  & 62.13 & 63.39 & 64.87 & -- & 66.18 & 59.47 & -- & 66.73 & -- \\
L-BERT\xspace      & 54.26 & 60.57 & 59.49 & -- & 61.52 & 58.98 & -- & 60.46 & -- \\

\bottomrule

\end{tabular}
}
\caption{Results on the WordNet test sets when using only English training data in WiC, either in zero-shot cross-lingual setting (top block) or translation-based settings (the lower two blocks). T-EN is a target language dataset, automatically constructed by translating English instances in WiC.
}
\label{tab:wordnet_results_en_train}
\end{table*}

\subsection{Evaluation settings}

We evaluated the baselines with different configuration setups, depending on the data used for training and tuning.

\paragraph{Cross-Lingual Zero-shot.}
This setting aims at assessing the capabilities of multilingual models in transferring knowledge captured in the English language to other languages.
As training set, we used the English training set of WiC.
As for tuning, depending on the setting, we either used the English development set of WiC or language-specific development sets of XL-WiC (Section \ref{multilingual-wordnet}).
We report results on all WordNet and Wiktionary datasets of XL-WiC\xspace, i.e., Bulgarian, Chinese, Croatian, Danish, Dutch, Estonian, Farsi, French, German, Italian, Japanese and Korean\xspace.

\paragraph{Multilingual Fine-Tuning.}
In this setting, models are first trained on the WiC's English training set, and then further fine-tuned on the development sets of the target languages in XL-WiC.
Depending on the training set used, we report results for two configurations: (i) \textit{EN+Target Language}, combining with WiC's training data and the language-specific WordNet development sets for each language, and (ii) \textit{EN+All Languages}, combining that with all WordNet development sets for all languages in XL-WiC, merged as one dataset.

\paragraph{Monolingual.}
In this setting, we trained each model on the corresponding training set of the target language only.
For the case of WordNet datasets (where no training sets are available), we used the development sets for training.
In this case we split each development set into two subsets with \num{9}:\num{1} ratio (for training and development).
As for the Wiktionary datasets, we used the corresponding training and development sets for each language (Section \ref{wiktionary}).

\paragraph{Translation.}
In this last setting we make use of existing neural machine translation (NMT) models to translate either the training or the test set, essentially reducing the cross-lingual problem to a monolingual one.
In particular, we used the general-domain translation models from the Opus-MT project\footnote{\url{github.com/Helsinki-NLP/Opus-MT}} \cite{TiedemannThottingal:EAMT2020} available for the following language pairs: English--Bulgarian, English--Croatian, English--Danish, English--Dutch, English--Estonian, and  English--Japanese. The models are trained on all OPUS parallel corpora collection \cite{tiedemann-lrec12}, using
the state-of-the-art 6-layer Transformer-based architecture \cite{vaswanietal:17}.\footnote{More details about the NMT models and their translation quality are given in the Appendix (Table \ref{tab:BLEU}).}
In this configuration, as the original target word may be lost during automatic translation, we view the task as context (sentence) similarity as proposed by \citet{pilehvar2019-wic}.\footnote{The Appendix (Table \ref{tab:translated_alignements}) includes another translation baseline that uses dictionary alignments to identify the target word as comparison, but performed worse overall.} Therefore, for each model, the context vector is given by the start sentence symbol. We note that while training custom optimized NMT models for each target language may result in better overall performance, this is beyond the scope of this work.

\paragraph{Evaluation Metrics.}
Since all datasets are balanced, we only report accuracy, i.e., the ratio of correctly predicted instances (true positives or true negatives) to the total number of instances.
\begin{table*}[!ht]
\centering
\setlength{\tabcolsep}{8.0pt}
\resizebox{\linewidth}{!}{
\begin{tabular}{l cccccccccc}
\toprule
\bf Model &\bf \lang{en} &\bf \lang{bg}  &\bf \lang{da} &\bf  \lang{et} &\bf \lang{fa} &\bf \lang{hr}&\bf \lang{ja} &\bf \lang{ko} &\bf \lang{nl} & \bf \lang{zh} \\
\midrule

\multicolumn{11}{r}{\textit{\textbf{Train}: EN ~~--~~ \textbf{Dev}: Target Language}}\\
\midrule
mBERT\xspace & -- & 59.18 &  64.59 &  63.08 & 70.38 &64.95 & 59.95 & 63.31 & 64.04 & 70.48 \\
XLMR-base\xspace & -- &60.74  & 64.80 &  60.77 & 67.75& 62.50 & 57.65 & 66.96 & 61.85 & 65.78 \\
XLMR-large\xspace & -- &\textbf{66.48} & \textbf{71.10}  &  \textbf{68.72} & \textbf{73.63} & \textbf{72.30} & \textbf{60.92} & \textbf{69.63} & \textbf{69.62} & \textbf{73.15} \\
\midrule
\multicolumn{11}{r}{\textit{\textbf{Train}: EN+Target Language ~~--~~ \textbf{Dev}: EN}}\\
\midrule
mBERT\xspace & -- & 71.72  & 62.62 & 63.08 & 69.38 & \textbf{72.30} &  60.92 & 70.91 & 62.95 & 76.72 \\
XLMR-base\xspace & -- & 64.51  & 64.45 &  60.00 & 65.38& 71.57 & 58.37 & 65.68 & 64.54 & 73.46 \\
XLMR-large\xspace & -- & \textbf{75.41}  & \textbf{70.52} & \textbf{68.97} & \textbf{73.75}& 69.61 & \textbf{63.11} & \textbf{73.47} & \textbf{74.50} & \textbf{77.52} \\
\midrule
\multicolumn{11}{r}{\textit{\textbf{Train}: EN+All Languages ~~--~~ \textbf{Dev}: EN}}\\
\midrule
mBERT\xspace & -- &73.03  & 65.09 & 62.31 &73.63& 72.30 &  65.53 & 71.01& 67.73 &76.53 \\
XLMR-base\xspace & -- &67.30  &67.62 & 59.49 &64.50&  66.18 & 57.77 &67.06& 66.33 &71.02 \\
XLMR-large\xspace & -- &\textbf{78.44}  & \textbf{71.49} &  \textbf{72.05} & \textbf{78.25} & \textbf{76.96}& \textbf{66.38} &\textbf{76.53}& \textbf{77.49}&\textbf{78.95} \\
\midrule
\multicolumn{11}{r}{\textit{\textbf{Train}: Target Language ~~--~~ \textbf{Dev}: Target Language}}\\
\midrule
mBERT\xspace & 66.71  & \textbf{82.30} & 62.13 & 58.21 & 63.75 & 77.45 & \textbf{61.04} &70.71 & 64.84 & 76.09\\
XLMR-base\xspace & 64.36 & 79.75  & 64.00 & \textbf{64.36} & 66.25& \textbf{79.17} & 58.86 &70.61 &66.33 &78.11 \\
XLMR-large\xspace & 70.14  & 82.05  & \textbf{66.53} & 59.23 & 68.00&76.72 &  55.22 & \textbf{73.08} & \textbf{69.42} & \textbf{81.83}\\
L-BERT\xspace & 69.60 & 81.23  &62.60 & 58.46 & \textbf{76.63} &76.47 & 56.07 & 58.68 & 68.73 & 77.36 \\
\bottomrule

\end{tabular}
}
\caption{Results on the WordNet test sets when using language-specific data, either for training or for tuning.}
\label{tab:wordnet_results_language_spec}
\end{table*}

\section{Results}
In this section, we report the results for the configurations discussed in the previous section on the XL-WiC\xspace benchmark.
We organize the experiments into two parts, based on the test dataset: WordNet (Section \ref{results:wordnet}) and Wiktionary (Section \ref{results:wiktionary}).

\subsection{WordNet datasets}
\label{results:wordnet}

\paragraph{Using English data only.}
Table \ref{tab:wordnet_results_en_train} shows results on the XL-WiC WordNet test sets, when only WiC's English data was used for training and tuning purposes.
Across the board, XLMR-large\xspace consistently achieves the best results, while mBERT\xspace and XLMR-base\xspace attain scores in the same ballpark.
Indeed, the massive pretraining and the number of parameters of XLMR-large\xspace play key roles behind this lead in performance.
As regards the translation-based settings (lower two blocks), the performance generally falls slightly behind the zero-shot cross-lingual counterpart.
This shows that the usage of good quality English data and multilingual models provide a stronger training signal than noisy automatically-translated data. This somehow contrasts with the observations made on other cross-lingual tasks in XTREME \cite{hu2020xtreme}, especially in question answering datasets \cite{artetxe2019cross,lewis2020mlqa,clark2020tydi}, where translating data was generally better.
This difference could perhaps be reduced with larger monolingual models or accurate alignment, but this would further increase the complexity, and extracting these alignments from NMT models is not trivial \cite{koehn-knowles-2017-six,ghader-monz-2017-attention,li-etal-2019-word}.

\paragraph{Utilizing language-specific data.}
Table \ref{tab:wordnet_results_language_spec} shows results for settings where target language-specific data was used for training or tuning.
Comparing the results in the top block (where target language data was used for tuning) with the middle two blocks (where target language data was instead used for training) reveals that it is more effective to leverage the target language data for training, rather than using it for tuning only.
Overall, it is clear that adding multilingual data during training drastically improves the results in all languages.
In this case, training (fine-tuning) is performed on a larger dataset which, despite having examples from different languages, provides a stronger signal to the models, enabling them to better generalize across languages.
On the contrary, when only using target language data for training and tuning (last block in the table), results drop for most languages.
This highlights the fact that having additional training data is beneficial, reinforcing the utility of multilingual models and cross-lingual transfer.

\begin{table}
\setlength{\tabcolsep}{9.0pt}
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{cl ccc}
\toprule
&\bf  Model &\bf  \lang{de} &\bf  \lang{fr} &\bf  \lang{it}   \\
\midrule

\multirow{3}{*}{\rotatebox[origin=c]{90}{\bf Z-Shot}}

& mBERT\xspace & 58.27 &56.00 &58.61  \\
& XLMR-base\xspace & 58.30 &56.13 & 55.91  \\
& XLMR-large\xspace &\textbf{65.83} & \textbf{62.50} & \textbf{64.86} \\
\midrule

\multirow{4}{*}{\rotatebox[origin=c]{90}{\bf Mono}}

& mBERT\xspace & 81.58 & 73.67 & 71.96  \\
& XLMR-base\xspace &80.84  & 73.06 & 68.58  \\
& XLMR-large\xspace & \textbf{84.03}& 76.16 &72.30   \\
& L-BERT & 82.90 & \textbf{78.14} & \textbf{72.64}  \\
\bottomrule
\end{tabular}
}
\caption{Results on the Wiktionary test sets in different training settings: zero-shot (Z-Shot) and monolingual training (Mono). L-BERT stands for language-specific models, i.e.,  BERT-de\xspace, CamemBERT-large\xspace and BERT-it\xspace for German, French and Italian, respectively.
}
\label{tab:wiktionary_results}
\end{table}

\subsection{Wiktionary Datasets}
\label{results:wiktionary}

In Table \ref{tab:wiktionary_results} we show results for the Wiktionary datasets.
Differently from the results reported for the WordNet datasets (Table \ref{tab:wordnet_results_en_train}), models are less effective in the zero-shot setting, performing from \num{10} to almost \num{20} points lower than their counterparts trained on data in the target language.
This can be attributed to the size of the available training data.
Indeed, while in the WordNet datasets we only have a very small amount of data at our disposal for training (see statistics in Table \ref{tab:stats}), Wiktionary training sets are much larger, hence providing enough data to the models to better generalize.
Once again, XLMR-large\xspace proves to be the best model in the zero-shot setting and a competitive alternative to the language-specific models (L-BERT\xspace  row) in the monolingual setting, performing \num{1.1} points higher in German and \num{2} and \num{0.3} lower in French and Italian, respectively.

\section{Analysis}

In this section, we delve into the performance of the models on XL-WiC\xspace and analyze relevant aspects about their behaviour.

\subsection{Seen and Unseen Words}
For this analysis, we aim at measuring the difference in performance when a given target word was seen (as a target word) at training time or not.
To this end, we evaluate our baselines when trained on the German, French, and Italian Wiktionary training sets and tested on two different subsets of the larger language-specific Wiktionary test sets: In-Vocabulary (\textbf{IV}), containing only the examples whose target word was seen at training time; and Out-of-Vocabulary (\textbf{OOV}), containing only the examples whose target word was not seen during training. We report the results in Table \ref{tab:oov}.

\begin{table}[t]
\centering
\setlength{\tabcolsep}{9.0pt}
\resizebox{\columnwidth}{!}{
\begin{tabular}{cl ccc}
\toprule
\bf
&\bf  Model &\bf  \lang{de} & \bf \lang{fr} &\bf  \lang{it} \\
\midrule

\multirow{4}{*}{\rotatebox[origin=c]{90}{\bf IV}}

& mBERT\xspace & 81.86 & 72.92 & 73.15  \\
& XLMR-base\xspace & 81.17 & 71.92 & 70.69 \\
& XLMR-large\xspace & \textbf{84.24} & 75.61 & \textbf{75.12} \\
& L-BERT & 83.23 & \textbf{77.62} & 73.89\\

\midrule

\multirow{4}{*}{\rotatebox[origin=c]{90}{\bf OOV}}
& mBERT\xspace & 70.08 & 71.24 & 68.54  \\
& XLMR-base\xspace & 71.31 & 71.14 & 62.36 \\
& XLMR-large\xspace & 72.54 & 73.93 & 65.17 \\
& L-BERT & \textbf{76.64} & \textbf{78.00} & \textbf{69.10}\\

\bottomrule
\end{tabular}
}
\caption{Results on the in-vocabulary ({\bf IV}) and out-of-vocabulary ({\bf OOV}) Wiktionary test sets. L-BERT stands for each language-specific model, i.e.,  BERT-base-de, camemBERT and BERT-base-xxl-it for German, French and Italian, respectively.
}
\label{tab:oov}
\end{table}
\begin{figure*}
        \includegraphics[scale=0.40]{figures/tsize.png}
        \caption{\label{fig:few-shot} The impact of training set size (\# of instances) on performance, for the Wiktionary datasets.}
\end{figure*}

In general, multilingual models are less reliable when classifying unseen instances, lagging between 1 and \num{12} points behind in performance, depending on the language and on the model considered. This can be attributed to the fact that their vocabulary is shared among several languages, and therefore may have less knowledge stored about particular words that do not occur often. The performance drop of language-specific models (L-BERT\xspace) is less pronounced, with the French architecture (CamemBERT-large\xspace) attaining even higher performance (\num{0.4} points more) on the OOV set.

\subsection{Few-shot Monolingual}
\label{sec:fewshot}
As an additional experiment, we investigate the impact of training size on performance.
To this end, we leveraged the Wiktionary datasets for German, French and Italian, which allow us to use varying-sized training sets, and created $7$ training sets with 10, 25, 50, 100, 250, 500, 1000 instances.\footnote{Bigger datasets are supersets of smaller ones.}

The results of this experiment are displayed in Figure \ref{fig:few-shot}.
When providing only 10 examples, most of the models perform similarly or even worse than random, i.e., $50\%$ accuracy.
In this setting, language-specific models (L-BERT\xspace) attain better results than their multilingual counterparts, showing better generalization capabilities when fewer examples are provided. This also goes in line with what we found in the previous experiment on seen and unseen words.
With less than $5\%$ of the training data (1000 instances in French and German and 50 instances in Italian), all models attain roughly $85\%$ of their performance with full training data, comparable to results reported for the zero-shot\xspace setting (Table \ref{tab:wiktionary_results}).

\section{Conclusions}

In this paper we have introduced XL-WiC, a large benchmark for evaluating context-sensitive models.
XL-WiC comprises datasets for a heterogeneous set of 13 languages, including the original English data in WiC \cite{pilehvar2019-wic}, providing an evaluation framework not only for contextualized models in those languages, but also for experimentation in a cross-lingual transfer setting.
Our evaluations show that, even though current language models are effective performers in the zero-shot cross-lingual setting (where no instances in the target language are provided), there is still room for improvement, especially for far languages such as Japanese or Korean.

As for future work, we plan to investigate using languages other than English for training (e.g., our larger French and German training sets) in our cross-lingual transfer experiments, since English may not always be the optimal source language \cite{anastasopoulos2020should}.
Finally, while in our comparative analysis we have focused on a quantitative evaluation for all languages, an additional
error analysis per language would be beneficial in revealing the weaknesses and limitations of cross-lingual models.

\section*{Acknowledgments}

We would like to thank Qincheng Zhang (Chinese), Janine Siewert (Danish), Houman Mehrafarin, Hossein Mohebbi, and Ali Modarresi (Farsi), \'{A}ngela Collados A\'{i}s (German), Asahi Ushio (Japanese), and Yunseo Joung (Korean) for their help with the manual evaluation of the datasets.

\begin{comment}
\vspace{1ex}
\noindent
\begin{minipage}{0.1\linewidth}
    \vspace{-10pt}
    \raisebox{-0.2\height}{\includegraphics[trim =32mm 55mm 30mm 5mm, clip, scale=0.2]{erc.png}} \\
    \raisebox{-0.25\height}{\includegraphics[trim =0mm 5mm 5mm 2mm,clip,scale=0.078]{eu.pdf}}
\end{minipage}
\hspace{0.01\linewidth}
\begin{minipage}{0.85\linewidth}
Alessandro gratefully acknowledges the support of the FoTran project, funded by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 771113), and the CSC – IT Center for Science, Finland, for computational resources.

Tommaso gratefully acknowledges the support of the ERC Consolidator Grant MOUSSE No. 726487 under the European Union’s Horizon 2020 research and innovation programme.
  \vspace{1ex}
  \end{minipage}
\end{comment}
  \noindent
\begin{figure}[!ht]
\begin{subfigure}{0.1\columnwidth}

    \raisebox{-0.2\height}{\includegraphics[trim =32mm 55mm 30mm 5mm, clip, scale=0.2]{erc.png}}
\end{subfigure}
\hspace{0.01\linewidth}
\begin{subfigure}{0.72\columnwidth}
  Alessandro gratefully acknowledges the support of the FoTran project, funded by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 771113), and the CSC – IT Center for Science, Finland, for computational resources.
    \\

    Tommaso gratefully acknowledges the support of the ERC Consolidator Grant MOUSSE No. 726487 under the European Union’s Horizon 2020 research and innovation programme.
\end{subfigure}
\hspace{0.01\linewidth}
\begin{subfigure}{0.05\columnwidth}
\raisebox{-0.25\height}{\includegraphics[trim =0mm 5mm 5mm 2mm,clip,scale=0.078]{eu.pdf}}
\end{subfigure}
\end{figure}


\section{Introduction}

 Many ground systems are influenced or disrupted by extreme space weather, such as power infrastructure, satellite communication, television and radio links, as well as GPS navigation systems \citep{allen1989effects, kan1979energy, ding2007large, board2009severe}. Forecasts of space weather events would provide early warning to these systems, allowing for precautions to be taken to mitigate the effects of otherwise damaging activity. One area of particular interest is the modeling of variations in Earth's magnetic field, or geomagnetic activity. There exist many different classes of geomagnetic activity, such as magnetic storms, magnetospheric substorms, quiet day variation, etc \citep{chapman1962geomagnetism}. The physical processes behind many of these phenomena begin in the interior of the sun, where pressure builds to a release of a steady stream of radiation and energetic particles known as the solar wind. Geomagnetic storms at Earth are often the result of periods of drastically intensified solar wind known as coronal mass ejections. In recent years, the problem of forecasting such storms has gained much attention. Geomagnetic indices were developed to measure the general intensity of space weather from the ground \cite{kivelson1995introduction}. Recent works focus on modeling a particular class of geomagnetic activity by studying the relationship between solar wind measurements obtained from space weather datasets or derived from tertiary datasets, and a proxy for magnetic activity such as magnetic indices which summarize specific geomagnetic phenomena.

  In this work, we seek to predict multiple magnetic indices such as the auroral electrojet indices (AE, AU, AL), the disturbance time index (Dst), the Solar Radio Flux at 10.7cm (F10.7) index, and the Planetary K (Kp) index several hours in advance. Where the majority of works have focused on predicting a single magnetic index, we demonstrate an approach to simultaneously forecast several indices multiple hours in advance. In our work, we frame the problem as a multivariate sequence-to-sequence learning task \citep{sutskever2014sequence} where the objective is to predict the four magnetic indices using solar wind measurements. We utilize long short-term memory networks which have been applied to myriad applications involving sequential data and are known for addressing vanishing gradient issues encountered with vanilla recurrent neural networks \citep{hochreiter1997long}.

\vspace{-.1in}
\section{Related Work}

\vspace{-.1in}
\subsection{Magnetic Index Forecasting}

Many works focus on modeling a particular class of geomagnetic activity by examining a single magnetic index. One example area of research is that of studying solar wind coupling where the objective is to compute some coupling function and solve a least-squares regression problem to predict a particular magnetic index using the time-lagged coupling function. \citet{mcpherron2015optimum} introduced a coupling function which partially linearizes the relationship between a set of commonly used solar wind variables and solved a least-squares regression problem to predict the AL index one hour in advance. They reported an average coefficient of determination ($R^2$) of $.67$ over two-month periods spanning 1963-2014 which improves upon other solar wind coupling functions \citep{newell2007nearly, rostoker1972response, clauer1986relationship}.

Other works which utilize neural networks have emerged in recent years. \citet{revallo2014neural} investigated forecasting the Dst index one hour in advance using neural networks, and \citet{boberg2000real} sought to predict the Kp index in real time using solar wind measurements. In \citet{maimaiti2019deep}, the authors derived a substorm classification dataset by applying the substorm identification criteria \citep{newell2011substorm} to the SuperMAG SML index \citep{gjerloev2012supermag}. The forecasting approach involves training variations of (previously) state-of-the-art convolutional networks, such as ResNet \citep{he2016identity} to predict the onset of a magnetospheric substorm within the next hour.

\vspace{-.1in}
\subsection{Solar Wind Forecasting}
Another area of research is the prediction of Solar Wind measurements at L1. NASA's solar dynamics observatory curates a database of ultraviolet images of the solar corona \citep{galvez2019machine} which has been used by many works to forecast measurements at L1. Works typically model complex coupling processes which occur between the Sun and the Earth's magnetic field in order to predict solar wind speed (V) or interplanetary magnetic field strength (B). \citet{upendran2020solar} used a CNN-LSTM architecture leveraging a GoogLeNet \citep{szegedy2015going} as a feature extractor to forecast daily averages of the solar wind multiple days in advance. \citet{chandorkar2019dynamic} proposed a new regression framework to model the varying time lag implicit to the coupling processes between the solar magnetic field and solar wind propagation around the heliosphere.

\vspace{-.1in}
\section{Data and Preprocessing}

\vspace{-.1in}
\subsection{Solar wind and IMF Measurements}
 We use measurements of the solar wind speed $V$, proton density $n$, and the components of the interplanetary magnetic field (IMF) $B_x, B_y, B_z$.  These measurements are taken by the ACE and WIND satellites located at L1. We also utilize historical values of the magnetic indices AE, AL, AU, Dst, F10.7, and Kp. This data is curated by NASA's Space Physics Data Facility and is available through the OMNIWeb database\footnote{OMNIWeb data available at https://omniweb.gsfc.nasa.gov/}. As additional features, we include calendar year, day of year, and hour of day signals. These signals are converted to sinusoids with periods 11 years (roughly equivalent to the average length of a solar cycle), 365 days, and 24 hours in order to account for cyclical variations in the data. All data available on the OMNIWeb database has been preprocessed and resampled to several different time resolutions. For our experiments, we utilize hourly resolution data.

\vspace{-.1in}
\subsection{SuperDARN Measurements}

We also utilize measurements derived from the SuperDARN network of radars \citep{greenwald1995darn}. Specifically, we used the spherical harmonic fit technique developed by \citet{ruohoniemi1989mapping} to produce a database of convection patterns covering the years from 2013 to 2017 (inclusive) on a five-minute cadence. These measurements were subsequently averaged and resampled to a 1 hour resolution. From these patterns we extracted two parameters, the Cross-Polar-Cap Potential (CCP), which parameterizes the degree to which solar wind energy is coupled into the Earth’s magnetosphere, (hence is one of the best parameters available for describing the level of global-scale magnetospheric activity), and the Polar Cap Radius (PCR), which is the average colatitude of the Convection Reversal Boundary. These measurements provide the best proxy for magnetic conditions at the poles aside from the auroral electrojet indices themselves. For a more detailed discussion, the reader is referred to \citep{bristow2007superposed}.

\vspace{-.1in}
\section{Model Training}

\vspace{-.1in}
\subsection{Dataset Assembly}
We treat this forecasting problem as a multivariate sequence-to-sequence learning problem. At some time $t$, the inputs to our model comprise the samples of the time series of all input features in the interval $[t - T_h, t]$ where $T_h \geq 0$ is the number of samples in the past considered. The target is the samples of the time series of the four magnetic indices in the interval $(t, t + T_p]$ where $T_p \geq 1$ is the number of samples in the future predicted. Figure 1 illustrates the structure of input/output pairs on a subset of features. To assemble our datasets, we simply slide this structure through the entire time series one sample at a time. For all of our experiments, we discard all datapoints with missing values. To avoid violating causality, we split the data into training, validation, and testing sets sequentially with sizes being roughly 60, 10, and 30 percent, respectively. We center and scale the input features and training labels by the mean and standard deviation of each indiviual feature in the training set. We perform no additional preprocessing or filtering by the magnitude of the magnetic indices.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.9 \textwidth]{pairStructure2.png}
  \caption{Structure of a single datum in our dataset. The blue shaded region between $t-T_h$ and $t$ comprises all samples in the input time series and the green shaded region between $t$ and $t + T_p$ comprises all samples in the target time series. In this example, $T_p = 6, T_h = 5$.}
\end{figure}

\vspace{-.1in}
\subsection{Model Training and Hyperparameter Tuning}
In this work, we consider a simple neural network consisting of a LSTM which embeds the input time series into a low dimensional space followed by a linear layer which performs a linear transformation on the final output hidden state of the LSTM to the dimensionality of the output time series. We use Mean Squared Error as our loss function, and optimize the weights of the network using the Adam optimizer \citep{kingma2014adam} with weight decay. We conduct all experiments using the PyTorch deep learning framework \citep{NEURIPS2019_9015}. We treat the optimizer learning rate, weight decay coefficient, batch size, hidden dimension, and the number of layers of the LSTM as training hyperparameters. For each experiment, we conduct a random search \citep{bergstra2012random} over 100 total models and train the models for 1350 epochs using the median early stopping rule. We select the best model ranked by the performance on the validation set for all experiments.

\vspace{-.1in}
\section{Results}
\textit{20 Year OMNIWeb Data} In this experiment, we use 20 contiguous years of OMNIWeb Data (without including SuperDARN data) ranging from the years 2000-2019. We use a time history and a lead time of $T_h = T_p = 6$ hours. Our training, validation, and testing datasets correspond roughly to the years 2000-2011, 2012-2013, and 2014-2019, respectively. Our baseline for these exeriments is a persistence forecast in which the prediction for the magnetic indices for all $t$ is equivalent to the most recent available observation in the historical indices. In this experiment, we report performance in terms of pearson corellation coefficient $\rho$ computed over the testing set. Table 1 illustrates the performance drop off over time for our network (mNet) and a persistence forecast (pers). The purpose of this experiment is to compare to a baseline persistent forecast, but it should be noted that we also compare to \citep{mcpherron2015optimum} by computing the coefficient of determination (or explained variance) for a 1 hour AL forecast using predictions from the same network. We find that we achieve an $R^2$ of $.744$ over our test set ranging from 2014-2019. For this experiment, the best performing model uses a one layer LSTM, a batch size of 502, hidden dimension of 58, an initial learning rate of .00368, and a weight decay parameter of 9.7E-8.

\begin{table}[h]
\small
\caption{Pearson correlation coefficent for mNet and baseline (pers) predictions with test labels.}
\label{sample-table}
\begin{tabular}{ccccccccccccc}
\hline
 & \multicolumn{2}{c}{\textbf{AE}} & \multicolumn{2}{c}{\textbf{AU}} & \multicolumn{2}{c}{\textbf{AL}} & \multicolumn{2}{c}{\textbf{Dst}} & \multicolumn{2}{c}{\textbf{f10.7}} & \multicolumn{2}{c}{\textbf{Kp}} \\ \hline
\textbf{Hrs} & \textbf{mNet} & \textbf{pers} & \textbf{mNet} & \textbf{pers} & \textbf{mNet} & \textbf{pers} & \textbf{mNet} & \textbf{pers} & \textbf{mNet} & \textbf{pers} & \textbf{mNet} & \textbf{pers} \\
\hline
1 & .901 & .478 & .885 & .491 & .874 & .422 & .976 & .790 & .995 & .993 & .937 & .658 \\
2 & .782 & .452 & .789 & .462 & .743 & .397 & .949 & .759 & .994 & .991 & .882 & .632 \\
3 & .696 & .429 & .715 & .437 & .656 & .375 & .918 & .730 & .993 & .990 & .825 & .605 \\
4 & .650 & .406 & .665 & .414 & .616 & .352 & .892 & .703 & .992 & .989 & .790 & .578 \\
5 & .622 & .382 & .627 & .394 & .594 & .328 & .870 & .678 & .991 & .988 & .757 & .557 \\
6 & .598 & .364 & .598 & .379 & .574 & .309 & .850 & .655 & .990 & .987 & .726 & .535 \\
\hline
\end{tabular}
\end{table}

\textit{SuperDARN measurements} We compare the performance of our model on three different datasets to examine the added predictive capability of SuperDARN measurements. We start with the original OMNIWeb dataset as described in the previous experiment (base), create an additional dataset by replacing the historical values of the magnetic indices with the SuperDARN measurements (sdrn), and finally create a baseline dataset where we only use solar wind measurements (sw). The hyperparameters for the selected models are given in Table 2. For a fair comparison, we discard all datum corresponding to hours which are not represented in all three datasets. Similar to the first experiment, we use a time history and a lead time of $T_h = T_p = 6$ hours. Figure 2 illustrates the performance of the three models in terms of the pearson correlation coefficient of the predictions with test labels.

\begin{figure}[b]
  \centering
  \includegraphics[scale=.28]{decorrFig.png}
  \caption{Pearson correlation for the predictions of the base model (blue/solid), sdrn model (orange/dotted), and sw model(red/dashed) with test labels.}
\end{figure}

We find that the SuperDARN measurements alone do not provide a substantial improvement over the base solar wind model, and thus we cannot conclude on their efficacy as a sufficient proxy for the magnetic indices. We also remark that the inconclusive results may be due to the limited data availability of SuperDARN measurements. To this end, we hope to increase the amount of SuperDARN measurements in future work.

\begin{table}[t]
\small
\centering
\caption{Hyperparameters for the models in the \textit{SuperDARN measurements} experiment.}
\begin{tabular}{cccccc}
\hline
\textbf{Model} & \textbf{LSTM Layers} & \textbf{Weight Decay} & \textbf{Hidden Dimension} & \textbf{Batch Size} & \textbf{Learning Rate} \\
\hline
\textbf{base} & 1 & 4.21E-8 & 59 & 1731 & 4.73E-2 \\
\textbf{sw} & 1 & 2.31E-7 & 44 & 1032 & 4.79E-2 \\
\textbf{sdrn} & 1 & 4.21E-2 & 48 & 1060 & 2.87E-6 \\ \hline
\end{tabular}
\end{table}

\vspace{-.1in}
\section{Conclusion}
\vspace{-.1in}

We have presented a sequence-to-sequence learning approach to forecasting several geomagnetic indices simultaneously. These results demonstrate the efficacy of such an approach to forecasting these magnetic indices, an improvement over the current state-of-the-art for magnetospheric substorm prediction as well as an ability to outperform a persistence forecast over a longer time horizon. We find that the addition of the currently available SuperDARN data does not improve the predictive power of our models substantially. We seek to include additional results after more SuperDARN data has been processed in the future. Much additional future work is planned. For example, a larger model class may result in improved forecasts. Many recent works have emerged for similar time series forecasting problems including the use of attention, one-dimensional convolutional networks, and other recurrent neural networks. We seek to investigate these by performing a model architecture search \citep{elsken2018neural} or by using pretrained models similar to the approach used in \citep{maimaiti2019deep}. Additionally, prior works have used the ultraviolet coronal image dataset curated by NASA to forecast the solar wind and magnetic conditions at L1 orbit point at a coarse granularity. One potential future direction is to utilize a similar approach to forecast the magnetic indices directly and model the Earth-sun system in an end-to-end fashion. Such an approach may benefit from a pretrained convolutional network. Lastly, we seek to investigate the application of physics-informed neural networks to this problem by exploiting domain knowledge as a regularizer, such as the relationship between the AE, AL, and AU indices.

\vspace{-.1in}
\section*{Broader Impact}
\vspace{-.1in}
Space weather is a broad term used to describe many potentially catastrophic phenomena for which we currently have limited forecasting capability. We know such events will happen, we know the consequences can be disastrous, but we do not have the sophistication to predict when or how a particular event will evolve. One example is a famous 1859 `Carrington Event' that occurred during which northern lights were visible from Central America. Communication infrastructure was not nearly as advanced as it is today, but early telegraph stations caught on fire from induced electrical activity. The same thing could happen today to our electric power grid. The National Academies of Sciences estimates that an event of similar magnitude occurring today could cause several trillion dollars, knocking out power for many countries for many months. Currently, alerts to power utility companies about potential space weather disruptions are based exclusively on these geomagnetic indices, thus improving forecasts of these indices would immediately provide a tangible benefit to society. Space weather forecasting remains in its infancy, in part due to the cacophony of physical processes that cascade from the Sun to the Earth when a solar storm occurs. For decades, physicists have investigated this problem and have made substantial advances in modeling individual components of these complex systems, but have struggled to model the system with analytical techniques in an end-to-end fashion. As such, the broad impacts of effectively applying machine learning to the space weather forecasting problem could be huge, as ML techniques have a unique ability to describe systems that cannot be modeled analytically.

\vspace{-.1in}
\begin{ack}
This work benefited directly from conversations with Jacob Bortnik and Ryan McGranaghan. We would like to thank them for their contributions.

This work was supported by the Defense Advanced Research Projects Agency (DARPA) under cooperative agreement D19AC00009 from the Department of the Interior to Georgia Tech. The first author is supported by the NDSEG fellowship.
\end{ack}

\bibliographystyle{unsrtnat}