\section{Generalization Experiments}
\label{sec:genexp}

We transformer-based text feature extractors for fake news
classification generalizability with several experiments.
We cover the datasets, architectures, and experiments below.

\PP{Datasets.}
We used 9 fake news datasets, consisting of blog articles,
news headlines, news content, tweets, social media posts,
and article headlines.
We have described our dataset below in \cref{tab:datasets}.

\begin{table}[t]
    \caption{Datasets used for our experiments. If splits were not available, we used a random class-balanced split. For Tweet datasets, sample counts are after rehydration, which removed some samples due to missing tweets.}
    \label{tab:datasets}
    \begin{tabular}{lrrr}
    \hline
    \textbf{Dataset} & \textbf{Training} & \textbf{Testing} & \textbf{Type}  \\ \hline
    k\_title \cite{kagglefn}          & 31k               & 9K               & Article Titles \\
    coaid \cite{coaid}                & 5K                & 1K               & News summary   \\
    c19\_text \cite{cov19fn}          & 2.5K              & 0.5K             & Articles       \\
    cq   \cite{covidcq}               & 12.5K             & 2K               & Tweets         \\
    miscov    \cite{miscov}           & 4K                & 0.6K             & Headlines      \\
    k\_text   \cite{kagglefn}         & 31k               & 9K               & Articles       \\
    rumor  \cite{covrumor}            & 4.5K              & 1K               & Social Posts   \\
    cov\_fn  \cite{covidfn}           & 4K                & 2K               & Tweets         \\
    c19\_title  \cite{cov19fn}        & 2.5K              & 0.5K             & Article Titles \\ \hline
    \end{tabular}
    \end{table}

Where possible, we have used the provided training and
validation sets; otherwise, we performed a random,
class-balanced 70-30 split for training and testing.
For \cite{covidcq} and \cite{covidfn} datasets, we performed tweet
rehydration, which removed some samples due to missing tweets.
We show example of label shift due to label overlap in \cref{fig:overlap}.
Here, samples from each dataset are passed through a pre-trained
BERT classifier.
The BERT embeddings are then reduced to 50 components with PCA
then to 2 components with tSNE.
There are several regions with label overlaps, where samples with positive
and negative labels occupy similar spaces.

\begin{figure}[t]
    \centering \includegraphics[width=3.4in]{figs/tfoverlap}
  \caption{\textbf{Overlap of label embeddings}: We show label embeddings for all 9 datasets here.
  %
  There are several regions where true news (blue) overlap with fake news (red) across different datasets.
  %
  This can make models for one dataset perform worse on unseen, but related text-based fake news datasets.}
  \label{fig:overlap}
  \end{figure}

\PP{Architectures.}
We use BERT and AlBERT architectures for our experiments \cite{bert, albert}.
Each transformer architecture converts input tokens to a classification feature vector.
We used pretrained architectures as starting points; our selections include
the BERT \cite{bert}, AlBERT \cite{albert}, and COVID-Twitter-BERT \cite{covbert}.

\PP{Experiments}
We performed 3 experiments to evaluate generalizability of covid fake
news detectors.
In each case, our starting point is a pre-trained foundation model, described in previous
section.
We then conduct the following experiments:

\squishlist
\item \textbf{Static-Backbone.} We freeze the pre-trained feature extractor backbone,
and train only the classifier head. This is analogous to using a static foundation
model.
\item \textbf{Static-Embedding.} We fine-tune the transformer part of the pre-trained
feature extractor along with the classifier head together with a single optimizer,
and freeze the embedding module
\item \textbf{Fine-Tuned Backbone.} We fine-tune the entire feature extractor backbone
along with the classifier head.
\end{list}

\PP{Evaluation.}
We average results across multiple runs of each transformer architecture.
To show results in limited space, we have provided complete evaluation results
for backbones using Covid-Twitter-BERT.
To test generalizability, we train each model on a single dataset, and evaluate on the
test-sets of the remaining, unseen datasets as well as its own testing dataset.
Our results are presented as a confusion matrix.
All approaches are trained for 5 epochs with an AdamW optimizer, with a
learning rate of 1e-4, with a batch size of 64.

\subsection{Generalizability Results}
We show generalizability results using the COVID-Twitter backbone in
for static-backbone training in \cref{fig:sbconf},
static-embedding training in \cref{fig:seconf},
and fine-tuned backbone training in \cref{fig:ftconf}.

\begin{figure}[t]
    \centering \includegraphics[width=3.4in]{figs/sbconf}
  \caption{\textbf{Confusion Matrix for Static Backbone}}
  \label{fig:sbconf}
  \end{figure}

  \begin{figure}[t]
    \centering \includegraphics[width=3.4in]{figs/seconf}
  \caption{\textbf{Confusion Matrix for Static-Embedding Backbone}}
  \label{fig:seconf}
  \end{figure}

  \begin{figure}[t]
    \centering \includegraphics[width=3.4in]{figs/ftconf}
  \caption{\textbf{Confusion Matrix for FIne-Tuned Backbone}}
  \label{fig:ftconf}
  \end{figure}

\PP{Static backbone vs Fine-Tuning.}
The confusion matrices show that upon fine-tuning,
each model increases accuracy on its corresponding test dataset.
For example, accuracy on `k\_short' increases from 0.90 to
0.97 between the static backbone and the fine-tuned backbone.
However, this fine-tuning comes at the cost of generalization in some cases:
across several datasets, model accuracy on unseen data suffers in
the fine-tuned backbone experiments.
For example, a model trained on `cov\_fn' achieves f1 of 0.72  when tested on `c19\_title' in the
static-backbone experiment in \cref{fig:sbconf}. On the fine-tuned experiment,
the same trained model achieves f1 of 0.57, approximately a 20\% drop.
SImilarly, a model trained on `miscov' and tested on `c19\_text' achieves f1 of 0.71
with static backbone, versus f1 of 0.43 with fine-tuned backbone.
This indicates once a model is fine-tuned on a specific covid dataset,
it loses some generalization information compared to the static-backbone version.
However, this is not consistent.
In some cases, generalization accuracy increases: `rumor' performs
better on `cov\_fn' after fine-tuning, with f1 of 0.52 on the static backbone,
versus an f1 of 0.75 after fine-tuning.
Furthermore, `rumor' achieves f1 of 0.67 when tested on `c19\_title`
on the static backbone, and f1 of 0.81 on the fine-tuned backbone (conversely, it performs \textit{worse} on
`coaid`, with f1 dropping from 0.83 to 0.40).

\begin{figure*}[t]
  \centering
  \begin{subfigure}{0.24\textwidth}
      \includegraphics[width=\textwidth]{figs/ometricstatic}
      \caption{Static-Backbone}
      \label{fig:sb}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.24\textwidth}
      \includegraphics[width=\textwidth]{figs/ometricembedding}
      \caption{Static-Embedding}
      \label{fig:se}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.24\textwidth}
      \includegraphics[width=\textwidth]{figs/ometricall}
      \caption{Fine-tuned}
      \label{fig:ft}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.24\textwidth}
      \includegraphics[width=\textwidth]{figs/variance}
      \caption{F1-Score Variance}
      \label{fig:variance}
  \end{subfigure}

  \caption{Overlap vs Accuracy for each of our experiment
  approaches. In \cref{fig:ft}, we have also provided the
  trendlines from the other experiments for comparison.}
  \label{fig:graphs}
  \end{figure*}

\PP{Static Embedding.}
A middle ground between complete fine-tuning and a fully static foundation
feature extractor is to freeze the embedding layer and
fine-tune the transformer layer of the backbone.
Recent work finds freezing the embedding layer during training
can reduce computation costs while achieving 90\% of the accuracy of a
fully fine-tuned model
\cite{sefreezing1, sefreezing2, sefreezing3, sefreezing4, seimportant}.
We find similar results, where freezing the embedding layer achieves accuracy
similar to the corresponding fine-tuned model on the testing dataset,
shown in \cref{fig:seconf}
However, on unseen data, accuracy drop has higher variance.
It is not immediately clear what impact embedding freezing has on unseen
data accuracy.
For this, we must explore the actual overlap between datasets.

\subsection{Data Overlap and Accuracy}
We have seen that there is label overlap between
datasets in \cref{fig:overlap}.
We have also seen accuracy variances on unseen data:
rather than a linear drop across unseen data, some models
perform better and some perform worse after fine-tuning.
These can be explained by directly measuring dataset overlap.

\PP{O-Metric.}
To compute overlap, we use the O-metric to calculate
point-proximity overlap from \cite{ometric}.
The O-metric computes overlap between 2 sets of points in
n-dimensional space using a distance-metric.
We find overlap as follows: given two datasets A and B,
we compute the fraction of points in each dataset where
the nearest neighbor is not from the same dataset.
So, for each point $x\in A$, first we obtain:

$$
O_A(x) = w_A(x) / b_A(x)
$$

where $w_A()$ is the distance to the nearest point to $x$ in A,
and $b_A()$ is the distance to the nearest point to $x$ in B.
Then we can compute the ratio $p_A=|O_A>1|/|A|$ to find overlap
of B in A.
$p_A$ is bounded in $[0,1]$; as $p_A$ approaches 1,
this indicates most points in $A$ are closer to a point in
$B$ than in $A$.
The O-metric is bidirectional in computing overlap
 and includes both $p_A$ and $p_B$:

$O = \frac{p_A + p_B}{2}$

Since we are interested in evaluating generalization, where
we want to see only the overlap of unseen data on training data,
we use a directional O-metric.
That is, we let $O = p_A$ for the final overlap value
 in a context where A is the training dataset
and B is the unseen dataset.
We compute the overlap value between each dataset pair using
cosine similarity on the embeddings of each data point.
So, for each model, we compute embeddings of every sample across
all 9 datasets, then compute the directional O-metric overlap
of each dataset on the model's training dataset [ALGO???].

\PP{Accuracy and Overlap}
We show in \cref{fig:graphs} the comparison of overlap to accuracy across
all 9 datasets on static-backbone, static-embedding, and fine-tuned backbone
experiments.
First, we see that as overlap increases, accuracy also increases
for the fine-tuned backbone in \cref{fig:ft}.
Second, and perhaps more striking, there is higher variance in
accuracy on the static-embedding versus fine-tuned backbone models.
This fits the observations from \cite{seimportant} that the
embedding layers capture more domain-specific information than
the deeper layers.
In this cases, because the embedding layers were frozen, they
never learned the domain-specific fake news information.
Consequently, the static-embedding models' accuracy on unseen data
sees significant variance, compared to the fine-tuned backbone
and static-backbone models, shown in \cref{fig:variance}.

\PP{Generalization and Overlap.}
Clearly, higher overlap between evaluation data and training data
is indicative of accuracy.
During testing, however, it may be difficult to calculate this
overlap on data for each sample.
Further, evaluation data changes continuously, so the overlap may
itself change due to concept drift.
Recent research has shown the importance of exploring a model's
feature space to identify embedding clusters \cite{foundation2, niceness}.
These embedding clusters signify regions of the data space a
model has captured.
Metrics such as probabilistic Lipschitzness show that accuracy on
embedding clusters
can be bounded using the smoothness, or gradient, in the embedding space
\cite{niceness}.
Further, LIGER \cite{foundation2} shows that nondeterministic label regions,
i.e. where labels overlap, indicate non-smoothness.
We extend these findings to present KMeans-Proxy - a plug-and-play pytorch layer.

\subsection{KMeans-Proxy}
\label{sec:kmeans}
Intuitively, if we can store the coverage of a model's embedding space,
then for any sample point, we can check if it falls inside the coverage.
Further, we can also check if a model's prediction on the sample matches
the prediction for the coverage.
We can pair this with a coverage radius, e.g. by computing $r$ that constitutes
coverage of all points in a cluster that are 1 standard deviation away from the
cluster center with respect to a distance metric, such as the l2 norm.
Then, if the predictions do not match or a point falls outside the single standard deviation
coverage radius, this is a strong abstention/reject signal.

\PP{Implementation.}
We can capture the coverage of the embedding space by
using embedding proxies.
Proxies are common in cluster and proxy NCA losses \cite{proxy1, proxy2}.
We adapt them as the KMeans cluster centroids by acting as proxies for
the cluster centers.
This allows our approach to extend to online or continuous learning
domains as well.
The findings in \cite{foundation2} suggest increased partitioning of the embedding space
can yield better local region coverage.
So, KMeans-Proxy is initialized with 2 parameters:
the number of classes $c$, and a proxy factor $k$.
Then, we then obtain $k\cdot c$ centers, with $k$ proxies for each class,
so that each cluster is a smaller, more representative local region.

\begin{lstlisting}
self.proxy = KMeansProxy(proxy_factor = 4, classes = 3, dimensions = 768)
\end{lstlisting}

During model training, KMeans-Proxy performs minibatch online KMeans clustering
 to obtain the embedding space proxies for cluster centers. Online clustering converges
 asymptotically, per \cite{kmeans}.

 \begin{lstlisting}
def forward(x):	# Proxy forward function
    if self.training:
        self.update_proxies(x)
        return x, None, None
    return x, self.nearest_proxy(x), self.nearest_proxy_label(x)
\end{lstlisting}

During prediction, a model using KMeans-Proxy can predection,
as well as nearest proxy label and nearest proxy.
A meta abstention policy can review for label flipping, or coverage radius.

\begin{lstlisting}
def forward(x):	 # Model forward function
    x = self.feature_extractor(x)
    x, proxy, proxy_label  = self.proxy(x)
    label = self.classifier(x)
    return label, proxy, proxy_label
\end{lstlisting}

\begin{table}[t]
    \caption{Generalization improvement with KMP for model trained on `coaid'.}
    \label{tab:coaid}
    \begin{tabular}{|lrrrr|}
    \hline
    \multicolumn{5}{|c|}{\textbf{Trained on `coaid'}}                                                                                \\ \hline
    \multicolumn{1}{|c|}{\multirow{2}{*}{\textbf{Testing Dataset}}} & \multicolumn{4}{c|}{\textbf{Approach}}                          \\ \cline{2-5}
    \multicolumn{1}{|c|}{}                                          & SB            & SE            & FT            & \textbf{FT+KMP} \\ \hline
    \multicolumn{1}{|l|}{cov\_fn}                                   & \textbf{0.57} & 0.44          & 0.51          & {\ul 0.53}      \\
    \multicolumn{1}{|l|}{k\_short}                                  & 0.49          & {\ul 0.57}    & 0.56          & \textbf{0.57}   \\
    \multicolumn{1}{|l|}{coaid}                                     & 0.85          & 0.95          & {\ul 0.97}    & \textbf{0.98}   \\
    \multicolumn{1}{|l|}{cq}                                        & 0.42          & \textbf{0.59} & 0.55          & {\ul 0.57}      \\
    \multicolumn{1}{|l|}{k\_long}                                   & 0.47          & 0.51          & \textbf{0.58} & {\ul 0.55}      \\
    \multicolumn{1}{|l|}{rumor}                                     & 0.34          & 0.31          & {\ul 0.55}    & \textbf{0.68}   \\
    \multicolumn{1}{|l|}{c19\_text}                                 & 0.43          & 0.64          & {\ul 0.79}    & \textbf{0.94}   \\
    \multicolumn{1}{|l|}{miscov}                                    & {\ul 0.54}    & 0.45          & 0.47          & \textbf{0.57}   \\
    \multicolumn{1}{|l|}{c19\_title}                                & 0.49          & 0.75          & {\ul 0.77}    & \textbf{0.90}   \\ \hline
    \end{tabular}
    \end{table}

\begin{table}[]
    \caption{Generalization improvement with KMP for model trained on `rumor'}
    \label{tab:rumor}
    \begin{tabular}{|lrrrr|}
    \hline
    \multicolumn{5}{|c|}{\textbf{Trained on `rumor'}}                                                                                \\ \hline
    \multicolumn{1}{|c|}{\multirow{2}{*}{\textbf{Testing Dataset}}} & \multicolumn{4}{c|}{\textbf{Approach}}                          \\ \cline{2-5}
    \multicolumn{1}{|c|}{}                                          & SB            & SE            & FT            & \textbf{FT+KMP} \\ \hline
    \multicolumn{1}{|l|}{cov\_fn}                                   & {\ul 0.76}    & 0.48          & 0.75          & \textbf{0.77}   \\
    \multicolumn{1}{|l|}{k\_short}                                  & 0.49          & {\ul 0.52}    & 0.52          & \textbf{0.54}   \\
    \multicolumn{1}{|l|}{coaid}                                     & 0.17          & 0.17          & {\ul 0.40}    & \textbf{0.76}      \\
    \multicolumn{1}{|l|}{cq}                                        & 0.47          & \textbf{0.59} & 0.42          & {\ul 0.58}      \\
    \multicolumn{1}{|l|}{k\_long}                                   & \textbf{0.59} & 0.52          & 0.43          & {\ul 0.53}      \\
    \multicolumn{1}{|l|}{rumor}                                     & 0.70          & 0.66          & {\ul 0.83}    & \textbf{0.86}   \\
    \multicolumn{1}{|l|}{c19\_text}                                 & 0.17          & 0.57          & {\ul 0.58}    & \textbf{0.73}   \\
    \multicolumn{1}{|l|}{miscov}                                    & 0.52          & 0.45          & {\ul 0.54}    & \textbf{0.54}      \\
    \multicolumn{1}{|l|}{c19\_title}                                & 0.74          & 0.57          & \textbf{0.81} & {\ul 0.76}      \\ \hline
    \end{tabular}
    \end{table}

\subsection{KMeans-Proxy Results}
We now show results from using KMeans-Proxy as a reject option
in \cref{tab:coaid} and \cref{tab:rumor}.
With rejection, we can increase generalization
accuracy on unseen data.
This is a faster approach than domain adaptation, since proxies are updated
during training.
Further, it is a plug-and-play solution, allowing for faster iteration on
overall model design.

With KMeans-Proxy, we are able to improve generalization performance across the board.
Here, we compare models trained on `coaid' and `rumor' in
\cref{tab:coaid} and \cref{tab:rumor}, respectively.
Models are compared across static backbone (SB), static embedding
(SE), fine-tuned (FT), and fine-tuned with KMeans-Proxy (FT+KMP).

For models trained on `coaid' (in \cref{tab:coaid}) and tested on all datasets,
incorporating KMeans-Proxy improves generalization performance.
In each case, FT+KMP is either the best or the runner-up model
by at most 0.05 f1 f1 points.
We see similar result for models trained on `rumor' in \cref{tab:rumor},
where KMeans-Proxy is either the best model or runner up for every
testing dataset.

\PP{Choice of Proxy Factor}
Increasing the proxy factor leads to better generalization performance.
\cref{tab:proxy} shows performance of a model trained on `c19\_text`
that has poor generalization without KMeans-Proxy (see \cref{fig:ftconf}).
As we increase the proxy factor, we gain better generalization
across testing datasets.
We test with different proxy factors and compare generalization performance
of each model.
We find that increasing the proxy factor leads to small, but measurable
increase in accuracy.

\begin{table}[]
    \caption{Impact of changing proxy factor: increasing the proxy factor increases accuracy, since more proxies allow tighter bounds on local coverage estimates.}
    \label{tab:proxy}
    \begin{tabular}{|l|rrrrrr|}
    \hline
    \multicolumn{1}{|c|}{\multirow{2}{*}{\textbf{Testing Dataset}}} & \multicolumn{6}{c|}{\textbf{Trained on c19\_text}}                                      \\ \cline{2-7}
    \multicolumn{1}{|c|}{}                                          & \textbf{FT} & \textbf{k=1} & \textbf{k=2} & \textbf{k=3} & \textbf{k=5} & \textbf{k=10} \\ \hline
    cov\_fn                                                         & 0.44        & 0.50         & 0.54         & 0.58         & 0.58         & \textbf{0.58} \\
    k\_short                                                        & 0.59        & 0.70         & 0.73         & 0.73         & 0.73         & \textbf{0.75} \\
    coaid                                                           & 0.76        & 0.88         & 0.84         & 0.89         & 0.89         & \textbf{0.90} \\
    cq                                                              & 0.51        & 0.56         & 0.58         & 0.58         & 0.60         & \textbf{0.63} \\
    k\_long                                                         & 0.60        & 0.70         & 0.72         & 0.73         & 0.73         & \textbf{0.73} \\
    rumor                                                           & 0.47        & 0.45         & 0.51         & 0.58         & 0.58         & \textbf{0.67} \\
    c19\_text                                                       & 0.97        & 0.98         & 0.99         & 0.99         & 0.99         & \textbf{0.99} \\
    miscov                                                          & 0.48        & 0.45         & 0.44         & 0.53         & 0.58         & \textbf{0.58} \\
    c19\_title                                                      & 0.58        & 0.61         & 0.64         & 0.69         & 0.68         & \textbf{0.73} \\ \hline
    \end{tabular}
    \end{table}

\subsection{Discussion}
There are several observations we can make from our
generalization studies and KMeans-Proxy experiments.
\PP{Generalization}
For fake news detection, fine-tuned models must be
used carefully to take advntage of learned parameters.
As we showed in the confusion matrices, fine-tuning
improves performance only on subsets of unseen data.
These subsets are regions of the data space where
the unseen data overlaps with training data.
On completely new regions of the data space, fine-tuned
models make mistakes.
These mistakes are because of label overlap.

We must make a distinction between label and data
overlap.
Data overlap means a model has coverage on the unseen
data, and can make predictions with higher confidence.
Label coverage, as we showed in Fig, indicates where
different labels occur close to each other in the
embedding space.
Both can coincide: unseen data points can have both
data \textit{and} label overlap.
For these points, fine-tuned models that have
better captured a local region with training data
are better poised to provide high-confidence labels.

\PP{KMeans-Proxy}
KMeans-Proxy allows us to identify these regions.
With KMeans-Proxy, we partition the data space into
clusters representing model coverage and labels.
Our inclusion of the proxy factor $k$, where we create $k$
clusters for each class label, allows fine-grained partitioning
of the embedding space.
This means we can better capture local characteristics
of the embedding space \cite{foundation2}.
In our experiments, we focus on 2 such characteristics:
(i) whether the label for an unseen point matches
the label for nearest proxy, and
(ii) whether this unseen point is within one
standard-deviation radius of the proxy.
In our experiments, we show that using these provides
improvements in generalizing to unseen data points.

Clearly, there is significant progress to be made in
capturing local characteristics.
For example, when using an ensemble of fine-tuned models,
local smoothness \cite{niceness, foundation2} can be computed for each
non-abstaining model to rank them on coverage.
There may also be advantages in using dynamic proxy allocation.
If prior class balance is known, then we could use a class-specific
proxy factor.

\section{Conclusion}
\label{sec:conclusion}
In this paper, we have presented generalizability
experiments and KMeans-Proxy.
We perform generalization studies across 9 fake
news datasets using several transformer-based
fake news detector models.
Our generalizability experiments show that fine-tuned
models generalize well to unseen data when
there is overlap between unseen and training data.
On unseen data that does not overlap, fine-tuned
models make mistakes due to poor coverage, label
flipping, and concept drift.

Using our observations and recent research into
local embedding regions, we develop and present
KMeans-Proxy, a simple online KMeans clusterer
paired with a proxy factor.
With KMeans-Proxy, we partition the embedding space into
local regions and use local characteristics to create a
reject option for models.
We show that KMeans-Proxy improves generalization accuracy
for fine-tuned models across all 9 fake news datasets.
We welcome future research in this area to better
explore the generalizability and fine-tuning tradeoff.

\section{Introduction}
\label{sec:intro}
The rapid spread of the Covid-19 virus has led
to a parallel surge in misinformation and disinformation \cite{misinfo}
This surge of false information, coined an `infodemic' by the
CDC \cite{infodemic} can be life-threatening,
destabilizing, and potentially dangerous
\cite{infolife}.
The infodemic is multimodal, meaning associated fake news
can take forms of social media posts, tweets,
articles, blogs, commentary, misrepresented titles and headlines,
videos, and audio content.

\PP{Current Research.}
There is significant progress on developing domain-specific
automated misinformation detection and classification tools
\cite{coaid, misinfo1, colies, fndetect, cosocial, generalization}.
Such tools analyze labeled datasets in aforementioned modalities
to classify fake news.
Recent approaches focus on transformer-based classifiers and language
modelers \cite{generalization, fndetect}.

Such fake news detectors are specific to the datasets
they are trained with \cite{colies, generalization}.
More recently, there is a focus on addressing generalizability
concerns in these models
\cite{generalization, generalization2, mcnnet, explainmisinfo, covidfn, fnft}.
For example, \cite{generalization} explores the impact of generalization
of 15 transformer models on 5 fake news datasets.
The results show there is a generalizability gap in
fake news detection: a fine-tuned model trained on
one fake news dataset performs poorly on other
unseen, but related, fake news datasets.

\PP{Generalization Study.}
In this paper, we study the generalizability and fine-tuning tradeoff
and present our findings for furthering the research interest.
We study several fake news detection architectures across
9 fake news text datasets of different modalities.
We find that fine-tuned models often have reduced accuracy
on any unseen dataset.
However, when paired with a reject option to abstain from
low-confidence predictions, fine-tuned models perform significantly better.
These abstention results can then be labeled with
active learning, crowdsourcing, weak label integration,
or a variety of other methods present in literature.

\PP{KMeans-Proxy}
Through observations on our generalizability results, we present a
simple `reject option' \cite{reject1, reject2, reject3} for fake news
detectors, called KMeans-Proxy.
KMeans-Proxy is based on KMeans clustering, and is inspired
by research into proxy losses \cite{proxy1, proxy2} and foundation models
\cite{foundation1, foundation2, foundation3}.
It is written as a PyTorch layer and requires only a
few lines of code to implement for most feature extractors.
We show in our results that KMeans-Proxy improves
generalization on fake news datasets by 0.1 to 0.2
f1 points across several experiments.

\PP{Contributions}
In summary, our contributions are:

\squishlist
\item	Extensive set of experiments across 9 fake news datasets on the generalizability/fine-tuning trade off.

\item KMeans-Proxy, a simple reject-option for feature extractors. KMeans-Proxy
uses cluster proxies from ProxyNCA to estimate embedding cluster centers
of the training data.
During prediction, KMeans-Proxy provides a reject option based on
label difference between sample prediction and nearest
training data cluster center.
\end{list}

Our code for running experiments and for KMeans-Proxy are provided\footnote{\url{https://github.com/asuprem/GLAMOR/blob/colabel-multicb/src/ednaml/utils/blocks/KMeansProxy.py}}.

\section{Related Work}
\label{sec:related}

\subsection{Generalizability and Fine-Tuning}
\label{sec:generalizability}
Since 2002, there are increasing numbers of Covid-19
fake news datasets and associated models for these datasets \cite{coaid, cosocial}.
Recently, there is increasing interest in gauging
the effectiveness of each of these fine-tuned models
on related, but unseen datasets \cite{generalization}.
The authors of \cite{generalization} conduct a generalization study
over 15 model architectures over 5 datasets, and find
that fine-tuning offers little advantage in
classification accuracy.
There is also an abundance of research in unsupervised
domain adaptation to recover accuracy under changing domains
or concept drift \cite{odin, drift1, drift2, drift3}.

\subsection{Concept Drift}
Concept drift occurs when testing or prediction data
exhibits distribution shift \cite{drift1}, either in the data domain,
or in the label domain \cite{drift3}.
Data domain shift can include introduction of new
vocabularies, disappearance of existing words,
and word polysemy \cite{litmus1}.
Label domain shift occurs when the label space
itself changes for the same type of data \cite{drift2, ebka}.
For example, when new types of misinformation are detected,
then the boundary between misinformation and true
information must be adjusted \cite{ebka}.
We show label shift in \cref{fig:overlap}, where subsets of true and
fake news occupy the same embedding space across datasets
due to fine-grained differences.

\subsection{Reject Options}
One drawback of classification models is that
they provide a prediction for every data point \cite{reject1},
regardless of confidence.
Reject options perform external or internal
diagnosing.
This can help detect either low confidence due to
low coverage or divergence from training data distributions
due to concept drift.
Several approaches are covered in a recent survey \cite{reject1}.
We present a reject option that uses recent
findings in \cite{foundation2} and \cite{niceness} on the topology
of the embedding space:
(i) find that local smoothness of the
label space is indicative of local accuracy and coverage \cite{niceness}, and
(ii) local label shift, where
nearby samples have different labels, is a good predictor
of local smoothness \cite{foundation2}.
Our reject option, described in \cref{sec:kmeans} is a clustering approach
that calculates cluster centers in the feature extractor embedding space
for the training data.
Then, during prediction, a model can provide a prediction as well
the label for the nearest training data cluster center.
Flipped, or different, labels can indicate reduced local smoothness,
confidence, and coverage, leading to a reject decision.

\subsection{Motivation}
It is well known that fine-tuned models suffer performance
degradation over time due to data domain shift \cite{ebka, argo, gft}.
Usually, this performance degradation is detected, and a new
model is trained on new labeled data.
Recently, the velocity and size of new data makes obtaining
labeled data quickly and at scale, very expensive \cite{mdaws}.
Updating models during data domain shift requires relying
on weak labels, authoritative sources, and hierarchical
models \cite{mdaws, snorkel, eews}.
In such cases, a team of prediction models is pruned and updated
with new training data \cite{ebka}.
However, we still need predictions in the period when data domain
shift is occurring, and new models have not been trained.
Our work, as well as recent research in generalizability \cite{generalization},
weak labeling \cite{snorkel}, foundation models \cite{foundation2}, and rapid fake
news detection \cite{mdaws} falls in this period.
Our generalizability experiments in \cref{sec:genexp} show that fine-tuned
models, while having lower performance on unseen data,
do have better accuracy on some subsets.
Our KMeans-Proxy solution finds these subsets where fine-tuned
models have higher accuracy.


\section{Introduction}
\noindent Two-dimensional superconductors, that come into being as superconducting thin films, differ crucially in their properties from bulk superconductors. First of all, global superconductivity, where the whole system becomes phase-coherent, sets in at temperatures below the Berezinskii-Kosterlitz-Thoules (BKT) transition, see\,\cite{Goldman2013,tinkham,minnhagen} for a review. In other words, `two-dimensional", or as we refer hereafter, ``planar superconductivity", arises due to a topological phase transition. Next, planar superconductors harbor the celebrated superconductor-insulator transition (SIT), see\,\cite{Goldman2010} for a review. The SIT has been demonstrated to be a topological phase transition, and the phases that emerge in the vicinity of the SIT were identified as topological phases\,\cite{dtv2019}.

The macroscopic physics of superconductors is commonly described by the Ginzburg-Landau (GL) theory\,\cite{tinkham} in which the Coulomb interaction is usually neglected.
Planar superconductivity, in films with the thickness $d$ comparable or smaller than the coherence length $\xi$, establishing at $T\leq T_{\rm\scriptscriptstyle BKT}$,  is still discussed within the GL framework with the order parameter having a fixed amplitude and a phase\,\cite{feigelman}. The bosonic model of the SIT typically treats fluctuations of the phase in the XY model\,\cite{minnhagen} and the Coulomb interaction is still neglected, since it is modelled by the same 1/r potential as in the 3D case. Here we demonstrate that the common treatment of planar superconductivity neglects crucial aspects distinguishing films from their bulk counterparts and outline the topological gauge theory replacing the standard approaches\,\cite{topsc, higgsless}.

In the vicinity of the SIT, superconducting films develop a huge normal-state dielectric constant $\varepsilon$ and the Coulomb interaction maintains the 2D Coulomb logarithmic form in the spatial scale range $d$$\ll$$r$$\ll$$\Lambda_{\mathrm c}$, where the 2D screening length $\Lambda_{\mathrm c}\simeq\varepsilon d$\,\cite{BV2013} reaches macroscopic scales (for TiN films with $d=5$\,nm, $\Lambda_{\mathrm c}\simeq 200\,\mu$m) (this is analogous to the behaviour of the finite-temperature Coulomb potential in 3D, with $\beta = 1/T$ playing the same role as $d$, see, e.g.,\,\cite{zarembo}).
This implies that, on the relevant spatial scales, one can consider electric interactions as genuinely 2D. As the strong logarithmic interactions cannot be neglected, the Ginzburg-Landau model has to be coupled to 2D electromagnetism. In the common approach, having absorbed the phase with the usual Anderson-Higgs mechanism, see e.g.,\,\cite{tinkham}, the gauge fields become massive and remain coupled to charge fluctuations of the condensate. Importantly, in thin films, the scale on which gauge fields are screened, i.e., the inverse of the Anderson-Higgs mass, is given by the Pearl length $\lambda_{\perp} = \lambda_{\rm\scriptscriptstyle L}^2/d$, which is also of a macroscopic scale ( for $d\simeq\xi$, the Pearl length in, for example, TiN films is $\approx$$100\,\mu$m, comparable to the characteristic sizes of the experimental systems).  This means that, on the relevant spatial scales, the gauge fields remain massless within the film.

Let us consider the perturbative expansion around the trivial gauge vacuum corresponding to vanishing fields. The resulting model is non-compact QED in (2+1) dimensions, which is famously infrared divergent\,\cite{polyakov, jackiw1}, since the perturbation parameter is $\propto {\rm log}\ R/a$ where $a$ is the ultraviolet (UV) cutoff, in our case the coherence length $\xi$, and $R$ the infrared (IR) cutoff, the system size. As a consequence the model becomes non-perturbative (and thus ill-defined) for system sizes $\xi \ll R \le \lambda_{\perp} $. Such superconductors epytomize what we call planar superconductors: in the formal limit $d\to 0$ every superconductor is in this class. This implies that for planar superconductors the Anderson-Higgs mechanism is ineffective. In the following we will describe how these planar superconductors cure their infrared divergences.

There are two ways in which gauge fields in (2+1) dimensions can get rid of their divergences. One is to generate a mass via magnetic monopole instantons, describing tunnelling between non-trivial topological gauge vacua \cite{polyakov}. The other is to generate a topological Chern-Simons (CS) mass \cite{jackiw1, jackiw2}. This second way, however entails breaking the discrete symmetries ${\cal P} $ and ${\cal T}$ of parity and time-reversal invariance. This undesired property can be circumvented, however, when the CS mass arises in a model with two gauge fields and a mixed topological term\,\cite{dst}. Depending on the value of $d$, the system chooses one of these possibilities. The former leads to superinsulators\,\cite{dtv1,dtv2,dtv3}, the latter results in planar superconductors that are the focus of present communication. Actually, there is a third possibility, which is eliminating completely any condensate, leading to Bose metal\,\cite{dst,das1,bm,dtv2021}. The topological CS mass comprises the product of two characteristic magnetic and electric length scales, so that the film thickness $d$ falls out. The resulting gauge screening length is not the Pearl length but the original bulk London penetration depth
\begin{equation}
\lambda_{\rm top} = {\cal O}\left( \lambda_{\rm\scriptscriptstyle L}\right)  \ .
\label{renlon}
\end{equation}
The topological gauge theory of superconductivity replaces the Ginzburg-Landau model for planar superconductors\,\cite{topsc, higgsless}. In this model there is no Higgs field and no Abrikosov vortices, only Josephson vortices.

\iffalse
where $\alpha $ is the fine structure constant, $g$ is the quantum conductivity, representing the ratio of magnetic to electric energies, and $\ell$ is a dynamically generated size, as we now discuss. Deep in the superconducting region, both $g$ and $\ell$ become large and the  screening length decreases. When $g$ approaches 1, the dynamical scale $\ell \approx \xi/\alpha$ and the screening length approaches the Pearl length: at this point the system switches to the monopole instanton solution to the infrared catastrophe and becomes a superinsulator.

\fi

Let us now describe in detail how does the topological gauge theory work. If we freeze instantons, the system has no other choice to regularize itself but breaking up the condensate into ``perturbative" droplets of the size ${\cal O}(\xi)$. These droplets are indeed observed in thin superconducting films \cite{sacepe}. Each of these droplets is characterized by an independent phase of the condensate and, thus, there exist topological configurations in which the circulation of these phases on neighbouring droplets is a multiple of $2\pi$. These constitute Josephson vortices see, e.g.,\,\cite{tinkham}, having a nontrivial phase structure but no normal-state core. Indeed, within the droplets, Abrikosov vortices cannot exist if the droplet size compares to $\xi$\,\cite{Likharev1979} (so-called Likharev vortex explosion phenomenon). The system forms global superconductivity as the Cooper pair tunneling between the droplets establish the global phase coherence.

The importance of droplet structure and vortices has already been noticed in very early studies of thin superconducting films. It was posited  that these vortices can condense\,\cite{fisher1, fisher2, fisher3, fazio, girvin} at the insulating side of the SIT, but that the vortex condensation is possible only because the vortices are Josephson ones, without a dissipative core, was never stressed. Two other crucial aspects of these vortices have been remaining underexplored. First, that near enough to the SIT the parameter window in which they become ballistic, thus moving without dissipation, gets really wide and diverges near the transition\,\cite{ballistic1}. Second, charges and vortices are subject to topological interactions leading to Aharonov-Bohm \cite{aharonovbohm} and Aharonov-Casher \cite{casher} phases. These topological interactions are the infrared-dominant ones and crucially influence the long-range properties of the system. This is ultimately why the treatment of thin superconducting films only in terms of a phase governed by the XY model is not correct since the most important interaction has been neglected.

\iffalse
 near the SIT were completely neglected moreover. First, for the same reason that they can condense, i.e. the absence of dissipation, the Josephson vortices are ballistic \cite{ballistic1, ballistic2} and their currents must be taken into account. Secondly, and even more importantly, charges and vortices are subject to topological interactions, which encode the Aharonov-Bohm \cite{aharonovbohm} and Aharonov-Casher \cite{casher} phases acquired by their respective wave functions when one excitation goes around the other. These mutual statistics phases have to be taken into account in the partition function of the system. This is particularly important, given that these topological interactions are the infrared-dominant ones and crucially influence the long-range properties of the system. This is ultimately why the treatment of thin superconducting films only in terms of a phase governed by the XY model is not correct: the most important interaction has been neglected.
\fi

Following \cite{dst, bm} we treat the condensate droplets of Cooper pairs and vortices in the continuum space-time with coordinates $x = (x_0, {\bf x})$  as the point-like objects by introducing the fields
\begin{eqnarray}
Q^{\mu} &&= \sum_I Q_I \int ds \ {dx_I^{\mu} \over ds} \ \delta^3 \left( x -x_I (s) \right) \ ,
\nonumber \\
M^{\mu} &&= \sum_J M_J \int dt \ {dx_J^{\mu} \over dt} \ \delta^3 \left( x -x_J(t) \right) \ ,
\label{pointsources}
\end{eqnarray}
with $x_I (s)$ and $x_J (t)$ parametrizing closed (for fluctuations) or infinitely long space-time trajectories so that $\partial_\mu Q^{\mu} = 0$, $\partial_\mu M^{\mu} = 0$. We will then introduce the droplet size as the necessary ultraviolet cutoff. The mutual statistics interactions are encoded in the Euclidean partition function by phases (we use natural units $c=1$, $\hbar =1$, $\varepsilon_0=1$)
\begin{equation}
S_{\rm top}  =  i \ 2\pi   \int d^3x \  Q_{\mu} \epsilon_{\mu \alpha \nu} {\partial_{\alpha} \over \nabla^2} M_{\nu} \ ,
\label{effec3}
\end{equation}
where $\epsilon^{\mu \alpha \nu}$ denotes the totally antisymmetric tensor.
Using the 3D Green function
\begin{equation}
{1\over -\nabla^2} \delta^3 (x) = {1\over 4\pi} {1\over |x|} \ ,
\label{3dG}
\end{equation}
one obtains
\begin{eqnarray}
S_{\rm top} \left( Q_I, M_J \right) &&=  i 2\pi  \sum_{IJ} Q_I M_J \Phi \left( C_I, C_J \right) \ ,
\nonumber \\
\Phi \left( C_I, C_J \right) &&= {1\over 4\pi} \int_0^1 ds \int_0^1 dt \ {dx_I^{\mu} \over ds} \epsilon_{\mu \alpha \nu}
{(x_I -x_J)^{\alpha} \over |x_I-x_J|^3} {dx_J^{\nu} \over dt} \ .
\label{gauss}
\end{eqnarray}
For closed Euclidean trajectories $C_I$ and $C_J$, the quantity $\Phi \left( C_I, C_J \right)$ represents the integer Gauss linking number, the simplest topological knot invariant (for a review see\cite{kaufmann}). If the excitations satisfy the quantization condition $Q_I M_J = {\rm integer} $ for all $I, J$, the above phase becomes trivial for closed trajectories, corresponding, e.g., to a Minkowski space-time fluctuation creating a charge and a hole that annihilate after having encircled a vortex. In general, however, Aharonov-Bohm-Casher (ABC) phases lead to non-trivial quantum interference effects that cannot be neglected.

Superficially, it looks like the topological interactions are non-local, see (\ref{effec3}), and thus difficult to treat. As was pointed out by Wilczek \cite{wilczek}, however, there exists a local formulation by coupling the charge and vortex trajectories to two gauge fields $a_{\mu}$ and $b_{\mu}$ with a mixed Chern-Simons interaction \cite{dst}, leading to the Euclidean partition function
\begin{eqnarray}
Z &&= \sum_{ \{ Q^{\mu}, M^{\mu} \} } \int {\cal D} a_{\mu} {\cal D} b_{\mu} {\rm e}^{-S} \ ,
\nonumber \\
S &&= \int d^3 x {i\over 2\pi} a_{\mu} \epsilon^{\mu \alpha \nu} \partial_{\alpha} b_{\nu} + i a_{\mu} Q^{\mu} + i b_{\mu} M^{\mu} \ .
\label{wilc}
\end{eqnarray}
This is the reason why the effective field theory of planar superconductors ${\it must}$ be formulated in terms of gauge fields.

While $Q_{\mu}$ represents the quantized charges on the droplets and their currents, the gauge invariant quantity
\begin{equation}
j^{\mu} = {1\over 2\pi} \epsilon^{\mu \alpha \nu} \partial_{\alpha} b_{\nu} \ ,
\label{chargefluc}
\end{equation}
describes the fluctuations of the droplet condensates and the tunnelling currents between them. $M_{\mu}$ and
\begin{equation}
\phi^{\mu} = {1\over 2\pi} \epsilon^{\mu \alpha \nu} \partial_{\alpha} a_{\nu} \ ,
\label{vortexfluc}
\end{equation}
are the dual quantities for vortices.

Once we have recognized that the effective field theory must be formulated in terms of gauge fields we can easily add the next-order terms in a derivative expansion, since they are completely dictated by gauge invariance,
\begin{equation}
S=\int d^3 x \  {i\over2 \pi} a_{\mu} \epsilon^{\mu \alpha \nu} \partial_{\alpha} b_{\nu}
+{1\over 2e^2_v} f_{\mu}f_{\mu}
+{1 \over 2e^2_q} g_{\mu} g_{\mu}
+i a_{\mu}  Q_{\mu} +i b_{\mu}  M_{\mu} \ ,
\label{nonrelac3}
\end{equation}
where
\begin{eqnarray}
f^{\mu} = {1\over 2} \epsilon^{\mu \alpha \nu}  f_{\mu \nu} = \epsilon^{\mu \alpha \nu} \partial_{\alpha} b_{\nu} \ ,
\nonumber \\
g^{\mu} = {1\over 2} \epsilon^{\mu \alpha \nu}  g_{\mu \nu} = \epsilon^{\mu \alpha \nu} \partial_{\alpha} a_{\nu} \ ,
\label{fields}
\end{eqnarray}
are the dual Maxwell field strength of the two gauge fields. The two parameters $e_{\mathrm q}$ and $e_{\mathrm v}$ encode the electric and magnetic scales of a droplet,
\begin{eqnarray}
e_{\mathrm q}^2 &&= {\cal O} \left( {e^2\over 2\pi d} \right) \ ,
\nonumber \\
e_{\mathrm v}^2 &&= {\cal O} \left( {\pi \over e^2 \lambda_{\perp} } \right) = {\cal O} \left( {\pi d \over e^2 \lambda_{\rm\scriptscriptstyle L}^2}\right)  \ .
\label{couplings}
\end{eqnarray}
Non-relativistic effects can be taken into account by considering the ``space component" $x_0 = vt$ where $v$ is the velocity of propagation in the medium and, correspondingly, all time derivatives with respect to $x_0$ and gauge field components with index ``0" as time derivatives with a pre-factor $1/v$.

The two parameters $e_{\mathrm q}^2$ and $e_{\mathrm v}^2$ can be traded for the topological CS mass \cite{jackiw1, jackiw2} $m=e_{\mathrm q}e_{\mathrm v}/2\pi v$, appearing in the dispersion relation of both charges and vortices,
\begin{equation}
E = \sqrt{m^2 v^4 + v^2 p^2} \ ,
\label{dis}
\end{equation}
and one dimensionless parameter $g=e_{\mathrm v}/e_{\mathrm q} = O(d/(\alpha \lambda_L))$, where $\alpha = e^2/4\pi$ is the fine structure constant. This parameter plays the role of a dimensionless conductivity and is the quantum parameter which ``selects" between the two infrared-catastrophe-avoiding models \cite{dst, bm}.

Given that $e_{\mathrm q}^2$ and $e_{\mathrm v}^2$ have canonical dimension [1/length] and, correspondingly, the two kinetic terms in (\ref{nonrelac3}) are infrared-irrelevant, one might be tempted to leave them out and consider anyway only the infrared-dominant mixed Chern-Simons theory. Unfortunately, this leads to wrong results and the origin of the problem lies exactly in the role of the droplet charges $Q_{\mu}$ and the Josephson vortices $M_{\mu}$. Indeed, the limit $m \to \infty$ does not commute with quantization since it involves a phase-space reduction\cite{csqm}. For physical applications (as opposed to purely mathematical ones concerning knot theory), the topological theory has {\it always} to be considered as the $m \to \infty$ limit of a topologically massive \cite{jackiw2} one since, otherwise, physical states are not normalizable. However, when two gauge fields are present, the ``topological limit" $e_{\mathrm q}^2 \to \infty$ and $e_{\mathrm v}^2 \to \infty$ is not well defined without specifying the value of $g$ in this limit. The behaviour of the charges and vortices depends crucially on this value and, as a consequence one obtains very different ground states when $g$ is varied \cite{dst, bm}.

We now minimally couple the charge current $j^{\mu}$ to the real electromagnetic field $A_{\mu}$ by adding to the action the term
\begin{equation}
S \to S + i\int d^3x \ A_{\mu} j^{\mu} \ ,
\label{mincoup}
\end{equation}
and we compute the effective action $S_{\rm eff}\left( A_{\mu}, Q_{\mu}, M_{\mu} \right)$ by integrating over the fictitious gauge fields $a_{\mu}$ and $b_{\mu}$. This gives $S_{\rm eff} = S_{\rm eff}\left( A_{\mu}, Q_{\mu} \right)$ as
\begin{eqnarray}
S_{\rm eff} &&= \int d^3x \left[
{e_{\mathrm q}^2 \over 2} Q_\mu { \delta_{\mu\nu}  \over -\Delta + m^2 v^2}  Q_\nu \right.\nonumber \\&& \left.+i 2 \pi  m^2v^2 Q_\mu { \epsilon_{\mu \alpha \nu} \partial_\alpha \over  \Delta (-\Delta + m^2 v^2 )} \left( M_\nu + {1\over 2\pi}  F_{\nu}\right)
+  \right. \nonumber  \\ && \left. +  {e_{\mathrm v}^2 \over 2} \left( M_\mu + {1\over 2\pi}  F_{\mu }\right) { \delta_{\mu\nu}  \over -\Delta + m^2 v^2}  \left( M_\nu + {1\over 2\pi}  F_{\nu} \right)
 \right]   \ ,
 \label{topact}
 \end{eqnarray}
with $ F_\mu$$=$$\epsilon_{\mu\nu\alpha} \partial_\nu A_\alpha$ being the real dual electromagnetic field strength.
Following the typical approach in lattice gauge theories\,\cite{kogut} we retain only the self-interaction terms in\,Eq.\,(\ref{topact}), replacing  the interaction kernel with its diagonal component
\begin{equation}
G(x-y) =  {1\over m^2v^2 - \Delta} \delta^3(x-y)  \to \ell^2 G(mv\ell) \ \delta^3(x-y) \ ,
\label{green}
\end{equation}
where the necessary UV cutoff $\ell $ represents, as anticipated, the droplet size. We also introduce the numerical parameter $\eta = (mv\ell) G(mv\ell)$. This is a numerical constant of ${\cal O}(1)$ depending on the dimensionless quantity $mv\ell$ and is the second parameter determining the quantum phase structure \cite{dst, bm}.
Eq. (\ref{topact}) become thus
\begin{eqnarray}
S_{\rm eff} &&= \int d^3x \left[
 {1\over 2} {2\pi \ell \eta  \over g} Q^2_\mu  +  {1 \over 2}  2\pi \ell \eta g \left( M_\mu + {1\over 2 \pi}  F_\mu \right)^2  + \right.  \nonumber \\  && \left.  +
 i 2 \pi  (mv\ell)  \eta Q_\mu { \epsilon_{\mu\nu\alpha} \partial_\nu \over  \Delta }  \left(M_\alpha + {1\over 2 \pi}  F_\alpha \right)   \right]   \ .
 \label{topactdiag}
 \end{eqnarray}

Let us now show how superconductivity emerges from the global condensation of the droplet charges in a phase with no vortices, $M_{\mu}=0$. This means that droplets get connected by quantum tunnelling, forming global phase coherence (therefore no vortices), i.e. a global condensate. To this end we integrate over the conserved currents $Q_{\mu}$ in (\ref{topactdiag}) by solving first the constraint $\partial_{\mu}Q_{\mu}=0$ with the introduction of a new gauge field $n_{\mu} $ defined as $Q_{\mu } = \epsilon_{\mu \alpha \nu} \partial_\alpha n_{\nu}$. This is a gauge field since shifts of $n_{\mu}$ by derivatives leave the conserved charge currents invariant. The effective action for the electromagnetic field $A_\mu$ becomes
\begin{equation}
S_{\rm eff} = \int d^3x  \ {1 \over 2} {2\pi \ell \eta \over g} H^2_\mu  + {1\over 2} { \ell \eta g \over 2\pi} F^2_\mu
- i  (mv\ell) \eta \ n_\mu \epsilon_{\mu\alpha\nu} \partial_\alpha  A_\nu  \ ,
\label{scinter}
\end{equation}
where we have introduced the dual field strength $H_\mu = \epsilon_{\mu\alpha \nu} \partial_\alpha  n_\nu$. As anticipated, this effective action is not a Ginzburg-Landau/Higgs model but, rather, a mixed Chern-Simons topological theory. The electromagnetic gauge field does not acquire its mass by the Anderson-Higgs mechanism but by a topological mass generation mechanism \cite{jackiw1, jackiw2} with gauge field mass $m = e_q e_v/2\pi $. The corresponding screening length $\lambda_{\rm top}=1/mv$ is the bulk London penetration depth (\ref{renlon}). Note also that the effective coupling of the Maxwell term in the action is $O(d\ell/e^2 \lambda_{\rm top})$, as can be seen using (\ref{couplings}). This shows that the the Coulomb coupling constant is renormalized to an effective coupling $e^2 \lambda_{\rm top}/\ell$: when $\ell \approx \lambda_{\rm top}$ we have the usual electron charge, when $\ell$ becomes small the effective Coulomb interaction increases.

\iffalse

By integrating also over the gauge field $n_{\mu}$ representing the global condensate fluctuations we obtain an effective action for $A_{\mu}$ alone,
\begin{equation}
S_{\rm eff} \left( A_{\mu} \right) = \int d^3 x \ {1\over 2\lambda_{\rm top}} A_{\mu} \left( \delta_{\mu \nu} -{\partial_{\mu} \partial_{\nu} \over \Delta}\right) A_{\nu} + \dots \ ,
\label{lond}
\end{equation}
with the dots denoting higher-order terms in the derivative expansion and $\lambda_{\rm top}$ given by (\ref{renlon}). This is the quantity entering the London equations and representing thus the effective London penetration depth of planar superconductivity.

\fi

In the dual phase with independent droplet phases and a vortex condensate we can set $Q_{\mu}=0$ while we keep the integration over the vortex degrees of freedom $M_{\mu}$. This phase is particularly interesting for materials characterized by very high dielectric constants, so that $v\ll1$ and we can neglect the magnetic components with respect to the electric ones in the effective action, which becomes
\begin{equation}
S_{\rm eff} \left( M_{\mu}, A_{\mu} \right) = \int d^3x \ {e_v^2  \ell^2 G \over v^2 8\pi^ 2 } \left(F_i + 2 \pi M_i  \right)^2 \ .
\label{polac}
\end{equation}
This is a non-relativistic version \cite{monjja} of Polyakov's compact QED\,\cite{polyakov} in which magnetic monopole instantons create a confining linear potential between probe charges and generate a photon mass. This dual phase is a superinsulator\,\cite{dst, dtv1}.

To conclude, planar superconductors cure their infrared divergences by realizing one of two possible phases with effective actions corresponding to the two known mechanisms of gauge field mass generation different from the Anderson-Higgs mechanism. This Higgsless, topological superconductivty \cite{topsc, higgsless} is the only possibility in 2D, but may be realized also in 3D bulk materials \cite{moncon, pseudo}. Indeed, 3D bulk materials with the emergent granularity typical of planar superconductors have been recently found\,\cite{parra}.

\smallskip

\textit{Acknowledgments--} The work of V.M.V. was supported by the Terra Quantum AG.


\section{Introduction}

Conventionally, Speech-to-text Translation (ST) task has been addressed through cascade approaches \citep{ney1999speech}, which consists of the concatenation of an Automatic Speech Recognition block (ASR), for the audio transcription, with another Machine Translation block (MT), for the translation of such transcription into the desired language. However, this approach ignores some information present in the audio, since it translates from the audio transcript, and is also vulnerable to error propagation, since an error in the ASR block automatically causes a mistranslation \citep{sperber-paulik-2020-speech,bentivogli-etal-2021-cascade}. Consequently, end-to-end alternatives based on an encoder-decoder structure and attention mechanisms have become increasingly popular in recent years \citep{anastasopoulos-etal-2016-unsupervised, duong-etal-2016-attentional, weiss2017sequence}. These are capable of translating the audio without the explicit need for transcription, thus avoiding the problems of the cascade approach and allowing unified optimization of the training parameters.

The advent of the Transformer \citep{vaswani2017attention} revolutionized the MT field, enabling models based on this architecture to achieve the state-of-the-art results. Nowadays, Transformer-based models are used to process all types of data, such as images \citep{parmar2018image} or speech \citep{dong_speech_transformer, di2019adapting}. However, due to its self-attention mechanism, the vanilla Transformer scales quadratically with the input sequence length, which makes it extremely inefficient when processing long sequences.

In speech tasks, it is common to extract audio features every 10 ms to build the input sequences, which causes them to be considerably longer than text sequences. Moreover, since the representation of a single phoneme requires several tokens \citep{igras2013length, inproceedingslength}, the presence of redundancy among the audio tokens is inferred. Therefore, state-of-the-art architectures propose the implementation of down sampling strategies prior to the model collapsing adjacent vectors in a fixed way \citep{berard2018end, di-gangi-etal-2019-enhancing, wang-etal-2020-fairseq}. Similarly, some studies propose to extract more informative sequences using pretrained compression modules \citep{salesky-etal-2019-exploring, zhang-etal-2020-adaptive, gaido-etal-2021-ctc}, obtaining considerable translation quality gains. While these achieve good results, we propose another approach, questioning the use of multi-head self-attention (MHSA), originally proposed for text, for the information extraction from speech sequences.

The closest work to ours was done by \citet{alastruey2021efficient}, who used Longformer's \citep{beltagy2020longformer} local attention pattern as an efficient alternative to self-attention for speech processing. However, they observed that, due to the scarcity of global context in the encoder output, the quality of the translations was slightly hindered. Recently, inspired by Linformer's \citep{wang2020linformer} attention, \citet{papi-etal-2021-speechformer} proposed ConvAttention as an attention mechanism that, by compressing keys and values, is more efficient and therefore able to directly process long sequences. However, this mechanism is not used as a replacement of the encoder self-attention, but as an extra input processing before a CTC-based compression module \citep{gaido-etal-2021-ctc}.

Our contribution to ST field is a new Transformer variant, the Multiformer, an architecture based on the S2T Transformer by \citet{wang-etal-2020-fairseq}. Our architecture enables the use of different attention mechanisms in the same encoder layer, by configuring individually the pattern of each head. With this approach, the Multiformer is able to apply efficient attention mechanisms, while maintaining the ability to learn both local and global content from speech sequences. This diversity among heads in a layer is also meant to stimulate a more varied information extraction and, therefore, reduce the presence of low-relevant heads \citep{voita-etal-2019-analyzing, michel2019sixteen, bian-etal-2021-attention, zhang-etal-2021-enlivening}. Furthermore, we explore the use of different head configurations for each encoder layer. This could help to adapt the attention mechanisms to the needs of each layer. To the best of our knowledge, this is the first study that allows this kind of head-wise configuration.

\section{Model}

In this section, we first introduce a new self-attention module that allows the use of multiple attention mechanisms in parallel (\S \ref{cap:MHMA}). Next, we explain the Multiformer (\S \ref{cap:Multiformer}), which replaces the Transformer encoder MHSA by the new proposed module.

\subsection{Multi-head Multi-attention}\label{cap:MHMA}

An increasing number of studies have observed the presence of redundant heads in multi-head self-attention \citep{michel2019sixteen, bian-etal-2021-attention, zhang-etal-2021-enlivening}. Moreover, \citet{voita-etal-2019-analyzing} even tried to prune them, and observed that the quality of the translations (in MT) was almost not affected. This suggests that the model does not exploit the full potential present in the use of attention heads. In addition, the quadratic time and memory complexity of Self-Attention with respect to the input sequence length makes it impossible to use it directly in Speech tasks. To address this challenge, end-to-end ST models are based on reducing the length of speech sequences, usually by a factor of 4, through compression techniques, so that they can be processed by the Transformer \citep{di-gangi-etal-2019-enhancing, wang-etal-2020-fairseq}. However, after this compression, the resulting sequences are still considerably longer and more redundant than their text counterparts. \citet{alastruey2021efficient} proposed the use of efficient Transformers for ST, but, as observed in different tasks by \citet{tay2020long}, they suffer from a drop in performance quality. The main reason for this deterioration is that most efficient Transformers propose strategies that deprive the model of the ability to learn all types of content from the input stream.\footnote{In efficient Transformers that approximate the $softmax()$ function \citep{choromanski2020rethinking} or the attention matrix \citep{wang2020linformer}, the quality drop can be attributed to an imperfection in these approximations.}

\begin{figure}[h]
    \centering
\includegraphics[scale=0.13]{Photos/MHMA5.png}
    \caption{Scheme of the MHMA with the representation of each of the attention mechanisms it incorporates. The function $g()$ denotes the downscaling and computation of the $softmax()$ function.}
    \label{fig:MHMA}
\end{figure}

To solve the aforementioned problems, we propose multi-head multi-attention (MHMA) (Figure \ref{fig:MHMA}) which, using heads with different attention mechanisms, is meant to force a more diversified learning, thus (i) hindering the presence of irrelevant heads and (ii) allowing the model to learn both local and global content from the input sequence, while applying efficient attention mechanisms. The MHMA module is manually set by selecting the type of attention mechanism for each head in each layer within the following ones:

\paragraph{ConvAttention.} Efficient attention mechanism proposed by \citet{papi-etal-2021-speechformer}. The ConvAttention compresses the keys and values by means of a convolutional layer, decreasing the size of the attention matrix by a factor of $\chi$, to reduce the original complexity to O($(\frac{n}{\chi})^2$). By not compressing the queries, they manage to maintain the dimensions of the input sequence at the output.

\paragraph{Local Attention.} Attention mechanism with a sliding window pattern \citep{beltagy2020longformer}. It only computes the product between queries and keys of nearby tokens within the same input sequence, so it is more efficient than the regular Self-Attention. In particular, given a fixed window size $w$, each token attends to $\frac{w}{2}$ tokens on each side, achieving a linear scaling (O($n\times w$)) of the module complexity. As in \citet{alastruey2021efficient}, this attention pattern is intended to force the learning of local relations, while being more efficient.

\subsection{Multiformer}\label{cap:Multiformer}

The Multiformer is a Transformer-based architecture inspired by \citet{wang-etal-2020-fairseq}. The original model consists on a regular Transformer, preceded by two 1D convolutional layers, that help to tackle speech-specific problems such as a longer sequence length or information redundancy in adjacent tokens. The Multiformer proposes to modify the self-attention module on each encoder layer by a MHMA, since we believe that this module could be helpful to deal with speech.

The introduction of MHMA allows the model to learn from different representational and contextual levels. This enables the construction of architectures capable of extracting different kinds of information from the input sequence, while performing more efficient attention mechanisms. In addition, the model is biased towards learning different types of token interactions, hindering the presence of irrelevant heads.

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.27]{Photos/Multiformer.png}
    \caption{Diagram of the Multiformer encoder. It comprises N layers, each one with C heads that can use different attention mechanisms.}
    \label{fig:Multiformer}
\end{figure}

However, the generation of attention diversity at the head level does not address the presence of redundancy between layers noted by \citet{dalvi-etal-2020-analyzing}, who, using linear Center Kernel Alignment \citep{kornblith2019similarity}, observed that, except for the last two layers, layer redundancy increases throughout the encoder. Moreover, the information processed by each layer differs, hence using the same MHMA configuration in all encoder layers may not be the optimal.

Therefore, the Multiformer (Figure \ref{fig:Multiformer}) allows the use of different MHMA configurations, which is meant to create architectures that process the speech sequence in a more progressive manner. This approach emphasizes the learning of different content along the encoder layers, while hampering information redundancy.

\begin{figure*}[h]
    \centering
\includegraphics[scale=0.185]{Photos/Analisi_heads_4.png}
    \caption{Head relevance for each layer of the proposed models. They have been computed using the median of $n$ contributions (equation \ref{eqn:head_contribution}) from 500 random samples of the en-de training partition. Heads marked in \textcolor{orange}{orange} use \textcolor{orange}{Local Attention} while those in \textcolor{blue!20!magenta}{purple} are using \textcolor{blue!20!magenta}{ConvAttention}.}
    \label{fig:Analisis_heads}
\end{figure*}

\section{Heads Contribution Analysis}\label{AnalsisTeorico}
MHMA allows the use of different attention mechanisms in parallel, therefore we wanted to evaluate the head contribution in each of the encoder layers.

In general, given an input sequence of $n$ tokens $\{\mathbf{x_{1}},...,\mathbf{x_{n}}\}\in \mathbb{R}^{n\times d}$ and a model with an embeddings dimension $d$ (head dimension $d_h$), the output of each attention-head $\mathbf{z^{h}_{i}}\in \mathbb{R}^{d_h}$  is

\begin{equation}{
    \mathbf{z}_{i}^{h} = \sum_{j}^{N}\mathbf{A}_{i,j}^h \mathbf{Wv}^{h} \; \mathbf{x}_j
    }
    \label{eqn:heads}
\end{equation}

where $\mathbf{A}_{i,j}^{h}$ is the attention weight of token $j$ on token $i$ and $\mathbf{Wv}^{h}\in \mathbb{R}^{d_h\times d}$ is the learned projection matrix of the values. The final output representation of the attention module $\mathbf{y_{i}} \in \mathbb{R}^{d}$ is:

\begin{equation}{
    \mathbf{y}_{i}= \mathbf{Wo} \; \text{Concat}\{ \mathbf{z}_{i}^{1}, ..., \mathbf{z}_{i}^{H} \} + \mathbf{b}_o
    }\label{eqn:attention_out}
\end{equation}

with $\mathbf{Wo} \in \mathbb{R}^{d \times H\cdot d_h}$ as the output projection matrix trained jointly with the model, and $\mathbf{b}_o \in \mathbb{R}^{d}$ referring to the bias. As previously done for interpretability research \citep{kobayashi-etal-2020-attention}, the above expression (equation \ref{eqn:attention_out}) can be rewritten as follows:

\begin{equation}{

    \mathbf{y}_{i} = \sum_{h}^{H}\mathbf{Wo}^{h} \; \mathbf{z}_{i}^{h} + \mathbf{b}_o
    }
    \label{eqn:output_layer_descomposed}
\end{equation}

where $\mathbf{Wo}^{h} \in \mathbb{R}^{d \times d_h}$ is the part of the output projection matrix corresponding to each head.
Note that from this last expression, it can be defined $\mathbf{\xi}^{h}_i = \mathbf{Wo}^{h}\mathbf{z}_{i}^{h}\in \mathbb{R}^{d}$ as the projected output vector of a head.

Inspired by \citet{kobayashi-etal-2020-attention}, for each layer, we define the contribution of each head to the attention output $\mathbf{y}_{i}$ as the Euclidean norm of the projected output vector of heads:

\begin{equation}{

    {c}_{i,h} = {\rvert \rvert \mathbf{\xi}^{h}_i \lvert \lvert}_2
    }
    \label{eqn:head_contribution}
\end{equation}

\section{Experiments}
In this section we first explain the training details (\S \ref{Settings}) in order to ensure reproducibility of experiments.\footnote{Code available: \url{https://github.com/mt-upc/fairseq/tree/multiformer}} Then we briefly describe Multiformer architectures and the procedure we followed (\S \ref{descripcion}).

\subsection{Experimental Settings}\label{Settings}

The Multiformer architectures have been trained on 3 different language directions of the MuST-C dataset \citep{cattoni2021must}. This corpus consists of audio, transcriptions, and translations of TED talks in English. MuST-C provides 8 language directions ranging in size from 385 (Portuguese) to 504 (Spanish) hours of transcribed and translated speech.

To ensure a faithful comparison with the baseline model, the small architecture of the S2T Transformer in Fairseq \citep{wang-etal-2020-fairseq}, all our models consist of 12 encoder layers, 6 decoder layers and 4 heads in each attention layer. The embedding dimension of the model is 256. Moreover, following the baseline architecture, we have kept the convolutional layers with downsampling prior to the model.

For the ConvAttention, we use a kernel size of $5$ and a stride of $2$, reducing the length of keys and values to the half. Regarding the Local Attention, as in \citet{alastruey2021efficient} we have chosen a window size of $64$ tokens. These hyperparameters have been employed in all Multiformer architectures. For a detailed description of training parameters, see appendix \ref{sec:DetailExp}.

\begin{table*}[t]
\centering
\small \begin{tabular}{l@{\hskip 4pt}| @{\hskip 7pt} c @{\hskip 5pt} c @{\hskip 5pt} c @{\hskip 4pt}| @{\hskip 7pt} l @{\hskip 5pt} c @{\hskip 5pt} c@{\hskip 4pt}| @{\hskip 7pt} l @{\hskip 5pt} c @{\hskip 5pt} c @{\hskip 4pt}| @{\hskip 4pt} c}
\hline
\multirow{2}{*}{\textbf{Model}} & \multicolumn{3}{c}{\textbf{en-de \space \space \space \space \space \space \space \space}} & \multicolumn{3}{c}{\textbf{en-fr \space \space \space \space \space \space \space \space}} & \multicolumn{3}{c}{\textbf{en-es}} & \multirow{2}{*}{\textbf{Avg}$(\Delta\%)$}\\
  & BLEU & $\Delta$BLEU & $\Delta\%$& BLEU & $\Delta$BLEU & $\Delta\%$ & BLEU  & $\Delta$BLEU & $\Delta\%$ \\
\hline
baseline & $22.65$ & - & - & $32.97$ & - & - & $26.99$ & - & - & - \\
\hline
local\_attention & $22.69$ & $+0.04$ & $+0.17$ & $33.00$ & $+0.03$ & $+0.09$ & $27.10$ & $+0.11$ & $+0.41$ & $+0.22$ \\
conv\_attention & $22.45$ & $-0.20$ & $-0.88$ & $33.07$ & $+0.10$ & $+0.30$ & $26.96$ & $-0.04$ & $-0.15$ & $-0.73$  \\
\hline
multiformer\_lc & $22.80$ &$+0.15$& $+0.66$ & $33.25$ &$+0.28$& $+0.85$ & $27.56$ & $+0.57$ & $+2.11$ & $+1.21$ \\
multiformer\_v1 & $\bf{23.16}$ &$\bf{+0.51}$& $\bf{+2.25}$& $33.10$& $+0.13$& $+0.39$& $\bf{27.68}$&$\bf{+0.69}$& $\bf{+2.56}$ & $\bf{+1.73}$\\
multiformer\_v2 & $22.98$ &$+0.33$& $+1.46$& $\bf{33.26}$ &$\bf{+0.29}$& $\bf{+0.88}$& $27.44$& $+0.45$& $+1.67$& $+1.34$ \\
\hline
\end{tabular}
\caption{\label{results}
BLEU results in 3 different language directions of the MuST-C dataset, English$\rightarrow$German (en-de), English$\rightarrow$French (en-fr) and English$\rightarrow$Spanish (en-es). Relative improvements are calculated with respect to the baseline \citep{wang-etal-2020-fairseq}.}
\end{table*}

\subsection{Experiments Description}\label{descripcion}

First, we trained two architectures based on a single attention mechanism (Local or ConvAttention), in order to obtain a comparison between models with and without diversity.

After this, we trained the first Multiformer architecture, the $multiformer\_lc$. It has a configuration of the MHMA with 2 heads of ConvAttention and 2 heads of Local Attention for all encoder layers. Then, we analyzed the contribution of each head following the methodology described in \S\ref{AnalsisTeorico}. This allowed us to better understand the needs of each layer, and to propose architectures based on this. From Figure \ref{fig:Analisis_heads}, it can be seen that in the first 3 layers, the $multiformer\_lc$ assigns low relevance to the representations extracted by one of the Local Attention heads, which could indicate the prioritization of the global context in the first layers. In the middle layers, a change in this trend is observed, with Local Attention heads acquiring more importance. As for the last layers, we see an equal relevance distribution between heads of both mechanisms.

These observations have motivated the training of the $multiformer\_v1$, which tries to correct the abandonment of Local Attention heads observed in the initial layers. It consists of substituting a Local Attention head for a ConvAttention head in the first six layers of the encoder.

Finally, the $multiformer\_v2$ is built more strictly from the analysis. It incorporates 3 different MHMA configurations. In the first 3 layers, it uses 1 head of Local Attention and 3 heads of ConvAttention. The next 5 layers (from the 4th to the 8th) use 3 Local Attention heads and 1 ConvAttention head, to finish the remaining 4 layers with 2 heads of each type.

In general, it is clear that, whereas the baseline uses few heads in most layers, Multiformer architectures\footnote{More details in Table \ref{tab:Arquitecturas} in the Appendices.} force the model to have a more uniformly distributed contribution between heads.

\section{Results}

First, it can be observed from Table \ref{results}, that the efficient architecture based only on Local Attention ($local\_attention$) already obtains the same results as the baseline, suggesting the presence of unnecessary computations in self-attention. Unlike previous works \citep{alastruey2021efficient}, this architecture maintains the convolutional layers, so the amount of global content within the attention mechanism is higher using the same window size. On the other hand, while the architecture based exclusively on ConvAttention ($conv\_attention$), manages to achieve baseline results in English$\rightarrow$French (en-fr) and English$\rightarrow$Spanish (en-es), its score in English$\rightarrow$German (en-de) drops $0.2$ BLEU, suggesting the need for a higher resolution extraction of representations for that language pair.

Secondly, analyzing the heads contribution of the baseline architecture, we can observe that the heads contribution tends to accumulate in few heads. This means we obtain similar conclusions than \citet{voita-etal-2019-analyzing}, but for the ST setting.
Furthermore, our work goes one step further, showing that those architectures where the heads contribution is uniformly distributed, obtain a higher performance. This finding confirms that, in ST, some heads on a regular Transformer tend to learn irrelevant information. This shows that MHSA might not be as capable as expected of extracting different kinds of patterns, unless it is biased on purpose towards doing so.

In particular, all Multiformer variants improve the results obtained by the baseline and the $local\_attention$ and $conv\_attention$ architectures.
However, these improvements are not equal in all languages pairs, and go from $0.15$ to $0.57$ BLEU for $multiformer\_lc$, from $0.13$ to $0.69$ BLEU for $multiformer\_v1$ and from $0.29$ to $0.45$ BLEU for $multiformer\_v2$, becoming the latter the architecture with the most robust gains.

\section{Conclusions}

In this paper, we present the Multiformer, the first Transformer-based model that allows to combine different attention mechanisms in the MHSA module. This helps the model extracting different types of token interactions from each head, hence preventing the appearance of irrelevant heads. By applying this diversity of attention patterns with efficient mechanisms, the model is able to maintain both local and global context across encoder layers while being more efficient. Experiments on 3 language pairs show that all Multiformer architectures outperform the results achieved by the S2T Transformer in the ST task, with an improvement up to $0.69$ BLEU for the English-Spanish direction in the $multiformer\_v1$.

\section{Acknowledgements}

This work was partially funded by the ADAVOICE project, PID2019-107579RB-I00 / AEI / 10.13039/501100011033, and the UPC INIREC scholarship nº3522. We would like to thank Javier Ferrando for his advice on the heads contribution analysis.