\section{Introduction}

Open-domain question answering (QA) is the task of finding answers to generic factoid questions. In recent literature, the task is largely approached in two ways, namely \emph{retrieve \& read} and \emph{parametric}. The former solves the problem by first retrieving documents relevant to the question from a large knowledge source and then reading the retrieved documents to find out the answer~\citep{orqa, realm, dpr, rag, fid}. The latter, also known as closed-book QA, generates the answer in a purely parametric end-to-end manner~\citep{gpt3, t5}.

\begin{figure}[t]
\centering
\includegraphics[width=0.5\textwidth]{figs/effqa.pdf}
\caption{\small System footprint vs. Exact Match (EM) accuracy on EfficientQA dev set.
System footprint is measured by the command \texttt{du -h /} inside the standalone docker container as stated in the EfficientQA competition guideline.
The red plot from left to right shows a path of reducing the size of an open-domain QA system with DPR from 77.5GB to 484.68MB by successively applying each of the strategies in Section~\ref{sec:method}.
The storage footprints of the baseline systems with T5 are calculated assuming the use of the lightweight docker image and post-training compression methods applied to our system.
}
\label{fig:main}
\end{figure}

While a parametric model enjoys the benefit in terms of system size that they do not require additional knowledge source like a retrieve \& read system does, its fundamental limitations are that their predictions are not so interpretable and they are not suitable for dynamic knowledge source as it is difficult to add, remove, or edit knowledge in the parametric model. These limitations are well-addressed by the retrieve \& read mechanism, which makes it often more suitable for real-world products. However, it is known to suffer from its large storage footprint due to its document corpus and index, especially compared to the parametric model that only needs to store the parameters~\citep{memory, r2d2, paq}.

Building an interpretable and flexible open-domain QA system and reducing its system size are both important in real-world scenarios; the system must be able to quickly adapt to the changes of the world and be deployed in a highly constrained serving environment such as edge devices. Hence, to get the best of both worlds, it is worthwhile to explore the trade-off between the storage budget and the accuracy of a retrieve \& read system.

Well-known approaches for reducing the size of a neural network include pruning~\citep{pruning}, quantization~\citep{q8bert}, and knowledge distillation~\citep{distillation}. In this paper, we utilize some of these generic approaches and combine them with problem-specific techniques to size down a conventional retrieve \& read system. We first train a passage filter and use it to reduce the corpus size (Section~\ref{sec:filtering}). We further apply parameter sharing strategies and knowledge distillation to make a single-encoder lightweight model that can perform both retrieval and reading (Section~\ref{sec:sharing}). In addition, we adopt multiple engineering tricks to make the whole system even smaller (Section~\ref{sec:compression}).

We verify the effectiveness of our methods on the dev set and test set of EfficientQA\footnote{A recently-hosted open-domain QA challenge at NeurIPS 2020 in which 18 teams participated. \\ \href{https://efficientqa.github.io}{https://efficientqa.github.io}}~\citep{efficientqa}. By applying our strategies to a recent extractive retrieve \& read system, DPR~\citep{dpr}, we reduce its size by 160x with little loss of accuracy, which is still higher than the performance of a purely parametric T5~\citep{t5} baseline with a comparable docker-level storage footprint. In Appendix~\ref{appendix:nq_tqa}, we also report the performance on two more open-domain QA datasets, Natural Questions~\citep{nq} and TriviaQA~\citep{tqa}, to test the generalizability of our methods and suggest a future research direction.

\section{Method}\label{sec:method}

In this section, we discuss three techniques for reducing the storage footprint of a generic retrieve \& read system, namely passage filtering (Section~\ref{sec:filtering}), unifying retriever and reader into a single model through parameter sharing (Section~\ref{sec:sharing}), and post-training compression (Section~\ref{sec:compression}).
We assume that the initial system takes the conventional composition of a trainable (neural) retriever with a question encoder and a passage encoder that create dense vectors used for search, a neural extractive reader (possibly with passage ranking), and a text corpus and the corresponding index that serve as the knowledge source.
Figure~\ref{fig:main} shows how we start from one such retrieve \& read system and apply each of the methods, in the order they are introduced in this section, to successively reduce its system footprint without sacrificing much accuracy.

\subsection{Passage Filtering}\label{sec:filtering}
Index and corpus files can take up a significant portion of the storage footprint of a retrieve \& read system if a large text corpus is utilized as the knowledge source.
Therefore, to drastically reduce the system size, we train a binary classifier and use it to exclude passages that are relatively unlikely to be useful for question answering.

Let the set of indices of all passages in the corpus be $J_\text{total}$. To create the training data, we split $J_\text{total}$ into two disjoint sets, $J^+_\text{train}$ and $J^-_\text{train}$, such that the former contains the indices of the passages we would like to include in the minimal retrieve \& read system.\footnote{All the details, including how we split the data, are in Appendix~\ref{appendix:filter}.}
Denoting $E(\cdot)$ as a trainable dense encoder which maps a passage to a $d$-dimensional embedding, such that $v_j = E(p_j) \in \mathbb{R}^{d}$, the score $s_j = v_j^\top w$, where $w \in \mathbb{R}^{d}$ as a learnable vector, represents how likely a passage $p_j$ would hold the answer to an input question.
The classifier is trained with binary cross entropy on the minibatches of half-positive and half-negative passages, $J^{+'}_\text{train}$ and $J^{-'}_\text{train}$ drawn from $J^+_\text{train}$ and $J^-_\text{train}$, respectively.

During training, we sample several checkpoints and evaluate them using the hit ratio on a validation set: $\text{hit}_\text{val} = |J^{[1:|J_\text{val}^+|]}_\text{val} \cap J^+_\text{val}| / |J^+_\text{val}|$, where $J^+_\text{val}$ is a set of indices of the ground truth passages that hold the answer for the questions in the validation set and $J^{[1:|J_\text{val}^+|]}_\text{val}$ is the set of indices $j$ of the passages whose inferred score $s_j$ is in the top-$|J_\text{val}^+|$ scores sorted in descending order, among all $s_j$ such that $j \in J^+_\text{val} \cup J^-_\text{val}$. $J^-_\text{val}$ is a disjoint set randomly sampled from $J^{-}_\text{train}$.

We select the checkpoint with the highest $\text{hit}_\text{val}$ and calculate $s_j$ for all $p_j$, where $j \in J_\text{total}$, using the selected checkpoint. Then, we retrieve $J_\text{subset} = J^{[1:n]}_\text{total}$, the set of indices of the $n$ top-scoring passages, to indicate the passages to include in our minimal retrieve \& read system.

\subsection{Retriever-Reader with Single Encoder}\label{sec:sharing}

In this subsection, we introduce how to obtain a unified retriever-reader with a single encoder (which results in a smaller system footprint) that can perform both retrieval and reading without much drop in accuracy. The unified retriever-reader is trained by successively applying (1) retriever encoder sharing, (2) distilling a reader into the retriever-reader network, and (3) iterative finetuning.

\paragraph{2.2.1 Lightweight Encoder and Embedding Dimension Reduction}\label{sec:lightweight}
To make the system small, we utilize a lightweight pretrained encoder. Specifically, MobileBERT~\citep{mobilebert} (4.3x smaller than BERT-base~\citep{bert}) is employed as the encoder of our retriever-reader model.

We use the dense embedding vectors of the passages in the knowledge source as the index. Therefore, reducing the embedding dimension results in a linear decrease in the index size. We use only the first 128 dimensions (out of 512) to encode the questions and passages.

\paragraph{2.2.2 Retriever Encoder Sharing}
Let $E_\psi(\cdot)$ and $E_\phi(\cdot)$ be the the question encoder and passage encoder of a retriever, where each of the encoders produces a vector for question $q$ and passage $p$.

We share the parameters of the encoders, so that $\psi = \phi = \theta$, and differentiate the question inputs from passages inputs using an additional input signal: different token type ids of 0 for questions and 1 for passages. The retrieval score for a pair of question $q$ and passage $p$ is calculated as $\text{sim}_\theta(q, p) = E_\theta(q, 0) \cdot E_\theta(p, 1)$.

We minimize the negative log-likelihood of selecting the passage which holds the answer, namely the positive passage, while training on mini-batches that consist of questions that are each paired with one positive passage and several negative passages. This procedure creates a retriever with a single encoder of parameters $\theta$ that can encode both questions and passages.\footnote{In a setting where the index is frozen (addition or editing of index items does not occur), the system does not need a passage encoder. However, we assume a self-contained system with the full ability to update the index, so the passage encoder is considered in the system composition.}

\paragraph{2.2.3 Unified Retriever-Reader Through Knowledge Distillation}

The previous subsection describes how to make a retriever that holds only one encoder. Here, we further train the parameters of the retriever $\theta$ so that it can also acquire the ability of a reader; we make a unified retriever-reader model that shares all the encoder parameters and eliminate the need for a separate reader. Specifically, using a fully trained reader of parameters $\xi$ as the teacher, we adopt knowledge distillation to transfer its reading ability to the unified retriever-reader network. The training starts after initializing the parameters of the retriever-reader as $\theta$, which is obtained from the retriever encoder sharing procedure described in the previous subsection.

Let $J_\text{read} \subset J_\text{subset}$ be the set of indices of the passages whose retrieval score $\text{sim}_\omega(q, p_j)$, calculated for question $q$ using a retriever with parameters $\omega$\footnote{The retriever with parameters $\omega$ is the retriever used with the teacher reader of parameters $\xi$.}, is among the top-$k'$ scores for all $j \in J_\text{subset}$.
$J_\text{read}$ serves as the candidate pool of the indices of the training set passages.

During training, for question $q$, a set of passages $P_q = \{p_i | 1 \leq i \leq m \}$ where $m \geq 2$ is sampled from $\{p_j | j \in J_\text{read}\}$ to construct a part of the training batch, such that only $p_1$ contains the answer to question $q$ among $p_i \in P_q$.

Then, we train the unified retriever-reader network with parameters $\theta$ using a multitask loss $\mathcal{L}_\text{read} + \mathcal{L}_\text{ret}$, such that the former is used to train the reader part of the network, and the latter is used to keep training the retriever part. The resulting retriever-reader model has the ability to perform both retrieval and reading.

$\mathcal{L}_\text{read}$ is designed to distill the knowledge of a reader teacher into the reader part of the retriever-reader student; the KL divergence between the sharpened and softmaxed answer span scores of the teacher and the student, $D_{KL}(\mathcal{P}^\text{span}_{\xi, q} || \mathcal{P}^\text{span}_{\theta, q})$. If the teacher reader additionally contains a passage ranker, distillation is also jointly done on the passage ranking scores ($m$-dim vector outputs).

Retrieval loss $\mathcal{L}_\text{ret}$ is jointly optimized in a multitask-learning manner to prevent the retriever part of the unified network from forgetting the retrieval ability while training the reader part. The loss can either be the negative log-likelihood described in the previous subsection or another knowledge distillation objective function with a fully trained retriever teacher. If the reader teacher used for $\mathcal{L}_\text{read}$ has a passage ranker, the passage ranking score of the teacher can serve as the distillation target~\citep{rdr}.

\paragraph{2.2.4 Iterative Finetuning of Unified Retriever-Reader}

We have observed that finetuning the unified retriever-reader for a few more epochs leads to better retrieval and reading performance. While the most simple method is to jointly train the model on the standard reader loss and retriever loss\footnote{The marginal negative log-likelihood of all the correct answer spans in the positive passage and the negative log-likelihood of positive passage $p_1$ being selected, respectively.}, we additionally try iterative finetuning of each of the retriever and reader part as described in Algorithm~\ref{alg:finetuning}. The motivation here is to apply a loose reconstruction constraint $\mathcal{L}_\text{recon}$ to keep the retrieval score as it is before and after the model is optimized for reading, with an assumption that this would be helpful to alleviate the train-inference discrepancy in the input distribution of the reader, created because the unified retriever-reader is not trained using a pipelined manner (training the reader on top of the retrieval result of a fixed retriever).

\begin{algorithm}[ht]
\small
   \caption{\small A single iterative finetuning step on the unified retriever-reader with parameters $\theta$ at time $t$}
   \label{alg:finetuning}

   \hspace*{\algorithmicindent} \textbf{Input}
    $\theta^{(t)}$ (parameters of the model at time $t$), knowledge distillation temperature $\tau$, and training batch of question $q$ and passages $P_q = \{p_i | 1 \leq i \leq m \}$ drawn from $J_\text{read}$ such that $m \geq 2$, $Y(q, p_1) = 1$, and $Y(q, p_i) = 0, \forall 2 \leq i \leq m$. (batch size of 1 is assumed here for a simple presentation) \\
    \hspace*{\algorithmicindent} \textbf{Output}
    Updated parameters $\theta^{(t+1)}$ \\
\begin{algorithmic}[1]
\STATE $\ell^{(t)} \gets [ E_\theta^{(t)} (p_1, 1), \cdots, E_\theta^{(t)} (p_m, 1) ]^\top E_\theta^{(t)} (q, 0)$
\STATE $\hat{\theta}^{(t)} \gets \text{GradientUpdate} ( \mathcal{L}_\text{read}( q, p_1, \cdots, p_m ); \theta^{(t)} )$
\STATE $\hat{\ell}^{(t)} \gets [ E_{\hat{\theta}}^{(t)} (p_1, 1), \cdots, E_{\hat{\theta}}^{(t)} (p_m, 1) ]^\top E_{\hat{\theta}}^{(t)} (q, 0)$
\STATE $\mathcal{L}_\text{recon} \gets D_\text{KL} \left( \text{softmax} (\ell^{(t)} / \tau) || \text{softmax} (\hat{\ell}^{(t)} / \tau) \right) $
\STATE $\mathcal{L}_\text{nll} \gets \text{CrossEntropy} ( \text{softmax} (\hat{\ell}^{(t)}), Y )$
\STATE $\theta^{(t+1)} \gets \text{GradientUpdate} ( \mathcal{L}_\text{recon} + \mathcal{L}_\text{nll} ; \hat{\theta}^{(t)} )$
\end{algorithmic}
\end{algorithm}
\vspace*{-.4cm}

\subsection{Post-Training Compression Techniques}\label{sec:compression}

In addition to the training methods to decrease the corpus, index, and model size, several post-training engineering tricks are applied to compress the system footprint further: (1) INT8 quantization of index items, (2) saving model weights as FP16, (3) resource compression, and (4) utilizing token IDs as the corpus instead of raw texts.

\paragraph{INT8 Quantization of Index Items}
The dense embeddings that serve as the items in the search index are of type FP32 in the default state. INT8 quantization can be applied to reduce the index size by four times with a little bit of drop in the accuracy. We make use of the quantization algorithm implemented in FAISS~\citep{faiss} IndexScalarQuantizer\footnote{\href{https://github.com/facebookresearch/faiss/blob/v1.5.2/\\IndexScalarQuantizer.cpp}{https://github.com/facebookresearch/faiss/blob/v1.5.2/\\IndexScalarQuantizer.cpp}}. During inference, the embeddings are de-quantized, and the search is performed on the restored FP32 vectors.

\paragraph{Saving Model Weights as FP16}
Half precision can be used to size down the model weights of originally FP32 tensors with almost no drop in accuracy. In PyTorch, this can be done by calling \texttt{.half()} on each FP32 tensor in the model checkpoint.

In TensorFlow, model graphs saved as the data type of FP16 may result in unacceptably slow inference according to the used hardware. We have found out that keeping the tensor types of the graph as FP32 but making the actual assigned values as FP16 enables a higher compression ratio when the model weights are compressed as described below.

\paragraph{Resource Compression}
Data compressors with a high compression ratio are effective at reducing the initial system footprint. Our observation is that bzip2 is better for binary files such as model weights or index of embedding vectors, whereas lzma is better for human-readable text files. System resources can also be compressed if necessary. We use \texttt{-9} option for both compressors.

\paragraph{Utilizing Token IDs as the Corpus}
A corpus file must be included in the system to get the actual text of the item retrieved by search (an embedding vector in our case). We have found out that using the file of the encoded token ids of the tokenized texts as the corpus, instead of the raw texts, is beneficial not only because it reduces the inference latency by preprocessing the texts, but also the compressed output size is often slightly smaller.

\section{Experiments}\label{sec:experiments}

\paragraph{Experimental Setup}
We apply our storage reduction methods to a recent extractive retrieve \& read system, DPR~\citep{dpr}, which consists of three different BERT-base encoders: question encoder of the retriever, passage encoder of the retriever, and encoder of the reader with a ranker.
All experiments are done on Naver Smart Machine Learning (NSML) Platform~\citep{nsml1, nsml2}.
The training and evaluation details are in Appendix~\ref{appendix:filter}, \ref{appendix:sharing}, and \ref{appendix:eval}.

\paragraph{Experimental Results}
Figure~\ref{fig:main} shows how each of the discussed strategies changes DPR's system size and Exact Match (EM) score on the EfficientQA dev set (see Table~\ref{table:ablations} and Table~\ref{table:overall} in Appendix for details). Our starting point is a standalone open-domain QA system with DPR whose estimated size is 77.5 GB: 1.4 (system) + 0.8 (retriever) + 0.4 (reader) + 61 (index) + 13 (text) GB. The red plot shows from left to right one path to successively apply each strategy to reduce the system footprint to 484.69MB, which is 160 times smaller.
Although the methods are described as sequential for easier presentation, the methods with filled markers and dotted lines are orthogonal to each other and thus can be applied in any other order. The methods with unfilled markers and solid lines are built on top of the previous method for each.

Sizing down the corpus from 21,015,325 to 1,224,000 (5.8\%) passages (\S 2.1) decreases the system footprint by a large margin of about 70.5GB with only 2.72\% of drop in EM.
Using a smaller passage embedding dimension of 128D (\S 2.2.1), changing the encoder to MobileBERT (\S 2.2.1), and sharing the encoders of the retriever (\S 2.2.2) save further 4.1GB of storage with little drop in accuracy of 1.28\%. The process of unifying the retriever and reader into a single model (\S 2.2.3) drops EM by 1.11, but the accuracy increases by 2.77\% (to 34.44\%) with iterative finetuning (\S 2.2.4).
In ablation studies on the three-step training procedure, omitting the knowledge distillation step drops EM by 1.5\%, and omitting $\mathcal{L}_\text{recon}$ drops EM by 0.38\%.

Applying post-training compression techniques further reduces the system footprint by a large margin while sacrificing little accuracy. EM changes to 34.39\% with INT8 quantization, and the rest of the tricks do not affect the accuracy. Converting the PyTorch checkpoint to a binary for TensorFlow Serving to reduce system library dependency and applying bzip2 compression on some of the system resources creates the final system of 484.69MB with an accuracy of 34.33\%.
Figure~\ref{fig:main} shows that this accuracy is higher than the performance of the parametric T5~\citep{t5} baseline with a comparable docker-level system footprint.\footnote{The accuracy of the T5 baselines are calculated using the SSM models finetuned on Natural Questions: \href{https://github.com/google-research/google-research/tree/master/t5_closed_book_qa\#released-model-checkpoints}{https://github.com/google-research/google-research/tree/ master/t5\_closed\_book\_qa\#released-model-checkpoints}.}

In Table~\ref{table:baseline}, we show the test set accuracy of our final system and other baselines. In summary, the performance of our system is higher than all of the parametric baselines, and the accuracy drop from DPR is only 2.45\% on the EfficientQA dev set and about 4\% on the test set while reducing the system footprint to about 0.6\% of the original size.

Our final system achieves the first place in the human (manual) evaluation and the second place in the automatic evaluation on ``Systems Under 500MB Track'' of the EfficientQA competition. While the accuracy of our system is 32.06\% on the EfficientQA test set in the automatic evaluation, which is 1.38\% behind the top-performing system~\citep{paq}, its accuracy is 42.23\% in the human evaluation which is 2.83\% higher than the other system. Interestingly, when possibly correct answers are also counted as correct, the accuracy rises to 54.95\% (7.58\% higher than the other system). Please refer to Table 2 of \citet{efficientqa} for more details.

In addition to the EfficientQA dataset, we also perform experiments on open-domain Natural Questions (NQ)~\citep{nq} and TriviaQA~\citep{tqa} to test the generalizability of the proposed methods. The results and detailed analysis are presented in Appendix~\ref{appendix:nq_tqa}.

\begin{table}[t]
    \setlength{\tabcolsep}{3.5pt}
    \centering
    \caption[Caption]{\small System size and Exact Match (EM) score of several standalone open-domain QA systems, reported on the test set of EfficientQA. The value of the baseline systems except for DPR are reported in the EfficientQA leaderboard\footnotemark. The system size of DPR is estimated as described in Section~\ref{sec:experiments}.}
    \label{table:baseline}
    \footnotesize
    \begin{threeparttable}
    \begin{tabular*}{0.5\textwidth}{l@{\extracolsep{\fill}}crc}
        \toprule
\textbf{Model}                     & \textbf{EM} & \textbf{System Size} & \textbf{Mechanism} \\
\midrule
T5-1.1-small+SSM & 18 & 486.61 MB & parametric       \\
T5-1.1-XL+SSM    & 28 & 5.65 GB   & parametric       \\
REALM            & 35 & 27.19 GB  & retrieve \& read \\
\midrule
DPR              & 36 & 77.5 GB   & retrieve \& read \\
+ Our Proposed Methods     & 32 & \textbf{484.69 MB} & retrieve \& read \\
        \bottomrule
        \vspace*{-.4cm}
    \end{tabular*}
    \end{threeparttable}
\end{table}
\footnotetext{\href{https://ai.google.com/research/NaturalQuestions/efficientqa}{https://ai.google.com/research/NaturalQuestions/ \\efficientqa}}
\vspace{-0.2em}

\section{Related Works}

There has recently been a line of work that targets to create storage-efficient open-domain QA systems, especially following the EfficientQA competition. Here, we introduce several approaches concurrent to ours that interested readers may refer to. \citet{memory} and \citet{r2d2} explore the trade-off between storage budget and accuracy, and their retrieve \& read systems take up only about 6GB with state-of-the-art performance. \citet{paq} propose a QA-pair retrieval system for open-domain QA, which enjoys the benefits of high flexibility and low latency.
Their retriever answers 1100 questions per second with 41.2\% accuracy on NQ, which rises to 47.7\% when equipped with a reranker. The variants optimized for small system footprint are the winning systems of two storage-constrained tracks at EfficientQA. \citet{efficientqa} review the EfficientQA competition with detailed analysis and summarize all of the top-performing systems.

\section{Conclusion}
We discuss several orthogonal approaches to reduce the system footprint of a retrieve-and-read-based open-domain QA system. The methods together reduce the size of a reference system (DPR) by 160 times with an accuracy drop of 2.45\% and 4\% on EfficientQA dev and test, respectively.
We hope that the presented strategies and results can be helpful for designing future retrieve-and-read systems under a storage-constrained serving environment.

\section*{Acknowledgements}

The authors would like to thank the members of NAVER Clova for proofreading this paper.
This work was supported by Institute of Information \& communications Technology Planning \& Evaluation (IITP) grant funded by the Korea government (MSIT)   (No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)).


\section{Introduction}

\subsection{Context}

Speech is one of the most important medium for human communication, not only conveying linguistic information but also rich and subtle para-linguistic information. In particular the affective prosody of speech conveys the emotions which play an important role in the perception of an utterance meaning. Therefore, speech emotion recognition (SER), which aims to recognize the actual emotional state of a speaker from the utterance he produced, has raised a great interest among researchers.

As pointed out by Scherer in \cite{Scherer1986VocalAE}, there is an undeniable contradiction between the apparent ease with which listeners judge emotions from speech and the intricacy of finding discriminative features in speech signal for emotion recognition. This contradiction, in part, lies in the role that individual difference variables play in the production of emotional speech \cite{Bachorowski1999}. In particular, the acoustic characteristics of the speaker voice as well as its speaking style, for a significant part, determine its way of expressing the emotions.

\subsection{Related works}

Formerly adressed using statistical methods and traditional learning techniques such as Hidden Markov Models (HMMs), Gaussian Mixture Models (GMMs) and Support Vector Machines (SVMs), the SER task has known significant improvements over the past years with the advent of deep neural networks (DNNs). Indeed such deep networks have shown excellent abilities to model more complex patterns within speech utterances by extracting high-level features from speech signal for better recognition of the emotional state of the speakers.

Mao et al. \cite{Mao2014} firstly introduced Convolutional Neural Networks (CNNs) for the SER task and obtained remarkable results on various datasets by learning affective-salient features. Recurrent neural networks (RNNs) has also been introduced for SER purpose with a deep Bidirectional Long Short-Term Memory (BLTSM) network proposed by Lee et al. \cite{Lee2015}. Several papers have then presented CNNs in combination with LSTM cells to improve speech emotion recognition, based on log Mel filter-banks (logMel) \cite{Keren2016} or raw signal in an end-to-end manner \cite{Trigeorgis2016}.

Recently, attention mechanisms have raised great interest in the SER research area for their ability to focus on specific parts of an utterance that characterize emotions. Mirasmadi et al. \cite{Mirsamadi2017} approached the problem with a reccurent neural network and a local attention model used to learn weighted time-pooling strategies. Neumann et al. \cite{Neumann2017} used an attentive CNN (ACNN) and showed the importance of the model architecture choice against the features choice. Ramet et al. \cite{Ramet2018} presented a review of attention models on top of BLSTMs and proposed a new attention computed from the outputs of an added BLSTM layer. Chen et al. \cite{Chen2018} proposed a 3-D Attention-based Convolutional Recurrent Neural Networks (ACRNN) for SER with 3-D log-Mel spectrograms (static, deltas and delta-deltas) as input features. They showed 3-D convolution can better capture more effective information for SER compared with 2-D convolution. Recently, Meng et al. outperformed this method by using dilated convolutions in place of a pooling layer and skip connection.

Attempts to inform emotion classification networks with extra-information involved in the description of emotions were proposed in the past years. Based on previous works \cite{Ververidis2004, Vogt2006, Zhang2018}, Li et al. \cite{Li2019} proposed a multitask learning framework that involves gender classification as an auxiliary task to provide emotion-relevant information leading to significant improvements in SER. Analogously, speaker identity has been used to inform emotion classification networks. The problem was approached by Sidorov et al. \cite{Sidorov2014} with speaker dependent models for emotion recognition. Recently, a method for speaker aware SER was introduced by Assunção et al. \cite{Assuncao2020}, a CNN model VGGVox \cite{Nagrani2017} is trained for speaker identification but is instead used as a front-end for extracting robust features from emotional speech. These first attempts have shown that teaching SER systems with additional signal-based information can greatly improve performances.

\subsection{Contributions of the paper}

In this paper, we assume that individuals may use different means to express emotions, and then that SER should be conditionned on the speaker identity information. Following this hypothesis, we propose a novel method based on previous work \cite{Chen2018}. The major contributions of this paper are summarized as:

\begin{enumerate}
    \item Conditioning emotion classification to speaker identity by using a key-query-value attention we called Self Speaker Attention (SSA) that allows to compute both self and cross-attribute (relation between speaker identity and emotions) attention scores to focus on emotion relevant parts of an utterance.
    \item Proposing a novel regularization by constraining the weights of the last fully connected layer of the network so as to avoid class collapse.
    \item Achieving an absolute increase of UAR by 13.10 \% for IEMOCAP compared with the best existing result.
\end{enumerate}

\section{Proposed Methodology}

In this section, we introduce our SSA-CRNN proposal for SER and a regularization answering class collapse issue. First, the computation of log-Mel spectrograms is described \ref{logmels}, then the basic ACRNN architecture from which our proposal is derived is presented in \ref{ACRNN}. Finally our contributions are detailed : the SSA-CRNN architecture is presented in \ref{SSA-CRNN} and our novel regularization in \ref{reg}.

\begin{figure*}[h!]
    \centering

    \includegraphics[width=\textwidth]{figs/Classifier.pdf}
    \caption{Self Speaker Attentive Convolutional Recurrent Neural Network (SSA-CRNN) for speaker attentive speech emotion recognition composed with two classifiers $C_{sp}$ (up) and $C_{em}$ (bottom) connected through the Self Speaker Attention (SSA) mechanism.}
    \label{SSA-CRNN}
\end{figure*}

\subsection{3-D Log-Mel spectrograms}
\label{logmels}

3-D Log-Mel spectrograms (with delta and delat-deltas), already used as input features for various tasks, were introduced for the SER task by Chen et al. \cite{Chen2018} as input of their ACRNN model and later used in \cite{Meng2019}. In this paper, the Log-Mel spectrograms are computed as presented in \cite{Meng2019}. The 3-D Log-Mel spectrogram consists of a three channel input. The first channel is the static of the Log-Mel spectrogram from 40 filterbanks, the second and third channels are respectively deltas and delta-deltas which can be considered as approximations of the first and second derivatives of the first channel. Once obtained, each 3-D input sample is normalized to have zero mean and unit variance for a given speaker.

\subsection{Architecture of (self) Attentive Convolutional Recurrent Neural Net (ACRNN)}
\label{ACRNN}

Given 3-D log-Mel spectrograms, CRNN is used in \cite{Chen2018} to extract high-level features for speech emotion recognition. The same architecture is used in our proposal to extract high-level features in both cases of SER and speaker recognition (SR).

\subsubsection{Architecture of CRNN}

The CRNN architecture consists of several 3-D convolution layers, one 3-D maxpooling layers, one linear layer and one LSTM layer. Specifically, the first convolutional layer has 128 feature maps, while the remaining convolutional layers have 256 feature maps, and the filter size of each convolutional layer is $5\times3$ , where 5 corresponds to the time axis, and 3 corresponds to the frequency axis. A max-pooling is performed after the first convolutional with pooling size is $2\times2$. The 3D features are reshaped to 2D, keeping time dimension unchanged and passed to a linear layer for dimension reduction before reaching the recurrent layer. As precised in \cite{Chen2018}, a linear layer of 768 output units is shown to be appropriate. These features are then processed through a bi-directional recurrent neural network with long short term memory cells (BLSTM) [19], with 128 cells in each direction,  for temporal summarization, which allows to get 256-dimensional high-level feature representations.

\subsubsection{Self Attention (SA) mechansim}

With a sequence of high-level representations, an attention layer is employed to focus on relevant features and produce discriminative utterance-level representations for classification, since not all frame-level CRNN features contribute equally to the representation of the attributes to recognize, in both cases of SER and speaker recognition.

Specifically, with the classifier's BLSTM output $\mathbf{h}_{att} = [h_{att}^{1}, ..., h_{att}^{T}] \in \mathbb{R}^{T \times d}$, a temporal vector $\alpha_{att} \in \mathbb{R}^{T}$, representing the contribution per frame to the attribute to recognize, is computed depending on learnt weights vector $W_{att} \in \mathbb{R}^{d}$. Then $\alpha_{att}$ is used to obtain an utterance-level representation by computing the weighted sum of temporal BLSTM internal states $c_{att}$ often called context vector.

\vspace*{-\baselineskip}

\begin{align}
    \alpha_{att} &= \frac{\exp(\mathbf{h}_{att}W_{att})}{\sum_{t=1}^{T}  \exp(W_{att}h_{att}^{t})} \in \mathbb{R^{T}} \\
    c_{att} &= \sum_{t=1}^{T} \alpha_{att}^{t}h_{att}^{t}
\end{align}

The attention layer is followed by a fully connected layer that determines the embedding size.

\subsection{Architecture of the proposed Self Speaker Attentive Convolutional Neural Net (SSA-CRNN)}
\label{SSA-CRNN}

Our system shown in Figure \ref{SSA-CRNN} is formed by two classifiers $C_{sp}$ and $C_{em}$. The first is an ACRNN and trained for SR. The second has the same architecture apart from its Self Speaker Attention (SSA) layer and is trained (once $C_{sp}$ has been trained) for SER, conditionally to the speakers embedding produced by $C_{sp}$ and through the SSA mechanism described further. This conditioning of SER to speaker identity is expected to help $C_{em}$ to better capture speakers individual strategies for emotion expression leading to better global SER performances.

\subsubsection{Self Speaker Attention (SSA) mechanism}

\begin{figure}[H]
    \centering
    \includegraphics[width=0.48\textwidth]{figs/SSA.pdf}
    \caption{Self Speaker Attention (SSA) Mechanism}
    \label{al}
\end{figure}

Inspired by \cite{Pan2020} that used multi-modal attention technique for SER, we employed the query-key-value representation to compute the attention from two inputs : $\mathbf{h}_{em}$ and $\mathbf{h}_{sp}$ respectively related to "self" and "speaker" aspects. We first compute the query of speech emotion $q_{em}$ through learnable weights $W_{em}^{q} \in \mathbb{R}^{T}$.

\vspace*{-\baselineskip}

\begin{equation}
    q_{em} = \mathbf{h}_{em}W_{em}^{q}
\end{equation}

The keys $K$ and values $V$ are computed using learnable
weights $W_{em}^{k}, W_{sp}^{k}, W_{em}^{v}, W_{sp}^{v} \in \mathbb{R}^{T}$ as follows.

\vspace*{-\baselineskip}

\begin{align}
    K &= concat\{\mathbf{h}_{em}W_{em}^{k}, \mathbf{h}_{sp}W_{sp}^{k}\} \\
    V &= concat\{\mathbf{h}_{em}W_{em}^{v}, \mathbf{h}_{sp}W_{sp}^{v}\}
\end{align}

The cross-attribute and self attention scores are computed by the product of the query $q_{em}$ and transposed keys $K^{T}$. It is then normalized, passed to a softmax activation layer and finally multiplied with values $V$ to obtain $\alpha_{em} \in \mathbb{R}^{T}$ which represents the interaction of speaker identity and speech emotion answering to speech emotion query. This temporal vector $\alpha_{em}$ is then used to compute the weighted sum of temporal BLSTM internal states $c_{em}$.

\vspace*{-\baselineskip}

\begin{align}
    \alpha_{em} &= softmax\left(\frac{q_{em}K^{T}}{\sqrt{T}}\right)V \\
    c_{em} &= \sum_{t=1}^{T} \alpha_{em}^{t}h_{em}^{t}
\end{align}

\subsection{Weights regularization}
\label{reg}

During preliminary experiments with ACRNN on the large database Att-HACK presented in \ref{datasets}, we faced a class collapse issue : the model tended to focus on one or two of the four classes at the expense of the others. That led to have one or several categories very badly recognized. In order to avoid this pitfall, we elaborated a training regularization by constraining the weights $W \in \mathbb{R}^{n_{em} \times 4}$ of the last fully connected layer $FC^{c}$, also called classification layer, just before the \textit{softmax} activation. Denoting $W^{c} \in \mathbb{R}^{n_{em}}$ the c-ieth column of $W$, $N$ the batch size, then for all input batch $\mathbf{x} = [x_{1}, ... , x_{N}]$, $FC^{c}$ is being fed with, the constraint ensuring all classes are equi-outputed  can be expressed as follows.

\begin{equation}
   \lVert W^{c} \rVert_{1} = N\left( 4\lVert\sum\limits_{i=1}^{N}x_{i}\rVert_{1}\right)^{-1}
\end{equation}

\section{Experiments}

\subsection{Datasets}
\label{datasets}

To evaluate the performance of our proposed model, we perform speaker-independent SER experiments on the speech database for speech social attitudes Att-HACK \cite{Le_Moine_2020} and the Interactive Emotional Dyadic Motion Capture database (IEMOCAP) \cite{IEMOCAP}. Att-HACK comprises 20 speakers interpreting  100  utterances in  4  social  attitudes  :  \textit{friendly}, \textit{distant}, \textit{dominant}  and  \textit{seductive}. With 3 to 5 repetitions each per attitude for a total of around 30 hours of speech, the database offers a wide variety of prosodic strategies in the expression of attitudes. IEMOCAP consists of 5 sessions and each session is displayed by a pair of speakers (female and male) in scripted and improvised scenarios. For this experiment, only improvised data and 4 emotions, \textit{happy}, \textit{angry}, \textit{sad} and \textit{neutral} were considered.

\subsection{Experiment Setup}
\label{Experimental Setup}
As for the feature extraction, we split the speech signal
into equal-length segments of 3 seconds for better parallel acceleration and adopt zero-padding for utterances not reaching the 3 seconds.

\begin{itemize}
    \item \textbf{ACRNN} :  Re-implemented strictly following \cite{Chen2018} apart from the embedding size that we increased from $64$ to $128$. Our proposed systems are derived from this baseline SER system $C_{em}$.
    \item \textbf{ACRNN-r} : Addition of a regularization (described in \ref{reg}) to $C_{em}$ model to avoid class collapse.

    \item \textbf{SSA-CRNN-r} : A first ACRNN model $C_{sp}$ (embedding size $64$) is trained for speaker recognition to learn high-level features that define the speaker identity, and from which our SSA mechanism will extract emotion salient information to inform the SER task. Once best accuracy is reached, the model is saved and fixed. Then a second ACRNN model $C_{em}$ (embedding size $128$) is trained conditionally to the BLSTM's outputs of $C_{sp}$ by mean of the SSA mechanism for SER task.
\end{itemize}

For evaluations, 10-fold cross validation technique is performed. Respectively for IEMOCAP and Att-HACK, 9 and 18 speakers are considered for training while the remaining ones are kept for validation. In the case of Att-HACK validation sets are balanced in gender (male-female couples) apart from 2 folds, as the database includes 12 females and only 8 males. As we aspire to perform emotion recognition independently of the speaker, we must consider two approaches for training the SSA-CRNN-r $C_{sp}$ module.

\begin{enumerate}
    \item \textbf{speaker-dependant approach} : a \textit{train/valid} split have been performed linguistically on the whole database, all speakers have been seen by $C_{sp}$. Consequently, the SER task is not performed independently of the speaker and there is no insurance the model will be able to generalize to speakers it has never seen.
    \item \textbf{speaker-independant approach} : a LOSO (Leave-One-Speaker-Out) protocol is followed so as to ensure that the speaker (or speakers for Att-HACK) on which the model is validated has never been seen by the model. For each fold $k \in [1, 10]$ the speaker $s_{k}^{v}$ (or speakers $[s_{k}^{v_{1}}, s_{k}^{v_{2}}]$) chosen to be in the validation set are left out of the database for the training and validation of $C_{sp}$ .
\end{enumerate}

Each module is optimized with respect to the cross-entropy objective function. Trainings are done with emotionally balanced mini-batches of 40 samples, using the Adam optimizer with Nestorov momentum. The initial learning rate is set to 0.0001 and the momentum is set to 0.9. The final model parameters are selected by maximizing the Unweighted Average Recall (UAR) on the validation set.

\subsection{Experiment Results}

Table \ref{results} is divided in two parts, the first part shows the results in UAR on IEMOCAP of state-of-the-art methods for SER, including the ACRNN system by \cite{Chen2018} and the gender-informed system by \cite{Zhang2018}. The second part shows the UAR results of our proposed methods on both Att-HACK and IEMOCAP databases. First, we compare our ACRNN re-implementation with the state-of-the-art method described in \cite{Chen2018}. Our re-implementation achieves slightly better performance for IEMOCAP with an absolute improvement of 1.64\% and a rather low standard deviation possibly due to the increased embedding size.

\vspace*{-\baselineskip}

\begin{table}[h!]
    \center
    \begin{tabular}[b]{lcc}

    & \multicolumn{2}{c}{} \\
    \hline
    \textbf{SER system} & \textbf{Att-HACK} & \textbf{IEMOCAP} \\
    \hline \hline
    Chen et al. \cite{Chen2018} & $-$ & $64.74 \pm 5.44$ \\
    Meng et al. \cite{Meng2019} & $-$ & $69.32 \pm 3.76$ \\
    Zhang et al. \cite{Zhang2018} & $-$ & $82.80 \pm -$ \\
    \hline
    ACRNN & $27.49 \pm 3.26$ & $66.38 \pm 2.72$  \\
    ACRNN-r & $66.36 \pm 5.68$ &  $86.35 \pm 3.21$ \\
    SSA-CRNN-r & $88.93 \pm 4.36$ & $96.12 \pm 4.27$ \\
    SSA-CRNN-r (LOSO) & $\mathbf{88.31} \pm \mathbf{5.60}$ & $\mathbf{95.90} \pm \mathbf{4.31}$ \\

    \end{tabular}
    \caption{SER results with 95\% confidence interval for different methods on Att-HACK and IEMOCAP in terms of UAR}
    \label{results}
\end{table}

\vspace*{-\baselineskip}

Next, we investigate the effectiveness of our proposed regularization. Compared with ACRNN, ACRNN-r obtains an absolute improvement of 38.47\% on Att-HACK and 19.97\% for IEMOCAP. This shows our regularization brings stability during training that allows the model to further recognize more subtle emotions. The confusion matrices depicted in Figure \ref{cm} also show the model is no longer over-focusing on one class, all the other classes are confused, with which was the case for dominant attitude in Att-HACK with the ACRNN.

To evaluate the addition of our proposed SSA mechanism, it is legitimate to chose the approach with LOSO protocol described in \ref{Experimental Setup}. Compared with ACRNN-r, this approach achieves an absolute improvement of 21.95 \% for Att-HACK and 9.55 \% for IEMOCAP. This shows combining emotion classification and speaker recognition through our SSA mechanism is relevant and lead to better capture speaker means of emotional expression. This approach achieves a global absolute improvement of 60.82 \% for Att-HACK and 29.52 \% for IEMOCAP compared with ACRNN. Compared with the gender-informed SER system by Zhang et al. \cite{Zhang2018}, it achieves 13.10 \% of UAR improvement on IEMOCAP. As expected, the speaker-dependant approach achieves slightly better performance than the LOSO-based one although it does not seem significant. It appears that $C_{sp}$ is able to generalize enough for new speakers so as the SSA can capture the speaker high-level features that are relevant in the representation of speech emotions.

\vspace{-2.6mm}

\begin{figure}[!htbp]

\begin{minipage}[c]{.49\linewidth}
     \begin{center}
             \includegraphics[width=2.5cm]{figs/Att-HACK_em_unreg.pdf}
         \end{center}
   \end{minipage} \hfill
   \begin{minipage}[c]{.49\linewidth}
    \begin{center}
            \includegraphics[width=2.5cm]{figs/Att-HACK_em_sp_attentive_reg_LOSO.pdf}
    \end{center}
 \end{minipage}
\begin{minipage}[c]{.49\linewidth}
     \begin{center}
             \includegraphics[width=2.5cm]{figs/IEMOCAP_em_unreg.pdf}
         \end{center}
   \end{minipage} \hfill
   \begin{minipage}[c]{.49\linewidth}
    \begin{center}
            \includegraphics[width=2.5cm]{figs/IEMOCAP_em_sp_attentive_reg_LOSO.pdf}
    \end{center}
 \end{minipage}
 \captionof{figure}{SER normalized confusion matrices where each row presents the class dependant prediction performance.}
 \label{cm}
\end{figure}

Finally, the t-SNE analysis in Figure \ref{tsne} tend to show both attitudes and emotions are well separated in their respective embeddings. Although this point would need to be deeper investigated, we can further expect to use these embeddings in a voice conversion context as proposed by Zhou et al. in \cite{Zhou2021}.

\vspace{-2mm}
\begin{figure}[!htbp]
\begin{center}
        \includegraphics[width=0.47\textwidth]{figs/TSNE.pdf}
\end{center}
\caption{T-SNE representations obtained from emotion embeddings, for SSA-CRNN-r (LOSO).}
\label{tsne}
\end{figure}

\vspace*{-\baselineskip}

\section{Conclusions}

In this paper, we propose a Self Speaker Attentive Convolutional Neural Network (SSA-CRNN) model to train the SER model operating on 3D Log-Mel spectrograms and a novel training regularization that allows to avoid class collapse. The evaluation on Att-HACK and IEMOCAP databases demonstrates that the proposed method outperforms the state-of-the-art methods. Notably, the evaluation on IEMOCAP achieves 13.10 \% absolute improvement of UAR compared with the gender-informed SER system by Zhang et al. \cite{Zhang2018}.

\section{Acknowledgements}
\label{sec:akn}
This research is supported by the MoVe project: ‘‘MOdelling of speech attitudes and application to an expressive conversationnal agent'', and funded by the Paris Region Ph2D grant.

\bibliographystyle{IEEEtran}