\section{Introduction}
\IEEEPARstart{R}{everberation} plays an important role in the perceived quality
of a sound signal produced in an enclosed environment. In highly reverberant environments, perceptual artifacts
such as coloration and echoes are added to the direct sound signal, thus drastically reducing speech signal intelligibility, particularly for the hearing impaired \cite{neuman_combined_2010}.  Automatic speech recognition (ASR) performance is also
severely affected, especially when reverberation is combined with additive noise \cite{yoshioka_making_2012}. To deal with
the distortions in such environments, several types of speech enhancement
systems have been proposed, ranging from single-channel systems based on simple
spectral subtraction \cite{boll_spectral_1979} to multistage systems which leverage
signals from multiple microphones \cite{cauchi_combination_2015}.

Deep neural networks (DNNs) are currently part of many large-scale ASR systems, both as separate acoustic and
language models as well as in end-to-end systems. To make these systems robust to reverberation, different strategies have been explored, such as feature enhancement during a preprocessing stage \cite{mimura_reverberant_2015} and DNN-based beamforming on raw multichannel speech signals for end-to-end solutions
\cite{sainath_factored_2016}.
On the other hand, the application of DNNs to more
general speech enhancement problems such as denoising
\cite{xu_regression_2015}, dereverberation \cite{han_learning_2015}, and source
separation \cite{weninger_discriminatively_2014} is comparatively in early
stages.

Recently, several works have explored deep neural networks for speech enhancement through two main approaches: spectral estimation and spectral masking. In the first, the goal of the neural network is to predict the magnitude spectrum of enhanced speech signal directly, while the latter aims at predicting some form of an ideal mask (either a binary or a ratio mask) to be applied to the distorted input signal.

Most of the published work in the area uses an architecture similar to the one first presented in \cite{han_learning_2015}: a relatively large feedforward neural network with three hidden layers containing several hundred units (1600 in \cite{han_learning_2015}), having as input a context window containing an arbitrary number of frames (11 in \cite{han_learning_2015}) of the log-magnitude spectrum and as target the dry/clean center frame of that window. The output layer uses a sigmoid activation function, which is bounded between 0 and 1, and normalizes targets between 0 and 1 using the minimum and maximum energies in the data. Moreover, an iterative signal reconstruction scheme inspired by \cite{griffin_signal_1984} is used to reduce the effect of using the reverberant phase for reconstruction of the enhanced signal.

In \cite{wu_study_2016}, the authors performed a study on target feature activation and normalization and their impacts on the performance of {DNN} based speech dereverberation systems. The authors compared the target activation/normalization scheme in \cite{han_learning_2015} with a linear (unbounded) activation function and output normalized by its mean and variance. Their experiments showed the latter activation/normalization scheme leads to higher PESQ and frequency-weighted segmental SNR (fwSegSNR) scores than the sigmoid/min-max scheme.

In a follow-up study \cite{wu_reverberation-time-aware_2017}, the same authors proposed a
reverberation-time-aware model for dereverberation that leverages knowledge of
the fullband reverberation time (T60) in two different ways. First, the step size of the short-time Fourier transform (STFT) is
adjusted depending on T60, varying from 2 ms up to 8 ms (the window size is
fixed at 32 ms). Second, the frame context used at the input of the network is
also adjusted, from 1 frame (no context) up to 11 frames (5 future and 5 past
frames). Since the input of the network has a fixed size, the context length is
adjusted by zeroing the unused frames. The model was trained using speech from
the TIMIT dataset convolved with 10 room impulse responses (RIR) generated at a room with fixed geometry
(6 by 4 by 3 meters), with T60 ranging from 0.1 to 1.0 s. The full training
data had about 40 hours of reverberant speech, but the authors also presented
results on a smaller subset with only 4 hours. The model proposed was a feedforward neural network with 3
hidden layers with 2048 hidden units each, trained using all the
different step sizes and frame context configurations in order to find the best
configuration for each T60 value. The authors considered the true T60 value to be
known at test time (oracle T60), as well as estimated using the T60 estimator
proposed by Keshavarz et al. \cite{keshavarz_speech-model_2012}.

The other well-known approach for speech enhancement using deep neural networks is to predict arbitrary ideal masks instead of the magnitude spectrum \cite{wang_training_2014}. The ideal binary mask (IBM) target transforms the speech enhancement problem into a classification problem, where the goal of the model is to predict which time-frequency cells from the input should be masked, and has been shown to improve intelligibility substantially. The ideal binary mask is defined quantitatively based on a local criterion threshold for the signal-to-noise ratio (SNR). Namely, if the SNR of a given time-frequency cell is lower than the threshold, that time-frequency cell is masked (set to zero). Alternatively, the ideal ratio mask is closely related to the frequency-domain Wiener filter with uncorrelated speech and noise. It is a soft masking technique where the mask value corresponds to the local ratio between the signal and the signal-plus-noise energies for each time-frequency cell. Recently, \cite{williamson_complex_2016} has proposed the complex ideal ratio mask, which is applied to the real and imaginary components of the STFT instead of just the magnitude. Models based on mask prediction usually include several different features at the input (such as amplitude modulation spectrograms, RASTA-PLP, MFCC, and gammatone filterbank energies), instead of using just the magnitude spectrum from the STFT representation like the works previously described here. Masks are also often predicted in the gammatone filterbank domain.

In \cite{xiao_speech_2016}, the authors present a model that predicts
log-magnitude spectrum and their delta/delta-delta, then perform enhancement by
solving a least-squares problem with the predicted features, which aims at
improving the smoothness of the enhanced magnitude spectrum. The method was
shown to improve cepstral distance (CD), SNR, and log-likelihood ratio (LLR), but caused slight degradation of the speech-to-reverberation modulation energy ratio (SRMR).
They also reported that the DNN mapping causes distortion for high T60.

Very few studies use architectures other than feed-forward for dereverberation. In \cite{kawahara_speech_2015}, the authors propose an architecture based on long short-term memory (LSTM) for dereverberation. However, they only report mean-squared error and word-error rates for a baseline ASR system (the REVERB Challenge evaluation system) and do not report its effect on objective metrics for speech quality and intelligibility. The method described in \cite{weninger_discriminatively_2014} also uses recurrent neural networks based on LSTMs for speech separation in the mel-filterbank energies domain. Although their system predicts soft masks, similar to the ideal ratio mask, they use a signal-approximation objective instead of predicting arbitrary masks (i.e., the target are the mel-filterbank features of the clean signal, not an arbitrarily-designed mask).

Most current models reported in the literature only explore one of two possible contexts from the reverberant signal. Feed-forward models with a fixed window of an arbitrary number of past and future frames only take into account the local context (e.g. \cite{wu_reverberation-time-aware_2017}) and are unable to represent the long-term structure of the signal. Also, since feed-forward models do not have an internal state that is kept between frames, the model is not aware of the frames it has predicted previously, which can lead to artifacts due to spectral discontinuities. LSTM-based architectures, on the other hand, are able to learn both short- and long-term structure. However, learning either of these structures is not enforced by the training algorithm or the architecture, so one cannot control whether the internal state will represent short-term, long-term context, or both.

In this paper, we propose a novel architecture for speech dereverberation that leverages both short- and long-term context information. First, fixed local context information is generated directly from the input sequence by a convolutional context encoder. We train the network to learn how to use long-term context information by using recurrent layers and training it to enhance entire sentences at once, instead of a single frame at a time. Additionally, we leverage residual connections from the input to hidden layers and between hidden layers. We show that combining short and long-term contexts, as well as including such residual connections, substantially improves the dereverberation performance across four different objective speech quality and intelligibility metrics (PESQ, SRMR, STOI, and POLQA), and also reduces the amount of perceived reverberation according to subjective tests.

\section{Proposed model}
The architecture of the proposed model can be seen in Fig.~\ref{fig:proposed}. As discussed previously, our model combines both short- and long-term context by using a convolutional context encoder to create a representation of the short-term structure of the signal, and a recurrent decoder which is able to learn long-term structure from that representation. The decoder also benefits from residual connections, which allow each of its recurrent stages to have access both to a representation of the input signal and the state of the previous recurrent layer. Each of the blocks is further detailed in the sections to follow.
\begin{figure}
    \centering
    \includegraphics[width=0.45\textwidth, height=.5\textheight]{img/model.pdf}
    \caption{Architecture of the proposed model}
    \label{fig:proposed}
\end{figure}
\subsection{Context encoder}
As shown in other studies, incorporating past and future frames can help on the
task of estimating the current frame for dereverberation. Most works, however,
use a fixed context window as the input
to a fully-connected layer \cite{han_learning_2015, wu_study_2016}. In this work, we decided to extract local context
features using 2D convolutional layers instead. By using a 2D convolutional
layer, these features encode local context both in the frequency and the time
axis. Our context encoder is composed by a single 2D convolutional layer with
64 filters with kernel sizes of $(21, C)$, where 21 corresponds to the number of frequency bins covered by the kernel and $C$ to the number of frames covered by the kernel in the time axis. In our implementation,
$C$ is always an odd number as we use an equal number of past and future
frames in the context window (e.g., $C = 11$ means the current frame plus 5
past and 5 future frames).  We report the performance of the model for
different values of $C$ in section IV-A. The convolution has a stride of 2 in the frequency
axis and is not strided in the time axis.

\subsection{Decoder}
Following the encoder, we have a stack of three gated recurrent unit (GRU) layers \cite{Cho14} with
256 units each.  The input to the first layer is the output of the convolutional context encoder with all of the channels concatenated to yield an input of shape $(F, T)$, where
\begin{equation*}
    F = 64 \times \left\lfloor\frac{B - N_{context} + 1}{2} +1 \right\rfloor,
\end{equation*}
\noindent 64 is the number of filters in the convolutional layer, $T$ is the number of STFT frames in a given sentence, $B$ is the number of FFT bins (257 in our experiments), and $\lfloor . \rfloor$ is
the \emph{floor} operation (rounds its argument down to an integer).

The outputs to the remaining GRU layers are a combination of affine
projections of the input and the states of the previous GRU layers. Consider
$x_{enc}(t)$ to be the encoded input at timestep $t$, and $h_1(t), h_2(t),
h_3(t)$ as the hidden state at timestep $t$ for the first, second, and third
GRU layers, respectively.  Then, the inputs $i_1(t), i_2(t), i_3(t)$ are as
follows:
\begin{align*}
    i_1(t) &= x_{enc}(t), \\
    i_2(t) &= f_{2}(x(t)) + g_{1,2}(h_1(t)), \\
    i_3(t) &= f_{3}(x(t)) + g_{1,3}(h_1(t)) + g_{2,3}(h_2(t)),
\end{align*}
\noindent where $f_{i}, g_{i,j}$ are affine projections from the input or
previous hidden states. The parameters of those projections are learned during
training, and all projections have an output dimension of 256 in order to match
the input dimension of the GRUs when added together.

Similarly, the output layer following the stacked GRUs has as its input the sum
of affine projections $o_i$ of $h_1, h_2, h_3$:
\begin{align*}
    i_{\text{out}}(t) &= o_1(h_1(t)) + o_2(h_2(t)) + o_3(h_3(t)).

\end{align*}

\noindent The parameters of those projections are also learned during training.
All of these projections have an output dimension of 256.

\section{Experimental setup}
\subsection{Datasets}
In order to assess the benefits of the proposed architecture for speech
dereverberation, we ran a series of experiments with both the proposed model
and two other models as baselines: the T60-aware model proposed in
\cite{wu_reverberation-time-aware_2017} and a similar model without T60
information that uses a fixed overlap of 16 ms and a fixed context of 11
frames (5 past and 5 future frames). For the T60 aware model, we extracted T60
values directly from the RIRs (oracle T60s) using a method similar to the one used for the ACE Challenge dataset \cite{AceChallenge, karjalainen_estimation_2002}. The
fullband T60 we used was computed as the average of the estimates for the bands
with center frequencies of 400 Hz, 500 Hz, 630 Hz, 800 Hz, 1000 Hz, and 1250
Hz.

For the single speaker experiments, a recording of the IEEE dataset uttered by a
single male speaker was used \cite{loizou_speech_2005}. The dataset consists of 72 lists with 10
sentences each recorded under anechoic and noise-free conditions. We used the
first 67 lists for the training set and the remaining 5 lists for testing.
Reverberant utterances were generated by convolving randomly selected subsets
of the utterances in the training set with 740 RIRs generated using a fast
implementation of the image-source method \cite{lehmann_diffuse_2010}, with T60
ranging from 0.2~s to 2.0~s in 0.05~s steps. Twenty different RIRs (with different
room geometry, source-microphone positioning and absorption characteristics)
were generated for each T60 value. Fifty random utterances from the training
set were convolved with each of these 740 RIRs, resulting in 37,000 files. A
random subset of 5\% of these files was selected as a validation set and used
for model selection and the remaining 35,150 files were used to train the
models. The test set was generated in a similar way, but using a different set
of 740 simulated RIRs and 5 utterances (randomly selected from the test lists)
were convolved with each RIR.

For the multi-speaker experiments, we performed a similar procedure but using the
TIMIT dataset \cite{timit} instead of the IEEE dataset. The default training
set (without the ``SA" utterances, since these utterances were recorded by all speakers) was used for generating the training and validation sets, and the test set (with the SA utterances removed as well) was
used for generating the test set. The training and test sets had a total of 462 and 168 speakers, respectively. The utterances were convolved with the same
RIRs used for the single speaker experiments. A total of 3696 clean utterances
were used for the training and validation set, and 1336 for the test set. As
with the single speaker dataset, 50 sentences were chosen at random from the training and validation sets (which include all of the utterances from all speakers) and convolved to each of the 740 simulated RIRs to generate the training/validation sets. The test set was generated following the same procedure as used for the single speaker dataset.

Additionally, in order to explore the performance of the proposed and baseline models on realistic settings, we tested the same single speaker models described above with sentences convolved with real RIRs from the ACE Challenge dataset
\cite{AceChallenge}. We used the RIRs corresponding to channels 1 and 5 of the
cruciform microphone array for all of the seven rooms and two microphone positions,
leading to a total of 28 RIRs. The test sentences were the same as for the
experiments with simulated RIRs. The interested reader can refer to \cite{AceChallenge} for more details about the ACE Challenge RIRs.

\subsection{Model Training}
Both the proposed model and the baselines were implemented using the PyTorch library \cite{pytorch} (revision \emph{e1278d4}) and trained using the Adam optimizer \cite{kingma2014adam} with
a learning rate of 0.001, $\beta_1 = 0.9$, and $\beta_2 = 0.999$. The models
were trained for 100 epochs and the parameters corresponding to the epoch with
the lowest validation error were used for evaluation. Since sequences in the
dataset have different lengths, we padded sequences in each minibatch and used masking to compute the MSE loss only for valid timesteps. The models used for evaluation were the ones with the lowest
validation loss amongst the 100 epochs.

\subsection{Objective evaluation metrics}
\label{sec:objmetrics}
We compare the performance of the models using four different objective metrics. The Perceptual Evaluation of Speech Quality (PESQ) \cite{itut_pesq_2001} and Perceptual Objective
Listening Quality Assessment (POLQA) \cite{POLQA_2011} are both ITU-T standards for intrusive speech quality measurement. The PESQ standard was designed for a very limited test scenario (automated assessment of band-limited speech quality by a user of a telephony system), and was superseded by POLQA. The two metrics work in a similar way, by computing and accumulating distortions, and then mapping them into a five-point mean opinion score (MOS) scale. Even though PESQ is not recommended as a metric for enhanced or reverberant speech, several works report PESQ scores for these types of processing and we report it here for the sake of completeness. POLQA is a more complete model and allows measurement in a broader set of conditions (e.g., super-wideband speech, combined additive noise and reverberation). The Short-Time Objective Intelligibility (STOI) \cite{taal2011algorithm} metric is an intrusive speech intelligibility metric based on the correlation of normalized filterbank envelopes in short-time (400 ms) frames of speech. Finally, the speech-to-reverberation modulation energy ratio (SRMR) is a non-intrusive speech quality and intelligibility metric based on the modulation spectrum characteristics of clean and distorted speech, which has been shown to perform well for reverberant and dereverberated speech \cite{Santos_NH_2014,falk_non-intrusive_2010}.

\subsection{Subjective listening tests}
\label{sec:listening}
To further gauge the benefits of the proposed architecture for speech enhancement, we performed a small-scale subjective listening test to assess how effective different methods are in reducing the amount of perceived reverberation. In particular, comparisons with the baseline model Wu2016 were performed as it resulted in improved performance with unmatched speakers relative to the benchmark Wu2017 model. The test protocol we used was the recently proposed MUSHRAR test \cite{mushrar}.

For the reverberation perception tests, five random samples of the TIMIT test dataset were chosen for each of simulated RIRs with T60s of 0.6, 0.9, 1.2, and 1.5~s and participants were asked to compare the outputs of all models and the reverberant signal to a reference signal. In addition to the model outputs and the reverberant signal, a hidden reference and anchor were used for both experiments: the reference signal was the anechoic signal convolved with an RIR with T60 of 0.2~s and the anchor was the anechoic signal convolved with a RIR with a T60 of 2.0~s. Users were asked to rate the amount of perceived reverberation on a 0-100 scale, with higher values corresponding to higher perceived reverberation. A total of nine participants took part in the test. Tests were performed using a web interface which is freely available online\footnote{Software available at \url{https://github.com/jfsantos/mushra-ruby}}.

\section{Experimental Results}
In this Section, we present the results for four different experiments, namely:
\begin{enumerate}[label=\Alph*.]
\item Evaluation of the effect of the context size (using the single-speaker, simulated RIR dataset)
\item Comparison between the proposed architecture and baselines on matched conditions (same speaker, simulated RIRs for training and testing)
\item Comparison between the proposed architecture and baselines on mismatched speakers (multispeaker dataset with simulated RIRs)
\item Comparison between the proposed architecture and baselines under realistic reverberation conditions (models trained on single-speaker, simulated RIR dataset, and tested on single-speaker, real RIR dataset).
\end{enumerate}

Since some of the models have a large difference in the number of
learnable parameters, we list the number of parameters for each of the models
that was used in our experiments in Table \ref{tab:nparams}.

\begin{table}
    \centering
    \caption{Number of parameters in each model}
    \label{tab:nparams}
    \begin{tabular}{cc}
        Model & \# of parameters \\
        \hline
        GRU & 4,458,753 \\
        Wu2016 and Wu2017 & 14,711,041 \\
        Proposed without context & 1,838,593 \\
        Proposed, context = 3 frames & 7,429,121 \\
        Proposed, context = 7 frames & 7,434,497 \\
        Proposed, context = 11 frames & 7,439,873
    \end{tabular}
\end{table}

\subsection{Effect of context size}
We first analyze the effect of the context size in the proposed model by
comparing models with context sizes of 3, 7, and 11. We also included a
baseline model based on GRUs without residual connections and a model without
the context encoder and residual connections in these tests. All the models
were trained and tested with single-speaker data convolved with simulated RIRs.
The results can be seen in Figure \ref{fig:exp2}. For this and all of the subsequent experiments in this section, we have subplots for the four objective metrics listed in Section \ref{sec:objmetrics}: (a) PESQ, (b) SRMR, (c) STOI and (d) POLQA in this order. The GRU-only model (without
the context encoder and residual connections) underperforms in all metrics, and
in the cases of PESQ and POLQA, even leads to lower scores than the reverberant
utterances. Adding only the residual connections and no context at all (which
allows the model to perform at real-time) significantly increases the
performance and leads to improvements in all metrics, except for reverberation
times under 400 ms (noticeable in both PESQ and POLQA). Adding the context
encoder, even with a short context of one previous and one future frame (shown
as ``3 frames" in the plots), leads to additional improvements. Adding more
frames (``7 frames" and ``11 frames") improves the performance even further.
However, we see diminishing returns around 7-11 frames, as these two models
have similar performances for SRMR and STOI and only a slight difference in
PESQ and POLQA. Since the model with a context of 11 frames (5 past and 5
future frames) showed the best performance amongst all context sizes, in the
next experiments we show only results with this context size for the proposed
model.

\begin{figure*}
    \centering
    \subfloat[(a)]{%
        \includegraphics[width=.49\textwidth]{img/exp2_pesq.png}}
        \hfill
    \subfloat[(b)]{%
        \includegraphics[width=.49\textwidth]{img/exp2_srmr.png}}
        \hfill
    \subfloat[(c)]{
        \includegraphics[width=.49\textwidth]{img/exp2_stoi.png}}
    \hfill
    \subfloat[(d)]{%
        \includegraphics[width=.49\textwidth]{img/exp2_polqa.png}}
    \hfill
    \caption{Effect of context size on scores: (a) PESQ, (b) SRMR,
    (c) STOI, and (d) POLQA.}
    \label{fig:exp2}
\end{figure*}

\subsection{Comparison with baseline models}
Figure \ref{fig:exp1} shows a comparison between our best model, Wu2016 (fixed
STFT window and hop sizes) and Wu2017 (STFT window and hop sizes dependent on
the oracle T60 values). The interested reader is referred to this manuscript’s supplementary material page to listen to audio samples generated by the proposed and benchmark algorithms \footnote{\url{http://www.seaandsailor.com/demo/index.html}}. A more complete audio demo can be found in \cite{figshare_demo}. It can be seen that our model outperforms these two feed-forward models in
most scenarios and metrics, except for SRMR with $\text{T60} < 0.7$~s, where the results
are very similar. It should be noted, however, that the SRMR metric is less accurate for lower T60s \cite{Santos_NH_2014}.
Even though it uses oracle T60 information, the model Wu2017
does not have a large improvement in metrics when compared to Wu2016,
especially in the STOI and POLQA metrics (which are more sensitive to the
effects of reverberation than PESQ). Using T60 information to adapt the STFT
representation seems to have a stronger effect for lower T60. It should also be
noted that these models lead to a reduction in PESQ scores for lower
reverberation times in the PESQ and POLQA metrics, which is probably due to the
introduction of artifacts. This is also observed for the proposed model, but
only in the PESQ metric and only for $\text{T60} < 0.4$~s. On the other hand, STOI
indicates all models lead to an improvement in intelligibility.

\begin{figure*}
    \centering
    \subfloat[(a)]{%
        \includegraphics[width=.49\textwidth]{img/exp1_pesq.png}}
    \hfill
    \subfloat[(b)]{%
        \includegraphics[width=.49\textwidth]{img/exp1_srmr.png}}
    \hfill
    \subfloat[(c)]{
        \includegraphics[width=.49\textwidth]{img/exp1_stoi.png}}
    \hfill
    \subfloat[(d)]{%
        \includegraphics[width=.49\textwidth]{img/exp1_polqa.png}}
    \hfill
    \caption{Single speaker, simulated RIR scores: (a) PESQ, (b) SRMR,
    (c) STOI, and (d) POLQA.}
    \label{fig:exp1}
\end{figure*}

\subsection{Unmatched vs. matched speakers}
Although testing dereverberation models with single speaker datasets allows
assessing basic model functionality, in real-world conditions, training a model
for a single speaker is not practical and has very limited applications. It is
important to evaluate the generalization capabilities of such models with
mismatched speakers, as this is a more likely scenario. Figure \ref{fig:exp3}
shows the results for Wu2016, Wu2017, the proposed model, as well as for the
reverberant files in the test set. Between the baseline models, we can see that
Wu2017 now underperforms Wu2016 in all metrics, and either reduces metric
scores (as for PESQ and POLQA for low T60) or does not change them. However,
the version of the model that does not depend on T60 leads to higher scores.
Although we do not have a clear explanation for this behaviour, we believe the
optimal scores for the STFT hop size and context might depend both on T60 and
speaker, but the Wu2017 model uses fixed values that depend only on T60. Since
the model has now less data from each speaker, it was not able to exploit these
adapted features properly.

The proposed model, on the other hand, outperforms both baselines in 3 out of 4 metrics, only
achieving similar scores in the SRMR metric. Compared to Wu2016, our model leads
to improvements of around 0.4 in PESQ, 0.1 in STOI, and 0.5 in POLQA.

\begin{figure*} \centering \subfloat[(a)]{%
    \includegraphics[width=.49\textwidth]{img/exp3_pesq.png}} \hfill
    \subfloat[(b)]{%
        \includegraphics[width=.49\textwidth]{img/exp3_srmr.png}} \hfill
    \subfloat[(c)]{
        \includegraphics[width=.49\textwidth]{img/exp3_stoi.png}} \hfill
    \subfloat[(d)]{%
        \includegraphics[width=.49\textwidth]{img/exp3_polqa.png}} \hfill
    \caption{Scores for experiment with unmatched speakers:
        (a) PESQ, (b) SRMR, (c) STOI, and (d) POLQA.}
    \label{fig:exp3}
\end{figure*}

\subsection{Real vs. simulated RIR}
In our last experiment, we test the generalization capability of the models
trained on simulated RIRs to real RIR. To that end, we used speech convolved
with real RIRs from the ACE Challenge dataset, as specified in Section IV. The
results are reported in Figure \ref{fig:exp4}. Note that the x-axis in that
figure does not have linear spacing in time, as we are showing the results for
each T60 as a single point uniformly spaced from its nearest neighbours in the
data. Note also the larger variability in scores for reverberant files, which
is due to the scores here not being averaged across many RIRs.

In this test, the proposed method achieves the highest scores in all metrics.
It is also the only method to improve PESQ and POLQA across all scenarios,
while the baselines either decrease or do not improve such metrics.  Although
the baselines do improve STOI in most cases, in some scenarios they actually
decrease STOI.

\begin{figure*}
    \centering
    \subfloat[(a)]{%
        \includegraphics[width=.49\textwidth]{img/exp4_pesq.png}}
    \hfill
    \subfloat[(b)]{%
        \includegraphics[width=.49\textwidth]{img/exp4_srmr.png}}
    \hfill
    \subfloat[(c)]{
        \includegraphics[width=.49\textwidth]{img/exp4_stoi.png}}
    \hfill
    \subfloat[(d)]{%
        \includegraphics[width=.49\textwidth]{img/exp4_polqa.png}}
    \hfill
    \caption{Scores for experiment with real RIRs: (a) PESQ, (b) SRMR,
    (c) STOI, and (d) POLQA.}
    \label{fig:exp4}
\end{figure*}

\subsection{Subjective Listening Tests}
The results of MUSHRAR tests are summarized in Table \ref{tab:mushrar}. The results for the original, unprocessed reverberant files are also reported. When rating how reverberant the enhanced stimuli were, participants rated the outputs of the proposed model as less reverberant than the baseline and reverberant signals for all T60 values, while the baseline model only achieved a lower reverberation perception for lower reverberation times.

\begin{table}
\centering
\label{tab:mushrar}
\caption{Results of the MUSHRAR test for reverberation perception for different T60 values. Lower is better.}
\begin{tabular}{ccccc}
Model & 0.6~s & 0.9~s & 1.2~s & 1.5~s \\
\hline
Reverb & 42.62 & 60.06 & 71.73 & 82.24 \\
Wu2016 & 32.27 & 38.08 & 46.27 & 43.89 \\
Proposed & 24.44 & 23.89 & 29.93 & 31.64 \\
\end{tabular}
\end{table}

\section{Discussion}
\subsection{Effect of the context size and residual connections}
In this paper, we propose the addition of a few components to the architecture of deep neural networks for dereverberation. Using a context window for speech enhancement is not a new idea, and most studies using deep neural networks use similar context sizes. The novelty in our approach, however, is to use a convolutional layer as a local context encoder, which we believe helps the model to learn how to extract local features both in the time and the frequency axes in a more efficient way than just using a single context window as the input for a feedforward model. This local context, together with the residual input connections, is used as an input for the recurrent decoder, which is able to learn longer-term features. Similar architectures (save for the residual connections) have been successfully used for speech recognition tasks \cite{hannun_deepspeech_2014} but, to the best of our knowledge, this is the first work where such an architecture is used with the goal of estimating the clean speech magnitude spectrum.

Regarding the effect of the context size, our results agree with the intuition that including more past and future frames would help in predicting the current frame. However, we also show that there is a very small difference between using 7 vs. 11 frames as context. A window of 7 frames encompasses a total of 144 ms, vs. 208 ms for 11 frames. Both sizes are still much smaller than most of the reverberation times being used for our study, but they already start allowing the model to easily extract features related to amplitude modulations with lower frequencies, which are very important for speech intelligibility. We believe the combination of short- and long-term contexts, by having both the short-term context encoder and longer-term features through the recurrent layers, allows our architecture to benefit even from a shorter context window at the input. Lower modulation frequencies can still be captured through the recurrent layers, although we cannot explicitly control or assess how long these contexts are since they are learned implicitly through training and might be input-dependant.

We also introduce the use of residual and skip connections in the context of speech enhancement. Residual connections have a very close corresponding method in speech enhancement, namely spectral subtraction. The output of a speech enhancement model is likely to be very similar to the input, save for a signal that has to be subtracted from it (e.g. in the case of additive noise). In our case, our model is not restricted to subtraction due to how our architecture was designed. Consider the architecture as shown in Figure 1. The input to each recurrent layer is the sum of projections of the corrupted input (via the linear layers $f_1$, $f_2$, and $f_3$), and the output of the previous layers (via the linear layers $g_1$, $g_2$, and $g_3$). One possible solution to the problem, given this architecture, is for each recurrent layer to perform a new stage of spectral estimation given the difference between what the previous stage has predicted and the input at a given time. The output layer uses skip connections to the output of each recurrent layer and combines their predictions, allowing each layer to specialize on removing different types of distortions or to successively improve the signal. Our architecture was inspired by the work on generative models by Alex Graves \cite{graves_generating_2013}, which uses a similar scheme of connections between recurrent layers. The recently proposed WaveNet architecture \cite{oord_wavenet:_2016}, which is a generative model for audio signals able to synthesize high-quality speech, also makes use of residual connections and skip connections from each intermediate layer and the output layer.

\subsection{Comparison with baselines}
As seen in the previous section, our model outperforms both baselines based on feed-forward neural networks. One clear advantage of our approach is that we do not need to predict T60, which is a hard problem in itself and adds another layer of complexity to the model, especially under the presence of other distortions such as background noise \cite{AceChallenge}.

Regarding model size, as seen in Table 1, it is important to note that despite having a significantly smaller number of parameters than the baseline models, our proposed model consistently outperforms them in most experiments. Our best model (shown in the last line of Table 1) has approximately half the number of parameters but uses these parameters more efficiently because of its architectural characteristics.

Although the authors of \cite{wu_reverberation-time-aware_2017} argue that their model leads to improvements in PESQ scores, the differences reported were not significant. Also, the authors have used PESQ
scores for selecting the best models; however, that might be an issue,
especially because PESQ is not a recommended metric for reverberant/dereverberated
speech. In our experiments we used the best validation
loss (MSE) for model selection for all the models, including the baselines.
Using a proper speech quality or intelligibility metric as a function for model selection would be a good, but costlier solution because one has to generate/evaluate samples using that metric for all epochs, while MSE can be computed directly from the output of the model.
\subsection{Generalization capabilities}
Another important aspect of our study, when compared to \cite{wu_reverberation-time-aware_2017}, is that we train and test the models under several different room impulse responses, both real and simulated. The results we report for Wu2017 have much lower performance than those reported in \cite{wu_reverberation-time-aware_2017} for similar T60 values; however, it must be noted we used random room geometries and random source and
microphone locations for our RIRs, while in their experiments the authors used
a fixed room geometry with different T60 values (corresponding to different
absorption coefficients in the surfaces of the room) and a fixed location.
Although they tried to show the model generalizes to different room sizes, they
tested generalization by means of two tests: (a) a single different room size
with the same source-microphone positioning and (b) and a single different
source-microphone positioning with fixed room geometry. We believe those
experiments were not sufficient to show the generalization capabilities of the
model, and experiments 1 and 3 in our work confirm that hypothesis. In our work, we tried to expose the model to several different room geometries and source-microphone positioning, since this is closer to real-world conditions and helps the model to better generalize to unseen rooms and setups.

\subsection{Anedoctal comparison with mask-based methods}
Although we did not compare our method to masking alternatives in this study, we would like to briefly mention the results reported in the most recent paper with a masking approach \cite{williamson_time-frequency_2017}, which used a similar single-speaker dataset (IEEE sentences uttered by a male speaker), simulated and real RIR for the dereverberation task. Although the range of T60s in our study and theirs is not similar, we can roughly compare the metrics of our model in the same range used in their study. The STOI improvements for our method are higher than the so-called cRM method proposed in that paper: our method has a STOI improvement of approximately 0.2 in all simulated T60s they report (0.3, 0.6, and 0.9~s), while their highest improvement is of 0.06 for 0.9~s. The cRM method, however, slightly decreased STOI for a T60 of 0.3~s.

\subsection{Study limitations}
Although we report both PESQ and SRMR scores in this work, these results should
be taken cautiously. PESQ has been shown to not correlate well with reverberant
and dereverberated speech. SRMR, on the other hand, is a non-intrusive metric,
so it is affected by speech- and speaker-related variability \cite{Santos_NH_2014}. It is also more sensitive to high reverberation times (0.8~s as shown in \cite{senoussaoui_speech_2017}), so measurements in low reverberation times
might not be accurate enough for drawing conclusions about the performance of a given model.

A characteristic common to all DNN-based models and also other algorithms based on spectral magnitude estimation is a higher level of distortion due to reusing the reverberant phase, especially in higher T60. This is added to magnitude estimation errors which are also higher in higher T60, since the input signal is very different from the target due to the combined effect of coloration and longer decay times. Although we do not try to tackle this issue here, there are recent developments in the field that could be applied jointly with our model, such as the method recently proposed in \cite{mayer_impact_2017}. This exploration is left for future work.

\section{Conclusion}
We proposed a novel deep neural network architecture for performing speech
dereverberation through magnitude spectrum estimation. We showed that this
architecture outperforms current state-of-the-art architectures and generalizes
over different room geometries and T60s (including real RIR), as well as to
different speakers. Our architecture extracts features both in a local context (i.e.,
a few frames to the past/future of the frame being estimated) as well as long-term
context. As future work, we intend to explore improved cost functions (e.g., incorporating sparsity in the outputs \cite{Jukic2017}) as well as applying the architecture to signals distorted with both additive noise and reverberation. We also intend to propose a multichannel extension of the architecture in the future. Finally, we intend to explore a number of solutions to the issue of reconstructing the signal using the reverberant phase.

\section*{Acknowledgement}
The authors would like to acknowledge funding from NSERC, FQRNT, and Google, as well as NVIDIA for the donation of the Tesla K40 GPUs used for the experiments.

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\bibliographystyle{IEEEtran}


\section{Introduction}\label{intro}
Understanding the baryon excitations is crucial to enhance our access to strong interactions
in the low energy domain. One of the multiple interesting light baryon excitations, is the $N^*(1440)$
resonance  - the first excitation of nucleon, also known as Roper resonance with $(I)J^P=(1/2)1/2^+$.
It has been elaborately studied and discussed in literature, but continues to be puzzling since
its discovery in 1964~\cite{Roper:1964zza}. This resonance predominantly decays to $N\pi$,
while decay modes such as $N\pi\pi$, $N\rho$ and $\Delta\pi$ are also observed collectively
to a significant fraction.

The most striking feature of this resonance is that it appears lower in mass with respect
to $N^*(1535)$, the lowest negative parity excitation of the nucleon with $(I)J^P=(1/2)1/2^-$. This
hierarchy is in contrast with results from most phenomenological studies that assumed the
$N^*(1440)$ resonance to be of three quark nature \cite{Isgur:1978xj,Capstick:1986bm}. A variety
of phenomenological suggestions that go beyond the  three quark picture
 followed these puzzling observations \cite{Crede:2013sze}.

There has been multiple lattice studies of the excited nucleon spectrum as well, most
of which indicated an inverted hierarchy with respect to the experiments
\cite{Liu:2014jua,Alexandrou:2013fsu,Alexandrou:2014mka,Engel:2013ig,Edwards:2011jj,Mahbub:2013ala,Roberts:2013ipa}.
Unlike other lattice investigations, Ref. \cite{Liu:2014jua} observed a low lying Roper resonance using fermion
discretization with good chiral properties and different eigenenergy extraction techniques. The question
on the importance of the role of chiral symmetry remains to be confirmed from future calculations. All
these previous lattice calculations utilized three-quark interpolating fields (assuming $N^*(1440)$ to
be stable) and identified the first excited lattice level with $N^*(1440)$. A recent lattice study that
included five quark interpolating fields used strictly local $qqqq\bar q$ interpolators \cite{Kiratidis:2016hda}.
However, either of these kinds of interpolating fields could suffer from lack of coupling with strongly decaying broad
resonance, such as $N^*(1440)$.

In this talk, we present the results from our recent lattice investigation of the excited nucleon spectrum within
an energy region below 1.65 GeV, where the Roper resonance is observed \cite{Lang:2016hnn}. In order to determine
the complete discrete spectrum we included for the first time $N\pi$ in $p$-wave as well as $N\sigma$ in $s$-wave
in order to account for their scattering, along with $qqq$ type interpolating fields. We investigate the possible
description of the Roper resonance based on our lattice results within elastic approximation of $N\pi$ scattering.
In Section \ref{sec-2}, we briefly describe the methods and technicalities in our calculation. We discuss the results
in Section \ref{sec:results} and then we summarize.

\section{Lattice methodology}\label{sec-2}
We investigate the excited nucleon spectrum on an $N_f=2+1$ ensemble generated by the PACS-CS collaboration
with lattice extension $V=32^3\times 64$, physical volume $L^3\simeq (2.9~$fm$)^3$ and $m_\pi=156(7)(2)~$MeV
\cite{Aoki:2008sm}. The valence and the sea quarks are realized using non-perturbatively improved
Wilson-clover fermions.

\textbf{Quark smearing width and distillation} :
For the  valence quarks sources and propagators we use full distillation \cite{Peardon:2009gh} with two different
smearing widths (narrow [n] and wide [w] using 48 and 24 eigenvectors of the discretized gauge-covariant Laplacian).
This technique aids us to build all the required Wick contractions, which are quite cumbersome for this study
involving as many as 113 distinct diagrams. Using two smearing widths we hope to get enhanced access to form radial
nodes in the wave function, which is favourable for the Roper resonance \cite{Roberts:2013ipa}.

\textbf{Interpolators} :
We employ the following nucleon (${\cal N}^i$), pion ($\pi$) and sigma meson ($\sigma$) operators in the dispersion relation
studies as well as in constructing the basis of baryon and baryon-meson interpolators used in extracting the excited
nucleon spectrum.
\begin{align}
\label{N}
{\cal N}^i_\mu(\mathbf{n})&\!=\!\sum_{\mathbf{x}} \epsilon_{abc} [u^{aT}(\mathbf{x},t)  \Gamma_2^i d^b (\mathbf{x},t)] ~[\Gamma_1^i q^c(\mathbf{x},t)]_{\mu}~\mathrm{e}^{i\mathbf{x\cdot n}\frac{2\pi}{L}}, \\
\label{pi}
\pi^+(\mathbf{n})& =\sum_{\mathbf x} \bar d({\mathbf x},t)\gamma_5 u({\mathbf x},t) \mathrm{e}^{i\mathbf{x\cdot n}\frac{2\pi}{L}},  \\
\pi^0(\mathbf{n}) &=\tfrac{1}{\sqrt{2}}\sum_{\mathbf x} [\bar d({\mathbf x},t)\gamma_5 d({\mathbf x},t)-\bar u({\mathbf x},t)\gamma_5 u({\mathbf x},t)] \mathrm{e}^{i\mathbf{x\cdot n}\frac{2\pi}{L}},\nonumber \\
\label{sigma}
\sigma(0)&=\tfrac{1}{\sqrt{2}}\sum_{\mathbf x} [\bar u({\mathbf x},t)u({\mathbf x},t)+\bar d({\mathbf x},t) d({\mathbf x},t)]~.
\end{align}
The proton field is given by ${\cal N}^i_\mu(\mathbf{n})|_{q=u}$, whereas the neutron is given by ${\cal N}^i_\mu(\mathbf{n})|_{q=d}$.
With three different choices of $(\Gamma_1^i,\Gamma_2^i)=(\mathbf{1},C\gamma_5),~(\gamma_5,C),~(i\mathbf{1},C\gamma_t\gamma_4)$, we employ
three nucleon fields in this study.

With these single hadron fields, we construct the following 10 baryon and baryon-meson annihilation operators with $(I)J^P=(1/2)1/2^+$
and total momentum zero.
\begin{align}
O_{1,2}^{N\pi}&=-\sqrt{\tfrac{1}{3}} ~\bigl[p^{1,2}_{-\frac{1}{2}}(-e_x) \pi^0(e_x)-p^{1,2}_{-\frac{1}{2}}(e_x) \pi^0(-e_x)
-i p^{1,2}_{-\frac{1}{2}}(-e_y) \pi^0(e_y)+i p^{1,2}_{-\frac{1}{2}}(e_y) \pi^0(-e_y)\nonumber\\
& \qquad+ p^{1,2}_{\frac{1}{2}}(-e_z) \pi^0(e_z)-p^{1,2}_{\frac{1}{2}}(e_z) \pi^0(-e_z)\bigr]
  +\sqrt{\tfrac{2}{3}} ~\bigl[\{p\to n, \pi^0\to \pi^+\}\bigr] \quad [n], \nonumber\\
O_{3,4,5}^{N_w}&=p^{1,2,3}_{\frac{1}{2}}(0)\quad [w], \quad\quad  O_{6,7,8}^{N_n}=p^{1,2,3}_{\frac{1}{2}}(0)\quad [n], \quad\quad
O_{9,10}^{N\sigma}=p^{1,2}_{\frac{1}{2}}(0) \sigma(0) \quad [n].
\label{O}
\end{align}
Here $e_x$, $e_y$, and $e_z$ refer to the unit lattice momentum vectors along $x, y$, and $z$ directions,
$n$ and $w$ inside square brackets refer to narrow and wide smearing respectively and the subscripts
$\pm\frac12$ to the proton fields refer to the spin projections. The basis in eqn (\ref{O}) contains all the baryon-meson
interpolators with non-interacting levels below $1.65~$GeV.

The $\sigma$ meson is a broad resonance and can decay into 2$\pi$; $O^{N\sigma}$ is expected to couple to
the $N\sigma$ as well as $N(0)\pi(0)\pi(0)$ levels. In this work, we implemented only the $O^{N\sigma}$ interpolator
with the expectation that it will effectively represent a mixture of $N\sigma$ and $N(0)\pi(0)\pi(0)$.

\textbf{Extracting the spectrum} :
Expanding the correlation matrix,
\begin{equation}
C_{ij}(t)=\langle \Omega|O_i(t+t_{src})\bar O_j (t_{src})|\Omega\rangle = \sum_n \langle \Omega | O_i|n\rangle \mathrm{e}^{-E_nt} \langle n|\bar O_j |\Omega\rangle = \sum_n  Z_i^n Z_j^{n*} \mathrm{e}^{-E_nt}
\label{C}
\end{equation}
one can see that the large-time behavior of the eigenvalue  $\lambda^{(n)}(t,t_0)$ provides $E_n$,
where as the eigenvectors $u_i^{(n)}$ are related to the operator overlaps $Z_i^n$. The spectrum is determined by performing non-linear fits to the eigenvalues $\lambda^{(n)}(t)$
extracted from the generalized eigenvalue problem \cite{Michael:1985ne,Luscher:1985dn} for the correlation matrices $C_{ij}(t)$
\begin{equation}
\label{gevp}
 C_{ij}(t)u_j^{(n)}=\lambda^{(n)}(t,t_0)C_{ij}(t_0)u_j^{(n)}\;,\ \  \lambda^{(n)}(t,t_0)\propto  \mathrm{e}^{-E_n (t- t_0)}
\end{equation}

\textbf{Extracting the resonance information} : In a scenario, when there is no meson-meson and baryon-meson interactions,
we expect to observe levels corresponding to $N(0)$, $N(0)\pi(0)\pi(0)$ and $N(1)\pi(-1)$ on the ensemble
we use. With non-trivial attractive interaction the extracted levels get affected and thus the observed levels and their
respective energies will have implications in the presence of resonances. Within the approximation of elastic $N\pi$
scattering, the interacting $N\pi$ levels (dotted yellow lines) gets shifted with respect to the respective non-interacting
positions (dashed orange lines) as shown in Fig. \ref{elasNpi}. The figure predicts the lattice spectrum as a function
of the lattice extension L given the experimental mass of the Roper (cyan band) and experimental phase shift information
as inputs to L\"uscher's relation  \cite{Luscher:1990ux,Luscher:1991cf}
\begin{equation}
\label{luscher}
\delta(p)=\mathrm{atan}\biggl[\frac{\sqrt{\pi} p L}{2\,Z_{00}(1;(\tfrac{pL}{2\pi})^2)}\biggr],\  E_{n.i.}=E_{N(p)}+ E_{\pi(p)}
\end{equation}
where $E_{H(p)}$ is the energy of the hadron.
\begin{figure}[bt]
\centering
\sidecaption
\includegraphics[width=72mm,clip]{fig1}
\caption{Lattice spectrum as a function of the lattice extension L, predicted by eqn. (\ref{luscher}) within
elastic $N\pi$ scattering. The experimental mass of the Roper (cyan band) and experimental phase shift information
have been used as inputs to determine the energy shifts on the lattice. $N(0)\pi(0)\pi(0)$ (dashed blue line)
and $N(0)\sigma(0)$ are assumed to be decoupled with the Roper and hence remain unaffected. }
\label{elasNpi
\end{figure}
The numerically extracted lattice spectrum from the correlation matrices we compute is compared with these analytic
predictions and explored in search of a possible description of $N^*(1440)$ feature as a conventional resonance
within the elastic approximation of $N\pi$ scattering.

\section{Results and discussion} \label{sec:results}

In Fig. \ref{fig:E_interp_set}, we present our results for the excited nucleon spectrum. These are
based on correlation matrices for subsets of the operators basis
$(O_1^{N\pi},~ O^{N_n}_3,~O^{N_w}_{6,8},~O_{9}^{N\sigma})$ as shown in the Fig.
\ref{fig:E_interp_set} legend. Basis 1 (within the red rectangle) represents the set with all
five operators in the above basis, which we call the complete set containing all types of
interpolators.  Including the rest of the operators in  (\ref{O}) makes the spectrum
noisier. The horizontal dashed blue and red lines denote the multi-hadron levels $m_N+2m_\pi$ and
$E_{N(1)}+E_{\pi(-1)}$ in the non-interacting limit.

\begin{figure}[htb]
\centering
\sidecaption
\includegraphics*[height=63mm,clip]{fig3}
\caption{Lattice excitation spectrum of the nucleon for various choices of the interpolator basis.
Basis 1 is referred to as the complete set, as it contains all types of interpolators and
is highlighted with a red rectangle in the figure. The non-interacting scattering levels
are shown as dashed horizontal lines (red for $N(1)\pi(-1)$ and blue for $N(0)\pi(0)\pi(0)$).}
\label{fig:E_interp_set}
\end{figure}

The main observation is that in basis 1 there are only three levels  below $1.65~$GeV. The ground
state represents the nucleon and the remaining two levels lie close to the non-interacting
positions of $m_N+2m_\pi$ and $E_{N(1)}+E_{\pi(-1)}$. As expected the ground state
representing the nucleon can be seen to be quite stable with respect to different bases used,
except for basis 6. In basis 6 only $O_{9}^{N\sigma}$ is used and hence the extracted level
could be a mixture of $N$, $N(0)\pi(0)\pi(0)$ and $N(0)\sigma(0)$, since the $\sigma$ meson can
also mix with the vacuum. The first excited state in basis 5, where only interpolators of type $qqq$
are used, appears significantly above $1.65~$GeV. This observation agrees with the previous lattice results,
where the quark fields are realized with clover fermions, using only $qqq$ type interpolators.
It is also to be noted that no interpolator basis renders  more than three eigenstates below
$1.65~$GeV. This indicates not only that the $qqq$ type interpolators have weak coupling
with the scattering levels, but also the first excited state with a strong $qqq$ Fock component
appears significantly above $1.65~$GeV. Considering the closeness of the first and second
excitations to the non-interacting energies of the scattering levels, we relate them with the
scattering channels $N(0)\pi(0)\pi(0)$ and the $N(1)\pi(-1)$ respectively. This identification is
supported by features of the spectrum for different choices of the operator basis as
demonstrated in Fig. \ref{fig:E_interp_set}. Exclusion of $O^{N(0)\sigma(0)}$ from basis 1 leads to the
disappearance of first excited state as can be seen in basis 4. Similarly the $N(1)\pi(-1)$
Fock component is observed to be crucial to determine the second excitation in basis 1, as can
be inferred from basis 3.

\begin{figure*}[!htb]
\centering
\sidecaption
\includegraphics*[height=72mm,clip]{fig2}
\caption{The normalized overlaps $\tilde Z_i^n=\langle \Omega| O_i|n\rangle$, extracted from the correlation
matrix (\ref{C}) based on the complete set (basis 1). The  ratios of overlaps $Z_i^n$ are with respect
to  the largest among $|Z_i^{m=1,...5}|$ for a given operator $i$. }
\label{fig:E_final}
\end{figure*}

Identification of the dominant nature of the levels from above procedures is further verified by
their signatures in the overlap factors, $Z_i^n$. We present the normalized overlap factors from the basis
1 in Fig. \ref{fig:E_final}. The overlap factors $Z_i^n$ for any operator $i$ are normalized with
the largest among $|Z_i^{m=1,...5}|$. The ground state is observed to couple strongly to all
interpolators. The operator $O^{N\pi}$ can be seen to overlap strongest with the second excitation,
implying that the state is related to $N(1)\pi(-1)$. $O^{N\sigma}$ couples strongest to the ground
state, which could be a result of the $\sigma$ meson mixing with the vacuum. However, the most interesting
observation in this figure is that the $O^{N\sigma}$ is found to have similar overlap with
first and second excited states. Furthermore $|\langle \Omega| O^{N\sigma} |n=2\rangle| \lesssim  |\langle \Omega|O^{N\sigma} |n=3\rangle|$
suggests that the scattering channels could be significantly coupled.

Extracting the phase shift information is not straightforward as the Roper channel is significantly
inelastic above the $N\pi\pi$ threshold. Numerical study of three-body channels has not been performed
in lattice QCD up to now, though analytic treatments exist \cite{Hansen:2015zga,Hansen:2016ync}. Hence
we restrict ourselves to the elastic approximation of $N\pi$ scattering. Within this approximation,
we determine the infinite volume phase shifts from the lattice energy levels using L\"uscher's relation
as in eqn. (\ref{luscher}). The energy level related to $N(1)\pi(-1)$ in basis 1 lies on top of the
non-interacting energy $E_{N(1)}+E_{\pi(1)}$, as can be seen from basis 1 in Figs. \ref{fig:E_interp_set}.
Hence the energy shift of this level with respect to the non-interacting level and the corresponding infinite
volume phase shift are consistent with zero within the sizable errors. This observation remains stable
with respect to various choices of dispersion relation to determine the non-interacting levels,
correlated or uncorrelated fits, etc. This indicates that $N\pi$ scattering in the elastic approximation
alone does not render a low lying Roper resonance.

We quantify this inference further by comparing the actual lattice data we compute with analytical predictions
for lattice data in Fig. \ref{fig:E_analytic}. These analytic predictions are based on L\"uscher's formulae
(assuming elasticity) with experimental inputs on the masses and the phase shifts related to the Roper
resonance. One can clearly see that below $E<1.65~$GeV numerical results indicate only three eigenlevels,
whereas analytic predictions indicate additional levels if the Roper can be described as a conventional
resonance within the elastic $N\pi$ scattering approximation.

\begin{figure}[htb]
\centering
\sidecaption
\includegraphics[width=63mm,clip]{fig4.pdf}
\caption{Energy values from our simulation (on the left pane) being compared with analytic
predictions based on L\"uscher's formulae (on the right pane). The relevant non-interacting
scattering levels are shown as dashed horizontal lines. }
\label{fig:E_analytic}
\end{figure}

Absence of signatures for the Roper resonance could be due to various reasons.
The Roper could be a dynamically coupled channel phenomenon involving multiple scattering channels
like $\Delta \pi$, N$\rho$ and $N\pi\pi$ in the interpolator basis. Or it could be a hybrid
phenomenon, where the gluonic degrees of freedom are excited \cite{Golowich:1982kx,Kisslinger:1995yw}.
It could also be that our basis lack some genuine interpolators like a pentaquark operator or
non-local $qqq$ type interpolators, which could actively scan radial and orbital excitations
within the $qqq$ structure. Lastly this work, like most other lattice studies, involved quark
fields that are not chiral at finite lattice spacing $a$. This is probably reflected in the contrasting
results from Ref. \cite{Liu:2014jua} with respect to other lattice investigations, including ours.

In Ref. \cite{Kiratidis:2016hda} a local pentaquark operator with color structure
$\epsilon_{abc} \bar q_a [qq]_b[qq]_c$ ($[qq]_c=\epsilon_{cde} q_cq_dq_e$) was used to explore the Roper
region, but a low-lying Roper state was not found. The local pentaquark
operator are related to baryon-meson interpolators via Fierz relations, with the important ones
being the  $N(1)\pi(-1)$ and $N(0)\sigma(0)$ in the Roper region. Since we include such
baryon-meson interpolators with good partial wave projection, we expect that our simulation
does incorporate good localized pentaquark operators as well. However, it remains to be a challenge
to include pentaquark interpolators with more complicated structures.

Interpolators of type $qqq$ with particularly designed non-local internal structure might also be important
and be better suited to scan the radial and orbital excitations rather than interpolators with varying
smearing widths. Most of the studies in the past with such non-local $qqq$ operators have also not yielded
low-lying state with mass below 1.6 GeV \cite{Edwards:2011jj}. However no work has been reported studying
the Roper channel with a large number of specially-designed non-local $qqq$ operators as well as relevant
baryon-meson interpolators in the basis.

Dynamically coupled channel phenomena to describe the Roper resonance have been explored recently in
the \cite{Liu:2016uzk} using Hamiltonian Effective Field Theory (HEFT). Various scenarios involving
(I) $N\pi-N\sigma-\Delta\pi$ coupled to a low-lying bare $qqq$ type interpolator representing the Roper,
(II) coupling only to $N\pi-N\sigma-\Delta\pi$ channels and (III) $N\pi-N\sigma-\Delta\pi$ coupled to
a low-lying bare $qqq$ type interpolator representing nucleon were explored using HEFT to explain
the experimentally observed phase shifts. Figure \ref{fig:HEFT} depicts the comparison of our lattice
spectrum with the lattice spectrum predicted from the above HEFT calculation. Excluding the stars in
the spectrum, which are related to the scattering levels that are not included in basis used in our work,
we compare the number of expected levels (squares and circles) between the lattice and the predictions
based on HEFT. Our results are compatible with the cases (II) and (III), where one can see that there
are only two states in the region between 1.2 to 1.7 GeV. However, scenario (I) is disfavored by our
results as it predicts three levels in this energy range. A recent follow up HEFT calculation, based
on constraints from our lattice QCD data, has claimed that the Roper resonance is best described as
a dynamical phenomenon through strongly coupled baryon-meson channels \cite{Wu:2017qve}.

\begin{figure}[!htb]
\centering
\sidecaption
\includegraphics*[width=0.49\textwidth,clip]{fig5.pdf}
\caption{Comparison of our results to predictions based on HEFT \cite{Liu:2016uzk} with different Hamiltonians.
Our results are consistent with the scenarios (II) and (III), whereas (I) is disfavored, as it predicts
additional levels in the energy region being investigated.}
\label{fig:HEFT}
\end{figure}

\section{Summary}\label{sec:Conc}

In this talk we present the results from our recent lattice QCD study of $N\pi$ scattering in the positive-parity
nucleon channel. This channel is interesting for the presence of a low lying first excitation of the nucleon,
called Roper resonance  $N^*(1440)$. Employing a set of interpolators that includes $qqq$-like, $N\pi$ in
$p$-wave and $N\sigma$ in $s$-wave, we systematically extract the excited nucleon spectrum up to an energy of
1.65 GeV. The extracted lattice spectrum disfavors the description of the  low lying Roper resonance as
a conventional resonance in the N$\pi$ scattering within the elastic approximation. The overlap factors
from our study indicate a possibility of Roper being a dynamical coupled channel phenomenon. Other possible
reasons for absence of signature of Roper resonance in our calculations were also discussed.

\section{Acknowledgements}

We thank the PACS-CS collaboration for providing the gauge configurations.  This work is
supported in part by the  Slovenian Research Agency ARRS, by the Austrian Science Fund
FWF:I1313-N27 and by  the Deutsche Forschungsgemeinschaft Grant No. SFB/TRR 55. The
calculations were performed on  computing clusters at the University of Graz (NAWI Graz)
and Theoretical Department at Jozef Stefan Institute, Ljubljana. M. P. acknowledges
support from EU under grant no. MSCA-IF-EF-ST-744659 (XQCDBaryons).

\clearpage