\section{Introduction}
The problem of visual attention prediction has been extensively studied in the past two decades. From the early bio-inspired attention models~\cite{itti1998model,harel2007graph,zhang2008sun} to subsequent shallow learning models \cite{MIT1003,zhao2012learning,kienzle2007nonparametric} as well as the latest deep learning models~\cite{kruthiventi2017deepfix,huang2015salicon,liu2018learning,pan2016shallow,wang2018deep}, their capability advances steadily in predicting attention regions reflected by the human fixations recorded in free-viewing eye-tracking experiments. To date, the performance of the latest model on the public saliency benchmark \cite{MIT300} seems to be sufficiently high and the problem seems to be well addressed. However, a compelling concern naturally arises: what do existing deep attention models actually predict?

\begin{figure}[t]
\begin{center}
   \includegraphics[width=1.00\linewidth]{motivation.png}
\end{center}
   \caption{Representative results of visual attention models on ground-level and solar images. The first row shows four representative images from existing visual attention datasets and two solar images.  The second row shows the ground-truth attention maps ({\em i.e.}, fixation density maps), and the third row shows the most attractive visual patterns. The last two rows show the results from the non-deep attention model SP~\cite{li2014visual} and the deep attention model SalNet~\cite{pan2016shallow}. When the non-deep model demonstrates consistent capabilities in handling ground-level and solar images, deep models fail on solar images, implying that they may only memorize specific visual patterns other than capture the inherent mechanism of human visual attention. }
\label{fig:motivation}
\end{figure}

To address this concern, we carefully inspect the images in many prevailing visual attention datasets such as MIT300~\cite{MIT300}, MIT1003~\cite{MIT1003}, CAT2000~\cite{CAT2000} and Toronto~\cite{Toronto}. We find that one common attribute shared by most images in these datasets is that they are captured on the Earth, fulfilled with repeatedly occurrence of attractive visual patterns from people, vehicle, building, etc. In most cases, the higher capability of memorizing these patterns and learning related representations, the better performance of an attention model (as shown in Fig.~\ref{fig:motivation}). However, one issue remains unclear even for the latest deep models: whether such capability sufficiently reflects the inherent mechanism of the human visual attention system?

To further investigate this issue, we propose to construct a new visual attention dataset composed of visual patterns that rarely appear in existing datasets. To maximize the diversity of visual patterns contained in the new dataset and existing ones, we refer to the solar images provided by the LSDO dataset~\cite{kucuk2017large} that are taken at specific wavelengths to record the solar activities. As shown in the last two columns of Fig.~\ref{fig:motivation}, visual patterns in these solar images are remarkably different from the classic patterns of people, vehicle, building and road signs. From the LSDO dataset, we uniformly sample 1070 images captured at the wavelength {171\AA} that perform the best in reflecting the solar active regions over three years ({\em i.e.}, 2012 to 2014). For these images, we design an eye-tracking experiment to record the visual attention of 16 subjects under free-viewing conditions. From Fig.~\ref{fig:motivation}, we can see that the human-being is still capable to quickly locate the most attractive regions-of-interest even they never seen the solar images before. The question is, whether the attention models, especially the deep models that have impressive capability on ground-level images, still work well on the solar images?

To answer this question, we benchmark the performance of 17 visual attention models over the new dataset (denoted as VASUN) by using five representative evaluation metrics. From the benchmark results, we find an interesting performance reversion over the non-deep and deep models. In most of the five evaluation metrics, the performances of many state-of-the-art deep models drop remarkably. Instead, many classic shallow models that are bio-inspired or non-deep learning-based still perform impressively on solar images and significantly outperform the deep models. From these results, we find that the performance advance of existing deep attention models may mainly come from their capabilities of memorizing and predicting the occurrence of some specific visual patterns other than learning the inherent mechanism of human visual attention. In other words, most of the existing deep models, despite their impressive performances on existing ground-level datasets, still lack the capability of modeling the inherent mechanism of human visual attention. Moreover, such capability can be quantitatively measured by their performance variations on ground-level images and the solar images in VASUN. In addition, we also train several baseline models on VASUN to demonstrate the feasibility and key issues of predicting visual attention on the sun. These baseline models, together with the proposed dataset, can be used to revisit the problem of visual attention prediction from a novel perspective that are complementary to existing ones.

Our contributions are summarized as follows: 1)~We propose a novel visual attention dataset on solar images, which can be used to measure the degree that an attention model captures the inherent mechanism of human visual attention; 2)~we conduct an extensive benchmark to provide a tentative guess on what do attention models really predict; 3)~we propose five baseline models to show that visual attention on the sun is predictable and conclude several key factors that can be reused in designing new models.

\section{Related Work}
There exist many reviews about visual attention models~\cite{borji2013state} and evaluation metrics~\cite{Bylinskii2018metric}. Here we briefly review related works from two perspectives: visual attention models and benchmark datasets as well as evaluation metrics.
\subsection{Visual Attention Models}
The development of visual attention models can be roughly divided into three stages characterized as bio-inspired, shallow learning and deep learning, respectively. It is widely acknowledged that the first bio-inspired stage originates from \cite{koch1987shifts,itti1998model}. In nearly a decade after the publication of \cite{itti1998model}, a widely adopted paradigm of designing attention models is to find neurobiological evidences or hypotheses first and then mimic them with computational attention models. Such evidences or hypotheses can be represented by features such as local contrast~\cite{harel2007graph} and global irregularity~\cite{zhang2008sun}. Based on these features, a bio-inspired attention model usually acts as a feature fusion or selection strategies. Generally speaking, each of these features and mapping strategies can reflect the mechanism human visual attention from a specific perspective but may suffer from the incompleteness.

In the second stage, researchers realize that the heuristically designed mapping strategies may be insufficient to combine various attention features. Therefore, they propose to find a better feature combination strategy through shallow learning algorithms. Most of the learning algorithms adopt the supervised framework such as Support Vector Machine~\cite{MIT1003} and Adaptive Boosting~\cite{zhao2012learning}. In addition, some algorithms adopt unsupervised frameworks such as sparse coding~\cite{hou2009dynamic} and statistical learning~\cite{li2014visual}. In particular, many datasets are proposed along with models in the second stage to meet the demand of machine learning algorithms.

With the cumulation of various visual attention datasets, models propose to direct learn a deep network that maps the input image to the output attention map. Compared with bio-inspired and shallow learning models, deep models such as eDN~\cite{eDN}, DeepGaze~\cite{kummerer2014deep}, DeepFix~\cite{kruthiventi2017deepfix}, SALICON~\cite{huang2015salicon}, SalNet~\cite{pan2016shallow}, DVA~\cite{wang2018deep} and SAM-ResNet\cite{SAMresnet} achieved much better results by learning better feature representations via novel deep learning architectures~\cite{wang2018deep}, multi-scale input strategy~\cite{li2015visual} and specially designed loss functions~\cite{jetley2016end}. From a positive perspective of view, these deep models do achieve impressive performance scores in existing benchmarks that can be hardly reached by bio-inspired and shallow learning models. From a negative perspective of view, however, it is not clear whether these deep models really capture inherent mechanism of human visual attention. Although some works have explore the relationships between deep networks with neural responses~\cite{yamins2014performance,khaligh2014deep}, a compelling concern still arises in the literature: what do existing deep attention models actually predict?

\begin{table}[t]
\caption{The subject number and image resolution of representative image datasets and our VASUN dataset. \#Img and \#Sub mean the number of images and subjects in the dataset.}
\centering{
\begin{tabular}{cccc}
\toprule
Dataset   & \#Img & \#Sub. & Max Res.\\
\midrule
MIT300~\cite{MIT300}      & 300     & 39  & {1024$\times{}$1024}\\
MIT1003~\cite{MIT1003}    & 1003    & 15  & {1024$\times{}$1024}\\
Toronto~\cite{Toronto}    & 120     & 20  & 511$\times$681\\
CAT2000~\cite{CAT2000}    & 2000    & 18  & 1920$\times{}$1080\\
SALICON~\cite{SALICON}    &  15000  & -   & 640$\times{}$480\\
PASCAL-S~\cite{PASCAL-S}  & 850     & 8   & {500$\times{}$500}\\
DUT-O~\cite{DUT-O}        & 5168    & 5   & {401$\times{}$401}\\
\hline
VASUN                     & 1070    & 16  & 1024$\times{}$1024\\
\bottomrule
\end{tabular}
}
\label{tab:dataset}
\end{table}
\subsection{Benchmark Datasets and Metrics}
The evaluation of visual attention models is also a challenging task since the consistency between human fixations and model predictions can be measured from many perspectives~\cite{Bylinskii2018metric}. As a result, many benchmark datasets have been proposed in the literature. Among these datasets, some of them are specifically designed for studying the human fixation prediction problem of more than a dozen subjects in the free-viewing conditions~({\em e.g.}, MIT300~\cite{MIT300}, MIT1003~\cite{MIT1003}, Toronto~\cite{Toronto} and CAT2000~\cite{CAT2000}), while the rest ones just record fixations of several subjects to facilitate subsequent annotations ({\em e.g.}, Pascal-S~\cite{PASCAL-S} and DUT-O~\cite{DUT-O}). The number of subjects and their image resolutions of these datasets can be found in Table~\ref{tab:dataset}.

Given a dataset, there are many ways to evaluate an attention model, leading to a dozen of evaluation metrics~\cite{riche2013saliency,li2015metric}. Here we only introduce the most representative five metrics that will be used in this study.

1)~Area Under the ROC Curve (AUC) and Shuffled AUC (sAUC). In computing AUC, the recorded fixations are selected as positives, while all the other locations are treated as negatives. After that, the true positive rates and false positive rates are computed at all probable thresholds to plot a ROC curve to compute the AUC score. The main difference between sAUC and AUC is that the negatives correspond to fixations shuffled from other images in the same dataset, leading to a more strict evaluation standard.

2)~Normalized Scanpath Saliency (NSS). NSS is considered to be one of the most effective metrics in many recent studies~\cite{riche2013saliency,Bylinskii2018metric}. It is computed by first normalizing the predicted attention map to zero mean and unit standard deviation, and then computes the average responses at fixations.

3)~Similarity (SIM) and Correlation Coefficient (CC). The ground-truth and predicted attention maps can be also treated as probability distributions. As a result, SIM can be computed as their intersection and CC describes their linear relationship.

\section{The VASUN Dataset}
In previous benchmarks of attention models
\cite{borji2013state}, the generalization ability of models are less discussed since almost all datasets are fulfilled with the same categories of scenarios ({\em e.g.}, people, vehicles, buildings and road signs). To measure the generalization ability, we need to construct a dataset that contains new visual patterns. Toward this end, we refer to the LSDO dataset~\cite{kucuk2017large}, which is a large scale dataset that contains 260K solar images captured in three years by the National Aeronautics Space Agency (NASA) in the Solar Dynamics Observatory (SDO) mission. This dataset consists of $4096\times{}4096$ solar images captured at three wavelengths ({\em i.e.}, 94 {\AA}, 171 {\AA} and 193 {\AA}). Neither such solar images nor their visual patterns have appear in any existing image attention datasets.

From the LSDO dataset, we construct a new image attention dataset to study visual attention on the sun (denoted as VASUN). Since the main objective of this dataset is to measure the generalization ability of attention models, we adopt the similar setting with MIT1003, the most widely used dataset with ground-level scenarios. That is, we sample 1070 images taken at the wavelength 171\AA ~from LSDO and down-sample them to the resolution of 1024$\times$1024. On these images, we record the visual attention of 16 subjects in eye-tracking experiments. These 16 subjects (10 males and 6 females) have normal or correct-to-normal visions. None of them has prior knowledge of astronomy and solar physics to avoid subjective bias.

In the experiment, each subject is asked to free-view every image for 4 seconds on a 22-inch color monitor. A chin rest is utilized to reduce head movements and enforce a viewing distance of 75 cm. During the free-viewing process, an eye-tracking apparatus with a sample rate of 500 Hz (SMI RED 500) is used to record various types of eye movements. Finally, we keep only the fixations and generate the ground-truth attention map $G$ for an image $I$ by overlaying Gaussian blobs centered at these fixations:
\begin{equation}
G(x,y)=\sum_{f=1}^{N}\frac{1}{2\pi{}\sigma^2}\exp\left(-\frac{(x-x_f)^2+(y-y_f)^2}{2\sigma^2}\right),
\end{equation}
where $\{(x_f,y_f),f=1,\ldots,N\}$ is the recorded fixations (each lasts about two milliseconds). $\sigma$ is a parameter computed according to the viewing distance and angle. Note that these ground-truth attention maps are then normalized into the dynamic range of $[0,1]$.

Some representative examples of the recorded fixations and ground-truth attention maps can be found in Fig.~\ref{fig:datasetexample}. From Fig.~\ref{fig:datasetexample}, we can see that most human visual attention is allocated to large active regions, making the visual attention prediction task on solar images a theoretically simple task. In addition, we also compare the average attention maps of VASUN with previous datasets in Fig.~\ref{fig:aam}. We can see that the distribution of fixations in VASUN is quite different from previous ground-level datasets. This may be caused by the fact that the solar images have no photographer bias that tend to place the target at image centers. This phenomena further validates that VASUN can be used to test the generalization ability of visual attention models.

\begin{figure}[t]
\begin{center}
   \includegraphics[width=1.00\linewidth]{datasetexample.png}
\end{center}
   \caption{Representative examples from VASUN. Red dots in the second row indicate the recorded fixation points, which are used to generate the ground-truth attention maps in the third row.}
\label{fig:datasetexample}
\end{figure}

\begin{figure}[t]
\begin{center}
   \includegraphics[width=1.00\linewidth]{aam.png}
\end{center}
   \caption{The average attention maps of various datasets. We can see that the distribution of human visual attention on VASUN is quite different from those in previous datasets, making a good dataset to test model generalization ability. }
\label{fig:aam}
\end{figure}

\section{Model Benchmark}
To test the generalization ability of existing visual attention models and find what they actually predict, we propose to measure their performance variations over VASUN and MIT1003. Note that MIT1003 contains 1003 images annotated by 15 subjects, which is quite similar to VASUN.

On these two datasets, we benchmark 17 visual attention models, which can be roughly divided into three groups: 1)~the \bl{BIO} group contains seven bio-inspired models, including SUN~\cite{zhang2008sun}, BMS~\cite{zhang2013saliency}, COV~\cite{erdem2013visual}, GBVS~\cite{harel2007graph}, CAS~\cite{CAS}, HFT~\cite{HFT} and AWS~\cite{Garciadiaz2012On}; 2)~the \bl{SL} group contains five shallow learning models, including ICL~\cite{hou2009dynamic}, SP~\cite{li2014visual}, SSD~\cite{SSD}, LDS~\cite{LDS} and FES~\cite{FES}; 3)~the \bl{DL} group contains five deep learning models, including  eDN~\cite{eDN}, iSEEL~\cite{R2016Exploiting}, SalNet~\cite{pan2016shallow}, SALICON~\cite{huang2015salicon} and SAM-ResNet~\cite{SAMresnet}. All these models have public source code on the Internet, and we use their default parameters to generate the attention maps. The predictions of these models are evaluated using five metrics, including AUC, sAUC, NSS, SIM and CC. The performance scores can be found in Table~\ref{tab:benchmarkScore}. Some representative results on VASUN can be found in Fig.~\ref{fig:benchmarkResult}.

From Table~\ref{tab:benchmarkScore}, we compare the performances of models from different groups. We find that on the MIT1003 dataset, deep attention models significantly outperform shallow learning models and bio-inspired models. However, on the VASUN dataset, we obtain a nearly reverse model ranking since all deep models suffer remarkable performance drops in terms of NSS. These results indicate that the significant performances of existing deep attention models may come from their capabilities in memorizing and detecting some specific visual patterns other than learning the inherent mechanism of human visual attention.

To further validate this point, we investigate the models in the three groups. For the bio-inspired group, the best-performing model over MIT1003 is GBVS, while the best model over VASUN changes to HFT. Note that GBVS is also the only bio-inspired model that suffers a performance drop when being applied on VASUN. This may be caused by the fact that the attention regions on solar images may have remarkably different sizes. As a result, the random walk of GBVS over graph nodes formed by fix-sized blocks may be insufficient to depict over-large and over-small attention regions. By contrast, HFT conducts scale-space analysis to select the most propriae scales, which enhances its capability to handle attention regions with various sizes.

\begin{table*}[t]
\caption{Benchmark of 17 models with default parameters. Bold means the highest score in each group.}
\centering{
\begin{tabular}{p{0.1cm}ccccccp{0.005cm}cccccc}
\toprule
\multicolumn{2}{c}{\multirow{2}*{Models}} & \multicolumn{5}{c}{MIT1003} && \multicolumn{5}{c}{VASUN}\\
\cline{3-7} \cline{9-13}
    & & AUC & sAUC & NSS & SIM & CC && AUC & sAUC & NSS & SIM & CC \\
\midrule
  \bl{\multirow{7}{*}{\rotatebox{90}{BIO}}}
    & SUN~\cite{zhang2008sun}      & 0.655 & 0.591 & 0.668 & 0.269 & 0.212 && 0.844 & 0.664 & 1.135 & 0.443 & 0.488\\
    & BMS~\cite{zhang2013saliency} & 0.790 & \bl{0.690} & 1.250 & 0.330 & 0.360 && 0.838 & 0.717 & 1.333 & 0.500 & 0.569\\
    & COV~\cite{erdem2013visual}   & 0.752 & 0.619 & 0.899 & 0.258 & 0.285 && 0.792 & 0.679 & 1.139 & 0.378 & 0.487\\
    & GBVS~\cite{harel2007graph}   & \bl{0.830} & 0.660 &\bl{1.380} & 0.360  & \bl{0.420} && 0.821 & 0.673 & 1.146 & 0.431 & 0.489\\
    & CAS~\cite{CAS}               & 0.758 & 0.632 & 0.992 & 0.320 & 0.310 && \bl{0.893} & 0.722 & 1.372 & 0.515 & 0.584\\
    & AWS~\cite{Garciadiaz2012On}  & 0.730 & 0.652 & 1.039 & 0.316 & 0.321 && 0.871 & 0.743 & 1.474 & \bl{0.537} & 0.629\\
    & HFT~\cite{HFT}               & 0.828 & 0.663 & 1.326 & {\bl{0.375}} & 0.419 && 0.887 & {\bl{0.749}} & {\bl{1.528}} & 0.522 & \bl{0.653}\\
\hline
  \bl{\multirow{5}{*}{\rotatebox{90}{SL}}}
    & ICL~\cite{hou2009dynamic}    & 0.605 & 0.515 & 0.422 & 0.244 & 0.130 && 0.874 & 0.767 & 1.564 & 0.556 & 0.665\\
    & SP~\cite{li2014visual}       & 0.827 & 0.663 & 1.372 & 0.401 & 0.434 && {\bl{0.893}} & {\bl{0.787}} & {\bl{1.768}} & {\bl{0.626}} & \bl{0.749}\\
    & SSD~\cite{SSD} & 0.787       & 0.651 & 1.342 & 0.398 & 0.427 && 0.858 & 0.717 & 1.501 & 0.560 & 0.633\\
    & LDS~\cite{LDS}               & {\bl{0.857}} & {\bl{0.680}} & {\bl{1.495}} & {\bl{0.434}} & {\bl{0.478}} && 0.860 & 0.734 & 1.553 & 0.571 & 0.656\\
    & FES~\cite{FES}               & 0.832 & 0.663 & 1.396 & 0.423 & 0.447 && 0.759 & 0.658 & 1.066 & 0.447 & 0.448\\
\hline
  \bl{\multirow{5}{*}{\rotatebox{90}{DL}}}
    & eDN~\cite{eDN}               & 0.851 & 0.657 & 1.292 & 0.304 & 0.408 && 0.837 & 0.698 & 1.078 & 0.378 & 0.464\\
    & iSEEL~\cite{R2016Exploiting} & 0.876 & \bl{0.743} & 1.870 & 0.460 & 0.587  && 0.841 & 0.648 & 0.987 & 0.451 & 0.425\\
    & SalNet~\cite{pan2016shallow} & 0.863 & 0.724 & 1.619 & 0.395 & 0.512 && 0.877 & 0.729 & 1.361 & 0.512 & 0.580\\
    & SALICON~\cite{huang2015salicon} & 0.870 & 0.740 & 2.120 & 0.600 & 0.740 && 0.857 & 0.705 & 1.400 & {\bl{0.537}} & 0.592\\
    & SAM-ResNet~\cite{SAMresnet}      & \bl{0.913} & 0.617 & {\bl{2.893}} & \bl{0.708} & \bl{0.768} && \bl{0.894} & \bl{0.743} & \bl{1.408} & 0.533 & \bl{0.599}\\
\bottomrule
\end{tabular}
}
\label{tab:benchmarkScore}
\end{table*}

\begin{figure*}[t]
\begin{center}
\includegraphics[width=1.00\textwidth]{benchmarkResult.png}
\end{center}
\caption{Representative results of 17 attention models on VASUN. GT means ground-truth.}
\label{fig:benchmarkResult}
\end{figure*}

Different from the bio-inspired models, shallow learning methods try to detect attention regions by learning the prior knowledge or fusing various pre-attentive features by machine learning methods based on thousands or even millions of images. As a result, models in the shallow-learning group generally perform better than the bio-inspired models on both datasets.  In addition, the best shallow learning model on MIT1003 is LDS, while SP becomes the best one over VASUN. This may be caused by the way of learning prior knowledge. LDS is a classic learning method that constructs a discriminative subspace on a small amount of images, while SP learns the foreground prior and the correlation prior from millions of images by modeling the spatial distributions and correlations of various visual stimuli. In this manner, the prior knowledge learned by SP becomes more robust than LDS and thus can be applied on various scenarios. In particular, SP outperforms all the other 16 models from the three groups, which further validates that the prior knowledge from big data is more robust and can improve the generalization ability of a model.

\begin{figure*}[t]
\begin{center}
   \includegraphics[width=1.00\textwidth]{baselines.png}
\end{center}
   \caption{The network architectures of five baseline models.}
\label{fig:baselines}
\end{figure*}

In the deep learning group, we find that many models suffer from a severe performance drop. In this group, the best performing model over MIT1003 is SAM-ResNet, whose performance drops remarkably on VASUN. This may be caused by the fact that the features in these deep models are learned other than being designed. While traditional features such as center-surround contrasts enhance the generalization ability of bio-inspired and shallow learning models, the deep features may over-fit to specific visual patterns that frequently appear in existing ground-level datasets ({\em e.g.}, faces, vehicles and texts). Such features can demonstrate impressive generalization ability over existing ground-level datasets since the training and testing data have similar distributions of such visual patterns. However, visual patterns in ground-level and solar images have remarkably different distributions. In this case, these features are likely to fail, leading to the poor performances of deep models on solar images.

To sum up, bio-inspired and shallow learning models have better generalization ability than deep models, and a better generalization ability may come from two aspects: 1)~bio-inspired pre-attentive features, 2)~robust prior knowledge for feature fusion. In particular, the prior knowledge unsupervisedly learned from millions of images demonstrates higher generalization ability. The success of deep models over ground-level datasets may mainly comes from their capability of learning feature representations and memorizing specific visual patterns, which is still far from the inherent mechanism of human visual attention. In other words, there is still a long way to go in the area of visual attention prediction even when the performance over existing ground-level datasets seems to be saturated in recent works.

\section{In-depth Analysis of Baseline Models}
In this section, we further provide an in-depth analysis of five baseline models to reveal two things: 1)~is the visual attention on the sun really predictable? and 2)~what key factors should be taken into account in designing a deep model for predicting visual attention on the sun.

The architecture of the five baseline models used in this study are shown in Fig.~\ref{fig:baselines}, from which we can see that each model slightly differs from the previous one to reveal the influence of a specific setting. All these baseline models adopt a sub-network $\mc{B}_{ini}$ with several convolution  layers to extract low-level features first, followed by the ResNet sub-network $\mc{B}_{res}$ to extract high-level features and de-convolution layers to derive attention maps. As shown in Fig.~\ref{fig:baselines}, the first baseline model $\mc{B}_1$ adopts 64 kernels with the size $3\times{}3$ in sub-network $\mc{B}_{ini}$ and 50 layers in sub-network $\mc{B}_{res}$, while the second and third baseline models $\mc{B}_2$ and $\mc{B}_3$ change the kernels to the size $7\times{}7$ and the kernel number to 256 in the sub-network $\mc{B}_{ini}$, respectively. The forth model $\mc{B}_4$ replace the sub-network $\mc{B}_{res}$ of $\mc{B}_1$ to 101 layers, while the last model $\mc{B}_5$ further takes two input streams with different resolutions. By measuring the performance variations of different baseline models, we can gradually find the key issues that should be considered in predicting visual attention on the sun.

\begin{table*}[t]
\caption{Performance of five baseline models on VASUN-testing before and after being fine-tuned on VASUN-training. Var (\%) is the NSS variation between VASUN and MIT1003.}
\centering{
\begin{tabular}{ccccccp{0.005cm}cccccc}
\toprule
{\multirow{2}*{Models}} & \multicolumn{5}{c}{Before Fine-tuning} && \multicolumn{5}{c}{After Fine-tuning} & \multirow{2}*{Var (\%)}\\
\cline{2-6} \cline{8-12}
    & AUC & sAUC & NSS & SIM & CC && AUC & sAUC & NSS & SIM & CC \\
\midrule
    Baseline $\mc{B}_1$ & 0.827 & 0.652 & 1.001 & 0.456 & 0.431 && 0.899 & 0.797 & 1.802 & 0.655 & 0.766 & +80.02\\
    Baseline $\mc{B}_2$ & 0.820 & 0.682 & 1.158 & 0.479 & 0.493 && 0.900 & 0.802 & 1.824 & 0.660 & 0.775 & +57.51\\
    Baseline $\mc{B}_3$ & 0.847 & 0.684 & 1.157 & 0.484 & 0.495 && 0.903 & 0.796 & 1.788 & 0.648 & 0.759 & +54.54\\
    Baseline $\mc{B}_4$ & 0.847 & 0.684 & 1.157 & 0.484 & 0.495 && 0.903 & 0.796 & 1.788 & 0.648 & 0.759 & +54.54\\
    Baseline $\mc{B}_5$ &\bl{0.866} &\bl{0.718} &\bl{1.275} &\bl{0.506} &\bl{0.515} &&\bl{0.958} &\bl{0.879} &\bl{2.223}&\bl{0.852}&\bl{0.947}& +74.35\\
\bottomrule
\end{tabular}
}
\label{tab:baselineScore}
\end{table*}

\begin{figure*}[t]
\begin{center}
\includegraphics[width=1.00\textwidth]{baselineResult.png}
\end{center}
\caption{Representative results of five baseline models before and after being fine-tuned on VASUN. GT means ground-truth.}
\label{fig:baselineResult}
\end{figure*}

In training the baseline model, we divide VASUN into a training subset (670 images), a validation subset (100 images) and a testing set (300 images). For the five baseline models, we first train them on the 10000 ground-level images from the SALICON dataset and then fine-tune them on VASUN. The performance scores of these five models on VASUN-testing are shown in Table~\ref{tab:baselineScore}, respectively. Some representative results of $\hat{\mc{B}}_i$ and $\mc{B}_i$ are simultaneously shown in Fig.~\ref{fig:baselineResult} to provide an intuitive comparison of what are actually predicted before and after the fine-tuning process on VASUN-training.

From Table~\ref{tab:baselineScore}, we find that all the five baseline models achieve a high performance on VASUN-testing after fine-tuning. As shown in Fig.~\ref{fig:baselineResult}, the predictions of all these five baseline models are very poor before being fine-tuned on VASUN-training. However, their AUC scores in these cases still reach up to 0.80-0.87, implying that the traditional AUC is not suitable for the performance evaluation when predicting visual attention on the sun. In particular, the baseline $\mc{B}_5$ that uses images at two-scales as the input can reach up to 0.96 in AUC and 0.95 in CC, implying perfect predictions in most cases. From these results, we can safely conclude that the visual attention on the sun is predictable, which also means the visual patterns, although quite different from those in ground-level images, are still learnable even with a simple deep network.

In addition, we conduct an analysis of these five models to show the key factors that should be considered in designing a deep model for predicting visual attention on the sun. By comparing the performance of $\mc{B}_1$ and $\mc{B}_2$, we find that $\mc{B}_2$ outperforms $\mc{B}_1$ before fine-tuning but the performances become comparable after fine-tuning. Considering that the only difference between $\mc{B}_1$ and $\mc{B}_2$ is the kernel size of the first convolution layer changes from $3\times3$ to $7\times7$. On solar images, a larger filter means that the low-level features take the contrast of bigger area into consideration, which can provide richer information for the high-level convolutional filters. Therefore, large-sized kernels should be considered with high priority in the problem of visual attention prediction on the sun.

The second comparisons are conducted between $\mc{B}_2$ and $\mc{B}_3$, which differ only in the number of kernels in the first convolution layer. Although the number of kernels in the first layer increases form 64 to 256, there is no obvious improvement in their performances on solar images. The reason is that the solar image in our dataset only has one channel, which means that 64 filters have sufficient capacity to represent the low-level features and the rest filters used in the first layer of $\mc{B}_3$ only have slight contribution during the training process. This means the problem of visual attention prediction is a simple problem with a limited number of attentive patterns. In addition, we find that $\mc{B}_2$ and $\mc{B}_3$ have the lowest performance variance before and after being fine-tuned on VASUN-training. This further validates that ground-level and solar images have different data distributions ({\em i.e.}, visual patterns) so that the generalization ability favors simple models and becomes inversely proportional to the model complexity.

The baseline $\mc{B}_4$ is the reflection of the influence caused by the depth of the network. Previous work shows that the semantic information will become more richer with the increase of the network depth and deeper models usually can get a better performance. However the results in Table~\ref{tab:baselineScore} suggest that when predicting visual attention on solar images, the deeper model $\mc{B}_4$ shows a slightly lower performance than $\mc{B}_2$ that only has a half number of layers. By analyzing the features of solar images, we find that the deeper parts in a network usually lose the information contained in the low-level feature maps, while such low-level features may be sufficient to the prediction of visual attention under free-viewing conditions.

The last factor that may influence the performance of the fixation prediction is handling multi-scale input images in $\mc{B}_5$. The results show that the performance of $\mc{B}_5$ has significant improvement compared with the base model $\mc{B}_2$. As shown in Fig.~\ref{fig:baselineResult}, we can see that the predictions of $\mc{B}_5$ are often cleaner than the other four baselines. This may be caused by two factors. First, a two-stream architecture can enhance the most attractive regions twice, leading to a cleaner map. Second, the multi-scale inputs have a better capability in capturing attractive regions with various sizes.

To sum up, visual attention on the sun is predictable and prefers large convolutional kernels, simple and shallow networks as well as multi-scale inputs in model design. The types of visual patterns and the data distribution in solar images are remarkably different from those in the ground-level images, making the VASUN a good benchmark dataset for generalization ability test.

\section{Conclusions}
In this paper, we revisit the problem of visual attention prediction from a novel perspective: the generalization ability. Instead of testing models across various ground-level datasets, we propose a new dataset of solar images, which contains many visual patterns that rarely appear in existing datasets. Be measuring the performance variations of massive visual attention models on ground-level and solar datasets, we find that classic attention models that adopt bio-inspired or shallow learning framework show stable performances, while the deep learning models suffer a remarkable performance drop. These results imply that the deep models only memorize visual patterns that frequently appear in existing datasets, and they are still far from capturing the inherent mechanism of human visual attention. In addition, we design five baseline models to step-wisely depict the key factors in designing visual attention models on the sun. From the results of these models, we find that visual attention on the sun is predictable and conclude key factors may be necessary to improve the performance and generalization ability in attention prediction on solar image.

{\small
\bibliographystyle{ieee}


\section{Introduction and Background}

Having a proper night of sleep and regular circadian rhythm is crucial for physical, mental and social well-being \cite{simon2018sleep,wulff2010sleep}. Sleep facilitates learning, memory-consolidation and emotion processing\cite{asfestani2018overnight,kurdziel2018sleep}. Identification of sleep stages is important not only in diagnosing and treating sleep disorders but also for understanding the neuroscience of healthy sleep. Polysomnography (PSG) is used in hospitals to study sleep and diagnose sleep disorders. It is considered the gold standard for sleep staging and involves recording of multiple electrophysiological signals from the body such as brain activity using EEG, heart rhythm through Electrocardiography (ECG), muscle tone through Electromyography (EMG) and eye-movement through Electrooculography (EOG). PSG is a tedious procedure which requires skilled sleep technologists in a laboratory setting. Since EEG is the most reliable signal for sleep staging, automation of PSG can be achieved through accurate classification of EEG\cite{lucey2016comparison}. Research in Deep Learning\cite{lecun2015deep,krizhevsky2012imagenet} has led to many efficient algorithms to classify different kinds of data including bio-medical and physiological signals. In this paper, we focus on developing a real-time sleep staging application using Time-distributed 1-D Deep Convolutional Neural Networks to classify the five sleep stages in a comfortable environment. As per the new AASM rules\cite{berry2012aasm}, these stages are\textemdash Wake, Rapid-Eye-Movement (REM) and Non-Rapid-Eye-Movement (N-REM) stages N1, N2, and N3. We make use of a single channel EEG recorded through a modified research version of the Muse headband\cite{krigolson2017choosing}. The headband is flexible and can be comfortably used while sleeping. It has 5 electrodes, namely, AF7, AF8, TP9, TP10 and reference Fpz, for brain activity measurements.
\paragraph{Importance of mobile systems for sleep staging}
 PSG requires a minimum of 22 wires attached to the body in order to monitor sleep activity. The complexity of this setup requires sleeping in a hospital or laboratory with an expert monitoring and scoring signals in real-time. This results in an unnatural and disturbed night of sleep for the subject which may not only affect the diagnosis but also causes sub-optimal utilization of time and energy resources for recording and scoring and is as such highly undesirable. There is significant development in research on automating sleep staging with wireless signals\cite{zhao2017learning} and more compact, wearable devices\cite{casson2010wearable,sano20170182}. Nevertheless, none of these systems implements a five-stage classification of sleep in real-time.

The goal of our research was to simplify and reliably automate PSG on-smart phone in just unit 30-second non-overlapping epochs for automatic real-time interventions during experiments on sleep stages and cognition as the minimum time-resolution of a single sleep score by an expert is 30 seconds. Automated classification is achieved through adaptation of a Time-Distributed Deep Convolutional Neural Network model. Simplification is achieved by developing TensorFlow Lite Android application that uses only a single channel recording from a wearable EEG. We have also designed a friendly user interface that visualizes sleep stages and raw EEG data with real-time statistics about accuracy. Our app connects via Bluetooth Low Energy (BLE) to the flexible EEG headband thus making it portable and not restricted to laboratory and hospital use.

\section{Related work}
Automatic analysis and sleep scoring using multi-layer Neural  Networks \cite{schaltenbrand1996sleep} was done as early as 1996 using 3 channels of physiological data, namely EEG, EOG and EMG. This involved power spectral density calculations for feature extraction from raw EEG which required a tedious laboratory setting to collect reliable data through these channels. More recent work has looked into creating portable sleep scoring systems, such as the work by Zhang et al. \cite{zhang2012rass}, that uses pulse, blood oxygen and motion sensors to predict sleep stages. In their paper, they do not detect sleep stages N1 and N2 separately, and N1 is usually the hardest one to predict. The authors already mention that these results cannot provide equally high accuracy as compared to the EEG and EOG signals of PSG. The same limitations apply to the work by Zhao et al\cite{zhao2017learning}. Our work achieves reliable accuracy by using only one channel from a wearable EEG, and overcomes the complexity of recording multiple signals.

Our model is based on Time-Distributed Deep Convolutional Neural Networks\cite{cvxtz2018eeg} and is inspired by the DeepSleepNet from Supratak et al.\cite{supratak2017deepsleepnet}. DeepSleepNet makes use of representation learning with a Convolutional Neural Network (CNN) followed by sequence residual learning using Bidirectional-Long Short Term Memory cells (Bi-LSTM). The major drawback of this network is that it requires 25 epochs of raw EEG data to be fed in together to obtain 25 labels. This is mainly because of the Bi-LSTM which relies on large temporal sequences to achieve better accuracy.

State-of-the art network model\textemdash SeqSleepNet\cite{phan2018seqsleepnet} processes multiple epochs and outputs the sleep labels all at once using end-to-end Hierarchical Recurrent Neural Networks. This uses all 3 channels\textemdash namely, EEG, EMG and EOG in order to give the best overall accuracy of 87.1\% on the MASS dataset\cite{o2014montreal}. CNN models by Sors et al.\cite{sors2018convolutional} and Tsinalis et al.\cite{tsinalis2016automatic}, as well as SeqSleepNet and DeepSleepNet all use longer temporal sequences for inference\textemdash 4, 5, 10 and 25 raw EEG epochs of 30 seconds respectively. We overcome this limitation by using Time-Distributed Deep CNN to predict single 30-second epochs with real-time adaptation from wearable EEG.

Flexibility of this wearable also makes it more preferable than the bulky system used by Lucey et al\cite{lucey2016comparison}. The smart-phone based nature of our sleep-staging application overcomes the need for a client-server architecture as used in Dreem headband\cite{patanaik2018end}. Our TensorFlow-Lite mobile application can also be adapted to other types of EEG devices for real-time settings.
\section{Methods and Materials}

\subsection{Dataset description and pre-processing}
We used the expanded Sleep-EDF database from Physionet-bank\cite{kemp2000analysis}. Single channel EEG (Fpz-Cz at 100Hz) of 20 subjects are divided into training set of 33 nights and validation set of 4 nights. Together, they contain non-overlapping nights of 19 subjects for 20 fold-cross validation. The non-overlapping test set contains 2 nights (1 subject).  We remove the extra wake states before and after half an hour of sleep as described in the DeepSleepNet\cite{supratak2017deepsleepnet}. We excluded \textit{MOVEMENT} and \textit{UNKNOWN} stages, and combined N4 and N3 to follow the five-stage classification as per the new AASM rules\cite{berry2012aasm}.

\subsection{Model architecture and training}
\begin{figure}[ht]
    \setlength{\fboxsep}{0pt}
    \setlength{\fboxrule}{1pt}
    \centering
    \includegraphics[width=12cm,height=0.77cm]{model_cnn.jpg}
    \caption{Model architecture used for training and inference}
    \label{model_cnn}
\end{figure}
Our model architecture is described in Figure \ref{model_cnn}. The Base-CNN has 3 repeated sets of two 1-D convolutional (Conv1D) layers, 1-D max-pooling and spatial dropout layers. This is followed by two Conv1D, 1-D global max-pooling, dropout and dense layers. We finally have a dropout layer as the output of Base-CNN. 30-second epochs of normalized raw EEG at 100Hz is fed into the Time-Distributed Base-CNN model\cite{cvxtz2018eeg} as described in the Figure \ref{model_cnn}. All Conv1D layers use Rectified-Linear-Units (ReLU) activation. The training uses an Adam optimizer of 0.001 with an initial learning rate of $e^{-3}$ which is reduced each time the validation accuracy plateaus using ReduceLROnPlateau Keras Callbacks.
\subsection{EEG adaptation and experiments} \paragraph{Pre-processing data from wearable EEG} Real-time brain activity from the flexible EEG headband is streamed via BLE to the smartphone application. Raw EEG from Af7 channel is down-sampled to 100Hz at the end of each 30-second epoch before feeding into the network.

\paragraph{EEG real-time adaptation}
Since the raw EEG recording instrument used for training is different from the testing instrument, we adapt EEG using a Z-score scaling with wake-stage standard deviation for calibration. The main reason for choosing standard deviation of EEG over any other metrics is highlighted in Figure \ref{sleep_features}. We estimate the mutual information for a discrete sleep stage variable. We use the set of statistical features shown in the diagram as input calculated every 30 seconds and the corresponding sleep-stage labels as output of the dataset. Final feature shown is Min-Max-Distance (MMD) as described by Aboalayon et al.\cite{aboalayon2016sleep}, calculated as a sum over euclidean distances between minimum and maximum EEG values with 1 second sliding window.

\begin{figure}[h!]
    \setlength{\fboxsep}{0pt}
    \setlength{\fboxrule}{1pt}
    \centering
    \includegraphics[width=13cm]{sleep_features_light.jpg}
    \caption{The heat-map shows the mutual importance of different statistical features of raw EEG. We choose the default of 3 nearest neighbours as the parameter for the mutual\_info\_classif function for automated feature-selection in scikit-learn machine learning library}
    \label{sleep_features}
\end{figure}

We calculated the statistical feature importance of raw EEG for 3 randomly selected nights. The standard deviation clearly has the major role with average relative importance of 53.33 percent over other features for the classification of sleep stages. Z-score calibration of wake stage employs this important feature, hence, using this method for 30-second epoch adaptation makes the raw-EEG both instrument-independent and subject-independent as long as signal is noise-reduced.

\section{Results}

Our model has an overall accuracy of 83.5\% for 20-fold cross validation of five-stage classification. The accuracy for nights from the test data ranges from 72\%(worst-case). This model achieves reliable accuracy given that the overall IRR (Inter-Rater-Reliability)\cite{danker2009interrater} among human experts scoring sleep recordings reported was about 80\% (Cohen’s $\kappa$ = 0.68 to 0.76).
\begin{table}[ht]
  \caption{Five-class classification report for sleep staging on 5 test nights}
  \label{class-table}
  \centering
  \begin{tabular}{llllll}
    \toprule
    Label    & Sleep Stage     & Precision     & Recall     & F1-score     & Support \\
    \midrule
    0    & Wake     & 0.83    & 0.96    & 0.89    & 730 \\
    1    & N1       & 0.47    & 0.42    & 0.44    & 337 \\
    2    & N2       & 0.87    & 0.83    & 0.85    & 2248 \\
    3    & N3       & 0.92    & 0.81    & 0.86    & 931 \\
    4    & REM      & 0.71    & 0.83    & 0.77    & 903 \\
    \midrule
    Micro    & average    & 0.82    & 0.82    & 0.82    & 5149 \\
    Macro    & average    & 0.76    & 0.77    & 0.76    & 5149 \\
    Weighted    & average    & 0.82    & 0.82    & 0.82    & 5149 \\
    \bottomrule
  \end{tabular}
\end{table}
Table\ref{class-table} describes the precision, recall, F1-score and support of all the five sleep stages on predictions from 5 test nights. The corresponding accuracy of 81.72\% and F1-score of 76.23\% was obtained. In addition, micro, macro and weighted average of these metrics are also calculated in order to give a better statistical understanding. The confusion matrix for the same night is shown in the left part of Figure \ref{sleep-multiple}. N1 stage shows the poorest agreement because of the absence of an occipital electrode\cite{lucey2016comparison}.
\begin{figure}[ht]
    \centering
    \includegraphics[height=6cm,width=9cm]{matrix_illustration.jpg}
    \caption{Confusion matrix (left). Snapshot of the headband and android application (right)}
    \label{sleep-multiple}
\end{figure}
The Figure \ref{hypnogram} shows the difference between the hypnogram of one full-night predicted by the model and the ground truth hypnogram as labeled in the dataset. The mobile application which deploys this deep learning model is built using TensorFlow Lite. The raw data from the wearable EEG is wirelessly streamed to the phone and updated every 30 seconds with its correspondent sleep stage and confidence value. We successfully validated real-time Rapid-Eye-Movement (REM) detection using our wearable headband by re-creating the same closed, lateral eye movement during wakefulness. We have also simulated and successfully validated jaw clenching and blinks during wakefulness.
\begin{figure}[h!]
    \centering
    \includegraphics[height=5cm,width=12cm]{hypnogram_compare_new.jpg}
    \caption{Comparison of the ground truth and predicted hypnograms of one full night}
    \label{hypnogram}
\end{figure}
\section{Conclusion and future scope}
This work demonstrates an end-to-end mobile pipeline for the fastest real-time sleep-staging by adaptation of a wearable EEG. With the development of the new mobile TensorFlow Lite application, we achieve automated sleep staging without the need of servers, in a portable way that can be used anywhere, including the home. The application is versatile as it can be adapted to take in single channel(Fpz-Cz) recordings from any wearable EEGs. We aim to use this work for real-time interventions using Brain Computer Interfaces (BCI) for applications in Human Computer Interaction (HCI), such as wearable olfactory interfaces \cite{amores2018bioessence,amores2017essence}, real-time audio-neural feedback\cite{schutze2015difficulty,dafna2018sleep} and sleep-based enhancement of learning and memory\cite{rasch2007odor,andrillon2017formation}.
\newpage
\bibliographystyle{plain}
\small{