\section{Introduction}
Semantic segmentation is one of the fundamental tasks in computer vision, with a wide range of applications such as image editing and scene understanding.
In order to obtain reliable models and achieve promising performance, recently deep neural network (DNN) based methods \cite{fcn_pami,deeplab,dilated} are learned from fully-supervised data that requires pixel-wise semantic annotations.
However, acquiring such pixel-wise annotations is usually time-consuming and labor-intensive, which limits the application potentials in the real world.
As a result, numerous approaches tackle this issue via training models only on weakly-annotated data, \textit{e.g.}, image-level \cite{ahn2018learning,kolesnikov2016seed,pathak2015constrained,pinheiro2015image}, bounding box \cite{papandreou2015weakly,dai2015boxsup,khoreva_CVPR17}, point-level \cite{Bearman_ECCV16}, scribble-based \cite{lin2016scribblesup,Vernaza_CVPR17}, or video-level \cite{Chen_IJCV_2020,Zhong_ACCV_2016,Tsai_ECCV_2016} labels.
In this paper, we focus on utilizing the image-level label, which is the most efficient scheme for weak annotations but also a challenging scenario.

Existing weakly-supervised semantic segmentation (WSSS) algorithms mainly operate with three steps: 1) localizing objects via a categorical response map, 2) refining the response map to generate pseudo annotations, and 3) training the semantic segmentation network using pseudo ground truths.
Recent methods \cite{ahn2018learning,huang2018weakly,wang2018weakly,wei2018revisiting} have achieved significant progress for WSSS, but most of them focus on improving the latter two steps.
Since the success of these sequential steps hinges on the quality of the initial response map generated in the first step, in this paper we present an effective solution to localize objects.

One common practice to produce the initial response map is using class activation map (CAM) \cite{zhou2016learning}. However, since CAM is typically supervised by a classification loss that could be sufficiently optimized through seeing only a small portion of objects, the generated response map usually only attends on partial regions (see Figure \ref{fig: teaser}(a)).
To tackle this issue, recent methods \cite{lee2019ficklenet,jiang2019integral} make efforts to improve the response map via using the dropout strategy that increases the model uncertainty or aggregating maps produced at different stages to see more object parts.
However, there remains a challenge whether there are better loss function designed to explicitly facilitate the model training and produce better response maps, which are not addressed in prior works.

In this paper, we propose a principled and end-to-end trainable network with loss functions designed to systematically control the generation of the response map.
First, inspired by the mixup data augmentation in \cite{zhang2017mixup}, we observe that including mixup could effectively calibrate the model uncertainty on overconfident predictions \cite{Thulasidasan2019mixup} and in return enables the model to attend to more object regions.
However, it is challenging to control the mixup augmentation process and the model uncertainty, due to non-uniform response distributions (see Figure \ref{fig: teaser}(b)), which may affect subsequent response refinement steps.
Therefore, we introduce another two loss terms to the mixup process by regularizing the class-wise uncertainty and the spatial response distribution.
We refer to our model as \textit{Mixup-CAM} and show that the produced response map is more complete and balanced across object regions (see Figure \ref{fig: teaser}(c)), which facilitates the latter response refinement and segmentation model training steps.

We conduct quantitative and qualitative experiments to demonstrate the effectiveness of the proposed Mixup-CAM method on the PASCAL VOC 2012 dataset \cite{PASCAL_VOC_2010_Data}.
To the best of our knowledge, our algorithm is the first to demonstrate that mixup could improve the WSSS task on complicated multi-labeled images, along with other designed loss functions to produce better response maps.
In addition, we present the ablation study and more analysis to validate the importance of each designed loss.
Finally, we show that our method achieves state-of-the-art semantic segmentation performance against existing approaches.

\begin{figure*}[t]
	\centering
	\includegraphics[width=0.95\linewidth]{teaser.pdf}\\
	\vspace{3mm}
	\caption{Comparisons of (a) the original CAM method; (b) CAM + the mixup data augmentation; and (c) the proposed Mixup-CAM framework that integrates the mixup scheme and the uncertainty regularization.
	%
	Compared to (a) and (b), our final response map (c) attends to other object parts with more uniformly distributed response.
	}
	\label{fig: teaser}
	\vspace{-5mm}
\end{figure*}

\section{Related Work}

\paragraph{Initial Prediction for WSSS.}
Initial cues are essential for segmentation tasks since they provide reliable priors to generate segmentation maps. The class activation map (CAM)  \cite{zhou2016learning} is a common practice for localizing objects. It highlights class-specific regions that are served as the initial cues. However, since the CAM model is trained by a classification network, it tends to attend to small discriminative parts of the object, leading to incomplete initial masks.
Several methods have been developed to alleviate this problem. Approaches like \cite{singh2017hide, wei2017object, zhang2018adversarial, li2018tell} deliberately hide or erase the regions of an object, forcing the model to look for more diverse parts. However, such strategies require iterative model training and response aggregation steps. After gradual expansion of the attention regions, non-object regions are prone to be activated, which leads to inaccurate attention maps. Other algorithms \cite{hou2018self, zeng2019joint} use both object and background cues to prevent the attention map from including more background regions, yet pixel-level saliency labels are used.

Instead of using the erasing scheme, recently the FickleNet approach \cite{lee2019ficklenet} introduces the stochastic feature selection to obtain a diverse combination of locations on feature maps. Moreover, the OAA method \cite{jiang2019integral} adopts an online attention accumulation strategy to collect various object parts discovered at different training stages. By aggregating the attention maps, it could obtain an initial cue that contains a larger region of the object.
Unlike methods that mitigate the problem by discovering complementary regions via iterative erasing steps or consolidating attention maps, our proposed approach aims at harnessing the uncertainty in end-to-end classification learning. In addition, by regularizing both class-wise uncertainties and spatial response distributions, our approach averts the attention from focusing on small parts of the semantic objects, hence producing much improved response maps.

\vspace{-3mm}
\paragraph{Response Refinement for WSSS.}
Various approaches \cite{ahn2018learning,fan2018cian, fan2018associating,huang2018weakly,kolesnikov2016seed,wang2018weakly,wei2018revisiting} are proposed to refine the initial cue by expanding the region of attention maps. Other methods \cite{ahn2018learning, fan2018cian, fan2018associating} are developed using affinity learning. The recent SSDD scheme \cite{shimoda2019self} proposes a difference detection module to estimate and reduce the gap between the initial mask and final segmentation results in a self-supervised manner.
However, the performance of these methods is limited as initial seeds are still obtained from CAM-like methods. If these seeds only come from the discriminative parts of the object, it is difficult to expand regions into non-discriminative parts. Moreover, the initial prediction may produce wrong attention regions, which would lead to even more inaccurate regions in subsequent refinement steps.

\vspace{-3mm}
\paragraph{Label-preserving \emph{vs.} Non-preserving Augmentations.}
Data augmentation is a common regularization technique in both supervised and unsupervised learning \cite{cubuk2019autoaugment,lim2019fast,he2019momentum}. Conventional data augmentation techniques such as scaling, rotation, cropping, color augmentation, and Gaussian noise can change the pixel values of an image without altering its labels. These label-preserving transformations are commonly applied in training deep neural networks to improve the model generalization capabilities.

Recent work has demonstrated that even non-label-preserving data augmentation can be surprisingly effective. Explicit label smoothing has been adopted successfully to improve the performance of deep neural models. The Mixup method \cite{zhang2017mixup} is proposed to train a neural network on a convex combination space of image pairs and their corresponding labels. It has been proven effective for the classification task and increases the robustness of neural networks. Numerous Mixup variants \cite{berthelot2019mixmatch, summers2019improved, Yun2019mixup, verma2018manifold, Thulasidasan2019mixup,guo2019mixup} have been proposed to extend mixup for better prediction of uncertainty and calibration of the DNNs. These methods exhibit shared similarity of producing better-generalized models.

\vspace{-3mm}
\paragraph{Entropy Regularization.}
Aside from mixup schemes for predictive uncertainty, another common uncertainty measure is entropy, which could act as a strong regularizer in both supervised and semi-supervised learning \cite{pereyra2017regularizing,grandvalet2005semi}.
In particular, \cite{pereyra2017regularizing} discourages the neural network from being over-confident by penalizing low-entropy distributions, while \cite{grandvalet2005semi} utilizes entropy minimization in a semi-supervised setting as a training signal on unlabeled data.
In this paper, we also adopt the entropy-based loss to regularize the uncertainty, coupled with the mixup data augmentation for producing better response maps on objects.

\begin{figure*}[t]
	\centering
	\includegraphics[width=0.95\linewidth]{framework.pdf}\\
	\vspace{3mm}
	\caption{Overview of Mixup-CAM. We perform mixup data augmentation on input images with their corresponding labels via \eqref{eq:mixup} and pass the mixed image through the feature extractor $E$ and the classifier $G$ to obtain the probability score $P^c$ for each category $c$. For loss functions, in addition to the classification loss $\mathcal{L}_{cls}$ on mixup samples, we design two terms to regularize class-wise entropy ($\mathcal{L}_{ent}$ via \eqref{eq:l_ent}) and spatial distribution on CAM ($\mathcal{L}_{con}$ via \eqref{eq:l_con}).
	}
	\label{fig: framework}
	\vspace{-5mm}
\end{figure*}

\section{Weakly-supervised Semantic Segmentation}
We first describe the overall algorithm and introduce details of the proposed Mixup-CAM framework with loss functions designed to improve the initial response map.
We then detail how to generate the final semantic segmentation results.

\subsection{Algorithm Overview}
One typical way to generate response maps for annotated object categories is to use
CAM \cite{zhou2016learning}.
However, these response maps tend to focus on discriminative object parts, which are less effective for the WSSS task.
One reason is that CAM relies on the classification loss, which only requires partial object regions to be activated during training. As a result, when the objective is already optimized with high confidence, the model may not attempt to learn other object parts.

In this paper, we propose to integrate the idea of mixup data augmentation \cite{zhang2017mixup}, thereby calibrating the uncertainty in prediction \cite{Thulasidasan2019mixup} as well as allowing the model to attend to other regions of the image.
Although we find that adding mixup could improve the response map, sometimes the response could diverge too much, resulting in more false-positive object regions.
To further regularize this uncertainty, we introduce two additional loss terms: the spatial loss and the class-wise loss.
We illustrate the overall model and loss designs in Figure~\ref{fig: framework} and provide more details in the next subsection.

After receiving the initial response map, we utilize the method in \cite{ahn2018learning} to expand and refine the response.
Finally, we generate pseudo ground truths from the refined response and train a semantic segmentation network to obtain the final segmentation output.
Note that while we focus on the first step of the initial response map in this paper, the succeeding two steps could be replaced with alternative modules or models.

\subsection{Mixup-CAM}
\paragraph{CAM Generation.}
We first describe the CAM method for producing the initial response map as our baseline (see Figure \ref{fig: teaser}(a)). The base network begins with a feature extractor $E$, followed by a global average pooling (GAP) layer and a fully-connected layer $G$ as the output classifier.
Next, given an input image $I$ with its image-level labels $Y$, the network is trained with a multi-label classification loss $\mathcal{L}_{cls}(Y, G(E(I)))$ following \cite{zhou2016learning}.
After training this classification network, the activation map $M^c$ for each category $c$ is obtained by applying the $c$-channel classifier weight $\theta_G^c$ on the feature map $f = E(I)$:
\begin{equation}
    M^c = \theta_G^{^c \top} f.
    \label{eq:cam}
\end{equation}
Finally, the response is normalized by the maximum value of $M^c$.

\vspace{-3mm}
\paragraph{Mixup Data Augmentation.}
Since the original classification network could easily obtain high confidence, in which the generated CAM only attends to small discriminative object parts, we utilize mixup data augmentation to calibrate the uncertainty in prediction \cite{Thulasidasan2019mixup}.
Given an image pair $\{I_1, I_2\}$ and its label $\{Y_1, Y_2\}$ randomly sampled from the training set, we augment an image $I'$ and its label $Y'$ via:
\begin{align}
    I' & = \lambda_{mix} I_1 + (1-\lambda_{mix}) I_2, \notag \\
    Y' & = \lambda_{mix} Y_1 + (1-\lambda_{mix}) Y_2,
    \label{eq:mixup}
\end{align}
where $\lambda_{mix}$ is sampled from the \textit{Beta}$(\alpha, \alpha)$ distribution following \cite{zhang2017mixup}.
Using this augmented data, we feed it into the classification network to minimize the loss $\mathcal{L}_{cls}(Y', G(E(I')))$ and follow the same procedure in \eqref{eq:cam} to produce the response map (see Figure \ref{fig: teaser}(b)).

Compared to the original CAM generation, our network no longer receives a pure image but a mixed image that could have multiple objects with their weights based on $\lambda_{mix}$ as in \eqref{eq:mixup}.
Therefore, the predictive uncertainty could be enhanced, leading to smoother output distributions and enforcing the model to pay attention to other regions in the image in order to satisfy the classification loss $\mathcal{L}_{cls}(Y', G(E(I')))$.
\vspace{-3mm}
\paragraph{Uncertainty Regularization.}
Although mixup could improve the response map by looking at other parts in the image, sometimes the response could become too divergent and thus attend to pixels non-uniformly, \textit{e.g.}, Figure \ref{fig: teaser}(b).
This is attributed to the difficulty in controlling the quality of mixed images, especially when the model faces more complicated images such as PASCAL VOC, \textit{e.g.}, an object could appear at various locations of the image with noisy background clutters.

To further facilitate the mixup process, we propose to self-regularize the uncertainty via class-wise loss and spatial loss terms.
The first term is to directly minimize the entropy in output prediction from the classifier to reduce uncertainty:
\begin{equation}
    \mathcal{L}_{ent}(G(E(I'))) = - \frac{1}{HW} \sum_{h,w} \sum_{c \in C} P^c(h,w) \log P^c(h,w),
    \label{eq:l_ent}
\end{equation}
where $C$ is the category number and $P^c \in \mathbb{R}^{H \times W}$ is the output probability for category $c$.
Since our classifier $G$ outputs multi-label probability, we concatenate the probabilities and normalize them by the maximum value, then calculate the final $P$ with \textit{softmax}.

Although the first term has the ability to minimize the uncertainty, it does not explicitly operate on the response map. To better balance the distribution on the response, we utilize a concentration loss similar to \cite{hung2019scops} and apply it directly on CAM for each category (\textit{i.e.}, $M^c$), which encourages activated pixels to be spatially close to the response center:
\begin{equation}
    \mathcal{L}_{con}(M) = \sum_{c \in \bar{C}} \sum_{h,w} ||\langle h,w\rangle - \langle \mu_h^c,\mu_w^c\rangle||^2 \cdot \hat{M}^c(h,w),
    \label{eq:l_con}
\end{equation}
where $\mu_h^c = \sum_{h,w} h \cdot \hat{M}^c(h,w)$ is the center in height for category $c$ (similarly for $\mu_w^c$), $\hat{M}^c$ is the normalized response of $M^c$ to represent a spatially distributed probability map.
Note that, here we only calculate the concentration loss on presented categories $\bar{C}$ as provided in the image-label $Y$ to avoid confusing the model with invalid categories.

\vspace{-3mm}
\paragraph{Overall Objective.}
We have described our proposed Mixup-CAM framework, including mixup data augmentation in \eqref{eq:mixup} and two regularization terms, \textit{i.e.}, \eqref{eq:l_ent} and \eqref{eq:l_con}. To train the entire model in an end-to-end fashion, we perform the online mixup procedure and jointly optimize the following loss functions:
\begin{equation}
    \mathcal{L}_{all} = \mathcal{L}_{cls}(I', Y') + \lambda_{ent} \mathcal{L}_{ent}(I') + \lambda_{con} \mathcal{L}_{con}(M).
    \label{eq:l_all}
\end{equation}
For simplicity, we omit the detailed notation inside each loss term. We also note that $M$ is produced online via computing \eqref{eq:cam} on valid categories in each forward iteration.

\subsection{Implementation Details}
\paragraph{Classification Network.}
Similar to \cite{ahn2018learning},
we use the ResNet-38 architecture \cite{wu2019wider} as our classification network, which consists of 38 convolution layers with wide channels, followed by a $3 \times 3$ convolution layer with 512 channels for better adaptation to the classification task, a global average pooling layer, and two fully-connected layers for classification.
In training, we adopt the pre-trained model on ImageNet \cite{deng2009imagenet} and finetune it on the PASCAL VOC 2012 dataset.
Typical label-preserving data augmentations, \textit{i.e.}, horizontal flip, random cropping, random scaling, and color jittering, are utilized on the training set.

We implement the proposed Mixup-CAM framework using PyTorch with a single Titan X GPU with 12 GB memory.
For training the classification network, we use the Adam optimizer \cite{kingma2014adam} with initial learning rate of 1e-3 and the weight decay of 5e-4.
For mixup, we use $\alpha = 0.2$ in the \textit{Beta}$(\alpha,\alpha)$ distribution.
For uncertainty regularization, we set $\lambda_{ent}$ as 0.02 and $\lambda_{con}$ as 2e-4. Unless specified otherwise, we use the same parameters in all the experiments.
In the experimental section, we show studies for the sensitivity of different parameters.
\vspace{-3mm}
\paragraph{Semantic Segmentation Generation.}

Based on the response map generated by our Mixup-CAM, we adopt the random walk approach via affinity \cite{ahn2018learning} to refine the response and produce pixel-wise pseudo ground truths for semantic segmentation.
In addition, similar to existing methods, we adopt dense conditional random fields (CRF) \cite{crf} to further refine the response and obtain better object boundaries.
Finally, we utilize the Deeplab-v2 framework \cite{deeplab} with the ResNet-101 architecture \cite{He_2016_CVPR} and train the segmentation network

\vspace{-2mm}
\section{Experimental Results}
In this section, we present our main results of the proposed Mixup-CAM method for the WSSS task.
First, we show that our approach achieves better initial response maps and further improves the subsequent refinement step.
Second, we demonstrate the importance of each designed component.
Finally, we provide evaluations on final semantic segmentation outputs in the PASCAL VOC dataset \cite{PASCAL_VOC_2010_Data} against the state-of-the-art approaches.
More results can be found in the supplementary material. We will make our code and models available to the public.

\vspace{-2mm}
\subsection{Evaluated Dataset and Metric}
We conduct experiments on the PASCAL VOC 2012 semantic segmentation benchmark \cite{PASCAL_VOC_2010_Data} with 21 categories, including one background class.
Following existing WSSS methods, we use augmented 10,528 training images \cite{hariharan2011semantic} to train our network.
For evaluation of response maps of the training set, we use the set without augmentation with 1,464 examples, following the setting in \cite{ahn2018learning}.
For final semantic segmentation results, we use 1,449 images in the validation set to compare our results with other methods\footnote{Although there is a test set that can be evaluated on the official PASCAL VOC website, by the submission deadline the website is still out of service for returning the evaluated performance.}.
In all experiments, the mean Intersection-over-Union (mIoU) ratio is used as the evaluation metric.

\begin{figure*}[t]
	\centering
	\includegraphics[width=0.95\linewidth]{response.pdf}
	\vspace{3mm}
	\caption{Sample results of initial responses. Our approach often produces the response map that covers more complete region of the object (i.e., attention on the body of the animal), while the initial cue obtained by CAM \cite{zhou2016learning} is prone to focus on small discriminative parts.}
	\label{fig: response_map}
	\vspace{-2mm}
\end{figure*}

\begin{table}[!t]
	\caption{IoU results of CAM and its refinement on the PASCAL VOC training set.}
	\vspace{3mm}
	\small
	\centering
	\renewcommand{\arraystretch}{1.1}
	\setlength{\tabcolsep}{6pt}
	\begin{tabular}{lcc}
		\toprule
		Method & CAM & CAM + Refinement \\
		\midrule

		AffinityNet \cite{ahn2018learning} & 48.0 & 58.1 \\
		Mixup $\mathcal{L}_{cls}$ & 49.3 & 60.5 \\
		Mixup $\mathcal{L}_{cls}$ + $\mathcal{L}_{ent}$ & 49.5 & 61.6 \\
		Mixup $\mathcal{L}_{cls}$ + $\mathcal{L}_{con}$ & 49.9 & 61.7 \\
		Mixup $\mathcal{L}_{cls}$ + $\mathcal{L}_{ent}$ + $\mathcal{L}_{con}$ & \textbf{50.1} & \textbf{61.9} \\

		\bottomrule
	\label{table: ablation_loss}
	\end{tabular}
	\vspace{-7mm}
\end{table}

\vspace{-2mm}
\subsection{Ablation Study and Analysis}

\paragraph{Improvement on Response Map.}

We first present results of the initial and refined response maps.
In Table \ref{table: ablation_loss}, we show the performance for the original CAM used by the baseline AffinityNet \cite{ahn2018learning}, our CAM using the mixup data augmentation (Mixup $\mathcal{L}_{cls}$), and our final Mixup-CAM with mixup and uncertainty regularization (Mixup $\mathcal{L}_{cls}$ + $\mathcal{L}_{ent}$ + $\mathcal{L}_{con}$).
In both results of CAM and its refinement, our IoU improvements are consistent after gradually adding the mixup augmentation and regularization.
In addition, Figure \ref{fig: response_map} shows some example results of the initial response, which illustrates that our Mixup-CAM is able to make the network attend to more object parts and produce more uniform response distributions on objects.

\begin{figure*}[t]
	\centering
	\includegraphics[width=0.95\linewidth]{rw_diff.pdf}\\
	\vspace{3mm}
	\caption{Enhancement on refinement. Our regularization enforces a more uniform response on objects, which can facilitate the refinement step. The examples illustrate that the IoU difference of the resultant refined map is significantly larger than the one of initial response.}
	\label{fig: rw_diff}
	\vspace{-3mm}
\end{figure*}

\begin{figure*}[t]
	\centering
	\includegraphics[width=0.95\linewidth]{para.pdf}
	\vspace{3mm}
	\caption{Sensitivity analysis for parameters. (a) $\alpha$ for mixup augmentation;
	(b) $\lambda_{ent}$ and (c) $\lambda_{con}$ for uncertainty regularization.
	}
	\label{fig: para}
	\vspace{-5mm}
\end{figure*}

\vspace{-3mm}
\paragraph{Effectiveness of Regularization.}

One interesting aspect we find is that adding regularization could enhance the effectiveness of the refinement step.
In Table \ref{table: ablation_loss}, compared to Mixup $\mathcal{L}_{cls}$, adding either $\mathcal{L}_{ent}$ (3rd row) or $\mathcal{L}_{con}$ (4th row) improves the CAM IoU by 0.2\% and 0.6\% respectively.
Nevertheless, with the refinement, the corresponding improvements in IoU is 1.1\% and 1.2\%, which are larger than the ones before refining the response.
This is because our regularization enforces a more uniform response on objects, which greatly facilitates the refinement step (\textit{e.g.}, via region expanding).
In addition, we illustrate one example in Figure \ref{fig: rw_diff}, where the IoU difference of initial response is relatively small, but the resultant refined map could differ significantly.

\vspace{-3mm}
\paragraph{Parameter Sensitivity.}
In this paper, we mainly study three parameters in our Mixup-CAM framework, i.e., $\alpha$ for mixup regularization and $\{\lambda_{ent}, \lambda_{con}\}$ in uncertainty regularization.
In Figure \ref{fig: para}(a), when increasing the $\alpha$ value, the \textit{Beta} distribution would become more uniform, which encourages a more uniform $\lambda_{mix}$ in \eqref{eq:mixup} and results in mixed images that are more challenging to optimize.
Nevertheless, the IoUs of Mixup $\mathcal{L}_{cls}$ under various $\alpha$ are consistently better than the CAM baseline.
For regularization terms, we fix $\lambda_{con} = 2e-4$ and adjust $\lambda_{ent}$ in Figure \ref{fig: para}(b), while fixing $\lambda_{ent} = 0.2$ and change $\lambda_{con}$ in Figure \ref{fig: para}(c).
Both figures show that these two parameters are robust to the performance under a wide range.

\subsection{Semantic Segmentation Performance}
After generating the pseudo ground truths using the refined response map, we use them to train the semantic segmentation network.
First, we compare our method with state-of-the-art algorithms using the ResNet-101 architecture or other similarly powerful ones in Table \ref{table: compare_sota}.
Note that, while our method focuses on improving the initial responses on the object, most methods aim to improve the refinement step or segmentation network training.
In Table \ref{table: compare_sota_detail}, we further present detailed performance for each category.
We show two groups of results without (top rows) or with (bottom rows) applying CRF \cite{crf} to refine final segmentation outputs.
Compared to the recent FickleNet \cite{lee2019ficklenet} approach that also tries to improve the initial response map, our proposed Mixup-CAM shows favorable performance in final semantic segmentation results.

\begin{table}[!t]
\caption{Comparison of WSSS methods using image-level labels on the PASCAL VOC 2012 validation set. $\checkmark$ indicates the methods that focus on improving the initial response. The result of $\dagger$ on AffinityNet is re-produced by training the same ResNet-101 as our pipeline.}
\vspace{3mm}
\label{tab:orbital_data}
\centering
\scriptsize
\renewcommand{\arraystretch}{1.1}
\setlength{\tabcolsep}{4pt}
\begin{tabular}{lccc}
\toprule
\text{Method} & {Backbone} & {Init. Resp.} & {IoU on Val} \\
\midrule
    MCOF \textsubscript{CVPR'18} \cite{wang2018weakly} & ResNet-101 &  & 60.3 \\

    DCSP \textsubscript{BMVC'17} \cite{chaudhry2017discovering} & ResNet-101 &  & 60.8 \\

    DSRG \textsubscript{CVPR'18} \cite{huang2018weakly} & ResNet-101 &  & 61.4 \\

    AffinityNet \textsubscript{CVPR'18} \cite{ahn2018learning} & Wide ResNet-38 & & 61.7 \\

    AffinityNet$^{\dagger}$ \textsubscript{CVPR'18} \cite{ahn2018learning} & ResNet-101 & & 61.9 \\

    SeeNet \textsubscript{NIPS'18} \cite{hou2018self} & ResNet-101 & \checkmark & 63.1 \\

    Zeng \textit{et al} \textsubscript{ICCV'19} \cite{zeng2019joint} &  DenseNet-169 &  & 63.3 \\

    BDSSW \textsubscript{ECCV'18} \cite{fan2018associating} & ResNet-101 & & 63.6  \\

    OAA \textsubscript{ICCV'19} \cite{jiang2019integral} & ResNet-101 & \checkmark & 63.9 \\

    CIAN \textsubscript{CVPR'19} \cite{fan2018cian} & ResNet-101 &  & 64.1 \\

    FickleNet \textsubscript{CVPR'19} \cite{lee2019ficklenet} & ResNet-101 & \checkmark & 64.9 \\

    SSDD \textsubscript{ICCV'19} \cite{shimoda2019self} & Wide ResNet-38 &  & 64.9 \\

    Ours & ResNet101 & \checkmark & \textbf{65.6} \\

\bottomrule
\label{table: compare_sota}
\end{tabular}
\vspace{-5mm}
\end{table}

\begin{table*}[!t]
	\caption{Semantic segmentation performance on the PASCAL VOC 2012 validation set. The bottom group contains results with CRF refinement, while the top group is without CRF.
	The best three results are in \textcolor{red}{red}, \textcolor{PineGreen}{green} and \textcolor{blue}{blue}, respectively.}

	\vspace{3mm}
	\footnotesize
	\centering
	\renewcommand{\arraystretch}{1.3}
	\setlength{\tabcolsep}{1pt}

	\scriptsize
	\begin{tabular}{lccccccccccccccccccccc|c}
		\toprule

		Method & \rotatebox{90}{bkg} & \rotatebox{90}{aero} & \rotatebox{90}{bike} & \rotatebox{90}{bird} & \rotatebox{90}{boat} & \rotatebox{90}{bottle} & \rotatebox{90}{bus} & \rotatebox{90}{car} & \rotatebox{90}{cat} & \rotatebox{90}{chair} & \rotatebox{90}{cow} & \rotatebox{90}{table} & \rotatebox{90}{dog} & \rotatebox{90}{horse} & \rotatebox{90}{motor} & \rotatebox{90}{person} & \rotatebox{90}{plant} & \rotatebox{90}{sheep} & \rotatebox{90}{sofa} & \rotatebox{90}{train} & \rotatebox{90}{tv} & mIoU \\
		\midrule

		AffinityNet  \cite{ahn2018learning} & 88.2 & 68.2 & 30.6 & \textcolor{PineGreen}{81.1} & 49.6 & 61.0 & 77.8 & 66.1 & 75.1 & 29.0 & 66.0 & 40.2 & 80.4 & 62.0 & \textcolor{blue}{70.4} & \textcolor{red}{73.7} & \textcolor{blue}{42.5} & 70.7 & 42.6 & \textcolor{blue}{68.1} & 51.6 & 61.7 \\

		Ours (w/o CRF) & 87.6 & 54.5 & 30.7 & 73.0 & 46.5 & \textcolor{PineGreen}{72.0} & \textcolor{PineGreen}{86.5} & \textcolor{PineGreen}{74.8} & \textcolor{PineGreen}{87.6} & \textcolor{blue}{31.3} & \textcolor{blue}{80.8} & \textcolor{PineGreen}{50.3} & \textcolor{blue}{82.6} & 74.5 & 67.2 & 68.7 & 39.6 & 79.2 & \textcolor{PineGreen}{44.8} & 64.9 & 51.1 & \textcolor{blue}{64.2} \\

		\midrule

		MCOF \cite{wang2018weakly}  & 87.0 &  \textcolor{red}{78.4} & 29.4 & 68.0 & 44.0 & 67.3 & 80.3 & 74.1 & 82.2 & 21.1 & 70.7 & 28.2 & 73.2 & 71.5 & 67.2 & 53.0 & \textcolor{PineGreen}{47.7} & 74.5 & 32.4 & \textcolor{PineGreen}{71.0} & 45.8 & 60.3 \\

		Zeng et al. \cite{zeng2019joint} &  \textcolor{red}{90.0} & \textcolor{PineGreen}{77.4} &  \textcolor{red}{37.5} & \textcolor{blue}{80.7} &  \textcolor{red}{61.6} & 67.9 & 81.8 & 69.0 & 83.7 & 13.6 & 79.4 & 23.3 & 78.0 & \textcolor{blue}{75.3} &  \textcolor{PineGreen}{71.4} & 68.1 & 35.2 & 78.2 & 32.5 & \textcolor{red}{75.5} & 48.0 & 63.3\\

		FickleNet \cite{lee2019ficklenet} &
		\textcolor{PineGreen}{89.5} & \textcolor{blue}{76.6} & \textcolor{PineGreen}{32.6} & 74.6 & \textcolor{blue}{51.5} & \textcolor{blue}{71.1} & \textcolor{blue}{83.4} & \textcolor{blue}{74.4} & 83.6 & 24.1 & 73.4 & 47.4 & 78.2 & 74.0 & 68.8 & \textcolor{PineGreen}{73.2} &  \textcolor{red}{47.8} &  \textcolor{blue}{79.9} & 37.0 & 57.3 & \textcolor{red}{64.6} & \textcolor{PineGreen}{64.9} \\

		SSDD \cite{shimoda2019self} & \textcolor{blue}{89.0} & 62.5 & 28.9 & \textcolor{red}{83.7} & \textcolor{PineGreen}{52.9} & 59.5 & 77.6 & 73.7 & \textcolor{blue}{87.0} & \textcolor{red}{34.0} & \textcolor{red}{83.7} & \textcolor{blue}{47.6} & \textcolor{PineGreen}{84.1} & \textcolor{PineGreen}{77.0} & \textcolor{red}{73.9} & 69.6 & 29.8 & \textcolor{red}{84.0} & \textcolor{blue}{43.2} & 68.0 & \textcolor{PineGreen}{53.4} & \textcolor{PineGreen}{64.9} \\

		Ours (w/ CRF) & 88.4 & 57.0 & \textcolor{blue}{31.2} & 75.2 & 47.8 & \textcolor{red}{72.4} & \textcolor{red}{87.2} & \textcolor{red}{76.0} & \textcolor{red}{89.2} & \textcolor{PineGreen}{32.7} & \textcolor{PineGreen}{83.1} & \textcolor{red}{51.1} & \textcolor{red}{85.4} & \textcolor{red}{77.3} & 68.4 &  \textcolor{blue}{70.2} & 40.0 & \textcolor{PineGreen}{81.5} & \textcolor{red}{46.2} & 65.4 & \textcolor{blue}{51.8} & \textcolor{red}{65.6} \\

		\bottomrule
	\label{table: compare_sota_detail}
	\end{tabular}
	\vspace{-5mm}
\end{table*}

\vspace{-3mm}
\section{Conclusions}

In this paper, we propose the Mixup-CAM framework to improve the localization of object response maps, as an initial step towards weakly-supervised semantic segmentation task using image-level labels.
To this end, we propose to integrate the mixup data augmentation strategy for calibrating the uncertainty in network prediction. Furthermore, we introduce another two regularization terms as the interplay with the mixup scheme, thereby producing more complete and uniform response maps.
In experimental results, we provide comprehensive analysis of each component in the proposed method and show that our approach achieves state-of-the-art performance against existing algorithms.


\section{Introduction}\label{sec:introduction}

This template includes all the information about formatting manuscripts for the ISMIR \conferenceyear\ Conference.
Please follow these guidelines to give the final proceedings a uniform look.
Most of the required formatting is achieved automatically by using the supplied
style file (\LaTeX) or template (Word).
If you have any questions, please contact the Program Committee (\texttt{ismir\conferenceyear-papers@ismir.net}).
This template can be downloaded from the ISMIR \conferenceyear\ web site (\texttt{http://ismir\conferenceyear.ismir.net}).
\section{Paper Length \& File Size}
We adopt a ``(6+n)-page policy'' for ISMIR \conferenceyear. That is, each paper may have a maximum of six pages of technical content (including figures and tables) \textcolor{red}{with additional optional pages that contain only references and acknowledgments. Note that acknowledgments should not be included in the anonymized submission.}

Paper should be submitted as PDFs and the \textcolor{red}{file size is limited to 4MB}. Please compress images and figures as necessary before submitting.

\section{Page Size}\label{sec:page_size}

The proceedings will be printed on
 \underline{portrait A4-size paper} \underline{(21.0cm x 29.7cm)}.
All material on each page should fit within a rectangle of 17.2cm x 25.2cm,
centered on the page, beginning 2.0cm
from the top of the page and ending with 2.5cm from the bottom.
The left and right margins should be 1.9cm.
The text should be in two 8.2cm columns with a 0.8cm gutter.
All text must be in a two-column format.
Text must be fully justified.

\section{Typeset Text}\label{sec:typeset_text}

\subsection{Normal or Body Text}\label{subsec:body}

Please use a 10pt (point) Times font. Sans-serif or non-proportional fonts
can be used only for special purposes, such as distinguishing source code text.

The first paragraph in each section should not be indented, but all other paragraphs should be.

\subsection{Title and Authors}

The title is 14pt Times, bold, caps, upper case, centered.
\textcolor{red}{Authors' names are omitted when submitting for double-blind reviewing.}
The following is for making a camera-ready version.
Authors' names are centered.
The lead author's name is to be listed first (left-most), and the co-authors' names after.
If the addresses for all authors are the same, include the address only once, centered.
If the authors have different addresses, put the addresses, evenly spaced, under each authors' name.

\subsection{First Page Copyright Notice}

Please include the copyright notice exactly as it appears here in the lower left-hand corner of the page.
It is set in 8pt Times. \textcolor{red}{After your paper is accepted, you will need to insert the appropriate author names and paper title in the copyright notice when submitting the camera-ready version.} For \LaTeX users, this will be handled by the template automatically. For Word users, this has to be done manually.

\subsection{Page Numbering, Headers and Footers}

Do not include headers, footers or page numbers in your submission.
These will be added when the publications are assembled.

\subsection{Line Numbers}

Line numbers should be included in your originally submitted manuscript, for reference during reviewing. \textcolor{red}{However, after your paper is accepted, you must remove all line numbers from the final camera-ready version.} This can be done in \LaTeX by commenting out the command \verb|\linenumbers| on Line 25. This can be done in Microsoft Word by selecting Layout > Line Numbers > None.

\section{First Level Headings}

First level headings are in Times 10pt bold,
centered with 1 line of space above the section head, and 1/2 space below it.
For a section header immediately followed by a subsection header, the space should be merged.

\subsection{Second Level Headings}

Second level headings are in Times 10pt bold, flush left,
with 1 line of space above the section head, and 1/2 space below it.
The first letter of each significant word is capitalized.

\subsubsection{Third and Further Level Headings}

Third level headings are in Times 10pt italic, flush left,
with 1/2 line of space above the section head, and 1/2 space below it.
The first letter of each significant word is capitalized.

Using more than three levels of headings is highly discouraged.

\section{Footnotes and Figures}

\subsection{Footnotes}

Indicate footnotes with a number in the text.\footnote{This is a footnote.}
Use 8pt type for footnotes. Place the footnotes at the bottom of the page on which they appear.
Precede the footnote with a 0.5pt horizontal rule.

\subsection{Figures, Tables and Captions}

All artwork must be centered, neat, clean, and legible.
All lines should be very dark for purposes of reproduction and art work should not be hand-drawn.
The proceedings are not in color, and therefore all figures must make sense in black-and-white form.
Figure and table numbers and captions always appear below the figure.
Leave 1 line space between the figure or table and the caption.
Each figure or table is numbered consecutively. Captions should be Times 10pt.
Place tables/figures in text as close to the reference as possible.
References to tables and figures should be capitalized, for example:
see \figref{fig:example} and \tabref{tab:example}.
Figures and tables may extend across both columns to a maximum width of 17.2cm.

\begin{table}
 \begin{center}
 \begin{tabular}{|l|l|}
  \hline
  String value & Numeric value \\
  \hline
  Hello ISMIR  & \conferenceyear \\
  \hline
 \end{tabular}
\end{center}
 \caption{Table captions should be placed below the table.}
 \label{tab:example}
\end{table}

\begin{figure}
 \centerline{\framebox{
 \includegraphics[width=\columnwidth]{figure.png}}}
 \caption{Figure captions should be placed below the figure.}
 \label{fig:example}
\end{figure}

\section{Equations}

Equations should be placed on separate lines and numbered.
The number should be on the right side, in parentheses, as in \eqnref{relativity}.

\begin{equation}\label{relativity}
E=mc^{2}
\end{equation}

\section{Citations}

All bibliographical references should be listed at the end,
inside a section named ``REFERENCES,'' numbered and in alphabetical order.
All references listed should be cited in the text.
When referring to a document, type the number in square brackets
\cite{Author:00}, or for a range \cite{Author:00,Someone:10,Someone:04}.

When the following words appear in the conference publication titles, please abbreviate them: Proceedings $\rightarrow$ Proc.; Record $\rightarrow$ Rec.; Symposium $\rightarrow$ Symp.; Technical Digest $\rightarrow$ Tech. Dig.; Technical Paper $\rightarrow$ Tech. Paper; First $\rightarrow$ 1st; Second $\rightarrow$ 2nd; Third $\rightarrow$ 3rd; Fourth/nth $\rightarrow$ 4th/nth.

\textcolor{red}{As submission is double blind, refer to your own published work in the third person. That is, use ``In the previous work of \cite{Someone:10},'' not ``In our previous work \cite{Someone:10}.'' If you cite your other papers that are not widely available (e.g., a journal paper under review), use anonymous author names in the citation, e.g., an author of the form ``A. Anonymous.''}

\section{Introduction}\label{sec:introduction}
Music is a heart-touching form of art that strikes a chord with people's emotions, joyful or sorrowful; intense or relieved, through the twists and turns of notes. Despite its ubiquity in our everyday lives, the composition and arrangement of music often requires substantial human effort. This is a major reason why automatic music composition is such a fascinating field of study. Over the years, researchers have sought strenuously ways for machines to generate well-formed music; such methods include meticulously designed non deep learning-based algorithms like the Markov chains \cite{anderson2013generative} and formal grammars \cite{groves2016automatic}; and, a proliferation of deep learning-based solutions in the past decade \cite{briot19book}. In this work, we exclusively study the extension and evaluation of Transformer-based models \cite{vaswani2017attention} for its claimed successes in natural language processing and music generation in recent years \cite{devlin18bert, radford19language, huang2018music, donahue2019lakhnes}.

The dataset chosen for our work is the Weimar Jazz Database (WJazzD) \cite{wjazzd,wjazzd_book}. As opposed to the commonly used piano MIDIs in recent works \cite{huang2018music, huang20remitransformer}, the choice of this dataset represents a fresh endeavor to train the Transformer on Jazz music, and grants us the unique opportunity to integrate structure-related events, precisely annotated in the WJazzD, to the model. However, such an attempt involves no short of technical challenges, including the quantization of the numerous short notes in Jazz improvisations; and, dealing with the complex chord representations used in the WJazzD. In Section \ref{sec:jazz-transformer}, we will elaborate on how these difficulties are tackled in a detailed manner.

Furthermore, while recent works in Transformer-based music generation often praised the model's capabilities, like being able to compose ``compelling'' music, or generate pieces with ``expressiveness, coherence, and clear structures'' as claimed in \cite{huang2018music} and \cite{huang20remitransformer} respectively, rarely do we admit that the machine is still far behind humans, as shown in our user study (Section \ref{sec:subj-study}), and take a step back to ``face the music'', in other words, to identify what exactly goes wrong in the model's compositions.

Therefore, the goal of the paper is two-fold. First, to deploy Transformers to a new, more complex music genre, Jazz, asking the model to compose melody lines, chord progression, and structures all at once. Second, to develop a set of objective metrics (Section \ref{sec:obj-metrics}) that evaluate the generated music's pitch usages, rhythmicity, consistency in chord progression, and structureness (see Sec. \ref{subsec:struct-indic} for definition), to discover the culprits behind the model's incompetence (Section \ref{sec:expr-result-discuss}).

Figure \ref{fig:pianoroll} shows an example of a composition generated by our model, in which we may find reasonable combinations of chords and melody; and, clear rests between phrases. Audio samples can be found in a Google Drive folder,\footnote{\url{https://drive.google.com/drive/folders/1-09SoxumYPdYetsUWHIHSugK99E2tNYD?usp=sharing}} which we encourage readers to listen to. We have also open-sourced our implementation of the Jazz Transformer\footnote{\url{https://github.com/slSeanWU/jazz_transformer}} and the proposed objective metrics.\footnote{\url{https://github.com/slSeanWU/MusDr}}

\section{Related Work}\label{sec:related-work}

There has been a great body of research work on computational analysis of human performance of Jazz \cite{abesser15ISMIR,gregorio2016PhraseLevelAS,abesser17AES,weissBAM18,eEremenkoDBS18}. One prominent example is the Jazzomat
Project \cite{abesser13CMMRa}, which established the WJazzD \cite{wjazzd} to study the creative processes underpinning Jazz solo improvisations \cite{wjazzd_book}. Weiss \emph{et al.} \cite{weissBAM18}, for instance, used the dataset to explore the evolution of tonal complexity of Jazz improvisations in the past century. See Sec. \ref{subsec:dataset} for more details of the dataset.

The use of Transformer-like architectures for training music composition models has drawn increasing attention recently. These works enhanced the Transformer's capability in modeling music through relative positional encoding schemes \cite{huang2018music, payne2019musenet}, cross-domain pre-training \cite{donahue2019lakhnes}, and event token design \cite{donahue2019lakhnes,huang20remitransformer}. To the best of our knowledge, this work represents the first attempt in the literature to employ Transformers to compose exclusively Jazz music.

Automatic composition of general lead sheets has been investigated lately, mostly based on recurrent neural network (RNN) models \cite{lim17ismir,boom20arxiv,liu18ismir_lbd}. As for inducing structures in the generated music, several RNN-based solutions have also been proposed \cite{medeotCKMASNW18,jhamtani19ml4md,raja20arxiv}. Since Transformers have been shown to outperform RNNs in various tasks \cite{huang2018music,chen19ismir,karita19asru}, we strive to be the forerunner in bringing them to these realms of research.

Relatively little work has been done to train a model for Jazz composition.  JazzGAN \cite{jazzgan} is a model employing a generative adversarial network (GAN) architecture for
chord-conditioned melody composition, using a dataset of only 44 lead sheets, approximately 1,700 bars. Another model presented in \cite{hung19apsipa} explores the use of recurrent variational auto-encoders for generating both the melody and chords of a lead sheet from scratch.

A number of objective metrics have been employed for measuring the performance of deep learning for music composition \cite{sturm17evaluation,chuan18aaai,musegan,yang18evaluation}. However, most of them focused on surface-level statistics only (e.g., pitch class histograms, note onset intervals, etc.). The introduction of structureness indicators and the MIREX-like metric (see Sec. \ref{subsec:struct-indic}--\ref{subsec:mirex-cont}) in this paper provide new insights into assessing music's quality at piece level, and evaluating the model's overall understanding of a certain music genre.

\section{The Jazz Transformer}\label{sec:jazz-transformer}
Transformers use self-attention modules to aggregate information from the past events when predicting the next events \cite{vaswani2017attention,dai2019transformer,katharopoulos2020transformers}. Accordingly, it is natural that we model music as a language, namely, to represent each composition by a sequence of event tokens. In this section, we will explain in detail how we break down the components of the WJazzD \cite{wjazzd_book} to construct the vocabulary of events, and how the pieces are converted into sequences that can be fed into a Transformer-like model for training.

\subsection{Dataset}\label{subsec:dataset}
The WJazzD dataset \cite{wjazzd,wjazzd_book}  comprises of 456 monophonic Jazz solos. Each solo is arranged in the lead sheet style and comes with two tracks: the melody track and the beat track. The melody track contains every note's pitch, onset time and duration (in seconds), with additional information on loudness (in decibels), phrase IDs and ``midlevel units'' (MLUs) \cite{frieler2016midlevelAO}, a structure of finer granularity than a phrase to capture the distinctive  short-time ideas in Jazz improvisations. The beat track contains the beat onsets (in seconds), chord progressions and form parts (or sections, e.g., \texttt{A1}, \texttt{B1}). The highlight of this dataset is that all the contents, including the notes, chords, metrical and structural markings, are human-annotated and cross-checked by the annotators \cite{wjazzd_book}, ensuring the data cleanliness that is often crucial for machine learning tasks. To simplify the subsequent processings, we retain only the pieces marked solely with 4/4 time signature, resulting in 431 solos. For objective analysis, we leave 5\% of the solos as the held-out validation data. See Table \ref{tab:dataset} for the statistics of the data.

\begin{table}[t]
\centering
\begin{tabularx}{\linewidth}{l R R R R}
\toprule
& \# solos & Total duration & Total \# events & Avg. \# events per solo\\
\midrule
\textbf{Train} & 409 &  11h 19m & 1,220 K & 2,983\\
\textbf{Val.} & 22 & 33m & 56 K & 2,548\\
\bottomrule
\end{tabularx}
\caption{Statistics of the dataset we compile from the WJazzD \cite{wjazzd_book}. See Section \ref{subsec:data-repr} for details of the ``events''.}\label{tab:dataset}
\end{table}

\subsection{Data Representation}\label{subsec:data-repr}
The event representation adopted here is a modified version of the ``REvamped MIDI-derived event representation'' recently proposed in \cite{huang20remitransformer}, extended to integrate the chord system and structural events of WJazzD. The resulting event encodings can be broken down into the following 4 categories: \textbf{note-related}---\textsc{Note-Velocity},~\textsc{Note-On},~\textsc{Note-Duration}; \textbf{metric-related}---\textsc{Bar},~\textsc{Position}, \textsc{Tempo-Class},~\textsc{Tempo}; \textbf{chord-related}---\textsc{Chord-Tone},~\textsc{Chord-Type},~\textsc{Chord-Slash}; and \textbf{structure-related}---\textsc{Phrase},~\textsc{MLU},~\textsc{Part},~\textsc{Repetition}.

\subsubsection{Note-related Events}\label{subsubsec:note-evs}
Each note in the melody is represented by three events, i.e., \textsc{Note-Velocity}, \textsc{Note-On}, and \textsc{Note-Duration}.

The \textsc{Note-Velocity} event decides how hard the note should be played. We derive it according to the estimated loudness (in decibels) provided by the dataset, and quantize it into 32 bins, corresponding to MIDI velocities $[3, 7, \dots, 127]$, through
$v = \floor{\big(80 + 3\cdot(dB - 65)\big)/4}$,
where $dB$ is the decibel value of the note, and $v$, clipped such that $v \in [1, 32]$, is the resulting \textsc{Note-Velocity($v$)} event. This mapping scheme comes in handy in the process of converting the model's compositions to MIDIs.

The \textsc{Note-On} events, ranging from $0$ to $127$, correspond directly to the MIDI numbers, indicating the note's pitch. The \textsc{Note-Duration} events represent the note's length in 64th note multiples, ranging from $1$ to $32$, obtained by taking the ratio of the note's duration (in seconds) to the duration of the beat (also in seconds) where the note situates. The reason why we use such a fine-grained quantum, while previous work mainly consider only 16th note multiples (e.g., \cite{huang2018music,huang20remitransformer}), is as follows. Most notes in WJazzD are quite short, with a significant portion being 32th and 64th notes (12.9\% and 2.7\% respectively). The quantum is chosen such that the coverage of the 32 \textsc{Note-Duration} events encompasses the most notes, which is 99.6\% with our choice of the 64th note.\footnote{All notes shorter than a 64th note are discarded and those longer than a half note are clipped.}

\subsubsection{Metric-related Events}\label{subsubsec:metric-evs}
To model the progression of time, we use a combination of \textsc{Bar} and \textsc{Position} events; as demonstrated in \cite{huang20remitransformer}, this combination leads to clearer rhythmic structure in the generated music compared to using \textsc{Time-Shift} events introduced in \cite{huang2018music}. In addition, the pace the music should be played at is set by \textsc{Tempo-Class} and \textsc{Tempo} events.

A \textsc{Bar} event is added at the beginning of each bar, and a bar is quantized into 64 subunits, each represented by a \textsc{Position} event; for example, \textsc{Position(16)} marks the start of the 2nd beat in a bar. A \textsc{Position} event occurs whenever there is a note onset, chord change, or tempo change.
It is worth mentioning that to minimize the quantization error, a note's onset position is justified with the beat it is in through the formula:
\begin{equation}\label{eqn:note-justification}
p_n = p_b + 16 \cdot \big(t_n - t_b\big) / d_b \,,
\end{equation}
where $p_b, t_b, d_b$ are the beat's position (note that $p_b \in \{0, 16, 32, 48\}$), onset time, and duration; and $t_n$ is the note's onset time. The resulting $p_n$ is then rounded to the nearest integer to determine the note's onset position.

The \textsc{Tempo-Class} and \textsc{Tempo} events always co-occur at every beat position. The 5 \textsc{Tempo-Class} events represent the general ``feeling'' of speed (i.e. fast, or slow) with interval boundaries of $[50, 80, 110, 140, 180, 320]$ beats per minute (bpm), while the 12 \textsc{Tempo} events assigned to each tempo class in evenly-spaced steps (within the interval, e.g., $50, 52.5, 55 \text{ bpm} ...)$ determine the exact pace. The events can be derived simply by taking the reciprocal of a beat's duration (provided by WJazzD). The frequent appearance of these tempo events facilitates smooth local tempo changes common in Jazz performances.

\subsubsection{Chord-related Events}\label{subsubsec:chord-evs}
Chord progressions serve as the harmonic foundation of Jazz improvisations \cite{jazz_improvise}, hence a complex chord representation system is used in the WJazzD dataset. If we were to treat each of the 418 unique chord representations present in the WJazzD as a token, the majority of chord tokens will have very few occurrences---in fact, 287 (69\%) of them appear in less than 5 solos, making it hard for the model to learn the meaning of those chords well; plus, the process of translating chords to individual notes during the conversion to MIDIs would be extremely cumbersome.

Fortunately, thanks to the detailed clarification provided in \cite{wjazzd_book}, we are able to decompose each chord into 3 events, namely, the \textsc{Chord-Tone}, \textsc{Chord-Type}, and \textsc{Chord-Slash} events, with the help of regular expressions (regex) and some rule-based exception handling.

The 12 \textsc{Chord-Tone} events, one for each note on the chromatic scale (i.e. \texttt{C}, \texttt{C\#}, \texttt{D}, ...), determine the root note, hence the tonality, of the chord. The 47 \textsc{Chord-Type} events affect the chord's quality and emotion by the different combination of notes played relative to the root note (or, \textit{key template} as we call it); e.g., the key template of a Dominant 7th chord (\textsc{Chord-Type}(\texttt{7})) is $[0, 4, 7, 10]$. Finally, the 12 \textsc{Chord-Slash} events allow the freedom to alter the bass note to slightly tweak the chord's quality. If a chord contains no slash, its \textsc{Chord-Slash} event will share the same key as its \textsc{Chord-Tone}. For instance, the chord \texttt{C7/G}, a C Dominant 7th over G, is represented by $[\text{\textsc{Chord-Tone}(\texttt{C}), \textsc{Chord-Type}(\texttt{7}), \textsc{Chord-Slash}(\texttt{G})}]$.

Note that after our decomposition, the number of unique chord-related events is greatly reduced to 71; and, the resulting set of events is still able to represent all 418 chords in WJazzD.
It is easy to use the manually-constructed key template accompanying each \textsc{Chord-Type}, together with the \textsc{Chord-Tone} and \textsc{Chord-Slash} events to map each chord to notes during the conversion to MIDIs.

\subsubsection{Structure-related Events}\label{subsubsec:struct-evs}
For the melodies, we prepend a \textsc{Phrase} event to the notes marked as the start of a phrase. The presence of phrases may be important as it informs the model to ``take a breath'' between streams of notes. And, we retain several common types and subtypes of midlevel units (e.g., \textit{line, rhythm, lick} etc.) as \textsc{MLU} events \cite{frieler2016midlevelAO}, likewise prepended to the starting note of each MLU, hoping that the model could capture the short-term note patterns described by the MLU types. \textsc{Part} and \textsc{Repetition} events are added to each beginning and end of a form part,\footnote{For example, the entire \texttt{A1} part is represented as $[$\textsc{Part-Start(A)}, \textsc{Repetition-Start(1)}, $\dots$ other events $\dots$, \textsc{Repetition-End(1)}, \textsc{Part-End(A)}$]$.} guiding the model to generate repetitive chord progression and coherent melody lines for the parts marked with the same letter.

\subsection{Model and Training Setups}\label{subsec:model}
Due to the large number of events per solo (check Table \ref{tab:dataset}), it is hard to feed the entire pieces into a Transformer at once because of memory constraint. Therefore, we choose as the backbone sequence model the Transformer-XL \cite{dai2019transformer}, an improved variant of the Transformer which introduces recurrence to the architecture. It remedies the memory constraint and the resulting context fragmentation issue by caching the computation record of the last segment,
and allowing the current segment to attend to the cache in the self-attention process. This allows information to flow across the otherwise separated segments, inducing better coherence in the generated music.

To evaluate the effectiveness of adding the structure-related events (cf. Section \ref{subsubsec:struct-evs}),
we consider the following two variants in our objective analysis:
\begin{itemize}
    \item \textbf{Model (A)}: trained with no  structure-related events.
    \item \textbf{Model (B)}: trained with the complete set of events.
\end{itemize}
They both consist of 12 layers, 8 attention heads and about 41 million learnable parameters. We train them on a single NVIDIA GTX 1080-Ti GPU (with 11 GB memory) with Adam optimizer, learning rate 1e$-$4, batch size 8, segment length 512 and 100\% teacher forcing. Besides, following the Music Transformer's data augmentation setting \cite{huang2018music}, we randomly transpose each solo in the range of $-$3 to $+$3 keys in every epoch. It takes roughly a full day for the negative log-likelihood losses of the models to drop to 0.25, a level at which they are able to produce music of distinctive Jazz feeling (see Section \ref{sec:expr-result-discuss} for justifications).

\section{Subjective Study}\label{sec:subj-study}
To discover how users feel about the Jazz Transformer's compositions, we set up a blind listening test in which test-takers listen to four one-minute long pieces, two from the Model (B)'s compositions (at loss level 0.25), and two from real data. We do not include Model (A) here to reduce the burden on the test-takers, assuming that Model (B) is better.
We inform them that the pieces are independent of one another, and they will be asked the same set of questions after listening to each piece, namely, to rate it in a five-point Likert scale on the following aspects:
\begin{itemize}
    \item \textbf{Overall Quality (O):} Does it sound good overall?
    \item \textbf{Impression (I):} Can you remember a certain part or the melody?
    \item \textbf{Structureness (S):} Does it involve recurring music ideas, clear phrases, and coherent sections?
    \item \textbf{Richness (R):} Is the music diverse and interesting?
\end{itemize}
We distribute five test suites to our social circles and collect responses from 59 anonymized subjects, of which 27 are classified as ``pros'' for they rate their musical background (in general, not restricted to Jazz) as $4/5$ or $5/5$ (i.e., also on a five-point scale). The result shown in Figure \ref{fig:user-study-all} indicates that the Jazz Transformer receives mediocre scores and falls short of humans in every aspect, especially in \textit{overall quality} \textbf{(O)} and \textit{structureness} \textbf{(S)}.
Moreover, performed \textit{one-tailed Z-tests for the difference of means} also suggests the significance of the gaps ($p < 0.05$ for all aspects), providing concrete evidence of the model's defeat.

\begin{figure}
 \centerline{
 \includegraphics[trim={0 7mm 0 0},clip,width=\linewidth]{user_study_all_numbers.pdf}}
\caption{Result of subjective study (\textbf{O:} Overall Quality, \textbf{I:} Impression, \textbf{S:} Structureness, \textbf{R:} Richness), comparing from-scratch compositions created by the proposed model with structure-related events (i.e., `Model (B)') against the real pieces from the WJazzD. We note that the gaps in all aspects are statistically significant ($p < 0.05$).}
 \label{fig:user-study-all}
\end{figure}

\section{Objective Evaluation Metrics}\label{sec:obj-metrics}
The result of our subjective study poses to us an intriguing question: If the machine is still inferior to humans in creating music, then what exactly are the causes? To unravel the mystery, we develop a set of objective metrics which enables us to scrutinize the Jazz Transformer's compositions from various perspectives, and make comparisons with real data. These metrics include the analyses of event distributions, namely, the pitch class histogram, the grooving pattern, and the chord progressions; assessing the structureness with the help of the fitness scape plot; and, judging the model's performance on a discriminative task through the MIREX-like continuation prediction challenge.

\subsection{Pitch Class Histogram Entropy}\label{subsec:pitch-hist-ent}
To gain insight into the usage of different pitches, we first collect the notes appeared in a certain period (e.g., a bar) and construct the 12-dimensional pitch class histogram $\vectorize {h}$, according to the notes' pitch classes (i.e. \texttt{C}, \texttt{C\#}, ..., \texttt{A\#}, \texttt{B}), normalized by the total note count in the period such that $\sum_i{h_i} = 1$. Then, we calculate the entropy of $\vectorize{h}$:
\begin{equation}\label{eqn:pitch-entropy}
        \mathcal{H}(\vectorize{h}) = - \displaystyle \sum_{i=0}^{11} h_i \log_2 (h_i) \,.
\end{equation}
The entropy, in information theory, is a measure of ``uncertainty'' of a probability distribution \cite{shannon1948mathematical}, hence we adopt it here as a metric to help assessing the music's quality in tonality. If a piece's tonality is clear, several pitch classes should dominate the pitch histogram (e.g., the tonic and the dominant), resulting in a low-entropy $\vectorize{h}$; on the contrary, if the tonality is unstable, the usage of pitch classes is likely scattered, giving rise to an $\vectorize{h}$ with high entropy.

\subsection{Grooving Pattern Similarity}\label{subsec:groove-sim}
The grooving pattern represents the positions in a bar at which there is at least a note onset, denoted by $\vectorize g$, a 64-dimensional binary vector in our setting.\footnote{For example, if a bar contains only two note onsets, at the beginning of the 1st beat and 2nd beat respectively, then the corresponding $\vectorize g$ will have $g_0, g_{16} = 1$, and the rest dimensions $0$.} We define the similarity between a pair of grooving patterns $\vectorize g^a$, $\vectorize g^b$ as:
\begin{equation}\label{eqn:groove-sim}
    \mathcal{GS}(\vectorize g^a, \vectorize g^b) = 1 - \frac{1}{Q} \displaystyle \sum_{i=0}^{Q-1} \text{XOR}(g_i^a, g_i^b) \,,
\end{equation}
where $Q$ is the dimensionality of $\vectorize g^a$, $\vectorize g^b$, and XOR($\cdot,\cdot$) is the exclusive OR operation. Note that the value of $\mathcal{GS}(\cdot, \cdot)$ would always lie in between $0$ and $1$.

The grooving pattern similarity helps in measuring the music's rhythmicity. If a piece possesses a clear sense of rhythm, the grooving patterns between pairs of bars should be similar, thereby producing high $\mathcal{GS}$ scores; on the other hand, if the rhythm feels unsteady, the grooving patterns across bars should be erratic, resulting in low $\mathcal{GS}$ scores.

\subsection{Chord Progression Irregularity}\label{subsec:chord-irreg}
To measure the irregularity of a chord progression, we begin by introducing the term \textit{chord trigram}, which is a triple composed of 3 consecutive chords in a chord progression; for example, (\texttt{Dm7}, \texttt{G7}, \texttt{CM7}). Then, \textit{the chord progression irregularity} ($\mathcal{CPI}$) is defined as the percentage of \textit{unique chord trigrams} in the chord progression of an entire piece. Please note that 2 chord trigrams are considered different if any of their elements does not match.

It is common for Jazz compositions to make use of 8- or 12-bar-long templates of chord progressions (known as the \textit{8-}, or \textit{12-bar blues}), which themselves can be broken down into similar substructures \cite{jazz_improvise, nelson01melodic}, as the foundation of a section, and more or less ``copy-paste'' them to form the complete song with, say, \texttt{AABA} parts. Therefore, a well-composed Jazz piece should have a chord progression irregularity that is not too high.

\subsection{Structureness Indicators}\label{subsec:struct-indic}
The \textit{structureness} of music is induced by the repetitive musical content in the composition. It can involve multiple granularities, ranging from an instant musical idea to an entire section. From a psychological perspective, the appearance of repeated structures is the essence of the catchiness and the emotion-provoking nature of music \cite{levitin06brain}.

The fitness scape plot algorithm  \cite{muller11fitness, muller12scapeplot} and the associated SM Toolbox \cite{muller14smtoolbox} offer an aesthetic way of detecting and visualizing the presence of repeating structures in music. The fitness scape plot is a matrix $S_{N \times N}$,\footnote{$N$ is the number of frames sampled from the audio of a piece, the 1st axis represents the segment duration (in frames), and the 2nd axis represents the center of segment (in frames).} where $s_{ij} \in [0, 1]$ is the \textit{fitness}, namely, the degree of repeat in the piece derived from the \textit{self-similarity matrix} (SSM) \cite{foote99visualizing}, of the segment specified by $(i, j)$.

Our \textit{structureness indicator} is based on the fitness scape plot and designed to capture the most salient repeat within a certain duration interval. For brevity of the mathematical representation, we assume the sampling frame rate of $S$ is $1$ Hz (hence $N$ will be the piece's duration in seconds), and define the structureness indicator as follows:
\begin{equation}\label{eqn:struct-indic}
    \mathcal{SI}_l^u(S) = \max_{\substack{l \leq i \leq u\\ 1 \leq j \leq N}} S \,,
\end{equation}
where $l, u$\footnote{If present, otherwise $l$ defaults to $1$, and $u$ defaults to $N$.} are the lower and upper bounds of the duration interval (in seconds) one is interested in. In our experiments, we choose the structureness indicators of $\mathcal{SI}_3^8$, $\mathcal{SI}_8^{15}$, and $\mathcal{SI}_{15}$ to examine the short-, medium-, and long-term structureness respectively.

\subsection{MIREX-like Continuation Prediction Challenge}\label{subsec:mirex-cont}

Being inspired by the ``Patterns for Prediction Challenge'' held as part of the Music Information Retrieval Evaluation eXchange~(MIREX)~2019 \cite{mirex-like,janssen19ismir},
we developed a method to test the model's capability to predict the correct continuation given a musical prompt. The challenge is carried out as follows: First, the model is fed with the beginning 8 bars of a piece, denoted by $\vectorize{s}$; then, it is presented with a set of four 8-bar continuations $\mathcal{X} = \{\vectorize{x}^0, \vectorize{x}^1, \vectorize{x}^2, \vectorize{x}^3\}$, in which one is the true continuation, and the rest are wrong answers randomly drawn from other pieces. The way the model attempts to answer the multiple choice question is by calculating the average probability of generating the events of each continuation:
\begin{equation}\label{eqn:multi-choice-prob}
    \mathcal{P}(\vectorize{x}^i) = \frac{1}{L}\displaystyle \sum_{j=0}^{L-1} p(x_j^i \mid \Tilde{x}_{j-1}, \dots, \Tilde{x}_0; \vectorize{s}), \; i \in \{0, 1, 2, 3\} \,,
\end{equation}
where $L$ is the length of the shortest given continuation (in \# events) in $\mathcal{X}$, $x_j^i$ is the $j$-th event token in $\vectorize{x}^i$, and $\Tilde{x}_{j-1}, \dots, \Tilde{x}_0$ are the events sampled from the model's output, hence the conditional probability $p(x_j^i)$ at each timestep can be obtained straightforward. Finally, the model returns $\argmax_i \mathcal{P}(\vectorize{x}^i)$ as its answer, of which the correctness we can check.

If the model can achieve high accuracy on this continuation prediction task, we may say it possesses a good overall understanding of Jazz music, enough for it to tell right from wrong when given multiple choices.

\begin{figure}
 \centerline{
 \includegraphics[trim={0 0 0 0},clip,width=\linewidth]{multi_choice_res_both.pdf}}
 \caption{Result of the MIREX-like continuation prediction challenge, each checkpoint is asked 100 questions. Notice that the accuracy of both Model (A) and (B) peaks at the loss level of 0.25, at 80\% and 83\% respectively.}
 \label{fig:multi-choice-res}
\end{figure}

\begin{table}[t]
\centering
\begin{tabularx}{\linewidth}{l | c c | c c c | C }
\toprule
& \multicolumn{2}{c |}{\textbf{Model (A)}} & \multicolumn{3}{c |}{\textbf{Model (B)}} & \textbf{Real} \\
\multicolumn{1}{r |}{\textit{loss}} & \textit{0.80} & \textit{0.25} & \textit{0.80} & \textit{0.25} & \textit{0.10} & - - \\
\midrule
$\mathcal{H}_1$ & 2.29 & 2.45 & 2.26 & 2.20 & \textbf{2.17} & 1.94\\
$\mathcal{H}_4$ & 3.12 & 3.05 & 3.04 & \textbf{2.91} & 2.94 & 2.87\\ \hline
$\mathcal{GS}$ & \textbf{0.76} & 0.69 & 0.75 & \textbf{0.76} & \textbf{0.76} & 0.86\\ \hline
$\mathcal{CPI}$ & 81.2 & 77.6 & 79.2 & \textbf{72.6} & 75.9 & 40.4\\ \hline
$\mathcal{SI}_3^8$ & 0.18 & 0.22 & 0.25 & \textbf{0.27} & 0.26 & 0.36\\
$\mathcal{SI}_8^{15}$ & 0.15 & 0.17 & \textbf{0.18} & \textbf{0.18} & 0.17 & 0.36\\
$\mathcal{SI}_{15}$ & 0.11 & \textbf{0.14} & 0.10 & 0.12 & 0.11 & 0.35\\
\bottomrule
\end{tabularx}
\caption{Results of objective evaluations.
$\mathcal{H}_1$, $\mathcal{H}_4$ are the 1-, and 4-bar pitch class histogram entropy (see Sec. \ref{subsec:pitch-hist-ent}); $\mathcal{GS}$ is the grooving pattern similarity (Sec. \ref{subsec:groove-sim}) measured on all pairs of bars within a piece; $\mathcal{CPI}$ is the chord progression irregularity (in \%; Sec. \ref{subsec:chord-irreg}); finally, $\mathcal{SI}_3^8$, $\mathcal{SI}_8^{15}$, and $\mathcal{SI}_{15}$ are the short-, medium-, and long-term structureness indicators (Sec.  \ref{subsec:struct-indic}).
\textbf{Bold} texts indicate the model checkpoint performing the closest to real data, which is considered to be the best.
It is observed that Model (B) (i.e., the model trained with structure-related events) with a loss of 0.25 outperforms its counterparts at other loss levels and Model (A) on most of the metrics.
Moreover, consistent with the result of the MIREX-like challenge (Fig. \ref{fig:multi-choice-res}), the performance of Model (B) plunges when the loss goes too low (0.1 in this case).}\label{tab:obj-res}
\end{table}

\section{Experiment Results and Discussions}
\label{sec:expr-result-discuss}
We begin with the evaluation
on the MIREX-like challenge (Section \ref{subsec:mirex-cont}). We pick 5 checkpoints of both Model (A) and Model (B) at different training loss levels to ask each of them 100 multiple choice questions (the prompt and continuation choices of each question are randomly drawn from the held-out validation data). The result shown in Figure \ref{fig:multi-choice-res} indicates that, similarly for both models, the accuracy steadily goes up as the training loss decreases, peaks at the loss level of 0.25, and drops afterwards. This shows that the models are gradually gaining knowledge about Jazz music along the training process until a certain point, where they potentially start to overfit.

Following the MIREX-like challenge, we pick several checkpoints of both Models (A) and (B) for objective evaluations described in Sections \ref{subsec:pitch-hist-ent}--\ref{subsec:struct-indic}. The chosen checkpoints are at loss levels 0.8, 0.25, and 0.1 (for Model (B) only, since in the MIREX-like challenge (Fig. \ref{fig:multi-choice-res}), its accuracy drastically drops when the loss reduces from 0.25 to 0.1). In the experiments, 50 32-bar-long from-scratch compositions from each checkpointed model are compared against the 409 pieces in the training dataset.

From the results (Table \ref{tab:obj-res}), we can summarize the model's deficiencies as follows: 1) the \textit{erraticity} of the generated musical events; and, 2) the \textit{absence} of medium- and long-term repetitive structures. Comparing with the real data, the first argument can be justified by the higher $\mathcal{H}_1$ and $\mathcal{H}_4$, manifesting the unstable usage of pitches at local scale; and, the lower $\mathcal{GS}$ and higher $\mathcal{CPI}$ of the entire pieces, marking the lack of consistency in rhythm and harmony from a global point of view; meanwhile, the second argument can be explained directly by the significantly lower values of structureness indicators $\mathcal{SI}_8^{15}$ and $\mathcal{SI}_{15}$, suggesting that while the model might be able to repeat some short fragments of music, creating structures of a longer time span is still beyond its capability.

Much to our delight, the introduction of structure-related events seems to be functional to some extent, noticeable from the numbers that Model (B) at 0.25 loss level is for most of the time the closest competitor to humans, with a substantial lead on the metrics focusing on shorter timespans (i.e., $\mathcal{H}_1$, $\mathcal{H}_4$, and $\mathcal{SI}_3^8$) when placed in comparison with Model (A). This suggests that the use of \textsc{Phrase} and \textsc{MLU} events provides some assistance to the model in modeling music. Furthermore, resonating with the accuracy trend in the MIREX-like challenge, the performance worsens when the loss is reduced to an overly low level.

To visualize the deficiency in structureness of the model's compositions, we choose the piece which scores the highest, 3.14, in the \textit{structureness} \textbf{(S)} aspect in our subjective study; and, a human composition of the same duration, receiving 3.54 in the aspect \textbf{S}, for a head-to-head comparison of their fitness scape plots. The rivalry (see Figure \ref{fig:scapeplot-cmp}) reveals the stark contrast between their fitness values across all timescales. In the model's work, all traces of repetitive structures disappear at the timescale of 10 seconds; whereas in the human composition, not only do the fitness values stay high in longer timespans, but a clear sense of section is also present, as manifested by the 2 large, dark ``triangles'' in its scape plot.

\begin{figure}
 \centerline{
 \includegraphics[trim={0 0 0 5}, clip,width=\linewidth]{scape_cmp_cropped.pdf}}
 \caption{The fitness scape plots of Model (B)'s best composition (according to the \textit{structureness} \textbf{(S)} score in our subjective study, see Sec. \ref{sec:subj-study}) versus a human composition in the WJazzD. Note that the piece by Model (B) contains almost no signs of repetition longer than 10 seconds, while the real piece's repetitive structures extend well into the 20--30 seconds range.}
 \label{fig:scapeplot-cmp}
\end{figure}

\section{Conclusion and Future Work}\label{sec:conclusion}
In this paper, we have presented the Jazz Transformer, whose incorporation of structure-related events has been shown useful here in enhancing the quality of machine-generated music. Moreover, we have proposed a series of objective metrics that shed light on the shortcomings of machine-composed pieces, including the erratic usage of pitch classes, inconsistent grooving pattern and chord progression; and, the absence of repetitive structures. These metrics not only show that the Transformer is in fact not that good a music composer, but also serve as effective quantitative measures for future efforts in automatic music composition to assess their models' performance, which by now still relies heavily on human evaluation.

In the future, we plan to carry out larger-scale studies to explicate the correlations between those quantitative metrics and the aspects of subjective evaluation; and, to continue working on inducing structures in machine-composed music; such endeavors may not stay on revamping events that fit into Transformers as done, but involve a complete redesign of the Transformer architecture, enabling it to read the structural information directly computable from data, say, the fitness scape plot, to grasp the blueprint of a piece before composing music at finer scales.

\section{Acknowledgements}\label{sec:ack}
The authors would like to express the utmost gratitude to \textbf{the Jazzomat Research Project} (University of Music FRANZ LISZT Weimar), for compiling the WJazzD dataset and making it publicly available; \textbf{Wen-Yi Hsiao} (Taiwan AI Labs), for rendering the MIDIs to audios for subjective study; and \textbf{Yi-Jen Shih} (National Taiwan University), for the help in arranging our open-source codes.