\section{Introduction}
Human beings are able to hear environmental sounds and discriminate  between the different sounds. This is due to the fact that  they are equipped with auditory systems that capture sounds and extract meanings from them in a discriminative way \cite{clap2000}. These meanings are necessary in informing how humans make decisions and  respond or behave based on the meaning of the sound extracted. Enabling machines to have sensing capabilities such as those of humans e.g. vision, hearing, touch, smell and taste is part of the goal of machine learning \cite{revaudio200}. To this end a number of machine learning  techniques have been developed that focus on giving  machines  auditory skills similar to those of human beings\cite{tool2000}, \cite{tool2001},\cite{tool2002}. This challenging problem is referred to as machine hearing \cite{revaudio200}. A given hearing machine will be faced with a wide variety of  sounds that they are required to perform an in-depth analysis of to extract their appropriate features that make them distinct. The process of extracting  distinctive features of an audio signal is referred to as feature extraction. Accurate and precise  feature extraction is crucial in machine hearing to guarantee the success of machine hearing applications. The features extracted from the sounds can then be used  in different tasks such as audio classification, detection, retrieval etc.
Recently research on making machines to hear  environmental sound and classify them correctly has gained traction \cite{TFCNN100}, \cite{pyramid100},\cite{Piczac100},\cite{Li2018}. Environmental sound classification (ESC), is geared to make machines identify environmental sounds  such as  siren, bird chirping, car horn etc and distinguish the sounds.
Compared  to music and speech audio, environmental sounds have
a number of distinctive  characteristics that present additional challenges to the machine hearing devices. Some of distinctive  features  between speech and audio vs environmental sounds include;
\begin{enumerate}
\item  In speech and music audio signals,  their respective phonemes and musical notes are combined so that the hearing human or machine can obtain a sequence of meanings such that they transmit a particular semantic message. However,  environmental sounds do not follow any predefined grammar and the semantic sequences remain unclear \cite{revaudio200}
\item Both  speech and music sounds are constructed  from a limited dictionary of phonemes and notes  respectively. On the other hand, environmental sounds are theoretically composed of infinite sounds from the environment since any occurring sound in the environment may be included in this category.
\item The environmental sounds depict a larger complexity of the spectrum  in the frequency domain as compared to the music and speech sounds \cite{revaudio200}.
\end{enumerate}
Based on these key differences, a number of  state of the art techniques \cite{pyramid100},\cite{Piczac100},\cite{Li2018} have been developed that focus solely on  environmental noise feature extraction and  classification. Most of the tools first design  techniques that extract audio signal features then  use the features to perform classification. All  of the tools reviewed have applied the supervised machine learning technique as the main technique of extracting the audio features. One of the  core objectives of deep learning is to learn useful representations of input data without annotated dataset \cite{con2000}. In the recent past, self-supervised methods have shown great success in domains such as computer vision \cite{visual200}, natural language processing \cite{electra2000} and speech recognition \cite{wav2vec200}. These self-supervised methods learn to extract distinctive features from the input dataset  without explicit annotations by recasting  the unsupervised representation learning problem into a supervised learning problem. Motivated by this, we also introduce the self-supervised contrastive technique into the environmental sounds domain. We specifically use the contrastive technique to learn how to extract representations of the different environmental sounds.
Concretely we make the following contributions;
\begin{enumerate}
\item Demonstrate that contrastive self-supervised technique can be used to extract  robust feature representations of different environmental sounds.
\item We adopt a shallow 1D CNN model to extract features of an environmental sound and evaluate  the effect of increasing  the depth of the  1D CNN model in the ESC accuracy.
\item We adopt mini-batch balancing during model training and evaluate its effect on  the classification accuracy of minority classes.
\item We exploit canonical correlation analysis to fuse two feature representations from raw normalised  waveform and spectrogram  inputs and evaluate the accuracy of the merged representation in the ESC.
\end{enumerate}
\section{Related Work}
Here, we review the different techniques that have been used in the classification of the environmental sounds. In each technique, we discuss, the type of input, the machine learning technique model for capturing features and how the model was trained  i.e. supervised or unsupervised. SB\_CNN \cite{Salamon2017} converts  raw audio into log-scaled Mel spectrogram representations. It then extracts 2D frame patches from the spectrograms which are then processed by a convolution neural network (CNN) model. The CNN model is connected to a 2 layer fully connected MLP with Softmax activation function in the output layer. The model is trained via a supervised training  to capture the  audio representation. The tool proposed in \cite{Piczac100} first  extracts log-scaled Mel spectrogram of the  audio then splits the spectrogram into two overlapping segments which act as the input of the proposed CNN model. It then performs  a cross validation supervised training to learn audio representations. Pyramid-Combined CNN \cite{pyramid100} first converts input audio waveforms into a spectrogram. It then uses average pooling and normalisation to  obtain the first transformed spectrogram. It again applies the same process of average pooling and normalisation on the first transformed spectrogram to obtain the second transformed spectrogram.  Combined with the original untransformed  spectrogram, three spectrograms are obtained. It then uses  learned weights of the pre-trained CNN models to extract  deep features from the three spectrograms. The obtained features are then fused into  a single representation where  normalisation is applied.  Relief method is then applied on the fused representation to  select the most discriminative features.  Finally, an SVM classifier is used to determine the class labels of the input waveforms. In ACRNN \cite{ACRNN100} first converts an audio to its spectrogram form, then uses a CNN model to extract high level feature representation of the spectrogram. The established features are then fed into a bidirectional gated recurrent neural network to learn the temporal correlation information. This is then passed to a fully connected layer for classification.  SoundNet \cite{Aytar2016} uses transfer learning technique . They leverage unlabeled video to learn a representation of sound. The tool uses CNN model to achieve a transfer learning between unlabeled video and raw waveform. From the features extracted from the CNN, a linear SVM is built for classification. The tool in \cite{verydeep100} uses raw waveform as the input to a 1D CNN. The 1D CNN is used to capture the features of the waveform.  It experiments with different depths of 1D CNN to compare performance differences. DS-CNN \cite{Li2018} uses both raw -waveform and spectrogram as its input. It uses two CNN models where one captures the features of the raw waveform and the other captures the features of the spectrogram. The features representation from the two inputs are combined and classification done based on the combined features. Table 1 summarises  the tools.
 \begin{table}[ht]
\centering
\caption{Summary of the state of the art tools for environmental sound classification}
 \begin{tabular}{ |c |c |c|c|}
   \hline
   \textbf{Tool} &\textbf{Technique}&\textbf{Type of Input}  &\textbf{Type of Training}  \\ \hline
   SB\_CNN \cite{Salamon2017} &2D CNN &spectrogram& Supervised \\ \hline
   Piczak-CNN \cite{Piczac100}&2D CNN& spectrogram& Supervised \\ \hline
    ACRNN \cite{ACRNN100}&2D CNN+GRU& spectrogram& Supervised \\ \hline
         SoundNet100\cite{Aytar2016} &2D CNN& raw waveform& Supervised \\ \hline
           \cite{verydeep100} &1D CNN& raw waveform& Supervised \\ \hline
             DS-CNN\cite{Li2018}&2 D CNN& raw waveform+spectrogram& Supervised \\ \hline
               Pyramid CNN \cite{pyramid100}& 2 D CNN& spectrogram& Supervised \\ \hline
TFCNN  \cite{TFCNN100}& 2 D CNN& spectrogram& Supervised \\ \hline
     \end{tabular}
     \end{table}

\section{Model}
Our model exploits the unsupervised contrastive learning  \cite{contrastive200},\cite{contrastive3000},\cite{contrastive400} to establish audio representations. The model shown in figure 1 is  composed of two key parts;
\begin{enumerate}
\item Input preprocessing.
\item Feature encoder
\end{enumerate}
\begin{figure}[ht]
	\centering
\includegraphics[scale=0.45,angle=0]{model1.png}
		\caption{Overall structure of the proposed self-supervised learning for  audio representation of environmental sound }
	\end{figure}
\subsection{Input  Pre-processing}
The feature encoder  proposed in this work  is able to accept two types of inputs.   The first type is the normalised waveform similar to the one adopted in \cite{wav2vec200}. Here, we first segment the raw waveform of length $t$ into two equal parts i.e. each waveform segment has  a length $t/2$. The  waveform  segments'  values are then normalised  to zero mean and unit variance. The normalised waveform acts as the first type of  input to the feature encoder can accept.  The second type of input that can be accepted by the feature encoder is the spectrogram patches. Here, we  segment raw audio of time $t $ into two segments  each of length $t/2$. From each segment we extract a sequence of 128-dimension log Mel spectrogram features  computed using a 25 ms Hamming window with  10 ms stride  yielding  a spectrogram  $X$ of size ${128\times 100t}$.  We then randomly select a number of rectangular patches  in time from the full Mel spectrogram to act as the input to the feature encoder.  Each $x_i$ patch has the shape  $128\times 100t$ where $t$ is the length of the patch.
  \subsection{Feature Encoder}
For the feature encoder,  we use a  1D convolution neural network (CNN). The 1D CNN has
been used successfully in other domains such as  in fault detection \cite{fault2000} and  patient electrocardiogram classification \cite{real200}.  It was utilised in these domains since it  has a  low computational complexity and it is able to  extract features of   dataset without any predetermined transformation. Since the proposed feature encoder can accept   both raw waveform  and spectrogram  as its input, 1D CNN  was considered ideal.  The main unique  architectural design of the 1D CNN is that each CNN layer  is equipped to perform both convolution and subsampling \cite{real200} \cite{fault2000}( see fig 2). To compute the input of a neuron $k$ at layer $l$, the output maps of the previous layer $l-1$ are convolved with their respective kernels and then summed to obtain  $x_k^l$  the input of neuron $k$ at layer $l$ (eqn 1).
 \begin{equation}
x_k^l=b_k^l+\sum_{i=1}^{N} Conv1D(w_{ik},s_i^{l-1})
\end{equation}
where  $w_{ik}^{l-1}$ is the weight connecting $i^{th}$ neuron at layer $l-1$ and $k$ neuron in  layer $l$.  $s_i^{l-1}$ is the output of the $i$ neuron at layer $l-1$, $b_k^l$ is the bias of the $k^{th}$ neuron at layer $l$. To compute the output $s_k^l$of  the $k^{th}$ neuron at layer $l$, $x_k^l$  processed via  a non-linear function $f$ ( eqn 2) then sub-sampling is applied (eqn 3).
\begin{equation}
y_k^l=f(x_k^l)
\end{equation}
\begin{equation}
s_k^l=Subsampling(y_k^l)
\end{equation}
   \begin{figure}[ht]
     \centering
\includegraphics[scale=0.5,angle=0]{1DCNN.png}
		\caption{1D CNN with four CNN layers }
	\end{figure}

 The 1D CNN architecture is also  adaptive and allows for any number of hidden layers. This is enabled by the fact that the last sub-sampling factor is dynamically set to be equal to the size of  its input map.
\section{Model Training with normalised waveform as the input}
The self-supervised contrastive framework for generating environmental sound representation is based on the idea that two  audio waveform segments $x_i$ and $x_j$ generated  from a given waveform $X$ are likely to contain some level of overlapping  information. This is based on our  analysis of UrbanSound8K \cite{urbandataset200} dataset  where the majority of the waveform visualisation has a repetitive nature hence exhibiting high periodicity ( see fig 3 for some visualisations of  selected raw waveforms). During training we seek to learn environmental sound audio representations by setting up a contrastive task  $\mathcal{C}$ which requires that a segment $s_i$ should be able to identify its sibling  from a set of other segments. To compute the loss $\mathcal{L}$,  we sample a random mini-batch of $M$  examples of raw waveforms and segment each into two equal parts resulting in $2M$ raw waveforms each of  size $t/2$. Each of the $2M$ raw waveforms are then normalised to zero mean and unit variance. The normalised waveforms serve as the input of the feature encoder. The feature encoder generates waveform representation $h$ which is then projected to an MLP network which produces final representation $z$ where contrastive loss is applied.  Denoting as $z_i$ and $z_i^\prime$ the two segment representations of  audio sounds of the $i^{th}$ audio input, the contrastive loss is defined  according to equation 4.
\begin{equation}
\mathcal{L}=\frac{1}{M} \sum_{i=1}^{i=M} L_i
\end{equation}
and
\begin{equation}
L_i=\log \frac{\exp( \nicefrac{sim(z_i,z_i^\prime)}{\tau})}{\sum_{k=1}^{i=2M}  \mathbb{1}_{[k\neq i]}  \exp(\nicefrac{sim(z_i,z_k)}{\tau})
}
\end{equation}
where $sim(u,v)=\frac{u^Tv}{||u||||v||}$
\begin{figure}[ht]
	\centering
\includegraphics[scale=0.45,angle=0]{waveform.png}
		\caption{Raw waveform of car horn, gun shot, street music and air conditioner respectively. }
	\end{figure}
\section{Model Training with Spectrogram as input}
Here,  we sample a random mini-batch of $M$  examples of raw waveforms and segment each into two equal parts resulting in $2M$ raw waveforms each of  size $t/2$. From each segment we extract  a sequence of 128-dimension log Mel spectrogram features as described in section 3.1. From the full Mel spectrogram of a segment, we then randomly select rectangular patches in time.  Each patch has a fixed duration of 1.5 s resulting in a rectangular patch of shape $P\in R^{128\times 150}$. The patches are then mapped to 1D  vector embedding  of size 768  through a trainable non-linear projection( see eqn 6) introduced just before the feature encoder. The embeddings are  then projected to the feature encoder and finally to the MLP where contrastive learning is applied according to equation 4 and 5.
\begin{equation}
k=f(W(P_1^TE+P_2^TE+\cdots+P_N^TE))
\end{equation}
where $E\in R^{128\times 768}$ and $N$ is the total number of randomly patches selected per Mel spectrogram of a segment and $W\in R^{1\times 150}$ and $f$ is a nonlinear function.

\section{Feature Fusion of  waveform and spectrogram representation}
Here both  the normalized raw waveform and  the extracted patches of spectrogram are fed into two trained  contrastive learners as shown in figure 1. The two feature representations generated by   the two contrastive learners are fused using canonical correlation analysis (CCA) \cite{Hardoon2003}, \cite{fusion2000} into a single global representation. We begin by giving a brief review of key intuition behind CCA. Consider two multivariate random vector ($\textbf{x}, \textbf{y})$. The idea behind CCA is  to find the basis vectors $\beta$ and $\alpha$ for two sets of variables $\textbf{x}$ and $\textbf{y}$ such that the correlation between the projections of  $\textbf{x}$ and $\textbf{y}$  onto the basis vectors $\beta$ and $\alpha$ i.e. $k_1=\beta^T x$ and $w_1=\alpha^T x$ are mutually maximized. If  $S_x=\{x_1,x_2,\cdots c_n\}$ and $S_y=\{y_1,y_2,\cdots y_n\}$,  CCA seeks to define a new direction  of $S_x$  by choosing a new direction $\beta$  and projecting the vector $S_x$ onto that direction i.e $S_x\beta=(\langle \beta,x_1 \rangle,\langle \beta,x_2 \rangle,\cdots,\langle \beta ,x_n\rangle)$. Similarly,  $S_y$ can be projected to a new directions by choosing a direction $\alpha$ i.e.
$S_y \alpha=(\langle \alpha,y_1 \rangle,\langle \alpha,y_2 \rangle,\cdots,\langle \alpha,y_n\rangle)$. The first task therefore is to select $\beta$ and $\alpha$ that maximise the correlation between the two vectors according to
\begin{equation*}
\sigma= \max_{\alpha,\beta} corr(S_x\beta S_y\alpha)=\max_{\alpha,\beta} \frac{\langle S_x\beta,S_y\alpha\rangle}{||S_x\beta||||S_y\alpha||}
\end{equation*}
Defining the empirical expectation  of a function $f(\textbf{x,y})$ as
\begin{equation*}
E[f(x,y)]=\frac{1}{m}\sum_{i=1}^{i=m} f(\textbf{x,y})
\end{equation*}
the expression of $\sigma$ can be rewritten as
\begin{equation*}
\sigma= \max_{\alpha,\beta} \frac{E[\langle \beta,\textbf{x}\rangle \langle \alpha, \textbf{y}\rangle]}{\sqrt{E[\langle \beta,\textbf{x}\rangle^2] E[\langle \alpha,\textbf{y}\rangle]^2]}}
\end{equation*}
\begin{equation*}
= \max_{\alpha,\beta} \frac{E[\langle \beta,\textbf{x}\rangle \langle \alpha, \textbf{y}\rangle]}{\sqrt{E[\beta^\prime \textbf{xx}^\prime \beta] E[\alpha^\prime\textbf{yy}^\prime \alpha)]}}
\end{equation*}
\begin{equation*}
= \max_{\alpha,\beta} \frac{\beta^\prime E[\textbf{xy}^\prime]\alpha^\prime}{\sqrt{\beta^\prime_x E[\textbf{xx}^\prime] \beta \alpha^\prime E[\textbf{yy}^\prime] \alpha)}}
\end{equation*}
Note that  the covariance matrix of (x,y) as
\begin{equation*}
cov(x,y)=E \left[ \begin{bmatrix}
          \textbf{x} \\
          \textbf{y} \\
       \end{bmatrix} \begin{bmatrix}
          \textbf{x} \\
          \textbf{y}\\
       \end{bmatrix}^\prime \right]=\begin{bmatrix}
          C_{\textbf{xx}} &  C_{\textbf{xy}} \\
          C_{\textbf{yx}} &   C_{\textbf{yy}}
        \end{bmatrix}
\end{equation*}
$cov(\textbf{x,y})$ matrix is composed of within set covariance $C_{\textbf{xx}}$ and $C_{\textbf{yy}}$ and the between the set covariance $C_{\textbf{xy}}=C_{\textbf{yx}}^\prime$
based on this, $\sigma$ can be rewritten as
\begin{equation}
\sigma=\max_{\alpha,\beta} \frac{\beta^\prime C_{\textbf{xy}} \alpha^\prime}{\sqrt{w^\prime_x C_{\textbf{xx}} \beta\alpha^\prime C_{\textbf{yy}} \alpha^\prime}}
\end{equation}
therefore canonical correlation is maximised based on the values of the basis vectors $\beta$ and $\alpha$
\par In this work, Given the two feature vectors $Z$  and $R$ , we exploit  CCA  to fuse the two features into a single representation that only keeps discriminant information and reduces redundancy between the two representations as much as possible. To fuse the two representations  we follow the following steps \cite{fusion2000};
\begin{figure}[ht]
	\centering
\includegraphics[scale=0.45,angle=0]{fused.png}
		\caption{Fusing representation from different input types. }
	\end{figure}
First use the trained contrastive learners to extract two feature vectors representation of the raw waveform and the spectrogram (see fig 4). We then perform the canonical correlation analysis (CCA) between the two representations  for fusion as follows;
\begin{enumerate}
\item Compute  the covariance matrices $S_{rr}$,$S_{zz}$ and     $S_{rz}$ representing covariance matrix of $R$, $Z$ and between  $Z$ and $R$ respectively.
\item Compute  $G1=S_{rr}^{-1/2}S_{rz}S_{zz}^{-1}S_{zr}S_{rr}^{-1/2}$, $G2=S_{zz}^{-1/2}S_{zr}S_{rr}^{-1}S_{rz}S_{zz}^{-1/2}$ and establishing their non-zero eigenvalues $\lambda_1^2\geq\lambda_2^2\geq\cdots\geq\lambda_r^2$ and corresponding orthonormal eigenvectors $u_i$ and $v_i$  with $ 1\leq  i\leq r$
\item Compute iteratively the canonical projection vectors (CPV)  $\alpha_i$  and $\beta_i$ where $ 1\leq  i\leq r$. $\alpha_i$ and $\beta_i$ are computed such that they maximise the correlation between the projections $a_i=\alpha_i^TR$ and $b_i=\alpha_i^TZ$. The projections $a_i$ and $b_i$ are referred to as a pair of canonical variates.
\item  Select the first $d$ pair of projections such that $R^*=(\alpha_1^Tz,\alpha_2^Tz,\cdots,\alpha_d^Tz)=(\alpha_1,\alpha_2,\cdots,\alpha_d)^T z=W_z^Tz$
and $Z^*=(\beta_1^Tr,\beta_2^Tr,\cdots,\beta_d^Tr)=(\beta_1,\beta_2,\cdots,\beta_d)^T z=W_r^Tr$.
\item Compute feature fusion as $T=X^*+Y^*$
\end{enumerate}
\section{Experimental Setup}
\subsection{Contrastive Training Details}
The 1D CNN was implemented with 4 convolution layers. For the three hidden layers we used 64,32 and 16 neurons respectively. The  kernel sizes used were  1,9,15 and 4 for the first, second, third and fourth CNN layers. We also use a sub-sampling value of 4 for the first three layers and the last CNN layer is set to 5 automatically. The output of the last CNN layer is fed into a 2 layer MLP for the final projection of the audio features to the 128-D representations. We use the Adam optimizer for the optimization, batch size of 64, learning rate of 0.001 and training steps of 400. ReLu activation is used in all layers. Further,  a dropout is  added to the fully connected layer with a probability of 0.5.
\subsection{Dataset}
We exploited  two datasets to evaluate the model proposed in this work. The first dataset is $ESC-50$ \cite{dataset2000} which has  50 unique classes of audio data. Each class has   40 audio data with a length of 5 s.  The $ESC-50$ dataset is balanced i.e. each class contains 40 audio data. The second dataset is the UrbanSound8K \cite{urbandataset200}  that contains annotated audio sounds collected from the urban setting. It contains  10 classes ( see table 1). The audio clips have varying lengths with the longest being 4 seconds. Further this dataset is class imbalanced i.e. some classes contain instances that are less than 1000. The distribution is shown in table 2.

\begin{table}[H]
\centering
\caption{Number of audio per class in UrbanSound8K dataset}
 \begin{tabular}{ |c | c | }
   \hline
   \textbf{Class} & \textbf{Number of instances} \\ \hline
   Street Music & 1000 \\ \hline
   Siren & 929 \\\hline
   Gun shot & 374 \\\hline
    Engine idling & 1000\\\hline
     drilling & 1000\\\hline
     dog bark & 1000\\\hline
      children playing  & 1000\\\hline
        car horn & 429\\\hline
         Air conditioner & 1000\\\hline
         \textbf{Total}&8732\\\hline
                 \end{tabular}
 \end{table}
\subsection{Mini-batch balancing}
The UrbanSound8K is unbalanced, therefore training the model with this unbalanced dataset may generate a model that shows weak generalisation ability for the minority classes. To mitigate against this, we adopt the balanced mini-batch training \cite{balanced2000}. We allow for an overlap selection of minority samples within the same epoch. Further, we restrict the number of samples from each class in a given mini-batch to be equal to the batch size divided by the number of classes.
\subsection{Evaluation objectives}
We conducted a number of experiments with a goal to establish if;
\begin{enumerate}
\item  The proposed method generates  environmental audio representation that results in accurate  classifications of the environmental sounds.
\item The input type significantly affect the quality of representations generated by the proposed method.
\item Increasing the depth of 1D CNN improves the quality of representations generated by the proposed model.
\item Mini-batch balancing has any effect in boosting the classifications of minority classes.
\end{enumerate}

\subsection{What is the quality of  audio representation generated by the proposed model }
The first experiment is to evaluate the ability  of representations generated by the proposed technique to capture the features of the audio dataset. To do this,  we set up a classification task by adding a 1  MLP  layer of size 10 for the UrbanSound8K  and 50 for  $ESC-50$ dataset on top of a trained contrastive learner. Our goal is to  establish the ability of the representations generated by the learner in classifying the environmental sounds correctly. We  evaluate the three possible configurations;
\begin{enumerate}
\item When the input to the contrastive learner is normalised raw waveform.
\item When the input to the contrastive learner is the spectrogram patches.
\item When there is fusion of both raw waveform and the spectrogram features.
\end{enumerate}
The results are shown in table 3.
\begin{table}[ht]
\centering
\caption{Performance of the three configurations proposed in this work.}
 \begin{tabular}{ |c | c | c|}
   \hline
   \textbf{Technique} & ESC-50 & UrbanSound8K \\ \hline

       Contrastive (raw waveform) & 94.42& 95.1 \\\hline
     Contrastive (spectrogram patches) & 94.9 & 95.43\\\hline
          Contrastive (fusion)  & 96.2 & 97.1\\\hline
   \end{tabular}
\end{table}
From the results in table 3, when raw waveform is the input, the model reports the lowest accuracy as compared to the other two. However,  the margin of results between  contrastive (raw waveform)  and contrastive (spectrogram patches) is marginal, signalling the ability of 1D CNN to capture the features of both untransformed and transformed waveforms well. The fused features between the  representations of raw waveform and spectrogram gives superior results demonstrating the need to capture feature representation from both the input types.
\section{What is the effect of  CNN depth on the quality of representations generated  }
Here, we experimented  with  different number of 1D CNN layers in the feature encoder  to establish the optimum depth of the 1D CNN model that produces audio representations that capture more accurate features of an audio input. We used a 1D CNN with 4,6,8 and 10 layers. The  number of neurons used in each layer is shown in table 4.  We mainly use  1,9,15 and 6 as the kernel sizes  and sub-sampling factors of 4 and 2.
\begin{table}[H]
\centering
\caption{ No. of neurons used in a given layer}
 \begin{tabular}{ |c | c | }
   \hline
   \textbf{Layer} & \textbf{Number of Neurons} \\ \hline
   2& 64 \\ \hline
   3 & 32 \\\hline
   4 & 16 \\\hline
    5& 64\\\hline
     6 & 32\\\hline
     7 & 16\\\hline
      8  & 62\\\hline
        9& 32\\\hline
         10& 16\\\hline
          \end{tabular}
          \end{table}

The results of the experiments are shown in table 5. From the results,  there is no definite trend with the varying number of layers. An increase in the depth of the 1D CNN does not offer significant benefit in terms of classification accuracy in all the model configurations. This may point to the ability of the 1D CNN to capture most features even when a shallow depth is used.
\begin{table}[H]
\centering
\caption{ Effects of varying CNN depth on the performance of the proposed model}
\begin{tabular}{|l|l|l|l|l|}
\hline

   & \textbf{Layers}& \textbf{ESC-50} &\textbf{UrbanSound8K}  \\ \hline
\multirow{4}{*}{\textbf{Contrastive(raw waveform)}} & 4 & 94.42 &95.1   \\\cline{2-4}
     & 6 & 94.67 &  95.04   \\ \cline{2-4}
  & 8 & 94.30 & 95.12    \\ \cline{2-4}
   & 10 & 94.16 & 95.28    \\ \hline
\multirow{4}{*}{\textbf{Contrastive(Spectrogram patches)}} & 4 & 94.9 & 95.43  \\ \cline{2-4}
 & 6 & 95.56 & 95.49  \\ \cline{2-4}
  & 8 &  94.8 & 95.83  \\ \cline{2-4}
  &10& 94.94&  95.92   \\ \hline
\multirow{4}{*}{\textbf{Contrastive(fusion) }} & 4& 96.2 & 97.1 \\ \cline{2-4}
  & 6 & 96.03 &97.41   \\ \cline{2-4}
 & 8 & 96.24 &  97.04   \\ \cline{2-4}
& 10 & 96.76 & 97.71    \\ \hline
\end{tabular}
\end{table}

\subsection{What is the effect of mini-batch balancing on the class prediction}
Here we evaluate if mini-balancing improves the classification of the minority classes in the UrbanSound8K. The evaluation uses  the contrastive(Spectrogram) model configuration. We first perform training when the mini-batch balancing technique has  not been applied during training. The results are shown in the confusion matrix figure 5. From the results the minority classes gunshot  and car horn have the lowest classification accuracy of 85\% and 89\% respectively. This shows that the model does not see enough instances  of the two classes to effectively capture their features. Figure 6 shows the confusion results when mini-batch balancing has been applied during training.
\begin{figure}[ht]
	\centering
\includegraphics[scale=0.4,angle=0]{CM1.png}
		\caption{Confusion matrix of evaluation results  without mini-batch balancing.}
	\end{figure}
When mini-batch balancing is applied during training, it improves the  classification accuracy of car horn and gunshot classes  to 91\%  and 93\%  respectively,  justifying the  need for  mini-batch balancing during training to boost the number of instances of the minor classes seen by the learner during training hence improving its ability to generalise well for these classes.
	\begin{figure}[H]
	\centering
\includegraphics[scale=0.4,angle=0]{CM2.png}
		\caption{Confusion matrix of evaluation results  with mini-batch balancing.}
	\end{figure}
\subsection{ Comparison to other state of the art tools}
Finally, we compared the results of the proposed technique in classifying environmental sounds and the existing state of the art tools for environmental sound classification. We compare the tools on both the ESC-50 and the UrbanSound8K dataset. The results are shown in table 6. On the ESC-50 the three configurations proposed in this work outperforms the existing methods. Contrastive (raw waveform) the least performing technique proposed outperforms the best performing tool TFCNN by a 10\% margin. This  demonstrates the robustness of the contrastive learning method and the ability of the 1D CNN to capture most of the audio features. On the UrbanSound8K dataset  Contrastive (raw waveform) achieves an accuracy of 95.1\% which is 2.1\% lower than TSCNN the highest performing tool on this dataset. Contrastive(fusion)  method outperforms all the existing techniques in both ESC-50 and UrbanSound8K. It records 12.08\% increase as compared to the best performing tool TFCNN on the $ESC-50$ dataset. On the UrbanSound8K it achieves an increase of 0.9\% as compared to TSCNN is the best performing tool on this dataset.
\begin{table}
\centering
\caption{ Effects of varying CNN depth on the performance of the proposed model}
 \begin{tabular}{ |c | c | c|}
    \hline
   \textbf{Tool} & \textbf{ESC-50}& \textbf{UrbanSound8K}\\ \hline
   Piczak-CNN \cite{Piczac100}& 65.0\%&73.7\% \\ \hline
   TFCNN \cite{TFCNN100} & 84	.4\%&93.1\% \\\hline
   Pyramid CNN\cite{pyramid100} & 81.4\%&78.1\% \\\hline
    SB\_CNN \cite{Salamon2017}& -&79.0\%\\\hline
     SoundNet \cite{Aytar2016}& 74.2\%&-\\\hline
     DS-CNN\cite{Li2018} & 82.8\%&92.2\%\\\hline
      TSCNN \cite{Su2019} &-&97.2\%\\\hline
         Contrastive (raw waveform) & 94.42& 95.1 \\\hline
     Contrastive (spectrogram patches) & 95.3 & 96.1\\\hline
          Contrastive (fusion)  & 97.2 & 98.1\\\hline
               \end{tabular}
                  \end{table}
      \section{Conclusion}
      This work proposes the use of self-supervised contrastive learning to train a learner which can  extract features from the environmental sounds. We exploit a shallow 1D CNN network to extract features of a given audio. We examine the effect of type of input on the quality of representations generated by the learner. Further, we examine the effect of fusing representations from two types of input. The mini-batch balancing is also performed to improve the generalisation of the minority classes by the learner. Overall the proposed technique records a significant improvement in accuracy in the classification task as compared to the existing methods. This demonstrated the robustness of the contrastive learning method and the ability of the 1D CNN to extract the features of waveform.
\bibliographystyle{unsrtnat}


\section{Introduction}~\label{sec:introduction}

The performance of Machine Learning models is highly dependent on the quality
of the training dataset used~\cite{Fenza2021, Halevy2009}. Specifically, the
presence of imbalanced and/or small datasets, target labels incorrectly
assigned, outliers and high dimensional input spaces reduce the prospects of a
successful machine learning model implementation~\cite{Halevy2009,
Domingos2012, Salman2019}.  Even though the performance of any classifier is
affected by the size of its training dataset, deep learning models have a
particularly inconsistent performance over unseen datasets even when trained
with large datasets~\cite{Hu2020, Xie2021}.  Conversely, deep learning models
are capable of quickly adapting (and overfitting) to the training dataset,
including when it contains label and/or complete pixel noise~\cite{Xie2021,
Zhang2021}.  Although the performance of these models can be improved through
regularization methods, they are still incapable of correcting label noise in
the training dataset~\cite{Zhang2021}.

Regardless of the machine learning model used, when the training set contains
significant limitations (regarding overall quality and size), the model's
performance on unseen data is generally going to be affected. Specifically,
when the training data is not representative of the true population, or the
model is over-parametrized, it becomes particularly prone to
overfitting~\cite{Bartlett2021}. There are different strategies to reduce
overfitting, known as regularization methods~\cite{Shorten2019}. Identifying
the appropriate regularization methods varies according to the use
case~\cite{Chun2020}. While some methods can only be applied on specific
classifiers, data types or domains, others may be applied at the data level,
independently from the classification problem. For example, methods such as
dropout/dilution, batch normalization and transfer learning/domain adaptation
are mostly applied on neural network architectures.  Pruning is applied on
decision trees. Early stopping can be used on learners trained iteratively,
making it a broader method.

Data augmentation techniques are used to increase the size (and hopefully the
diversity) of data in a training dataset through the production of artificial
observations~\cite{Van2001, Wong2016}. They are frequently used as
regularization techniques for various types of problems and classifiers, since
it is applied at the data level~\cite{Behpour2019}.
Figure~\ref{fig:data_augmentation_example} shows an example of data
augmentation, where the decision boundaries become clearer after the original
dataset is augmented. Data Augmentation methods can be divided into heuristic
and Deep Learning approaches~\cite{Shorten2019, Ratner2017}. Within these
approaches, they may be either domain specific or contextually independent.
For example, although both Synthetic Minority Oversampling Technique
(SMOTE)~\cite{Chawla2002} and Kernel Filters are heuristic approaches, SMOTE
may be used regardless of the context, while Kernel Filters are specific to
image data augmentation. The different types of Data Augmentation methods are
defined at a higher detail in Section~\ref{sec:data-augmentation}.

\begin{figure}[H]
	\centering
	\includegraphics[width=.9\linewidth]{analysis/data_augmentation_example}
    \caption{Example of data augmentation in a 2-dimensional binary
        classification setting. The left pane contains the original dataset,
        where the amount data is scarce and the gap among the two classes are
        wide, allowing for greater classification variability. The right pane
        contains the augmented dataset, where the gap among the two classes is
        narrower and the decision boundaries become easier to define.
    }~\label{fig:data_augmentation_example}
\end{figure}

In 2011, Jürgen Schmidhuber's group showed that a MLP ensemble architecture
can achieve state-of-the-art performance on computer vision benchmarks given
strong enough data augmentation~\cite{Meier2011, Ciresan2011}. Although the
state-of-the-art improved since then, two recent papers developed by Google
Brain and Facebook research teams support Schmidhuber's group's findings.
Specifically, in~\cite{Tolstikhin2021, Touvron2021} the authors discuss two
similar MLP ensemble architectures, showing that the proposed model attains a
comparable performance to convolutional neural networks and attention-based
networks. Another recent study also discusses a related MLP architecture with
similar findings, while suggesting that the strong performance of computer
vision models may be attributable mainly to the inducive bias produced by the
patch embedding and the carefully-curated set of training
augmentations~\cite{Melaskyriazi2021}.

Research on data augmentation methods gained significant popularity in recent
years. As such, there were some efforts in the past to establish a taxonomy
and distinction of the different types of data augmentations
methods~\cite{Shorten2019}. To the best of our knowledge, there is no analysis
on data augmentation research as a whole, as well as domains of application
and future directions. In this paper we focus on current and past research
trends of data augmentation methods, its different applications and use cases.
This is done with an extensive analysis of the title, keywords and abstract of
a large set of literature related to data augmentation, collected through the
\href{https://www.scopus.com/}{Scopus} database using Natural Language
Processing and Network Science techniques. The analysis contains 3 phases. We
started by performing an exploratory data analysis to identify the most
significant publications, journals and conferences within the field of data
augmentation. Then, we analysed the articles' author keywords by constructing
a network and extracting and identifying communities of keywords.  Finally we
used a text mining approach to extract additional applications and methods
using the articles' abstracts, as well as validate the findings discussed with
the keyword analysis.

The rest of this paper is structured as follows:
Section~\ref{sec:data-augmentation} describes the main methods and approaches
used in data augmentation, Section~\ref{sec:methodology} describes the
procedures defined throughout the different analyses.
Section~\ref{sec:results_discussion} presents and discusses the findings drawn
from the analyses, as well as research gaps and open questions in data
augmentation research. Section~\ref{sec:conclusion} summarizes the main
findings discussed throughout the study.

\section{Data Augmentation Methods}~\label{sec:data-augmentation}

Based on the literature found, a Data Augmentation method may be characterized
based on 3 criteria. The more common division is done between Heuristic and
Deep Learning approaches~\cite{Shorten2019}. Within these, several approaches
have been developed to produce artificial observations at the
input~\cite{Zhong2017}, feature~\cite{DeVries2017}, or output
space~\cite{Behpour2019}. Finally, we also distinguish them based on whether
their generation mechanism considers local (\textit{i.e.,} considers
partial/specific information within the dataset) or global (\textit{i.e.,}
it's based on the overall distribution/structure of the dataset) information
of the original dataset. Figure~\ref{fig:concept_map} depicts the concept map
with the different subdivisions of characteristics of data augmentation
methods. In this section, the analysis of the different types of data
augmentation will be based on their architectural approach. However, all the
methods mentioned may be divided using any of the definitions mentioned.

\begin{figure}[htb]
	\centering
	\includegraphics[width=\linewidth]{analysis/concept_map}
    \caption{%
        Data Augmentation concept map.
    }~\label{fig:concept_map}
\end{figure}

Heuristic methods use the information found in the input space to generate
new, relevant, non-duplicated observations by applying a predefined set of
rules, while incorporating a degree of randomness in the generation process.
Since data augmentation occurs in the input space, these are cost-effective
approaches to data augmentation. For this reason, heuristic methods are
simpler to implement and are particularly appealing for low dimensional
classification problems, especially when the computational power available is
limited.

Some Deep Learning methods, on the other hand, attempt to map the original
input space into a lower-dimensional representation, known as feature
space~\cite{DeVries2017}. The generation of artificial observations occurs at
the feature space level, before being reconstructed to the original input
space. This is commonly done with Convolutional Neural Networks (CNN) and
auto-encoder architectures~\cite{Shorten2019}. Since data augmentation is
performed in the feature space, this type of approach is particularly useful
for high-dimensional data types and results in more plausible synthetic
observations~\cite{DeVries2017}. However, deep learning approaches require
more computational power than heuristic approaches and the resulting feature
space is difficult to interpret.

The difference in classification performance of the two perspectives is still
unclear. Wen et al.~\cite{Wen2020} evaluate the impact of data augmentation on
time series for various classification and forecasting tasks. Although they
found that both heuristic and deep learning approaches improved the results
over the various experiments, there was no direct comparison among the
different methods. Wong et al.~\cite{Wong2016} compared both input and feature
space data augmentation methods for image data classification performance over
the MNIST dataset. They found that input space augmentation can lead to better
classification performance if plausible transforms on the data are known.
However, in~\cite{DeVries2017} the authors discuss that the effectiveness of
each data augmentation method generally depends on the domain. The lack of
research on effective, domain-agnostic data augmentation methods appears to be
a current research gap.

\subsection{Heuristic Approaches}

Various heuristic approaches depend on the data type. For example, image data
augmentation may be done via translation, cropping or random
erasing~\cite{Zhong2017}, among others~\cite{Shorten2019}. However, these
techniques depend on the context and may not be applicable to other data types
such as time-series~\cite{Wen2020, Iwana2021} or tabular data. In this
subsection we will focus on domain-agnostic data augmentation methods.

Heuristic approaches may be applied at the input or feature space. The
appropriate method to be applied also depends on the machine learning goal.
Specifically, heuristic methods are commonly used to address classification
problems where the frequency of the different target classes vary
significantly, a problem known as Imbalanced Learning~\cite{Chawla2004}. In
this context, the dataset contains one or multiple rare classes, which become
more difficult to predict. This happens because during the learning phase,
classifiers are trained in order to maximize one or few performance metrics.
Although, a poor choice of performance metrics (\textit{i.e.}, a metric
insensitive to class imbalance, such as overall accuracy) might lead to a poor
estimation of the model's actual performance, since minority classes
contribute less to the learning phase and to the estimation of the performance
metric~\cite{Fonseca2021}.

The problem of Imbalanced Learning is frequently addressed with oversampling
algorithms~\cite{Kaur2019}. Oversampling methods generate artificial
observations in order to balance class distributions using contextual
information based on the original dataset. These methods apply linear or
geometric interpolations between a random observation and one of its neighbors
to generate a new observation.

SMOTE~\cite{Chawla2002} is one of the most popular oversampling methods. It
generates an artificial observation along a line segment between a randomly
selected minority class observation and one of its nearest-neighbors. Since it
was first proposed, modifications at the neighbors parent observations
selection and data generation mechanisms of the original algorithm were
proposed. Borderline-SMOTE~\cite{Han2005} is an example of a modification of
SMOTE's data selection mechanism. Instead of selecting any random minority
class observation and one of its neighbors, the algorithm focuses in the
minority class observations closest to the decision boundary.
Geometric-SMOTE~\cite{Douzas2019} proposes a modification of the data
generation mechanism. Instead of generating data within a line segment, it
generates data within a hyper-spheroid between two parent observations.

Contrary to Deep Learning approaches, heuristic approaches can be applied at
the input space without the need of learning a feature space. This allows the
implementation of heuristic data augmentation with less computational power
and technical complexity. In addition, in contexts of limited data
availability (\textit{i.e.}, small datasets), deep learning approaches are not
appropriate since the amount of parameters to be tuned during the learning
phase often exceeds the number of observations in the dataset, making it
over-parametrized. The augmentation of small datasets using heuristic
approaches is explored in an Active Learning context
in~\cite{Fonseca2021active}. The authors found that significantly smaller
amounts of curated data using Active Learning, along with heuristic data
augmentation methods, achieved a classification performance comparable to
classifiers trained over the full dataset.

\subsection{Deep Learning Approaches}

Different deep learning approaches and architectures have been developed for
various domains. Deep learning data augmentation methods may be developed via
augmentation at the feature space (which involves learning a feature
space)~\cite{DeVries2017} or via a combination of a set number of observations
into a neural network in order to output a non-linear, non-geometric
combination of input observations~\cite{Wang2017}. Other domain-specific deep
learning methods also exist, such as style transferring techniques (specific
to image data)~\cite{Wang2017, Zhu2017}.

Data augmentation at the feature space is especially useful when dealing with
high-dimensional datasets with complex and/or discontinuous distributions.
For example, many heuristic data augmentation techniques cannot be applied in
handwritten digits classification problems since they may change the true
label of the generated image and generate noisy data. In this situation,
performing data augmentation at the input space may introduce
noise~\cite{Chu2020}, since the data is also subjected to the curse of
dimensionality (see Figure~\ref{fig:input_vs_feature_space}a). This allows
transformations of known observations in a lower dimensional space to generate
new, non-noisy artificial observations projected in the input space, as shown
in Figures~\ref{fig:input_vs_feature_space}b
and~\ref{fig:input_vs_feature_space}c.

\begin{figure}[htb]
	\centering
	\includegraphics[width=.9\linewidth]{analysis/input_vs_feature_space}
    \caption{%
        Example of a sparse input space and its corresponding feature space.
        Learning a manifold feature space facilitates the generation of
        non-noisy artificial data. In the original input space (a) the
        unseen/artificial observation marked with ``?'' is closer to a green
        observation and to the learnt red manifold space, which may lead to a
        noisy observation. When projected or generated in the feature space
        (b), the observation will better match the nearest manifold measure
        and avoid noisy data in the input space (c). Adapted
        from~\cite{Antoniou2017}.
    }~\label{fig:input_vs_feature_space}
\end{figure}

The utilization of autoencoders~\cite{Kramer1991} is particularly useful to
perform feature space data augmentation~\cite{Shorten2019}. Autoencoders are
composed by an encoder and a decoder, which map the input space to and from
the feature space, respectively. The autoencoder is trained by minimizing the
difference between the original observation and the reconstructed observation
(see Figure~\ref{fig:input_vs_feature_space}c). Once the training phase is
completed, heuristic methods are applied in the feature
space~\cite{DeVries2017}.

Generative Adversarial Network (GAN)~\cite{Goodfellow2014} architectures is a
deep learning approach frequently used as a data augmentation method. It
involves a generator and a discriminator. Although many different
architectures have been proposed, the generator in the vanilla GAN algorithm
may be seen as a decoder, which is trained based on the gradients calculated
via the discriminator. The discriminator attempts to distinguish true and
generated (fake) observations in order to assess the quality of the data
produced by the generator. The problem is better formulated as a minimax
decision rule, where the generator attempts to fool the discriminator by
producing observations that are difficult to classify as generated.

When the size of the training dataset is not sufficiently large to employ deep
learning approaches and other related datasets or unlabeled datasets are
available, one technique that may also be used is transfer learning. In this
context, the data augmentation model is trained on a secondary model and is
later adjusted to the training dataset. This allows the usage of deep learning
models in few-shot learning environments~\cite{Antoniou2017}.

\section{Methodology}~\label{sec:methodology}

In this section we describe the procedures defined for the literature
collection, data preprocessing and literature analysis. The analysis of the
literature was developed with 3 different approaches. Throughout the
analyses, data preprocessing and hyperparameter tuning was developed
iteratively. The procedure adopted in this manuscript is shown in
Figure~\ref{fig:slr_diagram}.

The literature collection procedure is described in
Subsection~\ref{sec:lit_collection}. The data and text preprocessing is
described in Subsection~\ref{sec:data_preprocessing}. The exploratory data
analysis described in Subsection~\ref{sec:journal_and_conference_analysis} was
done to understand which manuscripts, journals and conferences are most
significant within the field of Data Augmentation. The manuscripts' keywords
were used to construct a network of keywords (described in
Subsection~\ref{sec:keyword_analysis}) and study the different communities of
keywords found in the network. The topic modelling and parameter tuning is
described in Subsection~\ref{sec:topic_modelling}.

\begin{figure}[H]
	\centering
	\includegraphics[width=.85\linewidth]{analysis/slr_diagram}
    \caption{Diagram of the proposed literature analysis approach.
    }~\label{fig:slr_diagram}
\end{figure}

\subsection{Literature Collection}~\label{sec:lit_collection}

The focus of this literature analysis is to understand the different
algorithms, domains and/or tasks that employ data augmentation techniques.
Therefore, we search for documents containing the keyword ``data
augmentation'' in the search query. The results were then limited to
conference papers and journal articles written in English that were published
in the past 15 years.  Due to the large amount of results found, using solely
the \href{https://www.scopus.com/}{Scopus} database was found to be
sufficient. One of the goals during the search query design was to come up
with a simple and unbiased query. The resulting query is shown below:

\bigskip
\begin{verbatim}
KEY ("data augmentation") AND (LIMIT-TO (LANGUAGE, "English"))
AND (LIMIT-TO (DOCTYPE, "cp") OR LIMIT-TO (DOCTYPE, "ar"))
AND (
        LIMIT-TO (PUBYEAR, 2021) OR LIMIT-TO (PUBYEAR, 2020)
    OR  LIMIT-TO (PUBYEAR, 2019) OR LIMIT-TO (PUBYEAR, 2018)
    OR  LIMIT-TO (PUBYEAR, 2017) OR LIMIT-TO (PUBYEAR, 2016)
    OR  LIMIT-TO (PUBYEAR, 2015) OR LIMIT-TO (PUBYEAR, 2014)
    OR  LIMIT-TO (PUBYEAR, 2013) OR LIMIT-TO (PUBYEAR, 2012)
    OR  LIMIT-TO (PUBYEAR, 2011) OR LIMIT-TO (PUBYEAR, 2010)
    OR  LIMIT-TO (PUBYEAR, 2009) OR LIMIT-TO (PUBYEAR, 2008)
    OR  LIMIT-TO (PUBYEAR, 2007) OR LIMIT-TO (PUBYEAR, 2006)
)
\end{verbatim}
\bigskip

The search query resulted in 4281 documents. The resulting data
selection/filtering pipeline is shown in
Figure~\ref{fig:data_filtering_pipeline}. Due to the limitations in the Scopus
data export (maximum 2000 documents per export), the data was split in four
different time periods and exported separately: 2006 until 2018, 2019, 2020
and 2021, which produced four CSV files.

\subsection{Data Preprocessing}~\label{sec:data_preprocessing}

The data preprocessing stage and amount of documents dropped is represented in
Figure~\ref{fig:data_filtering_pipeline}. The data was first concatenated into
a single data frame. During this process, we found that one of the exported
references had a corrupted line, which caused the loss of one additional
document.  Since the DOI can be used as a unique identifier for intellectual
property~\cite{Paskin1999}, references without a DOI were disregarded from
further analysis, while the ones with the same identifiers are removed
(\textit{i.e.}, only one of the repeating entries is kept).

This dataset was kept to perform the analysis described in
Subsection~\ref{sec:journal_and_conference_analysis}. However, further
preprocessing was done for the remaining parts of the literature analysis.
References without any citations were excluded for the keyword network and
topic modelling analyses. Finally, only the documents containing keywords in
Scopus' database were used to prepare the network analysis.

\begin{figure}[H]
	\centering
    \includegraphics[width=.6\linewidth]{analysis/data_filtering_pipeline}
    \caption{Data filtering pipeline.
    }~\label{fig:data_filtering_pipeline}
\end{figure}

\subsection{Journal and Conference analysis}~\label{sec:journal_and_conference_analysis}

The exploratory analysis developed on the preprocessed dataset was targeted
towards the identification of the most significant works, journals and
conferences. We used the citation count as a proxy to understand the impact of
a specific manuscript within the research community.

The identification of the most significant conferences and journals is done by
sorting each type of publication according to the number of citations per
document. Conferences and journals with less than 10 papers published in the
area are not considered in this analysis.

\subsection{Keyword Analysis}~\label{sec:keyword_analysis}

The analysis of keywords is expected to uncover general trends in data
augmentation research and its applications. The keyword ``data augmentation''
was removed since it would link with all other keywords. Keywords are
connected based on their co-occurrence in each research paper to form the
edges of the network.  It consists of an undirected graph whose weights are
based on the total citation count for the papers containing a given keyword
pair and is calculated as $\textrm{weight} = \log(\textrm{citations}) + 1$ to
avoid a potential bias caused by highly cited research articles. The size of
the nodes were determined with a logarithmic transformation of each
node's page rank.

Keyword combinations showing up in only one document are removed from further
analysis. The keyword network is then analysed using Python and the
communities were found using the greedy modularity maximization algorithm
proposed in~\cite{Clauset2004}. The results of the analysis and community
detection were ported to Gephi to produce the final visualizations.

\subsection{Topic Modelling}~\label{sec:topic_modelling}

The extraction of topics was done using the publication's abstracts. The words
were tokenized and all tags, special characters, punctuation, multiple white
spaces, numeric values, stop words and words with size smaller than 4 were
removed. Finally, we enriched the corpus by constructing bi-grams and
tri-grams.

We used a Latent Dirichlet Allocation (LDA) model~\cite{Pritchard2000} to
infer the topics present in our research domain. The tuning of the parameters
was done through experimentation and qualitative interpretation of the results
achieved. Additionally, the coherence score curve was also used as a reference for
parameter tuning and the choice of parameters, which are described in
Table~\ref{tab:hyperparameters}.

\begin{table}[ht]
    \begin{center}
    \caption{Hyperparameters used.}~\label{tab:hyperparameters}
    \begin{tabular*}{.5\textwidth}{@{\extracolsep{\fill}}lllllll@{\extracolsep{\fill}}}
        \toprule
        Model   &   Hyperparameter  &   Value \\
        \midrule
        LDA     &   Num Topics      &   8     \\
                &   Chunk Size      &   2000  \\
                &   Passes          &   20    \\
                &   Alpha           &   0.1   \\
                &   ETA             &   auto  \\
        \bottomrule
    \end{tabular*}
    \end{center}
\end{table}

\subsection{Software Implementation}~\label{sec:software_implementation}

The analysis and modelling was developed using the Python programming
language, along with the
\href{https://scikit-learn.org/stable/}{Scikit-Learn}~\cite{Pedregosa2011},
\href{https://radimrehurek.com/gensim/}{Gensim}~\cite{Rehurek2010}, and
\href{https://networkx.org/}{Networkx}~\cite{Hagberg2008} libraries. The final
network analysis and visualization was done with
\href{https://gephi.org/}{Gephi}~\cite{Bastian2009}. All functions,
algorithms, analyses and results are provided in the
\href{https://github.com/joaopfonseca/ml-research}{GitHub repository of the
project}.

\section{Results \& Discussion}~\label{sec:results_discussion}

The popularity of research in data generation has grown significantly in the
past 5 years, as shown in Figure~\ref{fig:area_chart_cited_documents}. Despite
the significant amount of uncited publications, out of the ones published in
2020, 39\% have already been used in other works. Although most of the
research developed before 2016 was used in other works, the amount of cited
research increased significantly after that period.

\begin{figure}[ht]
	\centering
    \includegraphics[width=\linewidth]{analysis/area_chart_cited_documents}
    \caption{Annual number of publications containing the keyword ``data
        augmentation''.
    }~\label{fig:area_chart_cited_documents}
\end{figure}

\subsection{Journal and Conference Analysis}

The initial exploration of the bibliometric data allows us to assess which
journals focused in data augmentation more intensely over the past years, as
shown in Table~\ref{tab:top_journals}. Most of the top journals belong to
technical fields, predominantly from Statistics, Remote Sensing, Medical
Imaging and other domains of applications such as agriculture. In addition,
all these journals have a high impact in their respective fields (based on
\href{https://www.scimagojr.com/}{Scimago Journal \& Country Rankings}).

\begin{table}[ht]
    \begin{center}
    \caption{\label{tab:top_journals}
        Top journals focusing on data augmentation techniques, sorted by
        citations per document.
    }
    \begin{tabular*}{\textwidth}{@{\extracolsep{\fill}}lllllll@{\extracolsep{\fill}}}
        \toprule
        Source title & Publications & Citations & Average \\
        \midrule
        Journal of the American Statistical Association & 11 & 538 & 48.91 \\
        IEEE Geoscience and Remote Sensing Letters & 19 & 552 & 29.05 \\
        Neurocomputing & 35 & 808 & 23.09 \\
        Expert Systems with Applications & 14 & 283 & 20.21 \\
        Medical Image Analysis & 15 & 288 & 19.20 \\
        Neural Networks & 10 & 190 & 19.00 \\
        Journal of Computational and Graphical Statistics & 23 & 433 & 18.83 \\
        Computers and Electronics in Agriculture & 15 & 219 & 14.60 \\
        Biometrics & 13 & 163 & 12.54 \\
        IEEE Transactions on Medical Imaging & 10 & 123 & 12.30 \\
        \bottomrule
    \end{tabular*}
    \end{center}
\end{table}

Citation-wise, the publications coming from conference proceedings tend to
have a comparable impact in the research community, as shown in
Table~\ref{tab:top_conferences}. The most relevant conferences are positioned
in the computer science and information management fields. Research developed
in other areas of application, such as computer vision, speech recognition,
acoustic modelling, natural language processing and signal processing have
more activity in the form of conference proceedings publications. Conversely,
the domains most frequent in journal publications are not as active on
conference proceedings publications.

\begin{table}[ht]
    \begin{center}
    \caption{\label{tab:top_conferences}
        Top conferences focusing on data augmentation techniques, sorted by
        citations per document.
    }
    \begin{tabular*}{\textwidth}{@{\extracolsep{\fill}}lllllll@{\extracolsep{\fill}}}
        \toprule
        Source title & \# Pubs. & Cited & Avg \\
        \midrule
        Proceedings of the IEEE Computer Society Conference & 49 & 2111 & 43.08 \\
        \vspace{.2cm}on Computer Vision and Pattern Recognition &&& \\
        Lecture Notes in Computer Science (including subseries & 372 & 14946 & 40.18 \\
        Lecture Notes in Artificial Intelligence and Lecture Notes &&& \\
        \vspace{.2cm}in Bioinformatics) &&& \\

        \vspace{.2cm}Procedia Computer Science & 13 & 288 & 22.15 \\

        International Conference on Information and Knowledge & 10 & 180 & 18.00 \\
        \vspace{.2cm}Management, Proceedings &&& \\

        IEEE Computer Society Conference on Computer Vision & 23 & 314 & 13.65 \\
        \vspace{.2cm}and Pattern Recognition Workshops &&& \\

        ICASSP, IEEE International Conference on Acoustics, & 95 & 1153 & 12.14 \\
        \vspace{.2cm}Speech and Signal Processing - Proceedings &&& \\

        Proceedings - International Symposium on Biomedical & 30 & 346 & 11.53 \\
        \vspace{.2cm}Imaging &&& \\

        Proceedings of the International Conference on Document & 17 & 158 & 9.29 \\
        \vspace{.2cm}Analysis and Recognition, ICDAR &&& \\

        Proceedings of International Conference on Frontiers & 13 & 113 & 8.69 \\
        \vspace{.2cm}in Handwriting Recognition, ICFHR &&& \\

        2019 IEEE Automatic Speech Recognition and & 12 & 84 & 7.00 \\
        Understanding Workshop, ASRU 2019 - Proceedings &&& \\
        \bottomrule
    \end{tabular*}
    \end{center}
\end{table}

The papers with the highest citation count are listed in
Table~\ref{tab:top_papers}. We found that much of the research focused on
improving deep learning classification, segmentation or object detection
without a focus on a particular domain of application. Other papers centered
in the application of data augmentation methods for biomedical image
classification and segmentation, sound and speech recognition and remote
sensing.

\begin{table}[ht]
    \begin{center}
    \caption{\label{tab:top_papers}
        Top papers using data augmentation techniques, sorted by citation
        count.
    }
    \begin{tabular*}{\textwidth}{@{\extracolsep{\fill}}lllllll@{\extracolsep{\fill}}}
        \toprule
        Authors & Title & Year & Cited \\
        \midrule
        Ronneberger O., Fischer P., & U-net: Convolutional networks & 2015 & 13597 \\
        \vspace{.2cm}Brox T. & for biomedical image segmentation && \\

        Chatfield K., Simonyan K., & Return of the devil in the details: & 2014 & 1885 \\
        \vspace{.2cm}Vedaldi A., Zisserman A. & Delving deep into convolutional nets && \\

        Snyder D., Garcia-Romero D., & X-Vectors: Robust DNN Embeddings & 2018 & 636 \\
        \vspace{.2cm}Sell G., Povey D., Khudanpur S. & for Speaker Recognition && \\

        Shorten C., Khoshgoftaar T.M. & A survey on Image Data & 2019 & 590 \\
        \vspace{.2cm}                 & Augmentation for Deep Learning && \\

        Salamon J., Bello J.P. & Deep Convolutional Neural Networks & 2017 & 505 \\
                               & and Data Augmentation for \\
        \vspace{.2cm}          & Environmental Sound Classification \\

        Eitel A., Springenberg J.T., & Multimodal deep learning for robust & 2015 & 352 \\
        Spinello L., Riedmiller M., & RGB-D object recognition && \\
        \vspace{.2cm}Burgard W. &&& \\

        Ding J., Chen B., Liu H., & Convolutional Neural Network with & 2016 & 319 \\
        Huang M.                  & Data Augmentation for SAR Target && \\
        \vspace{.2cm}             & Recognition \\

        Wong S.C., Gatt A., & Understanding Data Augmentation & 2016 & 302 \\
        \vspace{.2cm}Stamatescu V., McDonnell M.D. & for Classification: When to Warp? && \\

        Frid-Adar M., Diamant I., & GAN-based synthetic medical image & 2018 & 296 \\
        Klang E., Amitai M., & augmentation for increased CNN && \\
        Goldberger J., Greenspan H. & performance in liver lesion && \\
        \vspace{.2cm}               & classification \\

        Bilen H., Vedaldi A. & Weakly Supervised Deep Detection & 2016 & 287 \\
        \vspace{.2cm}        & Networks \\

        \bottomrule
    \end{tabular*}
    \end{center}
\end{table}

\subsection{Keyword Analysis}

The keyword network shown in Figure~\ref{fig:keyword_network} revealed 8 main
communities of keywords, and 13 other small communities. The different
communities are distinguished by the type of algorithms used and/or the domain
of application. The main distinctive factor for the larger communities are the
types of generative models used, while the smaller communities are
distinguished according to the domain of application. The most significant
findings we found from this analysis are:

\begin{enumerate}
    \item The community marked with pink-colored nodes is characterized by the
        usage of neural network-based data augmentation methods in
        convolutional neural networks. The keyword ``deep learning'' is
        positioned as a central node (although not labelled in the figure to
        maintain readability). Other relevant keywords are related to
        machine/deep learning frameworks, deep learning classifiers and data
        augmentation algorithms, such as ``tensorflow'', ``keras'',
        ``convolutional neural network'' and ``generative adversarial
        networks''. Domain specific keywords are also present:
        \begin{itemize}
            \item Medical keywords located in this community cover a variety
                of applications. Relevant sub communities are [``hand
                writing'', ``parkinson's disease (pd)'', ``transfer
                learning''], [``breast cancer'', ``computer-aided
                detection''], [``melanoma'', ``skin cancer'', ``image
                processing'', ``googlenet''], [``chest x-ray'',
                ``computer-aided diagnosis'', ``tuberculosis'',
                ``segmentation''] and [``brain'', ``mri'', ``multiple
                sclerosis''].
            \item Remote sensing keywords are typically related to
                classification and object detection tasks. Relevant sub
                communities are [``object detection'', ``aerial image'',
                ``drone'', ``generative adversarial network'', ``semantic
                segmentation''], [``attributed scattering center (asc)'',
                ``synthetic aperture radar (sar)'', ``convolutional neural
                network (cnn)''], [``remote sensing'', ``road extraction'',
                ``transfer learning'', ``generative adversarial network''].
                Keywords such as ``hyperspectral imaging'' and ``weather
                classification'' are also scattered around the community.
            \item Facial recognition research is also represented in few sub
                communities: [``micro expression recognition'', ``small
                training data'', ``convolutional neural network (cnn)'', ``local
                binary pattern-three orthogonal planes (lbp-top)''] and
                [``training data augmentation'', ``sequence-to-sequence speech
                synthesis'', ``sequence-to-sequence speech recognition''].
            \item Fault detection studies also used data augmentation to deal
                with imbalanced datasets: [``fault diagnosis'', ``imbalanced
                data'', ``gan'']
            \item Data augmentation was also associated to regularization
                methods and feature extraction tasks, based on the presence of
                the sub communities [``overfitting'', ``dropout'' and ``cnn'']
                and [``feature extraction'', ``cnn'', ``svm''].
        \end{itemize}
    \item The community marked with blue-colored nodes is characterized by the
        usage of Markov Chain-based algorithms. The keywords ``markov chain'',
        ``data augmentation algorithm'' and ``monte carlo'' appear as central
        nodes. No application-specific sub-community was found.
    \item The community marked with green-colored nodes is characterized by
        the usage of Markov Chain and Bayesian-based algorithms. The keywords
        ``bayesian inference'', ``markov chain monte carlo'', ``mcmc'',
        ``bayesian analysis'', ``missing data'' and ``em algorithm''
        (expectation maximization algorithm). Application-specific keywords
        may be found sparsely distributed across the community, all of them
        related to biological applications. Specifically, the sub community
        [``ecological health'', ``stressor-response'', ``biological
        monitoring'', ``bayesian methods''] and the keyword ``camera
        trapping'' were found in this community.
    \item The community marked with orange-colored nodes is characterized by
        keywords specific to big data and data warehousing applications. The
        network is composed of the keywords ``big data'', ``data lake'',
        ``olap'', ``map reduce'', ``cmm'', ``data warehouse'',
        ``augmentation'' and ``dm''.
    \item The remaining communities consist mostly of data augmentation
        methods applied to specific domains. Specifically, the usage of
        temporal-dynamic neural network architectures with ``eeg
        (electroencephalogram)'', music information retrieval applications
        (e.g., ``chord recognition''), speech/ speaker recognition and
        embedding, time series forecasting of diabetes and natural language
        processing and text classification.
\end{enumerate}

\begin{figure}[ht]
	\centering
    \includegraphics[width=\linewidth]{analysis/keyword_network}
    \caption{Keyword network.
    }~\label{fig:keyword_network}
\end{figure}

\subsection{Topic Analysis}

The LDA topic extraction resulted in 8 different topics, whose distribution of
topics is shown in Figure~\ref{fig:lda_topics_sankey}. The main topics within
which most articles were included is topic 5, which is defined by the main
theoretical keywords related to image data augmentation. Rather, the secondary
topic is more useful for this analysis. It is found based on the topic
likelihood of each document, excluding the dominant topic. Documents belonging
to the same group across primary, secondary and/or tertiary topics had a
likelihood of zero of belonging to any other topic.

\begin{figure}[ht]
	\centering
    \includegraphics[width=\linewidth]{analysis/lda_topics_sankey}
    \caption{Distribution of documents over the different topics found. The
        left column represent the primary topics, the middle column represents
        the secondary topics and the right columns represents the tertiary
        topics.
    }~\label{fig:lda_topics_sankey}
\end{figure}

The topics found in the bibliometric data are shown in
Table~\ref{tab:topic_analysis}. A few topics seem to overlap each other,
although they are generally distinguishable. The primary domains of
application of data augmentation methods differ for each different topic
identified:

\begin{enumerate}
    \item Documents in Topic 1 frequently use the word ``yolov'', which refers
        to the YOLOvX family of deep learning object detection
        models~\cite{Redmon2015}, where X refers to the version of the model
        used (the most recent version is 5). Another relevant keyword is
        ``style\_transfer'', which refers to a specific technique of data
        augmentation.

        This topic has two primary domains of application. The keywords
        ``pest'' and ``coffe'' refer to data augmentation on agriculture
        research. The keywords ``biomed'', ``histolog'' and ``nodul'' refer to
        biomedical applications such as pulmonary nodule detection and
        histology image classification. Within these topics, a few
        domain-specific data augmentation algorithms were proposed. For
        example, in~\cite{Cicalese2020} the authors propose a style-transfer
        data augmentation method for histology image classification.

    \item Documents in Topic 2 are primarily associated to the study of
        applications that include image data augmentation. The dominant
        keyword, ``hyperspectr\_imag'', refers to the application of data
        augmentation on hyperspectral images, commonly used in remote sensing
        and medicine. Other classification tasks include license plate
        detection (``licens\_plate''), inpainting (``inpaint''), background
        subtraction (``illumin\_chang'') and cloud shadow
        detection/segmentation (``shadow'').

    \item Documents in topic 3 refer to the application of data augmentation
        to deal with censored data (a condition in which the value of an
        observation is only partially known) and/or supervised tasks on data
        structured as graphs. Other domains of application involve chest
        x-rays classification (``cxr''), epidemiology (``risk\_factor'') and
        few audio/music information retrieval (``sourc\_separ'') articles.

    \item Documents in topic 4 refer to the application of data augmentation
        methods on object detection tasks. Specifically fire and smoke,
        pedestrians and crowd counting. Other applications within this topic
        are focused on speech recognition and angiography
        segmentation/classification.

    \item Documents in topic 5 are focused on image segmentation
        and classification methods where data augmentation algorithms are
        involved. It includes common keywords present in a large set of
        articles. These articles are mainly focused on the development of
        different convolutional neural network architectures (``cnn'') and
        neural network-based data augmentation methods.

    \item Documents in topic 6 are focused on Bayesian-based algorithms and
        Markov Chain algorithms. This topic includes data augmentation on
        regression tasks and misclassification detection.

    \item Documents in topic 7 covers the application of data augmentation
        into various domains. Specifically, music information retrieval
        (``music''), fish/marine organisms recognition, gender bias, speech
        recognition, random erasing

    \item Documents in topic 8 contains remote sensing and biomedicine as the
        primary research domains. The keywords ``drone'' and ``aircraft''
        refer to the sources of data collected for remote sensing work,
        whereas ``pneumonia'' and ``chest\_rai\_imag'' refers to biomedicine
        research topics/image data.
\end{enumerate}

\begin{table}[ht]
    \begin{center}
    \caption{\label{tab:topic_analysis}
        Description of the main topics found in the literature.
    }
    \begin{tabular*}{\textwidth}{@{\extracolsep{\fill}}lllllll@{\extracolsep{\fill}}}
        \toprule
        Topic & Representative Paper & Papers & Words\\
        \midrule
        1 & GAN-based synthetic medical image & 440 & yolov, pest, style\_transfer, \\
          & augmentation for increased CNN && coffe, thermal, biomed, \\
          & performance in liver lesion && scene\_text, histolog, nodul, \\
        \vspace{.2cm}& classification && visibl \\

        2 & CVAE-GAN: Fine-Grained Image & 61 & hyperspectr\_imag, licens\_plate, \\
          & Generation through Asymmetric && command, inpaint, \\
          & Training && illumin\_chang, upper, restor, \\
        \vspace{.2cm}  &&& ann, foreign, shadow \\

        3 & A survey on Image Data & 401 & censor, markov\_chain, node, \\
          & Augmentation for Deep Learning && team, tree, cxr, risk\_factor, \\
        \vspace{.2cm}  &&&mass, largest, sourc\_separ\\

        4 & Return of the devil in the details: & 108 & smoke, pedestrian, transcrib, \\
          & Delving deep into convolutional nets && crowd, children\_speech, intent, \\
          &&&adult, auxiliari\_variabl, speech, \\
        \vspace{.2cm}  &&&angiographi \\

        5 & U-net: Convolutional networks for & 632 & imag, detect, gener, dataset, \\
          & biomedical image segmentation && classif, sampl, network, cnn, \\
        \vspace{.2cm}  &&&featur, augment\\

        6 & Deep Convolutional Neural Networks & 370 & tea, multivari, \\
          & and Data Augmentation for && markov\_chain\_mont\_carlo, \\
          & Environmental Sound Classification && bayesian, regress, misclassif, \\
        \vspace{.2cm}  &&& procedur, famili, illustr, mcmc \\

        7 & Weakly Supervised Deep Detection & 160 & music, fish, marin, gender, \\
          & Networks && vocal, random\_eras, low\_qualiti, \\
        \vspace{.2cm}  &&& crowd, prune, bengali \\

        8 & An Efficient Deep Learning Approach & 85 & drone, gait, aircraft, \\
          & to Pneumonia Classification in && gestur\_recognit, pneumonia, \\
          & Healthcare && chest\_rai\_imag, covid, walk, \\
          &&& onset, hidden\_layer \\
        \bottomrule
    \end{tabular*}
    \end{center}
\end{table}

The per-year popularity of the different topics is shown in
Figure~\ref{fig:topics_per_year}. Since 2015, topic 5 gained more research
momentum, whereas topic 6 lost much of its relative popularity within the
field. In the past 5 years topics 8 and 3 have become steady research streams
while topic 1 saw a significant growth in popularity.

\begin{figure}[H]
	\centering
    \includegraphics[width=\linewidth]{analysis/topics_per_year}
    \caption{Topic frequency per year.
    }~\label{fig:topics_per_year}
\end{figure}

\subsection{Research Gap Discussion}

Data augmentation mechanisms are often used as regularization methods for
deep learning classifiers. The study of data augmentation mechanisms in
ensembles of simple classifiers have achieved state-of-the-art performance not
only 10 years ago~\cite{Meier2011, Ciresan2011}, but also when compared to
modern deep learning architectures~\cite{Tolstikhin2021, Touvron2021,
Liu2021}. However, the implementation of different data augmentation methods
shows a promising path to improve the performance of simple classifiers
(and/or recent ensemble architectures) and requires further research.

A research application that was not frequently found in the literature was
small dataset augmentation. This is particularly useful for any complex
problem when the amount of labeled data available to use as training data is
scarce, which limits the usage of classification algorithms and especially
deep learning algorithms. In this context, techniques such as Active Learning
can be used to annotate a small amount of data, while maximizing the
classification performance~\cite{Su2020}. However, classifiers may not be
capable of generalizing with small training datasets and the ability to
reproduce and augment the labelled data available can further reduce
annotation cost and allow the usage of data intensive classifiers.

Another limitation found in the literature relates to the problem of
initialization on network-based data augmentation methods. The same data
augmentation algorithm trained with different initialization settings
(different random seed or training subset) may lead to different model
parameters and quality of the trained classifier.

The rapid development of data augmentation algorithms raises additional open
questions on how the data used and store for model training. Specifically, the
lower data storage and processing power available to the general public
(\textit{i.e.}, organizations and individuals) is a limitation for producing
state-of-the-art classifiers. Another problem arises from data privacy
concerns, since the usage of user data to train machine learning models
typically involve the storage of such information. However, if data
augmentation algorithms were continuously updated and capable of producing
reliable data on an as-needed basis, not only would storage requirements
decrease, but it would also become possible to work with fully artificial
data, without the need to store as much data. This would also facilitate the
sharing of datasets (in the form of an algorithm) without compromising
sensitive data.

\section{Conclusion}~\label{sec:conclusion}

Depending on the domain of application, data augmentation research differs in
the format of publication. On the one hand, domains like Statistics, Remote
Sensing and Medical Imaging seem more active on journal publications,
typically in journals with high impact factor. On the other hand, research
developed in the domains of computer vision, speech recognition, acoustic
modelling, natural language processing and signal processing seem to attribute
higher importance to conference papers. Many of the influential papers we found
were focused on deep learning methods for classification, segmentation, sound
and speech recognition and remote sensing.

We analysed the different communities of keywords formed using document
keywords, as well as topic analysis using a LDA analysis over the document's
abstracts. We found various distinctive areas of research, both regarding the
data augmentation methods used and the domain of application. We found that in
recent years research on augmentation methods using Bayesian-based algorithms,
as well as Markov Chain algorithms reduced its popularity, whereas data
augmentation methods based on neural networks and deep learning classifiers
have increased its popularity.

Data augmentation is most commonly applied/studied in the realm of computer
vision for tasks like image classification, segmentation, object detection,
inpainting and background subtraction tasks, even though it may be applied to
many other data structures. It is frequently used in studies within the
domains of biomedicine, agriculture, speech recognition, acoustic modelling,
remote sensing and computational creativity. It is also used alongside other
data preprocessing techniques, such as feature extraction and dimensionality
reduction.

Although data augmentation is a vibrant area of research, there are still
significant gaps to be addressed. Data augmentation methods are increasingly
used as regularization methods for deep learning. Although, recent research
shows that the same can be done for simpler classifier configurations in order
to achieve a classification performance comparable to that of state-of-the-art
deep learning, which require further confirmation, as well as the development
of less computational intensive data augmentation methods. Other less popular
topics, such as small data augmentation, appear to have a relevant practical
importance and require further research. In addition, other limitations of
data augmentation algorithms should be addressed. One problem commonly found
in the literature is the impact the weights initialization and training set
used have in the quality of the trained algorithm. In the future, using data
augmentation methods as a source of artificial datasets can address a variety
of concerns, such as data privacy, sharing and storage. Finally, exploring
data augmentation algorithms to complement or replace techniques such as
Active Learning may reduce the cost of data collection, although it is yet to
be explored.

\section*{Declarations}

\subsection*{Conflict of interest}

The authors declare that they have no conflict of interest or competing interest in
this work.

\subsection*{Authors' contributions}

Joao Fonseca collected all the research material and wrote the original text
of the paper. Fernando Bacao helped with the article preparation phase and
provided suggestions regarding the text, content and structure of the paper.

\subsection*{Availability of data and materials}

The data and the experiment's documentation is available at
\href{https://github.com/joaopfonseca/publications}{https://github.com/joaopfonseca/publications}.

\subsection*{Code availability}

The analyses and source code is available at
\href{https://github.com/joaopfonseca/publications}{https://github.com/joaopfonseca/publications}.