\section{Introduction}
\label{intro}
Deep Neural Networks (DNNs) have become vital to solve many tasks across domains including image/text/graph classification and generation, object recognition, segmentation, speech recognition, etc. As the applications of DNNs widen in scope, \textit{robustness} and \textit{interpretability} are two important parameters that define the goodness of a trained DNN model. While on one hand the deep network should be robust to imperceptible perturbations, on the other hand it should be interpretable enough to be trusted when practically used in domains like autonomous navigation or healthcare. Keeping in mind the vulnerability of deep networks to adversarial attacks \cite{szegedy2013intriguing}, efforts have been undertaken to make them more robust to these attacks. Among the proposed methodologies, Adversarial Training (AT) \cite{at@madry,at@goodfellow} has emerged as one of the best defenses wherein networks are trained on adversarial examples to better classify them at test time. On the other hand, in order to generate an interpretable explanation to a network prediction, many methods have been proposed lately, of which guided-backpropagation (GBP)\cite{gbp}, GradCAM++ \cite{grad-cam++}, Integrated Gradients (IG) \cite{integrated-gradients} and fine-grained visualisations (FGVis) \cite{fgvis} are popular to name a few. \par

Most work so far on interpretability and adversarial robustness have focused on either of them alone. Very recently, over the last few months, there has been a new interest in coupling the two notions and aiming to understand the connection between robustness and interpretability \cite{connection@etmann,imagenet-texture,robustness-odds,interpret-atcnn,attribute-regularization,jacobianAT}. The few existing efforts can be categorized broadly into three kinds: (i) Efforts \cite{interpret-atcnn,robustness-odds} which have shown that explanations generated by more robust models are more interpretable than their non-robust counterparts; (ii) Efforts that have attempted to theoretically analyze the relationship between adversarial robustness and interpretability (for e.g., \cite{connection@etmann} obtains an inequality between these notions which holds to equality in case of linear models, and \cite{relating-explanations} relates the two using a generalized form of hitting set duality); and (iii) Efforts \cite{attribute-regularization,jacobianAT}, more recent, that have attempted to improve robustness by training models with additional objectives that constrain the image explanations to be more interpretable and robust towards attribution attacks.
All of these efforts are recent, and more work needs to be done to explore the connections better. In this work, we provide a different, yet simple and effective, approach to leverage saliency maps for adversarial robustness.
We observe that adversarial perturbations correspond to class-discriminative pixels in later stages of training (see Figure \ref{fig:pert-variation}), and exploit this observation to use the saliency map (static) of a given image, whilst training to improve robustness. For datasets where bounding boxes and segmentation masks are provided, we demonstrate that one can exploit these, in lieu of saliency maps, to improve model robustness using our approach. Our work would be closest to the third category of methods described above, and we differentiate from these other recent efforts further in Section \ref{sec_related_work}.

Generally speaking, humans tend to learn new tasks in a robust, generalizable fashion when provided with explanations during their learning phase. For example, a medical student learns about a disease better when provided with explanations of how the disease acts inside the body, rather than just the description of external symptoms. A person may otherwise learn irrelevant relationships without knowledge of underlying explanations. Similarly, we opine that a DNN model that is trained with explanations is less easily fooled by adversarial perturbations.
Little has been explored in this direction, and we aim to leverage this relationship to provide an efficient methodology for adversarial robustness. We differ from earlier efforts in our primary objective that we use explanations to generate pseudo-adversarial examples instead of using them as a regularization term like \cite{attribute-regularization,jacobianAT} (more in Section \ref{sec_related_work}). In particular, we provide a simple yet effective methodology (as compared to a recent effort such as \cite{relating-explanations}, which is NP-hard) that improves model robustness using saliency maps or already provided bounding boxes or segmentation masks.
\par

Our key contributions can be summarized as follows: (i) We observe a tangible relationship between a saliency map and adversarial perturbations for a given image, and leverage this observation to propose a new methodology that uses the saliency map of the image to mimic adversarial training; (ii) We show through our empirical studies on widely used datasets: CIFAR-10, CIFAR-100, Tiny ImageNet and Flowers-17, to show the improvement in adversarial robustness through our proposed method; (iii) We show that the improvement becomes more pronounced when a finer and stronger saliency map is used, signifying a strong correlation between saliency maps and adversarial robustness; (iv) When bounding boxes or segmentation masks are already available in a dataset (e.g. Tiny ImageNet or Flowers), we demonstrate that our methodology improves adversarial robustness with no additional cost to generate perturbations for training; (v) Additionally, we show that integrating our method with adversarial training methods, such as PGD \cite{at@madry} and TRADES \cite{trades}, further improves their performance; and (vi) We perform detailed ablation studies to better characterize the efficacy of our proposed methodology. We believe that this work can contribute to opening up a rather new direction to enhance robustness of DNN models.

\section{Related Work}
\label{sec_related_work}
We review earlier efforts related to this work from multiple perspectives, as described below.

\noindent \textbf{Explanation methods:} Various methods have been proposed over the last few years to explain the decisions of a neural network. \textit{Backpropagation-based methods} find the importance of each pixel by backpropagating the class score error to the input image. An improved and popular version of this, known as Guided-Backpropagation \cite{gbp}, only keeps paths that lead to positive influence on the class score, leading to much cleaner-looking explanations. SmoothGrad \cite{smooth-grad}, VarGrad \cite{var-grad} and Integrated gradients \cite{integrated-gradients} refine the explanations by combining/integrating gradients of multiple noisy/interpolated versions of the image. Other backpropagation based methods like DeepLift \cite{deep-lift}, Excitation BackProp \cite{excitation-bp} and Layerwise Relevance Propagation \cite{lrp} generate explanations by utilizing topdown relevance propagation rules. PatternNet and PatternAttribution \cite{patternnet} yields explanations that are theoretically sound for linear models and produce improved explanations for deep networks. CAM \cite{cam}, Grad-CAM \cite{grad-cam}, Grad-CAM++ \cite{grad-cam++} form another variant of generating explanations known as \textit{Activation-based methods}. These methods use linear combinations of activations of convolutional layers, with the weights for these combinations obtained using gradients. \textit{Perturbation  based  methods} generate attribution maps by examining the change in prediction of the  model when the input image is perturbed \cite{pertexp1,pertexp2}. All the aforementioned methods however focus solely on explaining neural network decisions.

\noindent \textbf{Adversarial Attacks and Robustness:} With the advancement of newer adversarial attacks each year \cite{at@goodfellow,at@madry,spatial-adv-exmp,trades}, methods have been proposed to defend against them. Parseval Networks \cite{parseval-nets} train robust networks by constraining the Lipschitz constant of its layers to be smaller than 1. Another category of methods harness the susceptibility of latent layers by performing latent adversarial training (LAT) \cite{LAT}  or using feature denoising \cite{feature-denoising}. Other methods like DefenseGAN \cite{defense-gan} exploit GANs wherein they learn the distribution of unperturbed images and find the closest image for a given test image, to feed the network at inference time. TRADES \cite{trades}, more recently, presents a new defense method that provides a trade-off between adversarial robustness and vanilla accuracy by decomposing prediction error for adversarial examples (robust error) as the sum of natural (classification) error and boundary error, and providing a differentiable upper bound using theory based on classification-calibrated loss. Among the proposed defenses against adversarial attacks, Adversarial Training (AT) \cite{at@madry,at@goodfellow} has remained the most popular and widely used defense, where the network is trained on adversarial examples in order to match the training data distribution with that of adversarial test distribution. More recent efforts in this direction include \cite{free-AT,YOPO}, which aim to reduce adversarial training overhead by recycling gradients and accelerating via the maximal principle respectively. In this work, we focus on adversarial training, considering it still remains the most reliable defense against different attacks.

\noindent \textbf{Interpretability and robustness:} The last few months have seen a few efforts on associating the notions of robustness and interpretability. These efforts can be categorized into three kinds, as introduced briefly earlier: (i) The first kind centers around interpreting how adversarially trained convolutional neural networks (ATCNNs) recognize objects. Recent studies \cite{interpret-atcnn,robustness-odds} have shown that representations learned by ATCNNs are more biased towards image shape than its texture. Also, these ATCNNs tend to evince more interpretable saliency maps corresponding to their prediction than their non-robust equivalents. (ii) In the second kind, inspired by \cite{interpret-atcnn,robustness-odds}, Etmann et al. \cite{connection@etmann} recently quantified this behavior of ATCNNs by considering the alignment between saliency map and the image as the metric for interpretability. They confirmed that for a linear model, the alignment grows strictly with robustness. For non-linear models such as neural networks, they show that the their linearized robustness is loosely bounded by the alignment metric. \cite{relating-explanations}, which is rather new, provides a theoretical connection between adversarial examples and explanations by demonstrating that both are related by a generalized form of hitting set duality. (iii) Encouraged by the work of Etmann \cite{connection@etmann}, a third category of work more recently sought to answer the question: Do robust and interpretable saliency maps imply adversarial robustness ? Recent efforts \cite{explanations-manipulated,fragile-interpretation} have shown that the explanations of neural networks can also be manipulated by adding perturbations to input examples, which, instead of causing mis-classification, result in a different explanation. To tackle this problem, a recent effort, Robust Attribution Regularization (RAR) \cite{attribute-regularization}, aims to train networks by optimizing objectives with a regularization term on explanations in order to achieve robust attributions.

In contrast to the abovementioned methods, our work explores a different side to the connection between notions of interpretability and robustness. As explained in Section \ref{intro}, we aim to generate adversarial perturbations from a given saliency map to improve robustness while training a neural network. Our work is perhaps closest to RAR \cite{attribute-regularization} in using explanations (saliency maps) during training, we however differ from them in our primary objective and in that we are using explanations to generate adversarial examples, while RAR seeks to attain attributional robustness. In terms of approach, although one could consider the work by Ignatiev et al. \cite{relating-explanations} as related to ours, they had only preliminary results on MNIST, and focused on providing a theoretical connection between adversarial examples and explanations based on logic and constraint programming, which may not scale to large-scale datasets.
Our work attempts to show that explanations can yield robustness in standard large-scale benchmark datasets in an efficient way. We further hypothesize that, by advancement through our method, we can actually take advantage of weak explanations like bounding boxes and segmentation masks which may otherwise be left unexploited in a dataset, to train adversarially robust models.

\begin{figure*}[h]
\centering
\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{./images/Pert2.png}
\caption{Variation of adversarial perturbation with training epochs during 5-step PGD adversarial training of Resnet-34 on CIFAR-10. We observe that adversarial perturbations at later stages of training correspond to class-discriminative regions/pixels.}
\label{fig:pert-variation}
\end{figure*}

\section{Using Saliency Maps for Efficient Adversarial Training}
\label{methodology}
Our method is motivated by the observation that adversarial perturbations at later stages of training correspond to class-discriminative regions/pixels, as shown in Figure \ref{fig:pert-variation}. We begin describing our method with the necessary notations and preliminaries.
\vspace{3pt}

\noindent \textbf{3.1 Notations and Preliminaries.}
We denote a neural network as $\Phi(. \ ;  \ \theta) : \mathbb{R}^d \rightarrow \mathbb{R}^k $, parametrized by weights $\theta$, which takes an input $\textbf{x} \in \mathbb{R}^d $ and outputs a logit, $\Phi^i(\textbf{x})$, for each of $k$ classes, i.e. $i \in \mathcal{C} = \{1,\cdots,k\}$, $\mathcal{C}$ denoting the set of class labels.
Without loss of generality, considering an image classification setting, we define a saliency map $\textbf{s}$ corresponding to an input sample $\textbf{x}$ as $\textbf{s} \in [0,1]^d$, where the presence of an object of interest in input $\textbf{x}$ lies between 0 and 1. For a trained network $\Phi$, the unnormalized saliency map for an input $\textbf{x}$ can simply be given as:
$\nabla_{\textbf{x}} \Phi^{i^*}(\textbf{x})$, where $i^* = \argmax_i \Phi^i(\textbf{x})$. (In practice, this gradient is at times computed w.r.t feature maps of intermediate layers, but we leave our notations w.r.t $\textbf{x}$ for ease of understanding.)
\vspace{3pt}

\noindent \textit{Projected Gradient Descent (PGD) Attack:} Prior efforts on adversarial attacks \cite{at@goodfellow} include the Fast Gradient Sign Method (FGSM), a $l_{\infty}$-bounded single step attack which calculates an adversary as: $\textbf{x} + \epsilon \hspace{1mm} \text{sign}(\nabla_\textbf{x} \mathcal{L} \ (\Phi(\textbf{x},\theta),y))$.
A more powerful attack however is the popularly used multi-step variant, also called Projected Gradient Descent (PGD), given by:
\begin{align}
\label{eqn_pgd}
    \textbf{x}^0 = \textbf{x};
    \textbf{x}^{t+1} = \Pi_{\textbf{x}+ \textit{N}} \big( \textbf{x}^{t} + \alpha \hspace{1mm} \text{sign}(\nabla_{\textbf{x}} \mathcal{L} \ (\Phi(\textbf{x},\theta),y)) \big)
\end{align}
\noindent where $\alpha$ is the step size, $\Pi$ is the projection function, and $N$ is the space of possible perturbations. We use PGD as the choice of attack in our experiments, and later change to show (in Section \ref{ablations}) how our method performs against other attacks.
\vspace{3pt}

\noindent \textit{Adversarial Training (AT):} AT \cite{at@goodfellow} is generally used to make the models adversarially robust by matching the training distribution with the adversarial test distribution. Essentially, for AT, the optimal parameter $\theta^*$ is given by:
\begin{equation}
    \theta^* = \argmin_\theta \mathbb{E}_{(\textbf{x},y) \sim D} \ \left[ \ \max_{\delta \in N}\mathcal{L} \ (\Phi(\textbf{x} + \delta,\theta),y) \ \right]
\end{equation}
Here, the inner maximization $\max_{\delta \in N} \mathcal{L} \ (\Phi(\textbf{x} + \delta,\theta),y)$ is calculated using a strong adversarial attack such as PGD.\\

\noindent \textbf{3.2 Saliency-Based Adversarial Training: Motivation.}
An adversarial perturbation at input $\textbf{x}$ is given by:
\begin{equation}
\arg\inf_{e \ \in \ \mathbb{R}^d} \{\Vert e \Vert: \argmax_i \Phi^i(x+e) \neq \argmax_i \Phi^i(x) \}
\end{equation}
Etmann et al. \cite{connection@etmann} showed that for most multi-class neural networks, especially ones with ReLU or Leaky ReLU activation functions, which we consider in this work\footnote{In all our experiments, we train networks with ReLU activations which ensures that this assumptions is met. We follow the experiments in \cite{connection@etmann} in this regard, to ensure that our model is locally affine in a given data point's neighborhood.}, the network's score function, $\Phi$ is sufficiently locally-linear in relevant neighbourhood of input $\textbf{x}$, i.e.
\begin{equation}
    \Phi^i(\textbf{x} +  e) \approx \Phi^i(\textbf{x}) + e^T \cdot \nabla_\textbf{x}\Phi^i(\textbf{x})
\label{eqn_etmann_locally_linear}
\end{equation}
Leveraging this, an adversarial perturbation, $e$, which is intended as a perturbation to input $\textbf{x}$ which results in a change of predicted label, can be modeled as follows (given $i,j \in \mathcal{C}$):
\begin{align}
    & \argmax_i \Phi^i(x+e) \neq \argmax_i \Phi^i(x)\\
    \Longleftrightarrow \qquad & \exists j \neq i^* : \Phi^j(\textbf{x} + e) > \Phi^{i^*}(\textbf{x} + e)\\
    \Longleftrightarrow  \qquad & \exists j \neq i^* : e^T \cdot (\nabla_\textbf{x}\Phi^j(\textbf{x}) - \nabla_\textbf{x}\Phi^{i^*}(\textbf{x})) > \Phi^{i^*}(\textbf{x}) - \Phi^j(\textbf{x})
\end{align}
The third inequality (7) comes from combining inequality (6) and expression (4). The infimum over $\Vert e \Vert$, which provides a minimal perturbation to change the class label, is achieved by choosing $e$ as a multiple of $\nabla_\textbf{x}(\Phi^{j}(\textbf{x})- \Phi^{i^*}(\textbf{x}))$. (In general, note that the LHS of $e^T z > c$ where $c$ is a constant, is maximized by $e = \Vert e \Vert \frac{z}{\Vert z \Vert}$, leading to $\Vert e \Vert \Vert z \Vert > c$. The infimum of $e$ is then achieved by choosing it as a multiple of $z$).
The direction of adversarial perturbation then becomes:
\begin{equation}
\label{eqn_pert_direction}
\nabla_\textbf{x}(\Phi^{j}(\textbf{x}) - \Phi^{i^*}(\textbf{x}))
\end{equation}
This perturbation direction depends on two quantities: (i) $\nabla_\textbf{x} \Phi^{i^*}(\textbf{x})$ the saliency map for the true class $i^*$; and (ii) $\nabla_\textbf{x} \Phi^{j}(\textbf{x})$, the saliency map of $\textbf{x}$ for class $j$ for which the infimum of $e$ is attained. We now analyze the direction of adversarial perturbation separately for the binary and multi-class cases.
\vspace{3pt}

\noindent \textit{Binary case:} Let us consider a binary classifier $h: \textbf{x} \rightarrow \{-1,1\}$ given by: $h = \text{sign}(\Phi(\textbf{x},\theta))$, where $\Phi(\textbf{x},\theta))$ represents the logit of the positive class. Let $\Phi'(\textbf{x})$ denotes the logit of negative class. Applying a sigmoid activation function, we get the probability of the positive and negative class as: $P(y=+1|\textbf{x}) = \frac{1}{1 + \exp^{-\Phi(\textbf{x},\theta)}}$ and $P(y=-1|\textbf{x}) = \frac{1}{1 + \exp^{-\Phi'(\textbf{x},\theta)}}$ respectively.
It is simple to see that the probability of the negative class can also be written as:
\[P(y=-1|\textbf{x}) = 1 - P(y=+1|\textbf{x}) = \frac{1}{1 + \exp^{\Phi(\textbf{x},\theta)}}\]
Rather, the corresponding logit score of the negative class is $-\Phi(\textbf{x},\theta))$. In Eqn \ref{eqn_pert_direction} for a binary classifier, one can hence view $\Phi^{j}(\textbf{x}) =  - \Phi^{i^*}(\textbf{x})$, and define the adversarial perturbation direction in Eqn \ref{eqn_pert_direction} simply as $- \nabla_\textbf{x}(\Phi^{i^*}(\textbf{x}))$.
\vspace{3pt}

\noindent \textit{Multi-class case:} We extend a similar argument to the multi-class setting using an approximation. The direction of adversarial perturbation in the multi-class setting as give in Eqn \ref{eqn_pert_direction} would require finding the class $j$ for which the infimum of $\Vert e \Vert$ is attained. This requires computing the quantity in Eqn \ref{eqn_pert_direction} for all classes, and identifying the $j \neq i^*$ for which the quantity attains the least value. This is compute-intensive. To avoid this computational overhead, we rely on $\Phi^{i^*}$ alone (note that $i^*$, the ground truth label, is known to us at training), and simply propose the use of $- \nabla_\textbf{x}(\Phi^{i^*}(\textbf{x}))$ as the direction of perturbation (as in the binary case). We now argue that this approximation is a reasonable one. Considering the multi-class setting as $k$ binary classification problems, for the binary classifier corresponding to the ground truth class $i^*$, we would have:
\[P(y \neq i^*|\textbf{x}) = 1 - P(y = i^*|\textbf{x}) = \frac{1}{1 + \exp^{\Phi^{i^*}(\textbf{x},\theta)}}\]
In other words, the corresponding logit score of the negative class is $-\Phi^{i^*}(\textbf{x},\theta))$. Approximating the direction of the adversarial perturbation as the average of the directions of the perturbations across the $k$ binary classification problems, we would get the direction of the perturbation to be: $\nabla_\textbf{x}(\sum_{j \neq i^*}\Phi^{j}(\textbf{x}) - k \Phi^{i^*}(\textbf{x}))$. Assuming that each of the classes $l \neq i^* \in \mathcal{C}$ is equally likely to be the $j$ that minimizes $\Vert e \Vert$, it is evident that choosing $- \nabla_\textbf{x}(\Phi^{i^*}(\textbf{x}))$ as the direction of perturbation would be the most conservative option in the expected sense. We show through our experiments that this option works reasonably well.

Since we deal with $l_{\infty}$-bounded perturbations in this work, following PGD (Eqn \ref{eqn_pgd}), we use $- \text{sign} \ (\nabla_\textbf{x} \Phi^{i^*}(\textbf{x}))$ instead of $- \nabla_\textbf{x} \Phi^{i^*}(\textbf{x})$ itself as the perturbation direction.
We complete the above discussion by noting that $\nabla_\textbf{x} \Phi^{i^*}(\textbf{x})$ is the saliency map, \textbf{s}, defined at the beginning of this section.
In other words, the direction of adversarial perturbation can be obtained using a saliency map (as we also show in our experiments).

\begin{wrapfigure}[23]{l}{0.37\textwidth}
\vspace{-15pt}
\includegraphics[width=0.37\textwidth]{./images/sal_attack}
\vspace{-15pt}
\caption{Adversarial attack accuracy when perturbing input example using the direction of negative saliency on normally (Std.) and adversarially (Adv.) trained models on CIFAR-10 and CIFAR-100. Note that $\epsilon=0$ denotes clean accuracy (no attack), all other models face attacks by saliency maps and have lower accuracies.}
\label{fig:sal-attack}
\end{wrapfigure}
To study the above claim, we conducted experiments to check if the negative of the saliency map corresponding to the ground truth class, i.e. $-\nabla_\textbf{x}\Phi^{i^*}(\textbf{x})$, can indeed be used as a direction to perturb input. We created adversarial examples by perturbing original examples as: $\textbf{x} = \textbf{x} - \epsilon \cdot \text{sign}(\nabla_\textbf{x}\Phi^{i^*}(\textbf{x}))$ (using saliency maps) to attack a model. We attack a Resnet-10 model trained on CIFAR-10 and CIFAR-100 datasets, normally (Std.) and adversarially (Adv.) trained using 5-step PGD attack with 3 different magnitudes of $l_\infty$-norm perturbations, $\epsilon \in \{2/255, 4/255, 8/255\}$. We report our findings in Figure \ref{fig:sal-attack}. The figure shows that, examples generated by perturbing an input in the direction of negative saliency decreases the model's accuracy to a reasonable extent, affecting models both models trained normally and adversarially. We now describe how we leverage this relationship to perform Saliency-based Adversarial Training (SAT).\\

\noindent \textbf{3.3 Saliency-Based Adversarial Training: Algorithm.}
Algorithm \ref{mimic-AT} summarizes our methodology of using saliency maps for adversarial training. The saliency maps during training are obtained either through annotations provided in a dataset (such as bounding boxes or segmentation masks), or through a pre-trained model which is used only to get saliency maps. More details on obtaining saliency maps is discussed in Section \ref{results}.

We also observed in our studies that when training a network using adversarial training, during the initial phase of training when the weights are not optimal, the perturbations computed by the attack methods are random. But with training, as weights become optimal, they become more class-discriminative. Figure \ref{fig:pert-variation} illustrates this observation, where the perturbation is random in initial phases of training but eventually becomes class-discriminative as the model trains. We exploit this observation to complete our methodology.
In order to mimic the above behavior of the perturbation over training, we choose the direction of perturbation in a stochastic manner. We choose the $i^{th}$ component $\delta^t$[i] of perturbation $\delta^t$ at time $t$ as:
\begin{equation}
    \label{pert-calc}
    \delta^t[i] =
    \begin{cases}
      \textbf{z}[i], & \text{with probability } \alpha^{t} \\
      -\textbf{s}[i], & \text{with probability } 1-\alpha^{t}
    \end{cases}
 \end{equation}
\noindent where $\textbf{z} \in \{-1,1\}^d$ is sampled randomly, and 0 $< \alpha <$ 1. During initial epochs of training, when $\alpha^t$ is close to 1, $\delta^t$ will be dominated by random values. However, as training proceeds and $\alpha^t$ starts diminishing, $\delta^t$ smoothly transitions to $-\textbf{s}$ and will be influenced by the adversarial character of the saliency map.
\begin{wrapfigure}[21]{l}{0.63\textwidth}
\begin{minipage}{0.63\textwidth}
\vspace{-20pt}
\begin{algorithm}[H]
\SetAlgoLined
\textbf{Input:} Training Dataset $D$, Saliency Maps $S$, Model $\Phi(. \ ; \ \theta)$, SAT hyperparameter $\alpha$, Learning rate $\eta$, Maximum $l_\infty$ perturbation $\epsilon_0$ \\
\textbf{Output:} Optimal parameter $\theta^*$\\
Initialize model parameters as $\theta = \theta^0$. \\
 \For{$t \in \{1, 2, ..., n\}$}
    {
        Sample training data of size $B$ : $\{(\textbf{x}^i, y^i)\}$ from $D$. \\
        Pick out corresponding saliency maps : $\{\textbf{s}^i\}$ from $S$. \\
        Calculate $\delta^{ti}$ for each $\textbf{x}^i$ using Equation \ref{pert-calc}. \\
        Perturb the input examples : $\textbf{x}^i := \textbf{x}^i + \epsilon_0 \cdot \delta^{ti}$. \\
        Perform clipping to keep $\textbf{x}^i$ bounded : $\textbf{x}^i$ := $clip(\textbf{x}^i)$. \\
        Update model parameters : \\ $\theta^t := \theta^{t-1} - \eta \cdot \triangledown_\theta \ \frac{1}{B} \sum_{i=1}^{B} \mathcal{L} \ (\Phi(\textbf{x}^i,\theta^{t-1}),y^i) $
    }
 \caption{Saliency-based Adversarial Training (SAT) Methodology}
 \label{mimic-AT}
\end{algorithm}
\end{minipage}
\end{wrapfigure}
\\
When additional annotations such as bounding boxes or segmentation masks are available in a dataset, our approach considers these as weak saliency maps for the methodology. Usually, bounding boxes or segmentation masks are available as single channel images. To use them in our algorithm, we concatenate them along channel dimension to get the required dimension as image. After this pre-processing, we generate the weak saliency, $\tilde{\textbf{s}}$ from bounding boxes or segmentation masks as: \\

\begin{equation}
    \label{bbox-to-sal}
    \tilde{\textbf{s}}[i] =
    \begin{cases}
      1, & \text{if i\textsuperscript{th} pixel lies inside bbox or seg masks} \\
      -1, & \text{otherwise}
    \end{cases}
 \end{equation}
\noindent
As mentioned earlier, using these bounding boxes or segmentation masks as saliency maps allows us to perform adversarial training with no additional cost to compute perturbations, unlike existing adversarial training methods which can be compute-intensive.

\section{Experiments and Results}
\label{results}
In this section, we present our results using the proposed SAT method (Algorithm \ref{mimic-AT}) on multiple datasets with different variations of saliency maps. We begin with describing the datasets, evaluation criteria and implementation details.\\

\noindent {\large \textbf{4.1 Experimental Setup}}
\vspace{3pt}
\noindent \textbf{Datasets and Evaluation Criteria:} We perform experiments on well-known datasets: CIFAR-10, CIFAR-100 \cite{cifar-datasets}, Tiny ImageNet \cite{tiny-img} and FLOWER-17 \cite{flower}.
We evaluate the \textit{adversarial robustness} of trained models (model accuracy when perturbations from  following attacks are provided as inputs, as done in all earlier related efforts - also called \textit{adversarial accuracy}) using the popular and widely used PGD attack (described in Sec \ref{methodology}) with 5 steps and 4 different levels of $l_\infty$ perturbation $\epsilon_0 \in \{1/255, 2/255, 3/255, 4/255\}$. We also evaluate the robustness of our models against other attacks: TRADES \cite{trades} as well as uniform noise in Sec \ref{ablations}. We trained all our models for 5 trials and observed minimal variations in the values. We hence report the mean value of our experiments in our tables.
\vspace{3pt}

\noindent \textbf{Baselines:} We compare our method with multiple baseline methods, including those that train adversarially, as well as those that don't train adversarially:
\textit{(i) Original Model:} Model trained normally with no adversarial training; \textit{(ii) PGD-AT:} Model trained adversarially using 5-step PGD attack with max $l_\infty$ perturbation and $\epsilon = 8/255$, as in \cite{at@madry}; \textit{(iii) TRADES-AT:} Model trained adversarially using TRADES attack with max $l_\infty$ perturbation and $\epsilon = 8/255$, as in \cite{trades}; \textit{(iv) Original + Uniform Noise:} Model trained normally with no adversarial training, but training data is perturbed with uniform noise sampled from $[\frac{-8}{255},\frac{8}{255}]$ during training; \textit{(v) PGD + Uniform Noise:} PGD-AT, described in (ii), training data is additionally perturbed with uniform noise sampled from $[\frac{-8}{255},\frac{8}{255}]$ during training; \textit{(vi) TRADES + Uniform Noise:} TRADES-AT, described in (iii), training data is additionally perturbed with uniform noise sampled from $[\frac{-8}{255},\frac{8}{255}]$ during training.
The models are trained for 100 epochs with standard cross-entropy loss, minimized using Adam optimizer (learning rate = $1e-3$).
\vspace{3pt}

\noindent \textbf{Implementation Details:}
As in Sec \ref{methodology}, our method relies on being provided with a saliency map for the ground truth label of a given image, while training the model. For Tiny ImageNet and FLOWER-17, we simply use the bounding boxes and segmentation masks provided in the dataset (and left unexploited often) as `weak' saliency maps to train the model using the proposed SAT method.
This thus incurs no additional cost when compared to competing methods that perform adversarial training by generating perturbations (a costly operation when using methods such as PGD). For CIFAR-10 and CIFAR-100, since we do not have such information (bounding boxes/segmentation masks) provided, we obtain saliency maps using teacher networks. In particular, we train two teacher networks: Resnet-10 and Resnet-34 for this purpose. Our final model architecture in all these settings is a Resnet34 too.

\begin{wrapfigure}[9]{l}{0.5\textwidth}
\vspace{-20pt}
\includegraphics[width=0.5\textwidth]{./images/saliency3}
\vspace{-10pt}
\caption{Saliency maps of non-robust \textit{(top, model trained normally)} and robust \textit{(bottom, model trained adversarially)} variants of Resnet-10 with different explanation methods on CIFAR-10.}
\label{saliency}
\end{wrapfigure}
To go further, we also consider multiple variants of the above choices to study our method more carefully. It is believed that the quality of saliency maps generated by an adversarially trained model is better than its non-robust equivalent \cite{robustness-odds}. Figure \ref{saliency} supports the above claim. We hence train the aforementioned teacher models in two ways: Standard (denoted as \textit{Std} in the results) and Adversarial (denoted as \textit{Adv} in results) (regular PGD-based adversarial training), and use the saliency maps for the ground truth class. The saliency maps themselves are generated in two ways: we use bounding boxes obtained using Grad-CAM++ \cite{grad-cam++} as weak saliency maps, and we use the finer saliency maps generated as is by Guided Back-propagation (GBP) \cite{gbp} to train our student model. We set hyperparameters $\alpha = (0.6)^{\frac{1}{10}}$ and $\epsilon_0 = \frac{8}{255}$, and minimize cross-entropy loss using an Adam optimizer (learning rate = $1e-3$) for 200 epochs.

To complete the study, we also use variants of our method: \textit{PGD-SAT} and \textit{TRADES-SAT}, where we perturb the input randomly with either perturbations calculated by SAT or with PGD/TRADES attacks.
All these variants are named in a self-explanatory manner in our results: \textit{X | Y | Z}, where  \textit{X} denotes the teacher model (Resnet-10/-34), \textit{Y} denotes the mode of training (Std or Adv), and \textit{Z} denotes the saliency map method (GBP/ GradCAM++) used
In Sec \ref{ablations}, we also perform additional studies with other saliency methods like Guided-GradCAM++, Smooth-Grad and Integrated-Gradients to study the generalizability of our results on varying quality of saliencies.
\vspace{3pt}

\begin{table}
\begin{center}
\scalebox{0.7}{
\begin{tabular}{c|c|c|c|c|c|c|c|c}
\hline
\textbf{Method}  & \multicolumn{4}{c|}{\textbf{CIFAR-10}} & \multicolumn{4}{c}{\textbf{CIFAR-100}}\\

 & $\epsilon = 1/255$ & $\epsilon = 2/255$  & $\epsilon = 3/255$ & $\epsilon = 4/255$ & $\epsilon = 1/255$ & $\epsilon = 2/255$  & $\epsilon = 3/255$ & $\epsilon = 4/255$ \\
\hline \hline

                        Original & 47.71 & 10.36 & 1.39 & 0.28 & 25.83 & 7.76 & 3.35 & 1.94\\
                        Original + Uniform-Noise & 61.23 & 22.85 & 6.34 & 2.56 & 33.15 & 13.50 & 6.01 & 3.22\\
                        \cline{1-9}
                        \textbf{SAT (Weak saliency)} & & & & \\ \cline{1-1}
                          Resnet-10 | Std. | G.CAM++ & 59.61  & 22.12  & 6.16 & 1.77 & 31.98  & 11.93 & 5.48  & 2.89             \\
                          Resnet-10 | Adv. | G.CAM++ & 57.34  & 19.94 & 5.64 & 1.91 & 32.88  & 13.3 & 6.31 & 4.0      \\
                          Resnet-34 | Std. | G.CAM++  & 56.75 & 20.62 & 6.02 & 1.74 &  30.95 & 12.68 & 5.65 & 3.25           \\
                         Resnet-34 | Adv. | G.CAM++ & 60.0 & 22.94  & 6.58 & 1.96 & 32.96 & 12.07  & 5.2 & 3.04 \\
                           \cline{1-9}
                        \textbf{SAT (Fine saliency)} & & & & \\ \cline{1-1}
                         Resnet-10 | Std. | GBP & 10.87  & 0.95  & 0.0  & 0.0 & 20.53  & 7.52  & 3.5  & 2.12              \\
                         Resnet-10 | Adv. | GBP & \textbf{63.33}  & \textbf{26.79} & \textbf{9.62} & \textbf{3.69} & 34.29  & \textbf{14.73} & 6.84 & 4.22        \\
                         Resnet-34 | Std. | GBP  & 18.01 & 2.54 & 1.25 & 1.0 & 9.91 & 2.52 & 1.05 & 0.54             \\
                         Resnet-34 | Adv. | GBP & 62.67 & 23.76  & 5.82 & 1.28 & \textbf{34.71} & 14.32  & \textbf{7.08} & \textbf{4.29} \\

\hline
& & & & & & & & \\
\hline
                        PGD & 77.89 & 73.1 & 66.96 & 61.18 & 45.75 & 40.13 & 35.41 & 31.01 \\
                        PGD + Uniform-Noise & \textbf{82.23} & 74.97 & 65.61 & 55.04 & \textbf{42.67} & 36.10& 30.64 \\
                        \cline{1-9}
                        \textbf{PGD-SAT} & & & & \\
                        \cline{1-1}
                        Resnet-10 | Std. | GBP & 79.72 & 73.81 & 67.72 & 61.29 & 46.87 & 41.11 & 35.77 & 30.75   \\
                        Resnet-10 | Adv. | GBP & 80.72 & \textbf{75.07} & 68.68 & 62.49 & 48.33 & 42.2 & 36.33 & \textbf{31.66}         \\
                        Resnet-34 | Std. | GBP & 79.53 & 74.2 & 68.19 & 62.48 & 46.66 & 40.95 & 35.73 & 30.83 \\
                        Resnet-34 | Adv. | GBP & 80.15 & 74.60 & 68.47 & 62.53  & 47.38 & 42.0 & \textbf{36.34} & 31.53     \\
                        Resnet-10 | Adv. | G.CAM++ & 79.67 & 74.05 & 68.12 & 61.84  & 47.28 & 41.72 & 35.99 & 31.05      \\
                        Resnet-34 | Adv. | G.CAM++  &79.74 & 74.5 & \textbf{68.87} & \textbf{62.68} & 46.12 & 40.47 & 35.31 & 30.81\\
\hline
& & & & & & & & \\
\hline
                          TRADES & \textbf{84.0} & 73.25 &	59.79 &	47.03 & 47.21 & 42.2 & 37.03 & 32.96 \\
                          TRADES + Uniform-Noise & 81.69 & 74.96 &67.43 & 60.05 & \textbf{51.90} & 42.85 & 37.30 & 31.71\\
                          \cline{1-9}
                          \textbf{TRADES-SAT} & & & & \\ \cline{1-1}
                        Resnet-10 | Std. | GBP & 80.15 & 75.2 & 69.0 & 63.2 & 48.63 & 42.91 & 37.79 & 33.19 \\
                          Resnet-10 | Adv. | GBP & 80.65 & 75.38 & 69.28 & \textbf{63.46}  & 48.74 & 42.99 & 37.83 & 33.23 \\
                          Resnet-34 | Std. | GBP & 79.98 & 74.5 & 68.56 & 62.43 & 48.5 & 42.41 & 36.93 & 32.01 \\
                          Resnet-34 | Adv. | GBP & 80.26 & 74.87 & 68.75 & 62.74   & 48.76 & 42.83 & 37.36 & 32.87   \\
                          Resnet-10 | Adv. | G.CAM++ & 79.85 & 74.61 & 69.07 & 63.2 & 48.99 & 43.05 & 37.4 & 32.84      \\
                         Resnet-34 | Adv. | G.CAM++  & 83.17 & \textbf{77.18} & \textbf{70.27}	& 62.87 & 49.35 & \textbf{43.62} & \textbf{38.53} & \textbf{33.93}

                           \\ \hline

\end{tabular}

}
\end{center}

\caption{Results on CIFAR-10 and CIFAR-100 using saliency maps from different teachers and explanation methods. $\epsilon \in \{1/255,2/255,3/255,4/255\}$ denotes the maximum $l_\infty$ perturbation allowed in 5-step PGD attack (More the $\epsilon$, stronger the attack). GBP:  Guided-Backpropagation; G.CAM++: Grad-CAM++.}
\label{table:cifar-10}
\end{table}

\noindent {\large \textbf{4.2 Results}}

\noindent Tables \ref{table:cifar-10} and \ref{table:bbox} present our results on CIFAR-10 + CIFAR-100 and Tiny ImageNet + Flower-17 datasets respectively. We note again that the provided bounding boxes are used as saliency maps in Tiny ImageNet, and segmentation masks provided in Flower-17 are used as saliency maps. We note that with no additional cost of computing perturbations, we obtain improvement of $~2-4\%$ in robustness. Table \ref{table:cifar-10} shows the results for CIFAR-10 and CIFAR-100, and demonstrates the potential of using saliency maps for adversarial training. Barring 1-2 cases, the proposed SAT (or its combination with an adversarial training method such as PGD or TRADES) performs better than respective baselines across the results. As the adversarial attack gets stronger (larger $\epsilon$), adding SAT to existing methods significantly improves robustness performance (~1-3\% in adversarial accuracy).
\begin{table}
\begin{center}
\scalebox{0.9}{
\footnotesize
\begin{tabular}{c|c|c|c|c|c|c}
\hline
\textbf{Method}  & \multicolumn{3}{c|}{\textbf{Tiny-Imagenet}} & \multicolumn{3}{c}{\textbf{FLOWER-17}} \\
 & $\epsilon=1/255$ & $\epsilon=2/255$  & $ \epsilon=3/255$  & $\epsilon=1/255$ & $\epsilon=2/255$  & $ \epsilon=3/255$ \\
\hline \hline
                        Original & 1.04 & 0.4 & 0.0  & 63.2 & 48.01 & 34.2  \\
                        Original + Uniform-Noise & 9.45 & 2.32 & 0.77 & 64.56 & 50.43 & 36.2 \\
                        SAT & \textbf{9.79}  & \textbf{2.46} & \textbf{0.77} & \textbf{66.17}  &\textbf{ 52.94} & \textbf{38.93} \\
                        \cline{1-7}
                        PGD & 18.91 & 14.34 &11.37 & 72.38 & 70.4 & 70.3 \\
                        PGD + Uniform-Noise  & 19.57 & 15.49 & 11.66 & 73.52 & 72.79 & 72.71 \\

                        PGD-SAT  & \textbf{20.56} & \textbf{16.38} & \textbf{12.91} & \textbf{78.67} & \textbf{75.73} & \textbf{75.00} \\
                        \cline{1-7}
                        TRADES & 18.45 & 16.76 &	11.09& 74.56 & 73.89 &	73.67 \\
                        TRADES + Uniform-Noise & 19.96 & 16.13 & 12.58 & 76.47 & 74.26 & 74.0 \\
                        TRADES-SAT& \textbf{20.04} & \textbf{16.45} & \textbf{12.96} & \textbf{79.41} & \textbf{77.94} & \textbf{77.20} \\

\hline
\end{tabular}}
\end{center}
\caption{Results on Tiny-Imagenet and Flower-17 datasets where bounding boxes and segmentation masks provided with the dataset are used as saliency maps, respectively. $\epsilon \in \{1/255,2/255,3/255\}$ denotes the maximum $l_\infty$ perturbation allowed in 5-step PGD attack (More the $\epsilon$, stronger the attack).}
\vspace{-25pt}
\label{table:bbox}
\end{table}
It is also evident from Table \ref{table:cifar-10} that using finer saliency maps (GBP in our results) obtains better performance than weaker saliency maps (bounding boxes obtained from GradCAM++, in our case). This supports the inference that a stronger saliency map provides better adversarial robustness. We also notice, as pointed out earlier in this section (Fig \ref{saliency}), that using saliency maps obtained from an adversarially trained teacher leads to significant gains in robustness/adversarial accuracy performance. In other words, all our experiments point to the inference that better the saliency maps, better the adversarial robustness of the model trained using our approach. This further supports our inherent claim that saliency maps do provide adversarial robustness.

Table \ref{table:cifar-10} also shows that PGD-SAT and TRADES-SAT leads to improvement ($~1.5 - 3\%$) over vanilla PGD and TRADES adversarial training. This shows that adding our saliency-based method to existing adversarial training method further improves their performance. We observe here again that using saliency maps of an adversarially robust teacher model proves more useful than those obtained from a normal teacher model.

\begin{wrapfigure}[14]{r}{0.4\textwidth}
\includegraphics[width=0.4\textwidth]{./images/time3}
\caption{Training time for one epoch in seconds (averaged over 10 trials) for different methods considered in our results on Tiny-Imagenet and Flower-17.}
\label{fig:train-time}
\end{wrapfigure}
\noindent \textbf{Time Efficiency:}
We further analyzed the training efficiency of the proposed method. Figure \ref{fig:train-time} reports the average time taken by one epoch over 10 trials. As can be seen, PGD-SAT and TRADES-SAT require only $\approx 50-70\%$ of training time when compared to vanilla PGD and TRADES respectively, and at the same time, achieves superior performance (as in Table \ref{table:bbox}). In case of vanilla SAT, the behavior is desirable since we observe an increase in robustness without compromising much in training time. We assume in these results that explanations are provided to us, inspiring the demand for finer saliency map annotations to be included in vision datasets to the community. Similar results for CIFAR-10 and CIFAR-100 are included in the appendix.

\section{Discussions, Ablation Studies and Conclusions}
\label{ablations}
We carried out ablation studies to better characterize the efficacy of our methodology, and present these results in this section. Unless explicitly specified, these experiments are carried out on CIFAR-100 using GuidedBackprop and GradCAM++ explanations of a standard and adversarially trained Resnet-10 teacher.\\

\begin{wrapfigure}[11]{l}{0.5\textwidth}
\vspace{-20pt}
\includegraphics[width=0.5\textwidth]{./images/hyper}
\vspace{-20pt}
\caption{Variation of hyper-parameter $\alpha$ and $\epsilon_0$ on CIFAR-100 dataset.}
\label{fig:variation-alpha}
\end{wrapfigure}

\noindent \textbf{Varying Hyperparameters:} We studied the effect on the adversarial robustness of our trained model by varying the hyperparameters $\alpha$ and $\epsilon_0$ in our method. Fig \ref{fig:variation-alpha} shows these results, where all models are evaluated using a 5-Step PGD attack with $l_\infty$ norm and maximum perturbation $\epsilon \in \{\frac{1}{255}, \frac{2}{255}, \frac{3}{255} \}$.
We achieve higher robustness when $\alpha^{10}$ is set closer to 1 ($\alpha$ close to 1), which from Eqn \ref{pert-calc} indicates that the saliency map is useful to obtain perturbations in later stages of training, as explained before. Interestingly, when $\alpha^{10}$ is close to  0 ($\alpha$ close to 0), the training doesn't include any noise factor, and since we have a fixed saliency map for each image, it becomes equivalent to training on an additively shifted version of the original training data resulting in a less robust model.
While varying $\epsilon_0$ in Algorithm \ref{mimic-AT}, we observe a peak in robustness somewhere in the middle of the considered range of values. In Algorithm \ref{mimic-AT}, the distribution of our estimated perturbations and actual adversarial perturbations depend on the hyperparameter $\epsilon_0$. At $\epsilon_0 = 0$, the distributions are identical, and the distributions diverge as $\epsilon_0$ increases. Hence, at high values of $\epsilon_0$, when both distributions diverge, the perturbations used by SAT no longer resemble true adversarial perturbations, hence resulting in less robust models (when attacked adversarially using methods associated with the adversarial perturbations). In other words, the perturbations provided by saliency maps in our method has reasonable conjunction with standard adversarial perturbations, as shown in Fig \ref{fig:sal-attack}.

\begin{wraptable}[16]{l}{0.5\textwidth}
\vspace{-20pt}
\scalebox{0.65}{
\begin{tabular}{c|c|c|c|c}
\hline
\textbf{Method}  & \multicolumn{4}{c}{\textbf{PGD}}  \\
 & $\frac{1}{255}$ & $\frac{2}{255}$ & $\frac{3}{255}$ & $\frac{4}{255}$  \\
& & & &  \\
\hline \hline
Original  & 25.83 & 7.76 & 3.35 & 1.94 \\
Original + Uniform-Noise & 33.15 & 13.50 & 6.01 & 3.22 \\
\cline{1-5}
\textbf{SAT} & & & & \\ \cline{1-1}
Resnet-10 | Std. | GBP  & 20.53 & 7.52 & 3.5 & 2.12  \\
Resnet-10 | Std. | S.Grad  & \textbf{39.22} & \textbf{19.89} & \textbf{9.44} & \textbf{4.49}  \\
Resnet-10 | Std. | G.G.CAM++ & 21.46 & 8.00 & 3.53 & 2.23  \\
Resnet-10 | Std. | I.Grad & 36.2 & 5.43 & 7.28 & 3.37  \\

Resnet-10 | Adv. | GBP  & 34.29 & 14.73 & 6.84 & 4.22  \\
Resnet-10 | Adv. | S.Grad  & \textbf{40.01} & \textbf{21.2} & \textbf{10.96} & \textbf{4.85}  \\
Resnet-10 | Adv. | G.G.CAM++ & 34.07 & 13.18 & 5.85 & 3.09  \\
Resnet-10 | Adv. | I.Grad & 37.56 & 16.45 & 7.55 & 4.31 \\
                           \hline
\end{tabular}}
\caption{\footnotesize Results using saliency maps from Guided-Backprop (GBP), Smooth-Grad (S.Grad), Integrated-Gradients (I.Grad), and Guided-Grad-CAM++ (G.G.CAM++)}
\label{table:better-exp}
\end{wraptable}

\noindent \textbf{Robustness against Other Attacks:}
In Table \ref{table:cifar-10}, we evaluate our models using the widely used PGD attack. We now study how the proposed method works against other attacks - in particular, Uniform Noise attack, TRADES \cite{trades} attack and a Saliency-based adversarial attack, which is an attack using saliency maps generated by our method.
These results are shown in Table \ref{table:diff-pert}. It is clear that SAT and PGD-SAT outperform or have comparable performance to (in case of Uniform Noise) competing methods. Expectedly, the improvement is much more significant in case of Saliency-based adversarial attack, since the method is closely associated with the attack.

\begin{table}
\centering
\begin{center}
\scalebox{0.8}{
\footnotesize
\begin{tabular}{c|c|c|c|c|c|c|c|c|c}
\hline

\textbf{Method} &  \multicolumn{3}{c|}{\textbf{Uniform-Noise}} & \multicolumn{3}{c|}{\textbf{TRADES}} &
      \multicolumn{3}{c}{\textbf{Saliency Attack}} \\
         & $\epsilon=\frac{8}{255}$ & $\epsilon=\frac{16}{255}$ & $\epsilon=\frac{32}{255}$ & $\epsilon=\frac{2}{255}$ & $\epsilon=\frac{3}{255}$ & $\epsilon=\frac{4}{255}$  & $\epsilon=\frac{4}{255}$ & $\epsilon=\frac{8}{255}$ & $\epsilon=\frac{16}{255}$  \\
        &  &  &  & &  &  &  &  &  \\

\hline \hline
Original & 62.21 & 47.88 & 21.17 &  39.59
                            & 35.1 & 33.45  & 13.03 & 6.86 & 3.92
                            \\
Original + Uniform-Noise  & \textbf{67.42} & \textbf{60.75} & 28.9 &  \textbf{46.98}
                            & 39.37 & 37.8 & 19.63 & 10.48 & 6.1
                            \\
\hline
\textbf{SAT} &  & &  &  & & & & & \\ \cline{1-1}
Resnet-10 | Std | GBP & 56.63 & 57.4 & \textbf{46.9} & 36.45 & 33.35 & 32.46  & 51.78 & 49.02 & 42.75             \\

Resnet-10 | Adv. | GBP  & 66.63 & 57.93 & 21.92  & 45.93 & \textbf{40.80} & \textbf{37.86} &  62.99 & 56.48 & 43.49              \\

Resnet-10 | Adv. | G.CAM++  & 64.00 & 54.84 & 23.43 & 44.31 & 38.28 & 35.71 & \textbf{63.86} & \textbf{63.87} & \textbf{63.66}               \\
                           \hline \hline
PGD  & 50.96 & 51.0 & 48.69  & 49.03 & 47.63 & 45.34 & 44.03 & 34.77  & 21.04             \\
PGD + Uniform-Noise   & \textbf{55.6} & 52.45 & 49.20 & \textbf{52.32}
                            & 48.76 & 45.87  & 45.72 & 32.81 & 16.62
                            \\
\hline
\textbf{PGD-SAT} &  & &  &  & & & & &  \\ \cline{1-1}
Resnet-10 | Std | G.BP  & 52.78 & 52.56 & 49.21  & 50.43 & 47.69 & 46.24 &  47.25 & 46.74 & 44.95             \\
Resnet-10 | Adv. | G.BP   & 54.38 & \textbf{54.13} & \textbf{50.30}  & 51.58 & \textbf{49.15} & \textbf{46.83}   & 50.61 & 50   & 47.64            \\
Resnet-10 | Adv. | G.CAM++  & 53.30 & 52.76 & 49.74  & 50.87 & 48.42 & 46.31  &  \textbf{50.69} & \textbf{50.63}  & \textbf{50.54}            \\
                           \hline
\end{tabular}
}
\end{center}
\caption{Results against other attacks: Uniform-Noise, TRADES and Saliency. \textit{(GBP=  Guided-Backprop; G.CAM++=Grad-CAM++; $l_\infty $ perturbation $\in [-\epsilon, \epsilon]$.)}}
\vspace{-25pt}
\label{table:diff-pert}
\end{table}

\noindent \textbf{Using Other Saliency Maps:}
We also performed a study where we trained our models using stronger saliency maps obtained using methods such as SmoothGrad \cite{smooth-grad}, Guided Grad-CAM++ \cite{grad-cam++} and Integrated Gradients \cite{integrated-gradients} from teacher models. These models are evaluated using a 5-Step PGD attack with $l_\infty$ max-perturbation $\epsilon \in \{\frac{1}{255}, \frac{2}{255}, \frac{3}{255}, \frac{4}{255}\}$. As can be seen from Table \ref{table:better-exp}, we tend to achieve more robust models for better saliency maps.\\

\noindent \textbf{Conclusion.} In summary, this work explores the interesting connection between saliency maps and adversarial robustness to propose a Saliency based Adversarial training (SAT) method. SAT imitates adversarial training by using saliency maps to mimic adversarial perturbations. In particular, our methodology allows the use of annotations such as bounding boxes and segmentation masks to be exploited as weak explanations to improve model's robustness with no additional computations required to compute the perturbations themselves. Our results on CIFAR-10, CIFAR-100, Tiny ImageNet and Flowers-17 corroborate our claim. We further gain improvement over popular adversarial training methods by integrating SAT with them (PGD-SAT and TRADES-SAT). Further, our work shows how using better saliency maps leads to more robust models. Our effort opens rather a new direction to enhance robustness of DNNs by exploiting saliency maps, and inspires the need for strong saliency maps to be provided with vision datasets, which helps train adversarially robust models with little overhead. Our future work includes ways to improve SAT by reasoning about the class closest to the ground truth in terms of decision boundary, and improving our estimates of adversarial perturbations.

\section*{Acknowledgement}
\vspace{-3pt}
We are grateful to the Ministry of Human Resource Development, India; Department of Science and Technology, India; as well as Honeywell India for the financial support of this project through the UAY program. We thank the anonymous reviewers for their valuable feedback that helped improve the presentation of this work.

{\bibliographystyle{splncs04}
\section{Introduction}

Please follow the steps outlined below when submitting your manuscript to
the IEEE Computer Society Press.  This style guide now has several
important modifications (for example, you are no longer warned against the
use of sticky tape to attach your artwork to the paper), so all authors
should read this new version.

\subsection{Language}

All manuscripts must be in English.

\subsection{Dual submission}

Please refer to the author guidelines on the CVPR 2020 web page for a
discussion of the policy on dual submissions.

\subsection{Paper length}
Papers, excluding the references section,
must be no longer than eight pages in length. The references section
will not be included in the page count, and there is no limit on the
length of the references section. For example, a paper of eight pages
with two pages of references would have a total length of 10 pages.
{\bf There will be no extra page charges for CVPR 2020.}

Overlength papers will simply not be reviewed.  This includes papers
where the margins and formatting are deemed to have been significantly
altered from those laid down by this style guide.  Note that this
\LaTeX\ guide already sets figure captions and references in a smaller font.
The reason such papers will not be reviewed is that there is no provision for
supervised revisions of manuscripts.  The reviewing process cannot determine
the suitability of the paper for presentation in eight pages if it is
reviewed in eleven.

\subsection{The ruler}
The \LaTeX\ style defines a printed ruler which should be present in the
version submitted for review.  The ruler is provided in order that
reviewers may comment on particular lines in the paper without
circumlocution.  If you are preparing a document using a non-\LaTeX\
document preparation system, please arrange for an equivalent ruler to
appear on the final output pages.  The presence or absence of the ruler
should not change the appearance of any other content on the page.  The
camera ready copy should not contain a ruler. (\LaTeX\ users may uncomment
the \verb'\cvprfinalcopy' command in the document preamble.)  Reviewers:
note that the ruler measurements do not align well with lines in the paper
--- this turns out to be very difficult to do well when the paper contains
many figures and equations, and, when done, looks ugly.  Just use fractional
references (e.g.\ this line is $095.5$), although in most cases one would
expect that the approximate location will be adequate.

\subsection{Mathematics}

Please number all of your sections and displayed equations.  It is
important for readers to be able to refer to any particular equation.  Just
because you didn't refer to it in the text doesn't mean some future reader
might not need to refer to it.  It is cumbersome to have to use
circumlocutions like ``the equation second from the top of page 3 column
1''.  (Note that the ruler will not be present in the final copy, so is not
an alternative to equation numbers).  All authors will benefit from reading
Mermin's description of how to write mathematics:
\url{http://www.pamitc.org/documents/mermin.pdf}.

\subsection{Blind review}

Many authors misunderstand the concept of anonymizing for blind
review.  Blind review does not mean that one must remove
citations to one's own work---in fact it is often impossible to
review a paper unless the previous citations are known and
available.

Blind review means that you do not use the words ``my'' or ``our''
when citing previous work.  That is all.  (But see below for
techreports.)

Saying ``this builds on the work of Lucy Smith [1]'' does not say
that you are Lucy Smith; it says that you are building on her
work.  If you are Smith and Jones, do not say ``as we show in
[7]'', say ``as Smith and Jones show in [7]'' and at the end of the
paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:
\begin{quote}
\begin{center}
    An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of our
   previous paper [1], and show it to be inferior to all
   previously known methods.  Why the previous paper was
   accepted without this analysis is beyond me.

   [1] Removed for blind review
\end{quote}

An example of an acceptable paper:

\begin{quote}
\begin{center}
     An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of the
   paper of Smith \etal [1], and show it to be inferior to
   all previously known methods.  Why the previous paper
   was accepted without this analysis is beyond me.

   [1] Smith, L and Jones, C. ``The frobnicatable foo
   filter, a fundamental contribution to human knowledge''.
   Nature 381(12), 1-213.
\end{quote}

If you are making a submission to another conference at the same time,
which covers similar or overlapping material, you may need to refer to that
submission in order to explain the differences, just as you would if you
had previously published related work.  In such cases, include the
anonymized parallel submission~\cite{Authors14} as additional material and
cite it as
\begin{quote}
[1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324,
Supplied as additional material {\tt fg324.pdf}.
\end{quote}

Finally, you may feel you need to tell the reader that more details can be
found elsewhere, and refer them to a technical report.  For conference
submissions, the paper must stand on its own, and not {\em require} the
reviewer to go to a techreport for further details.  Thus, you may say in
the body of the paper ``further details may be found
in~\cite{Authors14b}''.  Then submit the techreport as additional material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem which you tested using a tool which
is widely known to be restricted to a single institution.  For example,
let's say it's 1969, you have solved a key problem on the Apollo lander,
and you believe that the CVPR70 audience would like to hear about your
solution.  The work is a development of your celebrated 1968 paper entitled
``Zero-g frobnication: How being the only people in the world with access to
the Apollo lander source code makes us a wow at parties'', by Zeus \etal.

You can handle this paper like any other.  Don't write ``We show how to
improve our previous work [Anonymous, 1968].  This time we tested the
algorithm on a lunar lander [name of lander removed for blind review]''.
That would be silly, and would immediately identify the authors. Instead
write the following:
\begin{quotation}
\noindent
   We describe a system for zero-g frobnication.  This
   system is new because it handles the following cases:
   A, B.  Previous systems [Zeus et al. 1968] didn't
   handle case B properly.  Ours handles it by including
   a foo term in the bar integral.

   ...

   The proposed system was integrated with the Apollo
   lunar lander, and went all the way to the moon, don't
   you know.  It displayed the following behaviours
   which show how well we solved cases A and B: ...
\end{quotation}
As you can see, the above text follows standard scientific convention,
reads better than the first version, and does not explicitly name you as
the authors.  A reviewer might think it likely that the new paper was
written by Zeus \etal, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been
contracted to solve problem B.
\medskip

\noindent
FAQ\medskip\\
{\bf Q:} Are acknowledgements OK?\\
{\bf A:} No.  Leave them for the final copy.\medskip\\
{\bf Q:} How do I cite my results reported in open challenges?
{\bf A:} To conform with the double blind review policy, you can report results of other challenge participants together with your results in your paper. For your results, however, you should not identify yourself and should not mention your participation in the challenge. Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\

\begin{figure}[t]
\begin{center}
\fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}

\end{center}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
\label{fig:long}
\label{fig:onecol}
\end{figure}

\subsection{Miscellaneous}

\noindent
Compare the following:\\
\begin{tabular}{ll}
 \verb'$conf_a$' &  $conf_a$ \\
 \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
\end{tabular}\\
See The \TeX book, p165.

The space after \eg, meaning ``for example'', should not be a
sentence-ending space. So \eg is correct, {\em e.g.} is not.  The provided
\verb'\eg' macro takes care of this.

When citing a multi-author paper, you may save space by using ``et alia'',
shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word.)
However, use it only when there are three or more authors.  Thus, the
following is correct: ``
   Frobnication has been trendy lately.
   It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
   Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''

This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...''
because reference~\cite{Alpher03} has just two authors.  If you use the
\verb'\etal' macro provided, then you need not worry about double periods
when used at the end of a sentence as in Alpher \etal.

For this citation style, keep multiple citations in numerical (not
chronological) order, so prefer \cite{Alpher03,Alpher02,Authors14} to
\cite{Alpher02,Alpher03,Authors14}.

\begin{figure*}
\begin{center}
\fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
\end{center}
   \caption{Example of a short caption, which should be centered.}
\label{fig:short}
\end{figure*}

\section{Formatting your paper}

All text must be in a two-column format. The total allowable width of the
text area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54
cm) high. Columns are to be $3\frac14$ inches (8.25 cm) wide, with a
$\frac{5}{16}$ inch (0.8 cm) space between them. The main title (on the
first page) should begin 1.0 inch (2.54 cm) from the top edge of the
page. The second and following pages should begin 1.0 inch (2.54 cm) from
the top edge. On all pages, the bottom margin should be 1-1/8 inches (2.86
cm) from the bottom edge of the page for $8.5 \times 11$-inch paper; for A4
paper, approximately 1-5/8 inches (4.13 cm) from the bottom edge of the
page.

\subsection{Margins and page numbering}

All printed material, including text, illustrations, and charts, must be kept
within a print area 6-7/8 inches (17.5 cm) wide by 8-7/8 inches (22.54 cm)
high.
Page numbers should be in footer with page numbers, centered and .75
inches from the bottom of the page and make it start at the correct page
number rather than the 4321 in the example.  To do this fine the line (around
line 23)
\begin{verbatim}
\setcounter{page}{4321}
\end{verbatim}
where the number 4321 is your assigned starting page.

Make sure the first page is numbered by commenting out the first page being
empty on line 46
\begin{verbatim}
\end{verbatim}

\subsection{Type-style and fonts}

Wherever Times is specified, Times Roman may also be used. If neither is
available on your word processor, please use the font closest in
appearance to Times to which you have access.

MAIN TITLE. Center the title 1-3/8 inches (3.49 cm) from the top edge of
the first page. The title should be in Times 14-point, boldface type.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and
adverbs; do not capitalize articles, coordinate conjunctions, or
prepositions (unless the title begins with such a word). Leave two blank
lines after the title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
and printed in Times 12-point, non-boldface type. This information is to
be followed by two blank lines.

The ABSTRACT and MAIN TEXT are to be in a two-column format.

MAIN TEXT. Type main text in 10-point Times, single-spaced. Do NOT use
double-spacing. All paragraphs should be indented 1 pica (approx. 1/6
inch or 0.422 cm). Make sure your text is fully justified---that is,
flush left and flush right. Please do not place any additional blank
lines between paragraphs.

Figure and table captions should be 9-point Roman type as in
Figures~\ref{fig:onecol} and~\ref{fig:short}.  Short captions should be centred.

\noindent Callouts should be 9-point Helvetica, non-boldface type.
Initially capitalize only the first word of section titles and first-,
second-, and third-order headings.

FIRST-ORDER HEADINGS. (For example, {\large \bf 1. Introduction})
should be Times 12-point boldface, initially capitalized, flush left,
with one blank line before, and one blank line after.

SECOND-ORDER HEADINGS. (For example, { \bf 1.1. Database elements})
should be Times 11-point boldface, initially capitalized, flush left,
with one blank line before, and one after. If you require a third-order
heading (we discourage it), use 10-point Times, boldface, initially
capitalized, flush left, preceded by one blank line, followed by a period
and your text on the same line.

\subsection{Footnotes}

Please use footnotes\footnote {This is what a footnote looks like.  It
often distracts the reader from the main flow of the argument.} sparingly.
Indeed, try to avoid footnotes altogether and include necessary peripheral
observations in
the text (within parentheses, if you prefer, as in this sentence).  If you
wish to use a footnote, place it at the bottom of the column on the page on
which it is referenced. Use Times 8-point type, single-spaced.

\subsection{References}

List and number all bibliographical references in 9-point Times,
single-spaced, at the end of your paper. When referenced in the text,
enclose the citation number in square brackets, for
example~\cite{Authors14}.  Where appropriate, include the name(s) of
editors of referenced books.

\begin{table}
\begin{center}
\begin{tabular}{|l|c|}
\hline
Method & Frobnability \\
\hline\hline
Theirs & Frumpy \\
Yours & Frobbly \\
Ours & Makes one's heart Frob\\
\hline
\end{tabular}
\end{center}
\caption{Results.   Ours is better.}
\end{table}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.  Please ensure that any point you wish to
make is resolvable in a printed copy of the paper.  Resize fonts in figures
to match the font in the body text, and choose line widths which render
effectively in print.  Many readers (and reviewers), even of an electronic
copy, will choose to print your paper in order to read it.  You cannot
insist that they do otherwise, and therefore must not assume that they can
zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use
\verb+\includegraphics+, and to specify the  figure width as a multiple of
the line width as in the example below
{\small\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.eps}
\end{verbatim}
}

\subsection{Color}

Please refer to the author guidelines on the CVPR 2020 web page for a discussion
of the use of color in your document.

\section{Final copy}

You must include your signed IEEE copyright release form when you submit
your finished paper. We MUST have this form before your paper can be
published in the proceedings.

Please direct any questions to the production editor in charge of these
proceedings at the IEEE Computer Society Press:
\url{https://www.computer.org/about/contact}.

{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}
\label{intro}
\vspace{-5pt}
Deep Neural Networks (DNNs) have become vital to solve many tasks across domains including image/text/graph classification and generation, object recognition, segmentation, speech recognition, etc. As the applications of DNNs widen in scope, \textit{robustness} and \textit{interpretability} are two important parameters that define the goodness of a trained DNN model. While on one hand the deep network should be robust to imperceptible perturbations, on the other hand it should be interpretable enough to be trusted when practically used in domains like autonomous navigation or healthcare. Keeping in mind the vulnerability of deep networks to adversarial attacks \cite{szegedy2013intriguing}, efforts have been undertaken to make them more robust to these attacks. Among the proposed methodologies, Adversarial Training (AT) \cite{at@madry, at@goodfellow} has emerged as one of the best defenses wherein networks are trained on adversarial examples to better classify them at test time. On the other hand, in order to generate an interpretable explanation to a network prediction, many methods have been proposed lately, of which guided-backpropagation (GBP)\cite{gbp}, GradCAM++ \cite{grad-cam++}, Integrated Gradients (IG) \cite{integrated-gradients} and fine-grained visualisations (FGVis) \cite{fgvis} are popular to name a few. \par

Most work so far on interpretability and adversarial robustness have focused on either of them alone. Very recently, over the last few months, there has been a new interest in coupling the two notions and aiming to understand the connection between robustness and interpretability \cite{connection@etmann,imagenet-texture,robustness-odds,interpret-atcnn,attribute-regularization,jacobianAT}. The few existing efforts can be categorized broadly into three kinds: (i) Efforts \cite{interpret-atcnn, robustness-odds} which have shown that explanations generated by more robust models are more interpretable than their non-robust counterparts; (ii) The second category of methods \cite{connection@etmann} have attempted to quantify the relationship between adversarial robustness and interpretability, by formulating inequalities between these notions which hold to equality in case of linear models; (iii) The last class of work \cite{attribute-regularization,jacobianAT}, more recently, have attempted to improve robustness by training models with additional objectives that constrain the image explanations to be more interpretable and robust towards attribution attacks. All of these efforts are recent, and more needs to be done to understand these relationship better. In this work, we explore an interesting connection between interpretabilty and robustness, by understanding the connection between saliency maps and adversarial perturbations. We observe that adversarial perturbations correspond to class-discriminative pixels in later stages of training (see Figure \ref{fig:pert-variation}), and exploit this observation to use the saliency map of a given image, whilst training to improve robustness. We further believe that for datasets where bounding boxes and segmentation masks are provided, one can exploit these, in lieu of saliency maps, to improve model robustness.\par

Speaking of natural learning, humans tend to learn new tasks in a more robust fashion when provided with the explanations during their learning phase. For example, a person assigned to learn a new task on his/her own may learn irrelevant relationships in the task without knowledge of the underlying explanations. The explanations guide the learning and help the person generalize the task skill to newer settings. Similarly, it would only be natural that a DNN model that is trained with explanations cannot be easily fooled by adversarial perturbations; providing one class label for the original image, and a different one for the same image with adversarial perturbations may not be easy when explanations have to be consistent in both cases. Surprisingly, little has been explored in this direction, and we aim to leverage this connection to provide an efficient methodology for adversarial robustness. In particular, we demonstrate a tangible relationship between saliency maps and adversarial perturbations, and leverage this observation - and hence saliency maps - to provide an easy-to-implement, efficient methodology that enhances model robustness when some kind of saliency maps (bounding boxes/ segmentation masks/ saliency maps) are available.
\par

Our key contributions can be summarized as follows:
\vspace{-5pt}
\begin{itemize}
\setlength\itemsep{-0.5em}
    \item We observe a tangible relationship between a saliency map and adversarial perturbations for a given image, and leverage this observation to propose a new, efficient methodology that uses the saliency map of the image to approximate adversarial perturbations and thus mimic adversarial training. We observe a significant improvement in model robustness by using visual explanation methods to obtain saliency maps, and then using these for `pseudo-adversarial training'.

    \item We also show that the improvement becomes more pronounced when a finer and more interpretable explanation (saliency map) is used instead, signifying a strong correlation between their interpretability and robustness of a given model.

    \item Additionally, we demonstrate that using an ensemble of saliency maps with adversarial perturbations leads to improvement over popular adversarial training strategies : PGD \cite{at@madry} and TRADES \cite{trades}, with just half the training time as the vanilla method.

    \item We perform detailed ablation studies to better characterize the efficacy of our proposed methodology, and discuss the scope for further research in this area, thus opening up a rather new direction to enhance robustness of DNN models.

\end{itemize}

\vspace{-7pt}
\section{Related Work}
\label{related-work}
\vspace{-5pt}
We review earlier efforts related to this work from multiple perspectives, as described below.

\vspace{-9pt}
\noindent \paragraph{Explanation methods:} Various methods have been proposed focusing on generating explanations to the decision of a neural network. \textbf{Backpropagation-based methods} find the importance of each pixel by backpropagating the class score error to the input image. An improved and popular version of this, known as Guided-Backpropagation \cite{gbp}, only keeps paths that lead to positive influence on the class score, leading to much cleaner-looking explanations. SmoothGrad \cite{smooth-grad} and VarGrad \cite{var-grad} refine the explanations by combining gradients of multiple noisy versions of the image. Other backpropagation based methods like DeepLift \cite{deep-lift}, Excitation BackProp \cite{excitation-bp} and Layerwise Relevance Propagation \cite{lrp} generate explanations by utilizing topdown relevancy propagation rules. CAM \cite{cam}, Grad-CAM \cite{grad-cam}, Grad-CAM++ \cite{grad-cam++} form another variant of generating explanations known as \textbf{Activation-based methods}. These methods use linear combinations of activations of convolutional layers, with the weights for these combinations obtained using gradients.

\vspace{-9pt}
\noindent \paragraph{Adversarial Attacks and Robustness:} With the advancement of newer adversarial attacks each year\cite{at@goodfellow, at@madry,adv-exmp-survey,spatial-adv-exmp,trades}, methods have been proposed to defend against them. Parseval Networks \cite{parseval-nets} train robust networks by constraining the Lipschitz constant of its layers to be smaller than 1. Another category of methods harness the susceptibility of latent layers by performing latent adversarial training (LAT) \cite{LAT}  or using feature denoising \cite{feature-denoising}. Other methods like DefenseGAN \cite{defense-gan} exploit GANs wherein they learn the distribution of unperturbed images and find the closest image for a given test image, to feed the network at inference time. TRADES \cite{trades} presents a new defense method that provides trade-off between adversarial robustness and accuracy by decomposing the prediction error for adversarial examples (robust error) as the sum of the natural (classification) error and boundary error, and providing a differentiable upper bound using the theory of classification-calibrated loss. Among the proposed defenses against adversarial attacks, Adversarial Training (AT) \cite{at@madry,at@goodfellow} has remained the most popular and widely used defense, where the network is trained on adversarial examples in order to match the training data distribution with that of adversarial test distribution. More recent efforts in this direction include \cite{free-AT,YOPO}, which aim to reduce adversarial training overhead by recycling gradients and accelerating via the maximal principle respectively.

\vspace{-9pt}
\noindent \paragraph{Explaining AT CNNs:} The last few months have seen a few efforts on associating the notions of robustness and interpretability. These efforts can be categorized into three kinds: (i) The first kind centers around interpreting how adversarially trained convolutional neural networks (ATCNNs) recognize objects. Recent studies \cite{interpret-atcnn, robustness-odds} have shown that representations learned by ATCNNs are more biased towards image shape than its texture. Also, these ATCNNs tend to evince more interpretable saliency maps corresponding to their prediction than their non-robust equivalents.
(ii) In the second kind, inspired by \cite{interpret-atcnn, robustness-odds}, Etmann et al. \cite{connection@etmann} recently quantified this behaviour of ATCNNs by considering the alignment between saliency map and the image as the metric for interpretability. They confirmed that for a linear model the alignment grows strictly with robustness. For non-linear models such as neural networks, they show that the their linearized robustness is loosely bounded by the alignment metric.
(iii) Encouraged by the work of Etmann \cite{connection@etmann}, a third category of work more recently sought to answer the question: Do robust and interpretable saliency maps imply adversarial robustness ? Recent efforts \cite{explanations-manipulated,fragile-interpretation} have shown that the explanations of neural networks can also be manipulated by adding imperceptible perturbations to input examples which instead of causing mis-classification, leads to a completely different explanation. To tackle this problem, recent works like Robust Attribute Regularization (RAR) \cite{attribute-regularization} aim to train networks by optimizing objectives with a regularization term on explanations in order to achieve robust attributions.

In contrast to all the abovementioned methods, our work explores a different side of the association of notions of interpretability and robustness. As explained in Section \ref{intro}, we aim to generate adversarial perturbations from a given saliency map (assumed to be available in prior) to improve robustness while training a neural network. Our work is perhaps closest to RAR \cite{attribute-regularization}, we however differ from them in our primary objective and in that we are using explanations to generate adversarial examples instead of optimizing them or using them as a regularization term. We further hypothesize that, by advancement through our method, we can actually take advantage of weak explanations like bounding boxes and segmentation masks which may otherwise be left unexploited in a dataset, to train more robust models.

\begin{figure*}[h]
\centering
\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{latex/images/Pert2.png}
\caption{Variation of adversarial perturbation with training epochs during 5-step PGD adversarial training of Resnet-34 on CIFAR-10. We observe that adversarial perturbations at later stages of training correspond to class-discriminative regions/pixels.}
\label{fig:pert-variation}
\end{figure*}

\vspace{-5pt}
\section{Using Saliency Maps for Efficient Adversarial Training}
\label{methodology}
\vspace{-5pt}
\paragraph{Notations and Preliminaries.}
We denote a neural network as $\Phi(. \ ;  \ \theta) : \mathbb{R}^d \rightarrow \mathbb{R}^k $, parametrized by weights $\theta$, which takes an input $\textbf{x} \in \mathbb{R}^d $ and outputs a logit, $\Phi^i(\textbf{x})$, for each of $k$ classes, i.e. $i \in \{1,\cdots,k\}$.
Without any loss of generality, we define a saliency map $\textbf{s}$ corresponding to an input sample $\textbf{x}$ as $\textbf{s} \in [0,1]^d$, where the presence of an object of interest in input $\textbf{x}$ lies between 0 and 1. For a trained network $\Phi$, the unnormalized saliency map for an input $\textbf{x}$ can simply be given as:
$\nabla_{\textbf{x}} \Phi^{i^*}(\textbf{x})$, where $i^* = \argmax_i \Phi^i(\textbf{x})$.\\

\noindent \textit{Projected Gradient Descent (PGD) Attack:} Prior efforts on adversarial attacks \cite{at@goodfellow} have proposed Fast Gradient Sign Method (FGSM), a $l_{\infty}$-bounded single step attack which calculates an adversary as:
\begin{equation}
\label{eqn_fgsm}
    \textbf{x} + \epsilon \hspace{1mm} \text{sign}(\nabla_\textbf{x} \mathcal{L} \ (\Phi(\textbf{x},\theta),y))
\end{equation}
\noindent A more powerful attack is the multi-step variant, also called Projected Gradient Descent (PGD), where:
\begin{align}
\label{eqn_pgd}
    \textbf{x}^0 & = \textbf{x}\\
    \textbf{x}^{t+1} & = \Pi_{\textbf{x}+ \textit{N}} \big( \textbf{x}^{t} + \alpha \hspace{1mm} \text{sign}(\nabla_{\textbf{x}} \mathcal{L} \ (\Phi(\textbf{x},\theta),y)) \big)
\end{align}
\noindent where $\alpha$ is the step size, $\Pi$ is the projection function, and $N$ is the space of possible perturbations.\\

\noindent \textit{Adversarial Training (AT):} AT \cite{at@goodfellow} is generally used to make the models robust by matching the training distribution with the adversarial test distribution. Essentially, for AT, the optimal parameter $\theta^*$ is given by:
\begin{equation}
    \theta^* = \argmin_\theta \mathbb{E}_{(\textbf{x},y) \sim D} \ \left[ \ \max_{\delta \in N}\mathcal{L} \ (\Phi(\textbf{x} + \delta,\theta),y) \ \right]
\end{equation}
Here, the inner maximization $\max_{\delta \in N} \mathcal{L} \ (\Phi(\textbf{x} + \delta,\theta),y)$ is calculated using a strong adversarial attack such as PGD.

\noindent \paragraph{Our Methodology: Saliency-Based Adversarial Training (SAT).}
We observe that when training a network using adversarial training, during the initial phase of training when the weights are not optimal, the perturbations computed by the attack methods are random. But with training, as weights become optimal, they become more class-discriminative. Figure \ref{fig:pert-variation} illustrates this observation. We exploit this observation to propose our methodology.

Etmann et al. \cite{connection@etmann} recently defined the \textit{linearized robustness} $\rho(\textbf{x})$ of a network $\Phi$ at image $\textbf{x}$ as:
\[
    \label{linearized-robustness}
    \rho(\textbf{x}) = \min_{j \neq i}\frac{\Phi^{i^*}(\textbf{x}) - \Phi^{j}(\textbf{x})}{\Vert \nabla_\textbf{x}(\Phi^{i^*}(\textbf{x}) - \Phi^{j}(\textbf{x})) \Vert}
\]
We note from the above expression that the linearized adversarial robustness, $\rho$, is inversely related to $\nabla_\textbf{x}(\Phi^{j}(\textbf{x}) - \Phi^{i^*}(\textbf{x}))$, which hence is the direction of the adversarial perturbation (more the perturbation, lesser the robustness. (One can elicit this as the direction of adversarial perturbation also from the proof of the above result in \cite{connection@etmann}, cf. proof of Lemma 1 in Appendix of \cite{connection@etmann}. \cite{connection@etmann} shows that this holds for most neural networks, especially ones with ReLU or Leaky ReLU activation functions, which we consider in this work. We describe this further in our Appendix.) We leverage this observation to hence denote the direction of adversarial perturbation by:
\begin{equation}
\label{eqn_pert_direction}
\nabla_\textbf{x}(\Phi^{j^*}(\textbf{x}) - \Phi^{i^*}(\textbf{x}))
\end{equation}
\noindent where $i^* = \argmax_i \Phi^i(\textbf{x})$ and $j^*$ minimizes Eqn \ref{linearized-robustness}. This perturbation depends on two quantities: (i) $\nabla_\textbf{x} \Phi^{i^*}(\textbf{x})$ which is  the saliency map for the true class $i^*$; and (ii) $\nabla_\textbf{x} \Phi^{j^*}(\textbf{x})$, the saliency map of $\textbf{x}$ for class $j^*$ which is closest to the actual class $i^*$ in terms of decision boundary. \par

Now, let us consider a binary classifier $h: \textbf{x} \rightarrow \{-1,1\}$ given by:
\begin{equation}
\label{eqn_binary_classifier}
h = \text{sign}(\Phi(\textbf{x},\theta))
\end{equation}
\noindent where $\Phi(\textbf{x},\theta))$ represents the logit of the positive class. We apply a sigmoid activation function, $\sigma$ to derive the probability of the positive class as:
\[P(y=+1|\textbf{x}) = \sigma(\Phi(\textbf{x},\theta)) = \frac{1}{1 + \exp^{-\Phi(\textbf{x},\theta)}}
\]
It is simple to see then that the probability of the negative class is:
\[P(y=-1|\textbf{x}) = 1 - P(y=+1|\textbf{x}) = \frac{1}{1 + \exp^{\Phi(\textbf{x},\theta)}}
\]
Rather, the corresponding logit score of the negative class is $-\Phi(\textbf{x},\theta))$. In the context of Eqn \ref{eqn_pert_direction} for a binary classifier, one can hence view $\Phi^{j^*}(\textbf{x}) =  - \Phi^{i^*}(\textbf{x})$, and define the adversarial perturbation direction in Eqn \ref{eqn_pert_direction} simply as $- \nabla_\textbf{x}(\Phi^{i^*}(\textbf{x}))$. \par

We can extend a similar argument to the multi-class setting by making an approximation. A multi-class classifier can be reasoned as a binary-classifier in one-vs-all setting and in $-\nabla_{\textbf{x}} \Phi^{i^*}(\textbf{x})$ positive values corresponds to presence of all classes except $i^*$ and negative to presence of true class $i^*$. Since its rather difficult to infer about class $j^*$ from just saliency maps, we decide to substitute the term $\nabla_{\textbf{x}} \Phi^{j^*}(\textbf{x})$ with $-\nabla_{\textbf{x}} \Phi^{i^*}(\textbf{x})$ which is an approximation to actual quantity but works well enough as we show in the experiments.

It is not hard to see that the direction of perturbation in Eqn \ref{eqn_pert_direction} can be approximated as that of a binary classifier above, which is nothing but the direction that causes maximum decrease in confidence of actual class $i^*$ (the logit $\Phi^{i^*}$). Since we deal with $l_{\infty}$-bounded perturbations in this work, following PGD (Eqn \ref{eqn_pgd}), we use $- \text{sign} \ (\nabla_\textbf{x} \Phi^{i^*}(\textbf{x}))$ instead of $- \nabla_\textbf{x} \Phi^{i^*}(\textbf{x})$ itself as the perturbation direction. \par

We now complete the above discussion by noting that $\nabla_\textbf{x} \Phi^{i^*}(\textbf{x})$ is the saliency map, \textbf{s}, defined at the beginning of this section. It is easy to see that by using thresholds, the saliency map $\textbf{s}$ can also be converted to a binary saliency ($\in \{-1,1\}$) where -1 and 1 denote the absence and presence of object of interest in $\textbf{x}$ respectively. In other words, the direction of adversarial perturbation is actually provided by the saliency map, which is available to us either as bounding box annotations/segmentation masks in a given dataset, or simply as the gradients/saliency map obtained using a teacher model (as we show in our experiments). We now describe how we leverage this relationship to perform Saliency-based Adversarial Training (SAT).

Analogous to our observation regarding AT earlier in this section and in Figure \ref{fig:pert-variation}, we can see that the perturbation, $- \text{sign} \ (\nabla_\textbf{x} \Phi^{i^*}(\textbf{x}))$ is random in initial phases of training but eventually becomes class discriminative as the model trains. In this work, we assume that we are provided with a saliency map of the ground truth class in a given image, which can be obtained easily as mentioned earlier. In order to mimic the above behavior of the perturbation over training, we choose the direction of perturbation in a probabilistic manner. We choose the $i^{th}$ component $\delta^t$[i] of perturbation $\delta^t$ at time $t$ as:
\begin{equation}
    \label{pert-calc}
    \delta^t[i] =
    \begin{cases}
      \textbf{z}[i], & \text{with probability } \alpha^{t} \\
      -\textbf{s}[i], & \text{with probability } 1-\alpha^{t}
    \end{cases}
 \end{equation}
\noindent where $\textbf{z}$ is a random vector of 1s and -1s and 0 $< \alpha <$ 1. So, during initial epochs of training, when $\alpha^t$ is close to 1, $\delta^t$ will be dominated by random values. However, as training proceeds and $\alpha^t$ starts diminishing, $\delta^t$ will smoothly transition to $-\textbf{s}$ and will be influenced by the adversarial character of the saliency map. Algorithm \ref{mimic-AT} summarizes our methodology. \par

\begin{algorithm}[t]

\SetAlgoLined
\textbf{Input:} Training Dataset $D$, Saliency Maps $S$, Model $\Phi(. \ ; \ \theta)$, SAT hyperparameter $\alpha$, Learning rate $\eta$, Maximum $l_\infty$ perturbation $\epsilon_0$ \\
\textbf{Output:} Optimal parameter $\theta^*$\\
Initialize model parameters as $\theta = \theta^0$. \\
 \For{$t \in \{1, 2, ..., n\}$}
    {
        Sample training data of size $B$ : $\{(\textbf{x}^i, y^i)\}$ from $D$. \\
        Pick out corresponding saliency maps : $\{\textbf{s}^i\}$ from $S$. \\
        Calculate $\delta^{ti}$ for each $\textbf{x}^i$ using Equation \ref{pert-calc}. \\
        Perturb the input examples : $\textbf{x}^i = \textbf{x}^i + \epsilon_0 \cdot \delta^{ti}$. \\
        Perform clipping to keep $\textbf{x}^i$ bounded : $\textbf{x}^i$ = $clip(\textbf{x}^i)$. \\
        Update model parameters : \\ $\theta^t = \theta^{t-1} - \eta \cdot \triangledown_\theta \ \frac{1}{B} \sum_{i=1}^{B} \mathcal{L} \ (\Phi(\textbf{x}^i,\theta^{t-1}),y^i) $
    }
 \caption{Saliency-based Adversarial Training (SAT) Methodology}
 \label{mimic-AT}
\end{algorithm}

\section{Experiments and Results}
\label{results}
In this section, we present our results using our method SAT (Algorithm \ref{mimic-AT}) on multiple datasets with different variations of saliency maps. We begin with describing the datasets, evaluation criteria and implementation details.
\vspace{-10pt}
\paragraph{Datasets:} We perform experiments on three well-known standard datasets: CIFAR-10, CIFAR-100 \cite{cifar-datasets} and Tiny ImageNet \cite{tiny-img}. \textbf{CIFAR-10} is a subset of 80 million tiny images dataset and consists of 60,000 $32\times32$ color images containing one of 10 object classes, with 6000 images per class. \textbf{CIFAR-100} is just like CIFAR-10, except that it has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class. \textbf{Tiny Imagenet} has 200 classes, with each class containing 500 training images, 50 validation images, and 50 test images.
\vspace{-10pt}
\paragraph{Evaluation Criteria:} We evaluate the adversarial robustness of trained models using the popular and widely used PGD attack (described in Sec \ref{methodology}) with 5 steps and 4 different levels of $l_\infty$ perturbation $\epsilon_0 \in \{1/255, 2/255, 3/255, 4/255\}$. We also evaluate the robustness of our models using other perturbations techniques: TRADES \cite{trades} as well as uniform noise in Sec \ref{ablations}. We trained all our models for 5 trials and observed minimal variations in the values. We hence report the mean value of our experiments in the tables. We compare our model with its original and noisy variants to evaluate its usefulness in each case.
\vspace{-10pt}
\paragraph{Implementation Details:}
As mentioned in Sec \ref{methodology}, our method relies on being provided with a saliency map for the ground truth label of a given image. We use two approaches to obtain this in our experiments: for CIFAR-10 and CIFAR-100, we obtain saliency maps using teacher networks. In particular, we train two teacher networks: Resnet-10 and Resnet-34 using two different techniques: Standard (Std.) and Adversarial (Adv.) (regular PGD-based adversarial training), and use the saliency maps generated by them for the ground truth class to train our student model (which is a Resnet34 too). For Tiny ImageNet, we simply use the bounding boxes provided in the dataset themselves (and left unexploited often) as `weak' saliency maps to train the model using the proposed SAT method.

For CIFAR-10 and CIFAR-100, we obtain saliency maps on the teacher model using Grad-CAM++ \cite{grad-cam++} as well as Guided Back-propagation (GBP) \cite{gbp}. One can use the explanations of Grad-CAM++ to obtain bounding regions (as shown in \cite{grad-cam++}), which we consider as weak saliency maps. GBP can be viewed as providing finer (or stronger) saliency maps in this context. In Sec \ref{ablations}, we also perform studies where we use other saliency methods like Guided-GradCAM++, Smooth-Grad and Integrated-Gradients to study the generalizability of our results.

It is believed that the quality of saliency maps generated by a adversarially trained model is better than its non-robust equivalent \cite{robustness-odds}. We hence have teacher networks trained normally (denoted as Std in the results), as well as adversarially (denoted as Adv in results) to study this observation. Figure \ref{saliency} shows the saliency maps of robust and non-robust model (Resnet-10) using some of the abovementioned explanation methods, which supports the above claim.

\begin{figure}[h]
\scalebox{0.55}{

\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{latex/images/saliency3.png}}
\caption{Saliency maps of robust and non-robust Resnet-10 with various explanation methods on CIFAR-10.}
\label{saliency}
\end{figure}
The teachers/student models, mentioned above, are trained by minimizing standard cross-entropy loss using the Adam optimizer (learning rate = $1e-3$) for 100 epochs.

\vspace{-10pt}
\paragraph{Baseline Models:} We compare the proposed SAT method against the following baseline methods:

\noindent \textit{1. Uniform Noise Training:} We perturb the model with uniform noise sampled from $[\frac{-8}{255},\frac{8}{255}]$ during training. The models are trained for 100 epochs with standard cross-entropy loss, minimized using Adam optimizer (learning rate = $1e-3$). This is denoted as `Original + Uniform Noise' in the results.

\noindent \textit{2. PGD-AT \& 3.  TRADES-AT:} We use 5-step PGD/TRADES adversarial training \cite{at@madry} with max $l_\infty$ perturbation, $\epsilon = 8/255$. The models are trained using Adam optimizer (learning rate = $1e-3$) for 200 epochs. These are denoted as PGD and TRADES in the results.

\noindent \textit{4. PGD-AT with Uniform Noise \& 5.  TRADES-AT with Uniform Noise:} The above methods, randomly interspersed with uniform noise perturbations, described above.

\noindent \textit{6. SAT:} Our method in Algorithm \ref{mimic-AT} is used to train the student models with a given saliency map. We set hyperparameters $\alpha = (0.6)^{\frac{1}{10}}$ and $\epsilon_0 = \frac{8}{255}$ for all the experiments. We minimize standard cross-entropy loss using Adam optimizer (learning rate = $1e-3$) for 100 epochs.

\noindent \textit{7. PGD-SAT \& 8. TRADES-SAT:} We perturb the input randomly with either perturbations calculated by SAT or with PGD/TRADES perturbations. We set hyper-parameters $\alpha = (0.6)^{\frac{1}{10}}$ and $\epsilon_0 = \frac{8}{255}$ for all the experiments. The models are trained using Adam optimizer (learning rate = $1e-3$) for 200 epochs.

\vspace{-9pt}
\noindent \paragraph{Saliency Maps for Adversarial Attacks:}
We first confirm that the negative of saliency i.e $-\nabla_\textbf{x}\Phi^{i^*}(\textbf{x})$ can be used as a direction to perturb input. To study this, we create adversarial examples by perturbing original examples as: $\textbf{x} = \textbf{x} - \epsilon \cdot \text{sign}(\nabla_\textbf{x}\Phi^{i^*}(\textbf{x}))$.
We used Resnet-10 and Resnet-34 models (both Standard and Adversarially trained) using Guided-Backpropagation \cite{gbp} with 3 different levels of $l_\infty$ perturbations, $\epsilon \in \{2/255, 4/255, 8/255\}$. We report our findings in Table \ref{table:sal-attack}. Evidently, examples generated by perturbing an input in the direction of negative saliency have led to a reasonable decrease in network accuracy, highlighting the presence of adversarial character in them.

\begin{table}[]
\begin{center}
\scalebox{0.7}{
\begin{tabular}{|c|c|c|c|c|c|c|}
\hline
\textbf{Dataset}           & \textbf{Model}             & \textbf{Training} & \textbf{\begin{tabular}[c]{@{}c@{}}Actual\\ Accuracy\end{tabular}} & $\epsilon = \frac{2}{255}$ & $\epsilon = \frac{4}{255}$ & $\epsilon = \frac{8}{255}$ \\
\hline \hline
\multirow{6}{*}{CIFAR-10}  & \multirow{3}{*}{Resnet-10} & Std.              & 90.5                                                               & 39.2           & 18.5           & 9              \\

                           &                            & Adv.              & 79.5                                                               & 75.67          & 70.46          & 56.67          \\ \cline{2-7}
                           & \multirow{3}{*}{Resnet-34} & Std.              & 92.57                                                              & 43.57          & 30.04          & 24             \\

                           &                            & Adv.              & 83.75                                                              & 80.6           & 75.3           & 63             \\ \cline{1-7}
\multirow{6}{*}{CIFAR-100} & \multirow{3}{*}{Resnet-10} & Std.  & 65.69 &  25.49 &  11.65 & 4.71             \\

                           &                            & Adv.  & 50.94 & 47.99  & 42.05  &  31.14              \\ \cline{2-7}
                           & \multirow{3}{*}{Resnet-34} & Std.  & 70.21 & 25.61  &  13.03 & 6.86               \\

                           &                            & Adv.  &  51.35 & 49.02  & 44.03  & 34.77      \\ \cline{1-7}
\end{tabular}
}
\end{center}
\caption{Adversarial attack accuracy when perturbing input example in direction of negative saliency. In Col 3, `Std' denotes a model trained normally, and '`Adv denotes a model trained adversarially. Each row of Cols 5-7 then shows the performance of adversarial attacks with $l_\infty$ perturbation $\epsilon$ on that model.}
\vspace{-6pt}
\label{table:sal-attack}
\end{table}

\vspace{-12pt}
\noindent \paragraph{Robustness Results of SAT of CIFAR-10 and CIFAR-100:} As described earlier, we train our student model using SAT (Algorithm \ref{mimic-AT}) on Grad-CAM++ explanations provided by teacher models described above. The results, shown in Tables \ref{table:cifar-10} and \ref{table:cifar-100} for CIFAR-10 and CIFAR-100 respectively, highlight the usefulness of SAT for adversarial training. As the adversarial attack gets stronger (larger $\epsilon$), the results show that adding SAT to existing methods significantly improves robustness performance.

\vspace{-12pt}
\noindent \paragraph{Robustness using Finer Saliency Maps:} Tables \ref{table:cifar-10} and \ref{table:cifar-100} also show the results of our method when a finer saliency map, Guided-Backprop \cite{gbp}, is used as the explanation. It is clear from the results that this improves the best possible performance, supporting the claim that a better saliency map provides better adversarial robustness. We also notice, however, that unlike using the saliencies of an adversarially trained model which leads to improvement in robustness, using saliencies of a standard model causes a decline. This observation can be explained by the fact mentioned in Sec \ref{results} that saliency maps of an adversarially trained model are more interpretable than that of a non-robust one. We thus believe that a saliency map which is more interpretable can further improve adversarial robustness of a model.

\begin{table*}[]
\begin{minipage}{.5\linewidth}
\begin{center}
\scalebox{0.8}{
\begin{tabular}{|c|c|c|c|c|}
\hline

\textbf{Method}  & $\epsilon = \frac{1}{255}$ & $\epsilon = \frac{2}{255}$  & $\epsilon = \frac{3}{255}$ & $\epsilon = \frac{4}{255}$ \\
& & & &  \\
\hline \hline
                        Original & 47.71 & 10.36 & 1.39 & 0.28 \\
                        Original + Uniform-Noise & 61.23 & 22.85 & 6.34 & 2.56 \\
                        \cline{1-5}
                        \textbf{SAT (Weak saliency)} & & & & \\ \cline{1-1}
                          Resnet-10 | Std. | G.CAM++ & 59.61  & 22.12  & 6.16 & 1.77             \\

                          Resnet-10 | Adv. | G.CAM++ & 57.34  & 19.94 & 5.64 & 1.91          \\
                          Resnet-34 | Std. | G.CAM++  & 56.75 & 20.62 & 6.02 & 1.74            \\

                         Resnet-34 | Adv. | G.CAM++ & 60.0 & 22.94  & 6.58 & 1.96  \\
                           \cline{1-5}
                        \textbf{SAT (Fine saliency)} & & & & \\ \cline{1-1}

                         Resnet-10 | Std. | GBP & 10.87  & 0.95  & 0.0  & 0.0              \\

                         Resnet-10 | Adv. | GBP & \textbf{63.33}  & \textbf{26.79} & \textbf{9.62} & \textbf{3.69}          \\
                         Resnet-34 | Std. | GBP  & 18.01 & 2.54 & 1.25 & 1.0             \\

                         Resnet-34 | Adv. | GBP & 62.67 & 23.76  & 5.82 & 1.28 \\

\Xhline{3\arrayrulewidth}
                        PGD & 77.89 & 73.1 & 66.96 & 61.18  \\
                        PGD + Uniform-Noise & \textbf{82.23} & 74.97 & 65.61 & 55.04 \\
                        \cline{1-5}
                        \textbf{PGD-SAT} & & & & \\
                        \cline{1-1}
                        Resnet-10 | Std. | GBP & 79.72 & 73.81 & 67.72 & 61.29   \\
                        Resnet-10 | Adv. | GBP & 80.72 & \textbf{75.07} & 68.68 & 62.49          \\
                        Resnet-34 | Std. | GBP & 79.53 & 74.2 & 68.19 & 62.48  \\
                        Resnet-34 | Adv. | GBP & 80.15 & 74.60 & 68.47 & 62.53      \\

                        Resnet-10 | Adv. | G.CAM++ & 79.67 & 74.05 & 68.12 & 61.84        \\
                        Resnet-34 | Adv. | G.CAM++  &79.74 & 74.5 & \textbf{68.87} & \textbf{62.68}

                           \\ \Xhline{3\arrayrulewidth}
                          TRADES & \textbf{84.0} & 73.25 &	59.79 &	47.03  \\
                          TRADES + Uniform-Noise & 81.69 & 74.96 &67.43 & 60.05 \\
                          \cline{1-5}
                          \textbf{TRADES-SAT} & & & & \\ \cline{1-1}
                        Resnet-10 | Std. | GBP & 80.15 & 75.2 & 69.0 & 63.2  \\
                          Resnet-10 | Adv. | GBP & 80.65 & 75.38 & 69.28 & \textbf{63.46}  \\
                          Resnet-34 | Std. | GBP & 79.98 & 74.5 & 68.56 & 62.43  \\
                          Resnet-34 | Adv. | GBP & 80.26 & 74.87 & 68.75 & 62.74     \\

                          Resnet-10 | Adv. | G.CAM++ & 79.85 & 74.61 & 69.07 & 63.2       \\
                         Resnet-34 | Adv. | G.CAM++  & 83.17 & \textbf{77.18} & \textbf{70.27}	& 62.87

                           \\ \hline
\end{tabular}
}

\end{center}
\caption{Results of improved robustness on CIFAR-10 dataset. GBP:  Guided-Backpropagation; G.CAM++: Grad-CAM++.}
\vspace{-10pt}
\label{table:cifar-10}
\end{minipage}
\hspace{0.8cm}
\begin{minipage}{.45\textwidth}
\begin{minipage}{.3\textheight}

\begin{center}
\scalebox{0.7}{
\begin{tabular}{|c|c|c|c|c|}
\hline
\textbf{Method}  & \multicolumn{4}{c|}{\textbf{PGD}}  \\
 & $ \epsilon =\frac{1}{255}$ & $\epsilon =\frac{2}{255}$ & $\epsilon =\frac{3}{255}$ & $\epsilon =\frac{4}{255}$  \\
& & & &  \\
\hline \hline
Original  & 25.83 & 7.76 & 3.35 & 1.94 \\
Original + Uniform-Noise & 33.15 & 13.50 & 6.01 & 3.22 \\
\cline{1-5}
\textbf{SAT} & & & & \\ \cline{1-1}
Resnet-10 | Std. | GBP  & 20.53 & 7.52 & 3.5 & 2.12  \\
Resnet-10 | Std. | S.Grad  & \textbf{39.22} & \textbf{19.89} & \textbf{9.44} & \textbf{4.49}  \\
Resnet-10 | Std. | G.G.CAM++ & 21.46 & 8.00 & 3.53 & 2.23  \\
Resnet-10 | Std. | I.Grad & 36.2 & 5.43 & 7.28 & 3.37  \\

Resnet-10 | Adv. | GBP  & 34.29 & 14.73 & 6.84 & 4.22  \\
Resnet-10 | Adv. | S.Grad  & \textbf{40.01} & \textbf{21.2} & \textbf{10.96} & \textbf{4.85}  \\
Resnet-10 | Adv. | G.G.CAM++ & 34.07 & 13.18 & 5.85 & 3.09  \\
Resnet-10 | Adv. | I.Grad & 37.56 & 16.45 & 7.55 & 4.31 \\
                           \hline

\end{tabular}
}
\end{center}
\caption{Improved robustness using Guided-Backpropagation (GBP), Smooth-Grad (S.Grad), Integrated-Gradients (I.Grad), and Guided-Grad-CAM++ (G.G.CAM++) saliencies.}

\label{table:better-exp}
\end{minipage}
\begin{minipage}{.3\textheight}
\scalebox{0.35}{
\includegraphics[width=22cm,height=12.5cm,keepaspectratio]{latex/images/hyper.png}
}
\caption{Variation of hyper-parameter $\alpha$ and $\epsilon_0$ on CIFAR-100 dataset.}
\vspace{-10pt}
\label{fig:variation-alpha}
\end{minipage}
\end{minipage}

\end{table*}

\begin{table*}[]
\begin{minipage}{.5\textwidth}

\begin{center}
\scalebox{0.8}{
\begin{tabular}{|c|c|c|c|c|}
\hline

\textbf{Method}  & $\epsilon = \frac{1}{255}$ & $\epsilon = \frac{2}{255}$  & $\epsilon = \frac{3}{255}$ & $\epsilon = \frac{4}{255}$ \\
 & & & & \\
\hline \hline
                        Original & 25.83 & 7.76 & 3.35 & 1.94 \\
                        Original + Uniform-Noise & 33.15 & 13.50 & 6.01 & 3.22 \\
                        \cline{1-5}
                        \textbf{SAT (Weak saliency)} & & & & \\ \cline{1-1}
                        Resnet-10 | Std. | G.CAM++ & 31.98  & 11.93 & 5.48  & 2.89              \\

                        Resnet-10 | Adv. | G.CAM++ & 32.88  & 13.3 & 6.31 & 4.0          \\
                        Resnet-34 | Std. | G.CAM++  &  30.95 & 12.68 & 5.65 & 3.25            \\

                         Resnet-10 | Adv. | G.CAM++ & 32.96 & 12.07  & 5.2 & 3.04                      \\
                           \cline{1-5}
                         \textbf{SAT (Fine saliency)} & & & & \\ \cline{1-1}

                        Resnet-10 | Std. | GBP & 20.53  & 7.52  & 3.5  & 2.12                \\

                        Resnet-10 | Adv. | GBP & 34.29  & \textbf{14.73} & 6.84 & 4.22          \\
                        Resnet-34 | Std. | GBP  & 9.91 & 2.52 & 1.05 & 0.54         \\

                        Resnet-34 | Adv. | GBP & \textbf{34.71} & 14.32  & \textbf{7.08} & \textbf{4.29}             \\

\Xhline{3\arrayrulewidth}
                        PGD & 45.75 & 40.13 & 35.41 & 31.01  \\
                        PGD + Uniform-Noise & \textbf{50.0} & \textbf{42.67} & 36.10& 30.64 \\
                        \cline{1-5}
                        \textbf{PGD-SAT} & & & & \\ \cline{1-1}
                        Resnet-10 | Std. | GBP & 46.87 & 41.11 & 35.77 & 30.75  \\
                        Resnet-10 | Adv. | GBP & 48.33 & 42.2 & 36.33 & \textbf{31.66}            \\
                        Resnet-34 | Std. | GBP & 46.66 & 40.95 & 35.73 & 30.83  \\
                        Resnet-34 | Adv. | GBP & 47.38 & 42.0 & \textbf{36.34} & 31.53      \\

                        Resnet-10 | Adv. | G.CAM++ & 47.28 & 41.72 & 35.99 & 31.05        \\
                        Resnet-34 | Adv. | G.CAM++  & 46.12 & 40.47 & 35.31 & 30.81

                           \\ \Xhline{3\arrayrulewidth}
                          TRADES & 47.21 & 42.2 & 37.03 & 32.96  \\
                          TRADES + Uniform-Noise & \textbf{51.90} & 42.85 & 37.30 & 31.71 \\
                          \cline{1-5}
                           \textbf{TRADES-SAT } & & & & \\ \cline{1-1}
                          Resnet-10 | Std. | GBP & 48.63 & 42.91 & 37.79 & 33.19  \\
                          Resnet-10 | Adv. | GBP & 48.74 & 42.99 & 37.83 & 33.23            \\
                          Resnet-34 | Std. | GBP & 48.5 & 42.41 & 36.93 & 32.01  \\
                          Resnet-34 | Adv. | GBP & 48.76 & 42.83 & 37.36 & 32.87     \\

                          Resnet-10 | Adv. | G.CAM++ & 48.99 & 43.05 & 37.4 & 32.84       \\
                          Resnet-34 | Adv. | G.CAM++  & 49.35 & \textbf{43.62} & \textbf{38.53} & \textbf{33.93}

                           \\ \hline
\end{tabular}
}
\end{center}
\caption{Results of improved robustness on CIFAR-100 dataset. GBP:  Guided-Backpropagation; G.CAM++: Grad-CAM++.}
\vspace{-10pt}
\label{table:cifar-100}
\end{minipage}
\hspace{1cm}
\begin{minipage}{.4\textwidth}
\begin{minipage}{0.3\textheight}

\begin{center}
\scalebox{0.87}{
\begin{tabular}{|c|c|c|}
\hline
\textbf{Method} & \multicolumn{2}{c|}{\textbf{Time (s)}}  \\
& CIFAR-10 & CIFAR-100 \\
\hline \hline
Original &  15.25  & 14.95\\
\cline{1-3}
\textbf{SAT} & & \\ \cline{1-1}
Resnet-10 | Std. | GBP &  18.25 & 18.11 \\
Resnet-10 | Adv. | GBP  &  18.18  & 18.26 \\
                          \Xhline{3\arrayrulewidth}
PGD  &  159.97 & 163.85 \\
\cline{1-3}
\textbf{PGD-SAT} & &  \\ \cline{1-1}
Resnet-10 | Std. | GBP &  85.15 & 92.83\\
Resnet-10 | Adv. | GBP &  85.08  & 93.99 \\
                           \Xhline{3\arrayrulewidth}
TRADES &  270.66  & 277.26 \\
\cline{1-3}
\textbf{TRADES-SAT} & & \\ \cline{1-1}
Resnet-10 | Std.|  GBP &  136.78 & 164.61 \\
Resnet-10 | Adv. | GBP &  138.19 & 163.80 \\
                           \hline

\end{tabular}
}
\end{center}
\caption{Training time for one epoch in seconds averaged over 10 trials.}
\label{table:train-time}
\end{minipage}
\vspace{0.8cm}
\begin{minipage}{0.3\textheight}
\begin{center}
\scalebox{0.8}{
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Method}  & \epsilon=$\frac{1}{255}$ & $\epsilon=\frac{2}{255}$  & $ \epsilon=\frac{3}{255}$ \\
& & &   \\
\hline \hline
                        Original & 1.04 & 0.4 & 0.0  \\
                        Original + Uniform-Noise & 9.45 & 2.32 & 0.77 \\
                        \cline{1-4}
                        \textbf{SAT (Weak Saliency)} & & &  \\ \cline{1-1}

                           Resnet-10 | Std. | G.CAM++ & 9.35  & 1.95 & 0.74                \\

                            Resnet-10 | Adv. | G.CAM++ & 9.68  & 2.30 & 0.80          \\

                           \cline{1-4}
                           \textbf{SAT (Fine Saliency)} & & &  \\ \cline{1-1}

                           Resnet-10 | Std. | GBP & 3.02  & 0.0 & 0.0                 \\
                            Resnet-10 | Adv. | GBP & \textbf{10.69} & \textbf{2.81} & \textbf{1.27}          \\

                        \Xhline{3\arrayrulewidth}
                          Bounding Boxes & 9.79  & 2.46 & 0.77 \\
\hline

\end{tabular}
}
\end{center}
\caption{Results of improved robustness on Tiny-Imagenet dataset. GBP:  Guided-Backpropagation; G.CAM++: Grad-CAM++.}
\vspace{-8pt}
\label{table:tiny-img}
\end{minipage}
\end{minipage}
\end{table*}

\vspace{-12pt}
\noindent \paragraph{Robustness by Ensembling of Adversarial Perturbations:}
Building upon previous observations, we further ask ourselves if we can ensemble our saliency-based perturbations with adversarial perturbation to improve upon existing adversarial training techniques. We now train our student model adversarially by randomly perturbing the input either with a standard perturbation method (using a 5-step PGD/TRADES attack, $\epsilon = 8/255$) or with our saliency-based map perturbation. We call these methods PGD-SAT and TRADES-SAT in the results. Tables \ref{table:cifar-10} and \ref{table:cifar-100} show that PGD-SAT and TRADES-SAT improve over vanilla PGD and TRADES \cite{trades} adversarial training. Also, we observe here too that in most cases, using saliencies of non-robust models (less interpretable) leads to less improvement than using saliencies of robust models (more interpretable), as stated earlier.

\vspace{-12pt}
\noindent \paragraph{Robustness Results on Tiny ImageNet:}
To complete this set of experiments, we use the bounding boxes in the Tiny ImageNet dataset as saliency maps in Algorithm \ref{mimic-AT}. We also train student models using GradCAM++ and Guided-Backprop saliencies from Resnet-10 standard and adversarially trained teachers for comparison. The models are evaluated using a 5-Step PGD attack with $l_\infty$ perturbation $\epsilon \in \{\frac{1}{255}, \frac{2}{255}, \frac{3}{255}\}$. Table \ref{table:tiny-img} shows our results (more detailed results are in the Appendix due to space constraints). As can be seen, using bounding boxes as saliencies achieves good improvement in robustness, and does as well as using weak saliency maps provided by GradCAM++.

\vspace{-12pt}
\noindent \paragraph{Time Efficiency:}
We further analyze the training efficiency of the proposed SAT method. We stress upon the assumption that the saliency maps used in above experiments are available to us a priori like any other explanations such as bounding-boxes and segmentation masks. Table \ref{table:train-time} reports the average time taken by one epoch over 10 trials. As can be seen, PGD-SAT and TRADES-SAT requires only half the training time than vanilla PGD and TRADES respectively, and at the same time achieves superior performance. In case of vanilla SAT, the behavior is desirable since we observe an increase in robustness without compromising much in training time. The analysis suggest that: in scenarios where explanations are available in form of either saliency maps or bounding boxes or segmentation masks, PGD-SAT and TRADES-SAT provides more robust models with half as much time. This further inspires the demand for finer and more interpretable saliency map annotations to be provided in current computer vision datasets.

\begin{table*}[]
\centering
\begin{center}
\scalebox{0.75}{
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
\hline

\textbf{Method} &  \multicolumn{4}{c|}{\textbf{Uniform-Noise}} & \multicolumn{4}{c|}{\textbf{TRADES}} &
      \multicolumn{4}{c|}{\textbf{Saliency Attack}} \\
         & $\epsilon=\frac{4}{255}$ & $\epsilon=\frac{8}{255}$ & $\epsilon=\frac{16}{255}$ & $\epsilon=\frac{32}{255}$ & $\epsilon= \frac{1}{255}$ & $\epsilon=\frac{2}{255}$ & $\epsilon=\frac{3}{255}$ & $\epsilon=\frac{4}{255}$ & $ \epsilon=\frac{2}{255}$ & $\epsilon=\frac{4}{255}$ & $\epsilon=\frac{8}{255}$ & $\epsilon=\frac{16}{255}$  \\
        &  &  &  & &  &  &  &  &  &  &  &   \\

\hline \hline
Original & 65.5 & 62.21 & 47.88 & 21.17 & 51.1 & 39.59
                            & 35.1 & 33.45 & 25.61 & 13.03 & 6.86 & 3.92
                            \\
Original + Uniform-Noise & \textbf{67.54} & \textbf{67.42} & \textbf{60.75} & 28.9 & 55.8 & \textbf{46.98}
                            & 39.37 & 37.8 & 37.05 & 19.63 & 10.48 & 6.1
                            \\
\hline
\textbf{SAT} &  & &  &  & & & & & &  &  & \\ \cline{1-1}
Resnet-10 | Std | GBP & 56.04 & 56.63 & 57.4 & \textbf{46.9} & 43.84 & 36.45 & 33.35 & 32.46 & 53.31 & 51.78 & 49.02 & 42.75             \\

Resnet-10 | Adv. | GBP& 66.55 & 66.63 & 57.93 & 21.92 & \textbf{55.93} & 45.93 & \textbf{40.80} & \textbf{37.86} & \textbf{65.56} & 62.99 & 56.48 & 43.49              \\

Resnet-10 | Adv. | G.CAM++ & 65.69 & 64.00 & 54.84 & 23.43 & 54.16 & 44.31 & 38.28 & 35.71 & 63.95 & \textbf{63.86} & \textbf{63.87} & \textbf{63.66}               \\
                           \Xhline{3\arrayrulewidth}
PGD & 51.04 & 50.96 & 51.0 & 48.69 & 50.6 & 49.03 & 47.63 & 45.34  & 49.02 & 44.03 & 34.77  & 21.04             \\
PGD + Uniform-Noise & \textbf{58.17}  & \textbf{55.6} & 52.45 & 49.20 & \textbf{55.66} & \textbf{52.32}
                            & 48.76 & 45.87 & \textbf{53.60} & 45.72 & 32.81 & 16.62
                            \\

\hline
\textbf{PGD-SAT} &  & &  &  & & & & & &  &  & \\ \cline{1-1}
Resnet-10 | Std | G.BP & 52.8 & 52.78 & 52.56 & 49.21 & 52.06 & 50.43 & 47.69 & 46.24 & 47.33 & 47.25 & 46.74 & 44.95             \\
Resnet-10 | Adv. | G.BP  & 54.22 & 54.38 & \textbf{54.13} & \textbf{50.30} & 53.72 & 51.58 & \textbf{49.15} & \textbf{46.83}  & 50.77 & 50.61 & 50   & 47.64            \\

Resnet-10 | Adv. | G.CAM++ & 53.27 & 53.30 & 52.76 & 49.74 & 52.43 & 50.87 & 48.42 & 46.31  & 50.66 & \textbf{50.69} & \textbf{50.63}  & \textbf{50.54}            \\
                           \hline

\end{tabular}
}
\end{center}
\caption{Improved robustness against other perturbations : Uniform-Noise, TRADES and Saliency. GBP:  Guided-Backpropagation; G.CAM++: Grad-CAM++. The range of $l_\infty$ perturbation is $[-\epsilon, \epsilon]$.}
\vspace{-10pt}
\label{table:diff-pert}
\end{table*}

\vspace{-5pt}
\section{Discussion and Ablation studies}
\label{ablations}
\vspace{-5pt}
We now carry out a few ablation studies to better characterize the efficacy of our methodology. Unless explicitly specified, the experiments below are carried out on CIFAR-100 using Guided-Backprop and G.CAM++ explanations of a standard and adversarially trained Resnet-10 teacher.

\vspace{-12pt}
\noindent \paragraph{Varying Hyperparameters:} We studied the effect on the robustness of our trained model by varying the hyperparameters $\alpha$ and $\epsilon_0$ in our method. Fig \ref{fig:variation-alpha} shows the graph of robustness by varying both hyperparameters. Models are evaluated using a 5-Step PGD attack with $l_\infty$ max-perturbation $\epsilon \in \{\frac{1}{255}, \frac{2}{255}, \frac{3}{255} \}$.
We achieve higher robustness when $\alpha^{10}$ is set closer to 1 ($\alpha$ close to 1), which from Eqn \ref{pert-calc} indicates that the saliency map is used to obtain perturbations in later stages of training, as explained before. Interestingly, when $\alpha^{10}$ is close to  0 ($\alpha$ close to 0), the training doesn't include any noise factor, and since we have a fixed saliency map for each image, it becomes equivalent to training on an additively shifted version of the original training data resulting in a less robust model.
While varying $\epsilon_0$ in Algorithm \ref{mimic-AT}, we observe a peak in robustness somewhere in the middle of the considered range of values (Table \ref{fig:variation-alpha}). Our algorithm works on the objective that the estimated perturbations used in SAT should have reasonable conjunction with standard adversarial perturbations. In algorithm \ref{mimic-AT}, the distribution of our estimated perturbations and actual adversarial perturbations depend on the hyperparameter $\epsilon_0$. At $\epsilon_0 = 0$, the distributions are identical but diverges as it increases. Hence, at high values of $\epsilon_0$, when both distributions diverge, the perturbations used by SAT no longer resembles true adversarial perturbations, hence resulting in less robust models.

\vspace{-12pt}
\noindent \paragraph{Robustness against Other Attacks:}
In Tables \ref{table:cifar-10} and \ref{table:cifar-100}, we evaluate our models using the widely used PGD attack. We now study how the proposed method works against other attacks - in particular, Uniform Noise attack, TRADES \cite{trades} attack and the Saliency-based perturbation described in Sec \ref{results} (under \textit{Saliency Maps for Adversarial Attacks}). These results are shown in Table \ref{table:diff-pert}. It is clear that SAT and PGD-SAT outperform or have comparable performance (in case of Uniform Noise) in most cases. The improvement is much more significant in case of Saliency-based perturbations.

\vspace{-12pt}
\noindent \paragraph{Using Better Explanation Methods}
We also performed a study wherein we trained our models using better saliency maps obtained using methods such as SmoothGrad \cite{smooth-grad}, Guided Grad-CAM++ \cite{grad-cam++} and Integrated Gradients \cite{IG} from teacher models. These models are evaluated using a 5-Step PGD attack with $l_\infty$ max-perturbation $\epsilon \in \{\frac{1}{255}, \frac{2}{255}, \frac{3}{255}, \frac{4}{255}\}$. As can be seen from Table \ref{table:better-exp}, we tend to achieve more robust models for better saliency maps.

\vspace{-5pt}
\section{Conclusion}
\vspace{-5pt}
This work explores the interesting connection between  interpretability and robustness by leveraging a tangible relation between saliency maps and adversarial perturbations to propose a Saliency based Adversarial training (SAT) method. SAT imitates adversarial training by generating adversarial perturbations that resembles true ones. We show how using finer and more interpretable saliencies leads to more robust models through experiments on CIFAR-10 and CIFAR-100. We further gain improvement over popular adversarial training methods by integrating SAT with them, thus giving rise to PGD-SAT and TRADES-SAT techniques. Lastly, we show how weak explanations like localization maps and bounding boxes can be exploited in the same manner to improve model's robustness with results on Tiny ImageNet. On basis of empirical studies, we believe that if we have better explanations at prior, one can use training techniques such as SAT to obtain a more robust model in lesser time. Our work opens rather a new direction to enhance robustness of DNNs by exploiting saliencies. Future work in this direction may aim to improve SAT by inferring about class closest to true one in terms of decision boundary and  approximating the directions better. The work also highlights the need for fine and interpretable saliency map annotations to be provided as part of computer vision datasets.
{\small
\bibliographystyle{ieee_fullname}

\section{Additional Results}
\begin{table}[h]
\begin{center}
\scalebox{0.9}{
\begin{tabular}{|c|c|c|c|}
\hline

\textbf{Method}  & $\epsilon = \frac{1}{255}$ & $\epsilon = \frac{2}{255}$  & $\epsilon = \frac{3}{255}$ \\
& & & \\
\hline \hline
                        Original & 1.04 & 0.4 & 0.0  \\
                        Original + Uniform-Noise & 9.45 & 2.32 & 0.77 \\
                        \cline{1-4}
                        \textbf{SAT (Weak Saliency)} & & &  \\ \cline{1-1}

                           Resnet-10 | Std. | G.CAM++ & 9.35  & 1.95 & 0.74                \\

                            Resnet-10 | Adv. | G.CAM++ & 9.68  & 2.30 & 0.80          \\

                           \cline{1-4}
                           \textbf{SAT (Fine Saliency)} & & &  \\ \cline{1-1}

                           Resnet-10 | Std. | GBP & 3.02  & 0.0 & 0.0                 \\
                            Resnet-10 | Adv. | GBP & \textbf{10.69} & \textbf{2.81} & \textbf{1.27}          \\

                        \cline{1-4}
                          Bounding Boxes & 9.79  & 2.46 & 0.77 \\
\Xhline{3\arrayrulewidth}
                        PGD & 18.91 & 14.34 &11.37 \\
                        PGD + Uniform-Noise & 19.57 & 15.49 & 11.66 \\
                        \cline{1-5}
                        \textbf{PGD-SAT} & & & \\
                        \cline{1-1}
                        Resnet-10 | Std. | GBP & 19.09 & 15.2 & 11.54 \\
                        Resnet-10 | Adv. | GBP & \textbf{20.6} & \textbf{16.38} & \textbf{12.96}         \\

                        Resnet-10 | Adv. | G.CAM++ & 19.32 & 15.38 & 11.76  \\

                        \cline{1-4}
                        PGD + Bounding Boxes & 20.56 & 16.38 & 12.91 \\
                                  \Xhline{3\arrayrulewidth}
                          TRADES & 18.45 & 16.76 &	11.09 \\
                          TRADES + Uniform-Noise & 19.96 & 16.13 & 12.58 \\
                          \cline{1-5}
                          \textbf{TRADES-SAT} & & & \\ \cline{1-1}
                        Resnet-10 | Std. | GBP & 19.57 & 16.22 & 11.82 \\
                          Resnet-10 | Adv. | GBP & 19.83 & 16.9 & 12.26\\

                          Resnet-10 | Adv. | G.CAM++ & \textbf{20.5} & 16.38& 12.93 \\

                        \cline{1-4}
                        TRADES + Bounding Boxes & 20.04 & \textbf{16.45} & \textbf{12.96}
                           \\ \hline
\end{tabular}
}
\end{center}
\caption{Results of improved robustness on Tiny-Imagenet dataset. GBP:  Guided-Backpropagation; G.CAM++: Grad-CAM++.}
\vspace{-10pt}
\label{table:tiny-img}
\end{table}
\paragraph{Tiny-Imagenet Results:} In addition to the results reported in Section 4 in the main paper, we report a tabulation of our results on Tiny-Imagenet [6] dataset on PGD-SAT and TRADES-SAT in Table \ref{table:tiny-img} (which we could not include in main paper due to space constraints). We use Guided-Backpropagation [21] as well as Grad-CAM++ [3] (written as G.CAM++) explanations of a Resnet-10 standard/adversarially trained teacher. We also report results when the bounding boxes available in Tiny-ImageNet are used as (weak) saliency maps to show the effectiveness of our methodology in scenarios where they are available. \footnote{We also trained SAT, PGD-SAT and TRADES-SAT models on datasets with segmentation masks as saliency maps. We obtain consistent and impressive results even in this case, but do not report them here due to the guidelines for the supplementary material.}

\paragraph{Robustness to Stronger PGD Attacks:}
As shown in the main paper (Section 4), our trained student models are robust against the 5-Step PGD [14] attack . We now study the effect of even stronger attacks on our trained student models. The models are evaluated using 20-Step and 40-Step PGD attacks with $l_\infty$ max-perturbation $\epsilon \in \{\frac{1}{255}, \frac{2}{255}, \frac{3}{255}, \frac{4}{255}\}$. The results are shown in Table \ref{table:strong-attack-robust}. As the results show, our models show similar improvements in robustness even on these stronger attacks.

\paragraph{Using Saliency Maps of Sparse Teachers}
We add another variation in the quality of saliency maps wherein we use the saliency maps of a sparse teacher model  (with 95\% of total weights as zero) whose performance is comparable to a dense equivalent model. We observe that saliency maps of sparse models are slightly better than standard ones but not better than that of adversarially trained models and expect similar behaviour when used in SAT. Therefore, we train SAT and PGD-SAT models on CIFAR-10 using Guided-Backpropogation [21] as the explanation of Resnet-10/Resnet-34 sparse models. Table \ref{table:sparse} shows our results. We notice that in most cases, using saliency maps of sparse teachers lead to a reasonable improvement as compared to saliency maps of the standard model but short of adversarially trained teachers.

\paragraph{Using Ensemble of Saliency Maps}
In the main paper (Section 4), we show how an ensemble of our estimated perturbations (SAT) and actual adversarial perturbations leads to further improvement. It seems obvious then to question about the effect of using more than one saliency maps in SAT. To study this, we assess the contribution of using an ensemble of two saliency maps in SAT. For getting two different saliency maps, we either change: \textbf{(1)} model architecture (Resnet-10, Resnet-34) or  \textbf{(2)} training procedure (Sparse, Adv). We therefore select the saliency map to be used in SAT (Algorithm 1 in main paper) randomly between these two maps. We call this Ensemble-SAT and report our findings in Table \ref{table:sparse} (Shown using '+' symbol) for CIFAR-10. We observe that using the ensemble of saliency maps leads to improvement in some cases. The cases where the improvement is more, include saliency maps which have different characteristics. This can be attributed to the fact that coming from slightly two different distributions, the saliency maps now cover different types of adversarial perturbations than their singular equivalents. Consequently, one can infer that if richer saliency maps, perhaps from different sources of reasonable divergence, are available, one can leverage them to improve performance further.

\begin{table*}
\begin{center}
\scalebox{1}{
\begin{tabular}{|c|c|c|c|c|c|}
\hline
\textbf{Method} &  \textbf{Attack} &$ \epsilon = \frac{1}{255}$ & $ \epsilon = \frac{2}{255}$ & $\epsilon = \frac{3}{255}$ & $\epsilon = \frac{4}{255}$ \\
& & & & & \\
\hline \hline
\multirow{2}{*}{Original}

                            & PGD-20 & 27.17 & 6.94 & 2.24 & 0.81             \\
                            & PGD-40 & 27.08 & 6.86 & 2.06 & 0.72             \\
                            \cline{1-6}
\multirow{2}{*}{Original + Uniform-Noise}

                            & PGD-20 & \textbf{36.39} & 12.22 & 5.25 & 2.29            \\
                            & PGD-40 & \textbf{36.45} & 12.1 & 5.14 & 2.18            \\
                            \cline{1-6}
\textbf{SAT} & & & &  &\\ \cline{1-1}
\multirow{2}{*}{Resnet-10 | Std | GBP}

                            & PGD-20 & 20.87 & 7.25 & 2.76 & 1.21            \\
                            & PGD-40 & 20.92 & 7.18 & 2.6 & 1.19             \\
                            \cline{1-6}

\multirow{2}{*}{Resnet-10 | Adv. | GBP}

                           & PGD-20 & 34.79 & \textbf{13.94} & \textbf{5.28} & \textbf{2.38}             \\
                            & PGD-40 & 34.97 & \textbf{13.97} & \textbf{5.2} & \textbf{2.27}             \\
                            \cline{1-6}

\multirow{2}{*}{Resnet-10 |Adv. | G.CAM++}

                            & PGD-20 & 33.46 & 12.55 & 5.06 & 2.37             \\
                            & PGD-40 & 33.52 & 12.46 & 4.98 & 2.27             \\
\Xhline{3\arrayrulewidth}
\multirow{2}{*}{PGD}

                            & PGD-20 & 45.75 & 40.09 & 35.19 & 30.67             \\
                            & PGD-40 & 45.76 & 40.09 & 35.18 & 30.65             \\
                            \cline{1-6}
\multirow{2}{*}{PGD + Uniform-Noise}

                            & PGD-20 & \textbf{50.76} & 41.61 & 35.05 & 30.58            \\
                            & PGD-40 & \textbf{50.75} & 41.63 & 35.03 & 30.52            \\
                            \cline{1-6}
\textbf{PGD-SAT} & & & &  &\\ \cline{1-1}
\multirow{2}{*}{Resnet-10 | Std | GBP}

                                & PGD-20 & 44.89 & 39 & 34.14 & 29.23             \\
                             & PGD-40 & 44.89 & 39.02 & 34.14 & 29.17             \\
                            \cline{1-6}

\multirow{2}{*}{Resnet-10 | Adv. | GBP}

                            & PGD-20 & 48.10 & \textbf{42.13} & \textbf{36.16} & \textbf{31.1}            \\
                            & PGD-40 & 48.11 & \textbf{42.14} & \textbf{36.14} & \textbf{31.07}             \\
                            \cline{1-6}

\multirow{2}{*}{Resnet-10 | Adv. | G.CAM++}

                           & PGD-20 & 47.29 & 41.69 & 35.98 & 30.96             \\
                         & PGD-40 & 47.29 & 41.69 & 35.92 & 30.93             \\
                            \cline{1-6}
                          \hline

\end{tabular}
}
\end{center}
\caption{Improved robustness to stronger PGD attacks on CIFAR-100. GBP: Guided-Backpropogation; G.CAM+: Grad-CAM++.}
\label{table:strong-attack-robust}
\end{table*}

\begin{table*}[]
\begin{center}
\scalebox{0.9}{
\begin{tabular}{|c|c|c|c|c|}
\hline

\textbf{Method}  & $\epsilon = \frac{1}{255}$ & $\epsilon = \frac{2}{255}$  & $\epsilon = \frac{3}{255}$ & $\epsilon = \frac{4}{255}$ \\
& & & &  \\
\hline \hline
                        Original & 47.71 & 10.36 & 1.39 & 0.28 \\
                        Original + Uniform-Noise & 61.23 & 22.85 & 6.34 & 2.56 \\
                        \cline{1-5}
                        \textbf{SAT} & & & & \\ \cline{1-1}
                        Resnet-10 | Std. | GBP & 10.87  & 0.95  & 0.0  & 0.0              \\
                         Resnet-10 | Adv. | GBP & \textbf{63.33}  & \textbf{26.79} & \textbf{9.62} & \textbf{3.69}          \\
                         Resnet-34 | Std. | GBP  & 18.01 & 2.54 & 1.25 & 1.0             \\
                         Resnet-34 | Adv. | GBP & 62.67 & 23.76  & 5.82 & 1.28 \\
                        Resnet-10 | Sparse | GBP & 52.43 & 26.79 & 5.82 & 1.97           \\
                        Resnet-34 | Sparse | GBP & 60.18& 23.56 & 8.58 & 2.76 \\

                          Resnet-10 | Adv. | GBP + Resnet-34 | Adv. | GBP &
                          63.2  & 24.2 & 6.58 & 1.71             \\
                          Resnet-10 | Adv. | GBP + Resnet-34 | Sparse | GBP & \textbf{71.01} & \textbf{39.94} & \textbf{19.0} & \textbf{8.64}           \\
                          Resnet-10 | Sparse | GBP + Resnet-34 | Adv. | GBP  &
                          66.47 & 35.19 & 15.0 & 6.11          \\
                          Resnet-10 | Sparse| GBP  + Resnet-34 | Sparse | GBP & 60.07 & 29.11 & 11.26 & 4.55 \\
\Xhline{3\arrayrulewidth}
                        PGD & 77.89 & 73.1 & 66.96 & 61.18  \\
                        PGD + Uniform-Noise & \textbf{82.23} & 74.97 & 65.61 & 55.04 \\
                        \cline{1-5}
                        \textbf{PGD-SAT} & & & & \\
                        \cline{1-1}
                        Resnet-10 | Std. | GBP & 79.72 & 73.81 & 67.72 & 61.29   \\
                        Resnet-10 | Adv. | GBP & 80.72 & \textbf{75.07} & \textbf{68.68} & 62.49          \\
                        Resnet-34 | Std. | GBP & 79.53 & 74.2 & 68.19 & 62.48  \\
                        Resnet-34 | Adv. | GBP & 80.15 & 74.60 & 68.47 & \textbf{62.53}      \\
                         Resnet-10 | Sparse | GBP & 79.48 & 73.97 & 67.72 & 61.77          \\
                         Resnet-34 | Sparse| GBP  & 79.42 & 74.38 & 68.48 & 62.48 \\
 \hline

\end{tabular}
}

\end{center}
\caption{Results of improved robustness on CIFAR-10 dataset using saliency of sparse teachers. GBP:  Guided-Backpropagation. '+' denotes ensemble of saliencies.}
\label{table:sparse}
\end{table*}

\begin{table*}

\end{table*}

\end{document}

\section{Introduction}
\label{intro}
Deep Neural Networks (DNNs) have become vital to solve many tasks across domains including image/text/graph classification and generation, object recognition, segmentation, speech recognition, etc. As the applications of DNNs widen in scope, \textit{robustness} and \textit{interpretability} are two important parameters that define the goodness of a trained DNN model. While on one hand the deep network should be robust to imperceptible perturbations, on the other hand it should be interpretable enough to be trusted when practically used in domains like autonomous navigation or healthcare. Keeping in mind the vulnerability of deep networks to adversarial attacks \cite{szegedy2013intriguing}, efforts have been undertaken to make them more robust to these attacks. Among the proposed methodologies, Adversarial Training (AT) \cite{at@madry,at@goodfellow} has emerged as one of the best defenses wherein networks are trained on adversarial examples to better classify them at test time. On the other hand, in order to generate an interpretable explanation to a network prediction, many methods have been proposed lately, of which guided-backpropagation (GBP)\cite{gbp}, GradCAM++ \cite{grad-cam++}, Integrated Gradients (IG) \cite{integrated-gradients} and fine-grained visualisations (FGVis) \cite{fgvis} are popular to name a few. \par

Most work so far on interpretability and adversarial robustness have focused on either of them alone. Very recently, over the last few months, there has been a new interest in coupling the two notions and aiming to understand the connection between robustness and interpretability \cite{connection@etmann,imagenet-texture,robustness-odds,interpret-atcnn,attribute-regularization,jacobianAT}. The few existing efforts can be categorized broadly into three kinds: (i) Efforts \cite{interpret-atcnn,robustness-odds} which have shown that explanations generated by more robust models are more interpretable than their non-robust counterparts; (ii) Efforts that have attempted to theoretically analyze the relationship between adversarial robustness and interpretability (for e.g., \cite{connection@etmann} obtains an inequality between these notions which holds to equality in case of linear models, and \cite{relating-explanations} relates the two using a generalized form of hitting set duality); and (iii) Efforts \cite{attribute-regularization,jacobianAT}, more recent, that have attempted to improve robustness by training models with additional objectives that constrain the image explanations to be more interpretable and robust towards attribution attacks.
All of these efforts are recent, and more work needs to be done to explore the connections better. In this work, we provide a different, yet simple and effective, approach to leverage saliency maps for adversarial robustness.
We observe that adversarial perturbations correspond to class-discriminative pixels in later stages of training (see Figure \ref{fig:pert-variation}), and exploit this observation to use the saliency map (static) of a given image, whilst training to improve robustness. For datasets where bounding boxes and segmentation masks are provided, we demonstrate that one can exploit these, in lieu of saliency maps, to improve model robustness using our approach. Our work would be closest to the third category of methods described above, and we differentiate from these other recent efforts further in Section \ref{sec_related_work}.

Generally speaking, humans tend to learn new tasks in a robust, generalizable fashion when provided with explanations during their learning phase. For example, a medical student learns about a disease better when provided with explanations of how the disease acts inside the body, rather than just the description of external symptoms. A person may otherwise learn irrelevant relationships without knowledge of underlying explanations. Similarly, we opine that a DNN model that is trained with explanations is less easily fooled by adversarial perturbations.
Little has been explored in this direction, and we aim to leverage this relationship to provide an efficient methodology for adversarial robustness. We differ from earlier efforts in our primary objective that we use explanations to generate pseudo-adversarial examples instead of using them as a regularization term like \cite{attribute-regularization,jacobianAT} (more in Section \ref{sec_related_work}). In particular, we provide a simple yet effective methodology (as compared to a recent effort such as \cite{relating-explanations}, which is NP-hard) that improves model robustness using saliency maps or already provided bounding boxes or segmentation masks.
\par

Our key contributions can be summarized as follows: (i) We observe a tangible relationship between a saliency map and adversarial perturbations for a given image, and leverage this observation to propose a new methodology that uses the saliency map of the image to mimic adversarial training; (ii) We show through our empirical studies on widely used datasets: CIFAR-10, CIFAR-100, Tiny ImageNet and Flowers-17, to show the improvement in adversarial robustness through our proposed method; (iii) We show that the improvement becomes more pronounced when a finer and stronger saliency map is used, signifying a strong correlation between saliency maps and adversarial robustness; (iv) When bounding boxes or segmentation masks are already available in a dataset (e.g. Tiny ImageNet or Flowers), we demonstrate that our methodology improves adversarial robustness with no additional cost to generate perturbations for training; (v) Additionally, we show that integrating our method with adversarial training methods, such as PGD \cite{at@madry} and TRADES \cite{trades}, further improves their performance; and (vi) We perform detailed ablation studies to better characterize the efficacy of our proposed methodology. We believe that this work can contribute to opening up a rather new direction to enhance robustness of DNN models.

\section{Related Work}
\label{sec_related_work}
We review earlier efforts related to this work from multiple perspectives, as described below.

\noindent \textbf{Explanation methods:} Various methods have been proposed over the last few years to explain the decisions of a neural network. \textit{Backpropagation-based methods} find the importance of each pixel by backpropagating the class score error to the input image. An improved and popular version of this, known as Guided-Backpropagation \cite{gbp}, only keeps paths that lead to positive influence on the class score, leading to much cleaner-looking explanations. SmoothGrad \cite{smooth-grad}, VarGrad \cite{var-grad} and Integrated gradients \cite{integrated-gradients} refine the explanations by combining/integrating gradients of multiple noisy/interpolated versions of the image. Other backpropagation based methods like DeepLift \cite{deep-lift}, Excitation BackProp \cite{excitation-bp} and Layerwise Relevance Propagation \cite{lrp} generate explanations by utilizing topdown relevance propagation rules. PatternNet and PatternAttribution \cite{patternnet} yields explanations that are theoretically sound for linear models and produce improved explanations for deep networks. CAM \cite{cam}, Grad-CAM \cite{grad-cam}, Grad-CAM++ \cite{grad-cam++} form another variant of generating explanations known as \textit{Activation-based methods}. These methods use linear combinations of activations of convolutional layers, with the weights for these combinations obtained using gradients. \textit{Perturbation  based  methods} generate attribution maps by examining the change in prediction of the  model when the input image is perturbed \cite{pertexp1,pertexp2}. All the aforementioned methods however focus solely on explaining neural network decisions.

\noindent \textbf{Adversarial Attacks and Robustness:} With the advancement of newer adversarial attacks each year \cite{at@goodfellow,at@madry,spatial-adv-exmp,trades}, methods have been proposed to defend against them. Parseval Networks \cite{parseval-nets} train robust networks by constraining the Lipschitz constant of its layers to be smaller than 1. Another category of methods harness the susceptibility of latent layers by performing latent adversarial training (LAT) \cite{LAT}  or using feature denoising \cite{feature-denoising}. Other methods like DefenseGAN \cite{defense-gan} exploit GANs wherein they learn the distribution of unperturbed images and find the closest image for a given test image, to feed the network at inference time. TRADES \cite{trades}, more recently, presents a new defense method that provides a trade-off between adversarial robustness and vanilla accuracy by decomposing prediction error for adversarial examples (robust error) as the sum of natural (classification) error and boundary error, and providing a differentiable upper bound using theory based on classification-calibrated loss. Among the proposed defenses against adversarial attacks, Adversarial Training (AT) \cite{at@madry,at@goodfellow} has remained the most popular and widely used defense, where the network is trained on adversarial examples in order to match the training data distribution with that of adversarial test distribution. More recent efforts in this direction include \cite{free-AT,YOPO}, which aim to reduce adversarial training overhead by recycling gradients and accelerating via the maximal principle respectively. In this work, we focus on adversarial training, considering it still remains the most reliable defense against different attacks.

\noindent \textbf{Interpretability and robustness:} The last few months have seen a few efforts on associating the notions of robustness and interpretability. These efforts can be categorized into three kinds, as introduced briefly earlier: (i) The first kind centers around interpreting how adversarially trained convolutional neural networks (ATCNNs) recognize objects. Recent studies \cite{interpret-atcnn,robustness-odds} have shown that representations learned by ATCNNs are more biased towards image shape than its texture. Also, these ATCNNs tend to evince more interpretable saliency maps corresponding to their prediction than their non-robust equivalents. (ii) In the second kind, inspired by \cite{interpret-atcnn,robustness-odds}, Etmann et al. \cite{connection@etmann} recently quantified this behavior of ATCNNs by considering the alignment between saliency map and the image as the metric for interpretability. They confirmed that for a linear model, the alignment grows strictly with robustness. For non-linear models such as neural networks, they show that the their linearized robustness is loosely bounded by the alignment metric. \cite{relating-explanations}, which is rather new, provides a theoretical connection between adversarial examples and explanations by demonstrating that both are related by a generalized form of hitting set duality. (iii) Encouraged by the work of Etmann \cite{connection@etmann}, a third category of work more recently sought to answer the question: Do robust and interpretable saliency maps imply adversarial robustness ? Recent efforts \cite{explanations-manipulated,fragile-interpretation} have shown that the explanations of neural networks can also be manipulated by adding perturbations to input examples, which, instead of causing mis-classification, result in a different explanation. To tackle this problem, a recent effort, Robust Attribution Regularization (RAR) \cite{attribute-regularization}, aims to train networks by optimizing objectives with a regularization term on explanations in order to achieve robust attributions.

In contrast to the abovementioned methods, our work explores a different side to the connection between notions of interpretability and robustness. As explained in Section \ref{intro}, we aim to generate adversarial perturbations from a given saliency map to improve robustness while training a neural network. Our work is perhaps closest to RAR \cite{attribute-regularization} in using explanations (saliency maps) during training, we however differ from them in our primary objective and in that we are using explanations to generate adversarial examples, while RAR seeks to attain attributional robustness. In terms of approach, although one could consider the work by Ignatiev et al. \cite{relating-explanations} as related to ours, they had only preliminary results on MNIST, and focused on providing a theoretical connection between adversarial examples and explanations based on logic and constraint programming, which may not scale to large-scale datasets.
Our work attempts to show that explanations can yield robustness in standard large-scale benchmark datasets in an efficient way. We further hypothesize that, by advancement through our method, we can actually take advantage of weak explanations like bounding boxes and segmentation masks which may otherwise be left unexploited in a dataset, to train adversarially robust models.

\begin{figure*}[h]
\centering
\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{./images/Pert2.png}
\caption{Variation of adversarial perturbation with training epochs during 5-step PGD adversarial training of Resnet-34 on CIFAR-10. We observe that adversarial perturbations at later stages of training correspond to class-discriminative regions/pixels.}
\label{fig:pert-variation}
\end{figure*}

\section{Using Saliency Maps for Efficient Adversarial Training}
\label{methodology}
Our method is motivated by the observation that adversarial perturbations at later stages of training correspond to class-discriminative regions/pixels, as shown in Figure \ref{fig:pert-variation}. We begin describing our method with the necessary notations and preliminaries.
\vspace{3pt}

\noindent \textbf{3.1 Notations and Preliminaries.}
We denote a neural network as $\Phi(. \ ;  \ \theta) : \mathbb{R}^d \rightarrow \mathbb{R}^k $, parametrized by weights $\theta$, which takes an input $\textbf{x} \in \mathbb{R}^d $ and outputs a logit, $\Phi^i(\textbf{x})$, for each of $k$ classes, i.e. $i \in \mathcal{C} = \{1,\cdots,k\}$, $\mathcal{C}$ denoting the set of class labels.
Without loss of generality, considering an image classification setting, we define a saliency map $\textbf{s}$ corresponding to an input sample $\textbf{x}$ as $\textbf{s} \in [0,1]^d$, where the presence of an object of interest in input $\textbf{x}$ lies between 0 and 1. For a trained network $\Phi$, the unnormalized saliency map for an input $\textbf{x}$ can simply be given as:
$\nabla_{\textbf{x}} \Phi^{i^*}(\textbf{x})$, where $i^* = \argmax_i \Phi^i(\textbf{x})$. (In practice, this gradient is at times computed w.r.t feature maps of intermediate layers, but we leave our notations w.r.t $\textbf{x}$ for ease of understanding.)
\vspace{3pt}

\noindent \textit{Projected Gradient Descent (PGD) Attack:} Prior efforts on adversarial attacks \cite{at@goodfellow} include the Fast Gradient Sign Method (FGSM), a $l_{\infty}$-bounded single step attack which calculates an adversary as: $\textbf{x} + \epsilon \hspace{1mm} \text{sign}(\nabla_\textbf{x} \mathcal{L} \ (\Phi(\textbf{x},\theta),y))$.
A more powerful attack however is the popularly used multi-step variant, also called Projected Gradient Descent (PGD), given by:
\begin{align}
\label{eqn_pgd}
    \textbf{x}^0 = \textbf{x};
    \textbf{x}^{t+1} = \Pi_{\textbf{x}+ \textit{N}} \big( \textbf{x}^{t} + \alpha \hspace{1mm} \text{sign}(\nabla_{\textbf{x}} \mathcal{L} \ (\Phi(\textbf{x},\theta),y)) \big)
\end{align}
\noindent where $\alpha$ is the step size, $\Pi$ is the projection function, and $N$ is the space of possible perturbations. We use PGD as the choice of attack in our experiments, and later change to show (in Section \ref{ablations}) how our method performs against other attacks.
\vspace{3pt}

\noindent \textit{Adversarial Training (AT):} AT \cite{at@goodfellow} is generally used to make the models adversarially robust by matching the training distribution with the adversarial test distribution. Essentially, for AT, the optimal parameter $\theta^*$ is given by:
\begin{equation}
    \theta^* = \argmin_\theta \mathbb{E}_{(\textbf{x},y) \sim D} \ \left[ \ \max_{\delta \in N}\mathcal{L} \ (\Phi(\textbf{x} + \delta,\theta),y) \ \right]
\end{equation}
Here, the inner maximization $\max_{\delta \in N} \mathcal{L} \ (\Phi(\textbf{x} + \delta,\theta),y)$ is calculated using a strong adversarial attack such as PGD.\\

\noindent \textbf{3.2 Saliency-Based Adversarial Training: Motivation.}
An adversarial perturbation at input $\textbf{x}$ is given by:
\begin{equation}
\arg\inf_{e \ \in \ \mathbb{R}^d} \{\Vert e \Vert: \argmax_i \Phi^i(x+e) \neq \argmax_i \Phi^i(x) \}
\end{equation}
Etmann et al. \cite{connection@etmann} showed that for most multi-class neural networks, especially ones with ReLU or Leaky ReLU activation functions, which we consider in this work\footnote{In all our experiments, we train networks with ReLU activations which ensures that this assumptions is met. We follow the experiments in \cite{connection@etmann} in this regard, to ensure that our model is locally affine in a given data point's neighborhood.}, the network's score function, $\Phi$ is sufficiently locally-linear in relevant neighbourhood of input $\textbf{x}$, i.e.
\begin{equation}
    \Phi^i(\textbf{x} +  e) \approx \Phi^i(\textbf{x}) + e^T \cdot \nabla_\textbf{x}\Phi^i(\textbf{x})
\label{eqn_etmann_locally_linear}
\end{equation}
Leveraging this, an adversarial perturbation, $e$, which is intended as a perturbation to input $\textbf{x}$ which results in a change of predicted label, can be modeled as follows (given $i,j \in \mathcal{C}$):
\begin{align}
    & \argmax_i \Phi^i(x+e) \neq \argmax_i \Phi^i(x)\\
    \Longleftrightarrow \qquad & \exists j \neq i^* : \Phi^j(\textbf{x} + e) > \Phi^{i^*}(\textbf{x} + e)\\
    \Longleftrightarrow  \qquad & \exists j \neq i^* : e^T \cdot (\nabla_\textbf{x}\Phi^j(\textbf{x}) - \nabla_\textbf{x}\Phi^{i^*}(\textbf{x})) > \Phi^{i^*}(\textbf{x}) - \Phi^j(\textbf{x})
\end{align}
The third inequality (7) comes from combining inequality (6) and expression (4). The infimum over $\Vert e \Vert$, which provides a minimal perturbation to change the class label, is achieved by choosing $e$ as a multiple of $\nabla_\textbf{x}(\Phi^{j}(\textbf{x})- \Phi^{i^*}(\textbf{x}))$. (In general, note that the LHS of $e^T z > c$ where $c$ is a constant, is maximized by $e = \Vert e \Vert \frac{z}{\Vert z \Vert}$, leading to $\Vert e \Vert \Vert z \Vert > c$. The infimum of $e$ is then achieved by choosing it as a multiple of $z$).
The direction of adversarial perturbation then becomes:
\begin{equation}
\label{eqn_pert_direction}
\nabla_\textbf{x}(\Phi^{j}(\textbf{x}) - \Phi^{i^*}(\textbf{x}))
\end{equation}
This perturbation direction depends on two quantities: (i) $\nabla_\textbf{x} \Phi^{i^*}(\textbf{x})$ the saliency map for the true class $i^*$; and (ii) $\nabla_\textbf{x} \Phi^{j}(\textbf{x})$, the saliency map of $\textbf{x}$ for class $j$ for which the infimum of $e$ is attained. We now analyze the direction of adversarial perturbation separately for the binary and multi-class cases.
\vspace{3pt}

\noindent \textit{Binary case:} Let us consider a binary classifier $h: \textbf{x} \rightarrow \{-1,1\}$ given by: $h = \text{sign}(\Phi(\textbf{x},\theta))$, where $\Phi(\textbf{x},\theta))$ represents the logit of the positive class. Let $\Phi'(\textbf{x})$ denotes the logit of negative class. Applying a sigmoid activation function, we get the probability of the positive and negative class as: $P(y=+1|\textbf{x}) = \frac{1}{1 + \exp^{-\Phi(\textbf{x},\theta)}}$ and $P(y=-1|\textbf{x}) = \frac{1}{1 + \exp^{-\Phi'(\textbf{x},\theta)}}$ respectively.
It is simple to see that the probability of the negative class can also be written as:
\[P(y=-1|\textbf{x}) = 1 - P(y=+1|\textbf{x}) = \frac{1}{1 + \exp^{\Phi(\textbf{x},\theta)}}\]
Rather, the corresponding logit score of the negative class is $-\Phi(\textbf{x},\theta))$. In Eqn \ref{eqn_pert_direction} for a binary classifier, one can hence view $\Phi^{j}(\textbf{x}) =  - \Phi^{i^*}(\textbf{x})$, and define the adversarial perturbation direction in Eqn \ref{eqn_pert_direction} simply as $- \nabla_\textbf{x}(\Phi^{i^*}(\textbf{x}))$.
\vspace{3pt}

\noindent \textit{Multi-class case:} We extend a similar argument to the multi-class setting using an approximation. The direction of adversarial perturbation in the multi-class setting as give in Eqn \ref{eqn_pert_direction} would require finding the class $j$ for which the infimum of $\Vert e \Vert$ is attained. This requires computing the quantity in Eqn \ref{eqn_pert_direction} for all classes, and identifying the $j \neq i^*$ for which the quantity attains the least value. This is compute-intensive. To avoid this computational overhead, we rely on $\Phi^{i^*}$ alone (note that $i^*$, the ground truth label, is known to us at training), and simply propose the use of $- \nabla_\textbf{x}(\Phi^{i^*}(\textbf{x}))$ as the direction of perturbation (as in the binary case). We now argue that this approximation is a reasonable one. Considering the multi-class setting as $k$ binary classification problems, for the binary classifier corresponding to the ground truth class $i^*$, we would have:
\[P(y \neq i^*|\textbf{x}) = 1 - P(y = i^*|\textbf{x}) = \frac{1}{1 + \exp^{\Phi^{i^*}(\textbf{x},\theta)}}\]
In other words, the corresponding logit score of the negative class is $-\Phi^{i^*}(\textbf{x},\theta))$. Approximating the direction of the adversarial perturbation as the average of the directions of the perturbations across the $k$ binary classification problems, we would get the direction of the perturbation to be: $\nabla_\textbf{x}(\sum_{j \neq i^*}\Phi^{j}(\textbf{x}) - k \Phi^{i^*}(\textbf{x}))$. Assuming that each of the classes $l \neq i^* \in \mathcal{C}$ is equally likely to be the $j$ that minimizes $\Vert e \Vert$, it is evident that choosing $- \nabla_\textbf{x}(\Phi^{i^*}(\textbf{x}))$ as the direction of perturbation would be the most conservative option in the expected sense. We show through our experiments that this option works reasonably well.

Since we deal with $l_{\infty}$-bounded perturbations in this work, following PGD (Eqn \ref{eqn_pgd}), we use $- \text{sign} \ (\nabla_\textbf{x} \Phi^{i^*}(\textbf{x}))$ instead of $- \nabla_\textbf{x} \Phi^{i^*}(\textbf{x})$ itself as the perturbation direction.
We complete the above discussion by noting that $\nabla_\textbf{x} \Phi^{i^*}(\textbf{x})$ is the saliency map, \textbf{s}, defined at the beginning of this section.
In other words, the direction of adversarial perturbation can be obtained using a saliency map (as we also show in our experiments).

\begin{wrapfigure}[23]{l}{0.37\textwidth}
\vspace{-15pt}
\includegraphics[width=0.37\textwidth]{./images/sal_attack}
\vspace{-15pt}
\caption{Adversarial attack accuracy when perturbing input example using the direction of negative saliency on normally (Std.) and adversarially (Adv.) trained models on CIFAR-10 and CIFAR-100. Note that $\epsilon=0$ denotes clean accuracy (no attack), all other models face attacks by saliency maps and have lower accuracies.}
\label{fig:sal-attack}
\end{wrapfigure}
To study the above claim, we conducted experiments to check if the negative of the saliency map corresponding to the ground truth class, i.e. $-\nabla_\textbf{x}\Phi^{i^*}(\textbf{x})$, can indeed be used as a direction to perturb input. We created adversarial examples by perturbing original examples as: $\textbf{x} = \textbf{x} - \epsilon \cdot \text{sign}(\nabla_\textbf{x}\Phi^{i^*}(\textbf{x}))$ (using saliency maps) to attack a model. We attack a Resnet-10 model trained on CIFAR-10 and CIFAR-100 datasets, normally (Std.) and adversarially (Adv.) trained using 5-step PGD attack with 3 different magnitudes of $l_\infty$-norm perturbations, $\epsilon \in \{2/255, 4/255, 8/255\}$. We report our findings in Figure \ref{fig:sal-attack}. The figure shows that, examples generated by perturbing an input in the direction of negative saliency decreases the model's accuracy to a reasonable extent, affecting models both models trained normally and adversarially. We now describe how we leverage this relationship to perform Saliency-based Adversarial Training (SAT).\\

\noindent \textbf{3.3 Saliency-Based Adversarial Training: Algorithm.}
Algorithm \ref{mimic-AT} summarizes our methodology of using saliency maps for adversarial training. The saliency maps during training are obtained either through annotations provided in a dataset (such as bounding boxes or segmentation masks), or through a pre-trained model which is used only to get saliency maps. More details on obtaining saliency maps is discussed in Section \ref{results}.

We also observed in our studies that when training a network using adversarial training, during the initial phase of training when the weights are not optimal, the perturbations computed by the attack methods are random. But with training, as weights become optimal, they become more class-discriminative. Figure \ref{fig:pert-variation} illustrates this observation, where the perturbation is random in initial phases of training but eventually becomes class-discriminative as the model trains. We exploit this observation to complete our methodology.
In order to mimic the above behavior of the perturbation over training, we choose the direction of perturbation in a stochastic manner. We choose the $i^{th}$ component $\delta^t$[i] of perturbation $\delta^t$ at time $t$ as:
\begin{equation}
    \label{pert-calc}
    \delta^t[i] =
    \begin{cases}
      \textbf{z}[i], & \text{with probability } \alpha^{t} \\
      -\textbf{s}[i], & \text{with probability } 1-\alpha^{t}
    \end{cases}
 \end{equation}
\noindent where $\textbf{z} \in \{-1,1\}^d$ is sampled randomly, and 0 $< \alpha <$ 1. During initial epochs of training, when $\alpha^t$ is close to 1, $\delta^t$ will be dominated by random values. However, as training proceeds and $\alpha^t$ starts diminishing, $\delta^t$ smoothly transitions to $-\textbf{s}$ and will be influenced by the adversarial character of the saliency map.
\begin{wrapfigure}[21]{l}{0.6\textwidth}
\begin{minipage}{0.6\textwidth}
\vspace{-30pt}
\begin{algorithm}[H]
\SetAlgoLined
\textbf{Input:} Training Dataset $D$, Saliency Maps $S$, Model $\Phi(. \ ; \ \theta)$, SAT hyperparameter $\alpha$, Learning rate $\eta$, Maximum $l_\infty$ perturbation $\epsilon_0$ \\
\textbf{Output:} Optimal parameter $\theta^*$\\
Initialize model parameters as $\theta = \theta^0$. \\
 \For{$t \in \{1, 2, ..., n\}$}
    {
        Sample training data of size $B$ : $\{(\textbf{x}^i, y^i)\}$ from $D$. \\
        Pick out corresponding saliency maps : $\{\textbf{s}^i\}$ from $S$. \\
        Calculate $\delta^{ti}$ for each $\textbf{x}^i$ using Equation \ref{pert-calc}. \\
        Perturb the input examples : $\textbf{x}^i := \textbf{x}^i + \epsilon_0 \cdot \delta^{ti}$. \\
        Perform clipping to keep $\textbf{x}^i$ bounded : $\textbf{x}^i$ := $clip(\textbf{x}^i)$. \\
        Update model parameters : \\ $\theta^t := \theta^{t-1} - \eta \cdot \triangledown_\theta \ \frac{1}{B} \sum_{i=1}^{B} \mathcal{L} \ (\Phi(\textbf{x}^i,\theta^{t-1}),y^i) $
    }
 \caption{Saliency-based Adversarial Training (SAT) Methodology}
 \label{mimic-AT}
\end{algorithm}
\end{minipage}
\end{wrapfigure}
\\

When additional annotations such as bounding boxes or segmentation masks are available in a dataset, our approach considers these as weak saliency maps for the methodology. Usually, bounding boxes or segmentation masks are available as single channel images. To use them in our algorithm, we concatenate them along channel dimension to get the required dimension as image. After this pre-processing, we generate the weak saliency, $\tilde{\textbf{s}}$ from bounding boxes or segmentation masks as:
\begin{equation}
    \label{bbox-to-sal}
    \tilde{\textbf{s}}[i] =
    \begin{cases}
      1, & \text{if i\textsuperscript{th} pixel lies inside bbox or seg masks} \\
      -1, & \text{otherwise}
    \end{cases}
 \end{equation}
\noindent
As mentioned earlier, using these bounding boxes or segmentation masks as saliency maps allows us to perform adversarial training with no additional cost to compute perturbations, unlike existing adversarial training methods which can be compute-intensive.

\section{Experiments and Results}
\label{results}
In this section, we present our results using the proposed SAT method (Algorithm \ref{mimic-AT}) on multiple datasets with different variations of saliency maps. We begin with describing the datasets, evaluation criteria and implementation details.\\

\noindent {\large \textbf{4.1 Experimental Setup}}
\vspace{3pt}
\noindent \textbf{Datasets and Evaluation Criteria:} We perform experiments on well-known datasets: CIFAR-10, CIFAR-100 \cite{cifar-datasets}, Tiny ImageNet \cite{tiny-img} and FLOWER-17 \cite{flower}.
We evaluate the \textit{adversarial robustness} of trained models (model accuracy when perturbations from  following attacks are provided as inputs, as done in all earlier related efforts - also called \textit{adversarial accuracy}) using the popular and widely used PGD attack (described in Sec \ref{methodology}) with 5 steps and 4 different levels of $l_\infty$ perturbation $\epsilon_0 \in \{1/255, 2/255, 3/255, 4/255\}$. We also evaluate the robustness of our models against other attacks: TRADES \cite{trades} as well as uniform noise in Sec \ref{ablations}. We trained all our models for 5 trials and observed minimal variations in the values. We hence report the mean value of our experiments in our tables.
\vspace{3pt}

\noindent \textbf{Baselines:} We compare our method with multiple baseline methods, including those that train adversarially, as well as those that don't train adversarially:
\textit{(i) Original Model:} Model trained normally with no adversarial training; \textit{(ii) PGD-AT:} Model trained adversarially using 5-step PGD attack with max $l_\infty$ perturbation and $\epsilon = 8/255$, as in \cite{at@madry}; \textit{(iii) TRADES-AT:} Model trained adversarially using TRADES attack with max $l_\infty$ perturbation and $\epsilon = 8/255$, as in \cite{trades}; \textit{(iv) Original + Uniform Noise:} Model trained normally with no adversarial training, but training data is perturbed with uniform noise sampled from $[\frac{-8}{255},\frac{8}{255}]$ during training; \textit{(v) PGD + Uniform Noise:} PGD-AT, described in (ii), training data is additionally perturbed with uniform noise sampled from $[\frac{-8}{255},\frac{8}{255}]$ during training; \textit{(vi) TRADES + Uniform Noise:} TRADES-AT, described in (iii), training data is additionally perturbed with uniform noise sampled from $[\frac{-8}{255},\frac{8}{255}]$ during training.
The models are trained for 100 epochs with standard cross-entropy loss, minimized using Adam optimizer (learning rate = $1e-3$).
\vspace{3pt}

\noindent \textbf{Implementation Details:}
As in Sec \ref{methodology}, our method relies on being provided with a saliency map for the ground truth label of a given image, while training the model. For Tiny ImageNet and FLOWER-17, we simply use the bounding boxes and segmentation masks provided in the dataset (and left unexploited often) as `weak' saliency maps to train the model using the proposed SAT method.
This thus incurs no additional cost when compared to competing methods that perform adversarial training by generating perturbations (a costly operation when using methods such as PGD). For CIFAR-10 and CIFAR-100, since we do not have such information (bounding boxes/segmentation masks) provided, we obtain saliency maps using teacher networks. In particular, we train two teacher networks: Resnet-10 and Resnet-34 for this purpose. Our final model architecture in all these settings is a Resnet34 too.

\begin{wrapfigure}[14]{l}{0.5\textwidth}
\vspace{-20pt}
\includegraphics[width=0.5\textwidth]{./images/saliency3}
\vspace{-10pt}
\caption{Saliency maps of non-robust \textit{(top, model trained normally used to compute saliency)} and robust \textit{(bottom, model trained adversarially used to compute saliency)} variants of Resnet-10 with different explanation methods on CIFAR-10.}
\label{saliency}
\end{wrapfigure}
To go further, we also consider multiple variants of the above choices to study our method more carefully. It is believed that the quality of saliency maps generated by an adversarially trained model is better than its non-robust equivalent \cite{robustness-odds}. Figure \ref{saliency} supports the above claim. We hence train the aforementioned teacher models in two ways: Standard (denoted as \textit{Std} in the results) and Adversarial (denoted as \textit{Adv} in results) (regular PGD-based adversarial training), and use the saliency maps for the ground truth class. The saliency maps themselves are generated in two ways: we use bounding boxes obtained using Grad-CAM++ \cite{grad-cam++} as weak saliency maps, and we use the finer saliency maps generated as is by Guided Back-propagation (GBP) \cite{gbp} to train our student model. We set hyperparameters $\alpha = (0.6)^{\frac{1}{10}}$ and $\epsilon_0 = \frac{8}{255}$, and minimize cross-entropy loss using an Adam optimizer (learning rate = $1e-3$) for 200 epochs.

To complete the study, we also use variants of our method: \textit{PGD-SAT} and \textit{TRADES-SAT}, where we perturb the input randomly with either perturbations calculated by SAT or with PGD/TRADES attacks.
All these variants are named in a self-explanatory manner in our results: \textit{X | Y | Z}, where  \textit{X} denotes the teacher model (Resnet-10/-34), \textit{Y} denotes the mode of training (Std or Adv), and \textit{Z} denotes the saliency map method (GBP/ GradCAM++) used
In Sec \ref{ablations}, we also perform additional studies with other saliency methods like Guided-GradCAM++, Smooth-Grad and Integrated-Gradients to study the generalizability of our results on varying quality of saliencies.
\vspace{3pt}

\begin{table}
\begin{center}
\scalebox{0.7}{
\begin{tabular}{c|c|c|c|c|c|c|c|c}
\hline
\textbf{Method}  & \multicolumn{4}{c|}{\textbf{CIFAR-10}} & \multicolumn{4}{c}{\textbf{CIFAR-100}}\\

 & $\epsilon = 1/255$ & $\epsilon = 2/255$  & $\epsilon = 3/255$ & $\epsilon = 4/255$ & $\epsilon = 1/255$ & $\epsilon = 2/255$  & $\epsilon = 3/255$ & $\epsilon = 4/255$ \\
\hline \hline

                        Original & 47.71 & 10.36 & 1.39 & 0.28 & 25.83 & 7.76 & 3.35 & 1.94\\
                        Original + Uniform-Noise & 61.23 & 22.85 & 6.34 & 2.56 & 33.15 & 13.50 & 6.01 & 3.22\\
                        \cline{1-9}
                        \textbf{SAT (Weak saliency)} & & & & \\ \cline{1-1}
                          Resnet-10 | Std. | G.CAM++ & 59.61  & 22.12  & 6.16 & 1.77 & 31.98  & 11.93 & 5.48  & 2.89             \\
                          Resnet-10 | Adv. | G.CAM++ & 57.34  & 19.94 & 5.64 & 1.91 & 32.88  & 13.3 & 6.31 & 4.0      \\
                          Resnet-34 | Std. | G.CAM++  & 56.75 & 20.62 & 6.02 & 1.74 &  30.95 & 12.68 & 5.65 & 3.25           \\
                         Resnet-34 | Adv. | G.CAM++ & 60.0 & 22.94  & 6.58 & 1.96 & 32.96 & 12.07  & 5.2 & 3.04 \\
                           \cline{1-9}
                        \textbf{SAT (Fine saliency)} & & & & \\ \cline{1-1}
                         Resnet-10 | Std. | GBP & 10.87  & 0.95  & 0.0  & 0.0 & 20.53  & 7.52  & 3.5  & 2.12              \\
                         Resnet-10 | Adv. | GBP & \textbf{63.33}  & \textbf{26.79} & \textbf{9.62} & \textbf{3.69} & 34.29  & \textbf{14.73} & 6.84 & 4.22        \\
                         Resnet-34 | Std. | GBP  & 18.01 & 2.54 & 1.25 & 1.0 & 9.91 & 2.52 & 1.05 & 0.54             \\
                         Resnet-34 | Adv. | GBP & 62.67 & 23.76  & 5.82 & 1.28 & \textbf{34.71} & 14.32  & \textbf{7.08} & \textbf{4.29} \\

\hline
& & & & & & & & \\
\hline
                        PGD & 77.89 & 73.1 & 66.96 & 61.18 & 45.75 & 40.13 & 35.41 & 31.01 \\
                        PGD + Uniform-Noise & \textbf{82.23} & 74.97 & 65.61 & 55.04 & \textbf{42.67} & 36.10& 30.64 \\
                        \cline{1-9}
                        \textbf{PGD-SAT} & & & & \\
                        \cline{1-1}
                        Resnet-10 | Std. | GBP & 79.72 & 73.81 & 67.72 & 61.29 & 46.87 & 41.11 & 35.77 & 30.75   \\
                        Resnet-10 | Adv. | GBP & 80.72 & \textbf{75.07} & 68.68 & 62.49 & 48.33 & 42.2 & 36.33 & \textbf{31.66}         \\
                        Resnet-34 | Std. | GBP & 79.53 & 74.2 & 68.19 & 62.48 & 46.66 & 40.95 & 35.73 & 30.83 \\
                        Resnet-34 | Adv. | GBP & 80.15 & 74.60 & 68.47 & 62.53  & 47.38 & 42.0 & \textbf{36.34} & 31.53     \\
                        Resnet-10 | Adv. | G.CAM++ & 79.67 & 74.05 & 68.12 & 61.84  & 47.28 & 41.72 & 35.99 & 31.05      \\
                        Resnet-34 | Adv. | G.CAM++  &79.74 & 74.5 & \textbf{68.87} & \textbf{62.68} & 46.12 & 40.47 & 35.31 & 30.81\\
\hline
& & & & & & & & \\
\hline
                          TRADES & \textbf{84.0} & 73.25 &	59.79 &	47.03 & 47.21 & 42.2 & 37.03 & 32.96 \\
                          TRADES + Uniform-Noise & 81.69 & 74.96 &67.43 & 60.05 & \textbf{51.90} & 42.85 & 37.30 & 31.71\\
                          \cline{1-9}
                          \textbf{TRADES-SAT} & & & & \\ \cline{1-1}
                        Resnet-10 | Std. | GBP & 80.15 & 75.2 & 69.0 & 63.2 & 48.63 & 42.91 & 37.79 & 33.19 \\
                          Resnet-10 | Adv. | GBP & 80.65 & 75.38 & 69.28 & \textbf{63.46}  & 48.74 & 42.99 & 37.83 & 33.23 \\
                          Resnet-34 | Std. | GBP & 79.98 & 74.5 & 68.56 & 62.43 & 48.5 & 42.41 & 36.93 & 32.01 \\
                          Resnet-34 | Adv. | GBP & 80.26 & 74.87 & 68.75 & 62.74   & 48.76 & 42.83 & 37.36 & 32.87   \\
                          Resnet-10 | Adv. | G.CAM++ & 79.85 & 74.61 & 69.07 & 63.2 & 48.99 & 43.05 & 37.4 & 32.84      \\
                         Resnet-34 | Adv. | G.CAM++  & 83.17 & \textbf{77.18} & \textbf{70.27}	& 62.87 & 49.35 & \textbf{43.62} & \textbf{38.53} & \textbf{33.93}

                           \\ \hline

\end{tabular}

}
\end{center}

\caption{Results on CIFAR-10 and CIFAR-100 using saliency maps from different teachers and explanation methods. $\epsilon \in \{1/255,2/255,3/255,4/255\}$ denotes the maximum $l_\infty$ perturbation allowed in 5-step PGD attack (More the $\epsilon$, stronger the attack). GBP:  Guided-Backpropagation; G.CAM++: Grad-CAM++.}
\label{table:cifar-10}
\end{table}

\noindent {\large \textbf{4.2 Results}}

\noindent Tables \ref{table:cifar-10} and \ref{table:bbox} present our results on CIFAR-10 + CIFAR-100 and Tiny ImageNet + Flower-17 datasets respectively. We note again that the provided bounding boxes are used as saliency maps in Tiny ImageNet, and segmentation masks provided in Flower-17 are used as saliency maps. We note that with no additional cost of computing perturbations, we obtain improvement of $~2-4\%$ in robustness. Table \ref{table:cifar-10} shows the results for CIFAR-10 and CIFAR-100, and demonstrates the potential of using saliency maps for adversarial training. Barring 1-2 cases, the proposed SAT (or its combination with an adversarial training method such as PGD or TRADES) performs better than respective baselines across the results. As the adversarial attack gets stronger (larger $\epsilon$), adding SAT to existing methods significantly improves robustness performance (~1-3\% in adversarial accuracy).
\begin{table}
\begin{center}
\scalebox{0.9}{
\footnotesize
\begin{tabular}{c|c|c|c|c|c|c}
\hline
\textbf{Method}  & \multicolumn{3}{c|}{\textbf{Tiny-Imagenet}} & \multicolumn{3}{c}{\textbf{FLOWER-17}} \\
 & $\epsilon=1/255$ & $\epsilon=2/255$  & $ \epsilon=3/255$  & $\epsilon=1/255$ & $\epsilon=2/255$  & $ \epsilon=3/255$ \\
\hline \hline
                        Original & 1.04 & 0.4 & 0.0  & 63.2 & 48.01 & 34.2  \\
                        Original + Uniform-Noise & 9.45 & 2.32 & 0.77 & 64.56 & 50.43 & 36.2 \\
                        SAT & \textbf{9.79}  & \textbf{2.46} & \textbf{0.77} & \textbf{66.17}  &\textbf{ 52.94} & \textbf{38.93} \\
                        \cline{1-7}
                        PGD & 18.91 & 14.34 &11.37 & 72.38 & 70.4 & 70.3 \\
                        PGD + Uniform-Noise  & 19.57 & 15.49 & 11.66 & 73.52 & 72.79 & 72.71 \\

                        PGD-SAT  & \textbf{20.56} & \textbf{16.38} & \textbf{12.91} & \textbf{78.67} & \textbf{75.73} & \textbf{75.00} \\
                        \cline{1-7}
                        TRADES & 18.45 & 16.76 &	11.09& 74.56 & 73.89 &	73.67 \\
                        TRADES + Uniform-Noise & 19.96 & 16.13 & 12.58 & 76.47 & 74.26 & 74.0 \\
                        TRADES-SAT& \textbf{20.04} & \textbf{16.45} & \textbf{12.96} & \textbf{79.41} & \textbf{77.94} & \textbf{77.20} \\

\hline
\end{tabular}}
\end{center}
\caption{Results on Tiny-Imagenet and Flower-17 datasets where bounding boxes and segmentation masks provided with the dataset are used as saliency maps, respectively. $\epsilon \in \{1/255,2/255,3/255\}$ denotes the maximum $l_\infty$ perturbation allowed in 5-step PGD attack (More the $\epsilon$, stronger the attack).}
\vspace{-25pt}
\label{table:bbox}
\end{table}
It is also evident from Table \ref{table:cifar-10} that using finer saliency maps (GBP in our results) obtains better performance than weaker saliency maps (bounding boxes obtained from GradCAM++, in our case). This supports the inference that a stronger saliency map provides better adversarial robustness. We also notice, as pointed out earlier in this section (Fig \ref{saliency}), that using saliency maps obtained from an adversarially trained teacher leads to significant gains in robustness/adversarial accuracy performance. In other words, all our experiments point to the inference that better the saliency maps, better the adversarial robustness of the model trained using our approach. This further supports our inherent claim that saliency maps do provide adversarial robustness.

Table \ref{table:cifar-10} also shows that PGD-SAT and TRADES-SAT leads to improvement ($~1.5 - 3\%$) over vanilla PGD and TRADES adversarial training. This shows that adding our saliency-based method to existing adversarial training method further improves their performance. We observe here again that using saliency maps of an adversarially robust teacher model proves more useful than those obtained from a normal teacher model.

\begin{wrapfigure}[14]{r}{0.4\textwidth}
\includegraphics[width=0.4\textwidth]{./images/time3}
\caption{Training time for one epoch in seconds (averaged over 10 trials) for different methods considered in our results on Tiny-Imagenet and Flower-17.}
\label{fig:train-time}
\end{wrapfigure}
\noindent \textbf{Time Efficiency:}
We further analyzed the training efficiency of the proposed method. Figure \ref{fig:train-time} reports the average time taken by one epoch over 10 trials. As can be seen, PGD-SAT and TRADES-SAT require only $\approx 50-70\%$ of training time when compared to vanilla PGD and TRADES respectively, and at the same time, achieves superior performance (as in Table \ref{table:bbox}). In case of vanilla SAT, the behavior is desirable since we observe an increase in robustness without compromising much in training time. We assume in these results that explanations are provided to us, inspiring the demand for finer saliency map annotations to be included in vision datasets to the community. Similar results for CIFAR-10 and CIFAR-100 are included in the appendix.

\section{Discussions, Ablation Studies and Conclusions}
\label{ablations}
We carried out ablation studies to better characterize the efficacy of our methodology, and present these results in this section. Unless explicitly specified, these experiments are carried out on CIFAR-100 using GuidedBackprop and GradCAM++ explanations of a standard and adversarially trained Resnet-10 teacher.\\

\begin{wrapfigure}[11]{l}{0.5\textwidth}
\vspace{-20pt}
\includegraphics[width=0.5\textwidth]{./images/hyper}
\vspace{-20pt}
\caption{Variation of hyper-parameter $\alpha$ and $\epsilon_0$ on CIFAR-100 dataset.}
\label{fig:variation-alpha}
\end{wrapfigure}

\noindent \textbf{Varying Hyperparameters:} We studied the effect on the adversarial robustness of our trained model by varying the hyperparameters $\alpha$ and $\epsilon_0$ in our method. Fig \ref{fig:variation-alpha} shows these results, where all models are evaluated using a 5-Step PGD attack with $l_\infty$ norm and maximum perturbation $\epsilon \in \{\frac{1}{255}, \frac{2}{255}, \frac{3}{255} \}$.
We achieve higher robustness when $\alpha^{10}$ is set closer to 1 ($\alpha$ close to 1), which from Eqn \ref{pert-calc} indicates that the saliency map is useful to obtain perturbations in later stages of training, as explained before. Interestingly, when $\alpha^{10}$ is close to  0 ($\alpha$ close to 0), the training doesn't include any noise factor, and since we have a fixed saliency map for each image, it becomes equivalent to training on an additively shifted version of the original training data resulting in a less robust model.
While varying $\epsilon_0$ in Algorithm \ref{mimic-AT}, we observe a peak in robustness somewhere in the middle of the considered range of values. In Algorithm \ref{mimic-AT}, the distribution of our estimated perturbations and actual adversarial perturbations depend on the hyperparameter $\epsilon_0$. At $\epsilon_0 = 0$, the distributions are identical, and the distributions diverge as $\epsilon_0$ increases. Hence, at high values of $\epsilon_0$, when both distributions diverge, the perturbations used by SAT no longer resemble true adversarial perturbations, hence resulting in less robust models (when attacked adversarially using methods associated with the adversarial perturbations). In other words, the perturbations provided by saliency maps in our method has reasonable conjunction with standard adversarial perturbations, as shown in Fig \ref{fig:sal-attack}.

\begin{wraptable}[16]{l}{0.5\textwidth}
\vspace{-20pt}
\scalebox{0.65}{
\begin{tabular}{c|c|c|c|c}
\hline
\textbf{Method}  & \multicolumn{4}{c}{\textbf{PGD}}  \\
 & $\frac{1}{255}$ & $\frac{2}{255}$ & $\frac{3}{255}$ & $\frac{4}{255}$  \\
& & & &  \\
\hline \hline
Original  & 25.83 & 7.76 & 3.35 & 1.94 \\
Original + Uniform-Noise & 33.15 & 13.50 & 6.01 & 3.22 \\
\cline{1-5}
\textbf{SAT} & & & & \\ \cline{1-1}
Resnet-10 | Std. | GBP  & 20.53 & 7.52 & 3.5 & 2.12  \\
Resnet-10 | Std. | S.Grad  & \textbf{39.22} & \textbf{19.89} & \textbf{9.44} & \textbf{4.49}  \\
Resnet-10 | Std. | G.G.CAM++ & 21.46 & 8.00 & 3.53 & 2.23  \\
Resnet-10 | Std. | I.Grad & 36.2 & 5.43 & 7.28 & 3.37  \\

Resnet-10 | Adv. | GBP  & 34.29 & 14.73 & 6.84 & 4.22  \\
Resnet-10 | Adv. | S.Grad  & \textbf{40.01} & \textbf{21.2} & \textbf{10.96} & \textbf{4.85}  \\
Resnet-10 | Adv. | G.G.CAM++ & 34.07 & 13.18 & 5.85 & 3.09  \\
Resnet-10 | Adv. | I.Grad & 37.56 & 16.45 & 7.55 & 4.31 \\
                           \hline
\end{tabular}}
\caption{\footnotesize Results using saliency maps from Guided-Backprop (GBP), Smooth-Grad (S.Grad), Integrated-Gradients (I.Grad), and Guided-Grad-CAM++ (G.G.CAM++)}
\label{table:better-exp}
\end{wraptable}

\noindent \textbf{Robustness against Other Attacks:}
In Table \ref{table:cifar-10}, we evaluate our models using the widely used PGD attack. We now study how the proposed method works against other attacks - in particular, Uniform Noise attack, TRADES \cite{trades} attack and a Saliency-based adversarial attack, which is an attack using saliency maps generated by our method.
These results are shown in Table \ref{table:diff-pert}. It is clear that SAT and PGD-SAT outperform or have comparable performance to (in case of Uniform Noise) competing methods. Expectedly, the improvement is much more significant in case of Saliency-based adversarial attack, since the method is closely associated with the attack.

\begin{table}
\centering
\begin{center}
\scalebox{0.8}{
\footnotesize
\begin{tabular}{c|c|c|c|c|c|c|c|c|c}
\hline

\textbf{Method} &  \multicolumn{3}{c|}{\textbf{Uniform-Noise}} & \multicolumn{3}{c|}{\textbf{TRADES}} &
      \multicolumn{3}{c}{\textbf{Saliency Attack}} \\
         & $\epsilon=\frac{8}{255}$ & $\epsilon=\frac{16}{255}$ & $\epsilon=\frac{32}{255}$ & $\epsilon=\frac{2}{255}$ & $\epsilon=\frac{3}{255}$ & $\epsilon=\frac{4}{255}$  & $\epsilon=\frac{4}{255}$ & $\epsilon=\frac{8}{255}$ & $\epsilon=\frac{16}{255}$  \\
        &  &  &  & &  &  &  &  &  \\

\hline \hline
Original & 62.21 & 47.88 & 21.17 &  39.59
                            & 35.1 & 33.45  & 13.03 & 6.86 & 3.92
                            \\
Original + Uniform-Noise  & \textbf{67.42} & \textbf{60.75} & 28.9 &  \textbf{46.98}
                            & 39.37 & 37.8 & 19.63 & 10.48 & 6.1
                            \\
\hline
\textbf{SAT} &  & &  &  & & & & & \\ \cline{1-1}
Resnet-10 | Std | GBP & 56.63 & 57.4 & \textbf{46.9} & 36.45 & 33.35 & 32.46  & 51.78 & 49.02 & 42.75             \\

Resnet-10 | Adv. | GBP  & 66.63 & 57.93 & 21.92  & 45.93 & \textbf{40.80} & \textbf{37.86} &  62.99 & 56.48 & 43.49              \\

Resnet-10 | Adv. | G.CAM++  & 64.00 & 54.84 & 23.43 & 44.31 & 38.28 & 35.71 & \textbf{63.86} & \textbf{63.87} & \textbf{63.66}               \\
                           \hline \hline
PGD  & 50.96 & 51.0 & 48.69  & 49.03 & 47.63 & 45.34 & 44.03 & 34.77  & 21.04             \\
PGD + Uniform-Noise   & \textbf{55.6} & 52.45 & 49.20 & \textbf{52.32}
                            & 48.76 & 45.87  & 45.72 & 32.81 & 16.62
                            \\
\hline
\textbf{PGD-SAT} &  & &  &  & & & & &  \\ \cline{1-1}
Resnet-10 | Std | G.BP  & 52.78 & 52.56 & 49.21  & 50.43 & 47.69 & 46.24 &  47.25 & 46.74 & 44.95             \\
Resnet-10 | Adv. | G.BP   & 54.38 & \textbf{54.13} & \textbf{50.30}  & 51.58 & \textbf{49.15} & \textbf{46.83}   & 50.61 & 50   & 47.64            \\
Resnet-10 | Adv. | G.CAM++  & 53.30 & 52.76 & 49.74  & 50.87 & 48.42 & 46.31  &  \textbf{50.69} & \textbf{50.63}  & \textbf{50.54}            \\
                           \hline
\end{tabular}
}
\end{center}
\caption{Results against other attacks: Uniform-Noise, TRADES and Saliency. \textit{(GBP=  Guided-Backprop; G.CAM++=Grad-CAM++; $l_\infty $ perturbation $\in [-\epsilon, \epsilon]$.)}}
\vspace{-25pt}
\label{table:diff-pert}
\end{table}

\noindent \textbf{Using Other Saliency Maps:}
We also performed a study where we trained our models using stronger saliency maps obtained using methods such as SmoothGrad \cite{smooth-grad}, Guided Grad-CAM++ \cite{grad-cam++} and Integrated Gradients \cite{integrated-gradients} from teacher models. These models are evaluated using a 5-Step PGD attack with $l_\infty$ max-perturbation $\epsilon \in \{\frac{1}{255}, \frac{2}{255}, \frac{3}{255}, \frac{4}{255}\}$. As can be seen from Table \ref{table:better-exp}, we tend to achieve more robust models for better saliency maps.\\

\noindent \textbf{Conclusion.} In summary, this work explores the interesting connection between saliency maps and adversarial robustness to propose a Saliency based Adversarial training (SAT) method. SAT imitates adversarial training by using saliency maps to mimic adversarial perturbations. In particular, our methodology allows the use of annotations such as bounding boxes and segmentation masks to be exploited as weak explanations to improve model's robustness with no additional computations required to compute the perturbations themselves. Our results on CIFAR-10, CIFAR-100, Tiny ImageNet and Flowers-17 corroborate our claim. We further gain improvement over popular adversarial training methods by integrating SAT with them (PGD-SAT and TRADES-SAT). Further, our work shows how using better saliency maps leads to more robust models. Our effort opens rather a new direction to enhance robustness of DNNs by exploiting saliency maps, and inspires the need for strong saliency maps to be provided with vision datasets, which helps train adversarially robust models with little overhead. Our future work includes ways to improve SAT by reasoning about the class closest to the ground truth in terms of decision boundary, and improving our estimates of adversarial perturbations.

{\bibliographystyle{splncs04}