\section{Introduction}

\begin{figure}
\centering
\includegraphics[width=0.95\columnwidth]{figure/schema2_reduced.pdf}
\caption{Illustration of the proposed approach.}
\label{fig:schema-approach}
\end{figure}

Deep Neural Networks (DNNs) have caught a lot of attention in recent years thanks to their capability to solve efficiently various tasks, especially in computer vision \cite{Dargan2019ALearning}. However, a common pitfall of these models is that they are vulnerable to adversarial examples, i.e. misclassified examples that result from slightly altering a well-classified example at test time. For instance, it was shown that adding carefully chosen, elusive perturbations to a correctly classified image may cause misclassification \cite{Biggio2013, Szegedy2013}. This constitutes a critical security threat for any system embedding DNNs, as a malicious third party may exploit this property to enforce some desired outcome.

Such \emph{adversarial attacks} have been primarily designed in white-box settings, where the attacker is assumed to have complete knowledge of the target DNN (incl. its weights). While studying such worst-case scenarios is essential for proper security assessment, in practice the attacker should have limited, or even no knowledge of the target model. In such a case the adversarial attack is applied to a surrogate model, with the hope that the produced adversarial examples \emph{transfer to} (i.e., are also misclassified by) the target DNN.

Achieving transferability remains challenging, though. This is because adversarial attacks were designed to exploit specific information of the surrogate model (e.g., the gradient of its loss function \cite{Goodfellow2014ExplainingExamples, Kurakin2019AdversarialWorld}), which are different in the target model. As a result, \citet{Liu2017DelvingAttacks} improved transferability by attacking an \emph{ensemble} of architectures. The key intuition is that adversarial examples that fool a diverse set of models are more likely to generalize.

While ensemble-based attacks typically report significantly higher success rates than their single-model counterparts (e.g. \citet{Liu2017DelvingAttacks}), their computational cost is prohibitive due to the necessity to independently train several surrogate models (to form a diverse ensemble).

In this paper, we propose a new method to improve the transferability of adversarial examples by approximately sampling from the posterior distribution of neural network weights -- and do so with lesser computation overhead compared to ensemble-based methods. Our approach, shown in Figure \ref{fig:schema-approach}, leans upon recent results in Bayesian Deep Learning. More precisely, we apply a cyclical variant of Stochastic Gradient Markov Chain Monte Carlo (i.e., \emph{cSGLD} \cite{Zhang2020CyclicalLearning}) during model training to sample from the posterior distribution on model parameters. We perform efficient Bayesian model averaging during the attack, while the cyclical nature of cSGLD can sample from different modes of the weight posterior distribution \cite{Zhang2020CyclicalLearning}.

Another advantage of our method is that it requires minimal modifications of the adversarial attack algorithms and the surrogate model. Thus, it can be applied on top of any attack. We show, moreover, that our approach is complementary to other heuristics to improve transferability.

We evaluate our approach on the CIFAR-10 and the ImageNet datasets (with 5 DNN architectures each), four adversarial attack algorithms, and three test-time transformations. Overall, our results indicate that applying cSGLD significantly improves the transfer rate compared to training single DNNs and outperforms classical ensemble-based attacks both in terms of transferability rate and computation cost. Deep Ensemble requires at least 2.51 times more flops to achieve the same success rates than cSGLD when the targeted architecture is known. This can represent, on ImageNet, a saving of 3.56 exaflops (2.36 vs 5.92). At the same computation budget, our method increases the intra-architecture transfer rates between 12.1 and 49.2 percentage points and the inter-architecture transfer rates between -2.3 and 62.1 percentage points at constant computation costs. cSGLD always raises the effectiveness of test-time techniques designed for transferability between 3.8 and 56.2 percentage points. Applied alone, it is more effective than these techniques applied to a single DNN in 105/120 cases.

To summarize, our contributions are:
\begin{itemize}
    \item We relate epistemic uncertainty and transfer-based attacks to propose the first method based on a Bayesian Deep Learning technique to generate transferable adversarial examples. Our approach differs from previous research as it can approximately sample from the posterior distribution within a single training process and does not require post-training modification of the surrogate model.
    \item Our generic method paves the way for improving various adversarial methods by considering the training cost. It applies to any DNN architecture and attack algorithms. Moreover, it is compatible with other heuristics to improve the efficiency and transferability of adversarial attacks (e.g., \citet{Liu2017DelvingAttacks}).
    \item Our evaluation on CIFAR-10 and ImageNet reveals significant improvements over the single-DNN and Deep Ensemble baselines. Compared to classical ensembles trained with the same computation budget, our approach increases transfer rate by up to 49.2 (intra-architecture) and 62.1 (inter-architecture) percentage points.
    \item We explore the use of other techniques to improve the training surrogate and show that cSGLD is a strong competitor, though other techniques open promising avenues.

\end{itemize}
\section{Background and Related work}
\label{sec:related}

\paragraph{Adversarial Attacks.} We consider 4 gradient-based attacks which aims to maximise the prediction loss $L(x, y, \theta)$ with a $p$-norm constraint: $ \argmax_{\| \delta \|_p \leq \varepsilon} L(x + \delta, y, \theta) $. FGSM \cite{Goodfellow2014ExplainingExamples} is a L$\infty$ single-step attack defined by $\delta_{\text{FGSM}} = \varepsilon \, \text{sign}(\nabla_x L(x, y, \theta)) $. FGM is its L$2$ equivalent: $\delta_{\text{FGM}} = \varepsilon \, \frac{\nabla_x L(x, y, \theta)}{ \| \nabla_x L(x, y, \theta) \|_2 } $. The resulting adversarial example is then clipped in $[0,1]$. I\nobreakdash-FGSM \cite{Kurakin2019AdversarialWorld} applies iteratively the fast gradient attack with a small step-size $\alpha$: $\delta_0 = 0$ and $ \delta_{i+1} = \text{proj}_{B_\varepsilon}(\delta_i + \alpha \, \text{sign}(\nabla_x L(x, y, \theta))) $, where $\text{proj}_{B_\varepsilon}(\bullet)$ projects the perturbation in the L$p$ ball of radius $\varepsilon$. The L$2$ I-FGM attack is derived similarly. MI-FGSM attack \cite{Dong2018} adds a momentum term with decay factor $\mu$ to the previous attack: $g_{i+1} = \mu \, g_t + \frac{\nabla_x L(x, y, \theta)}{ \| \nabla_x L(x, y, \theta) \|_1 }$ and $ \delta_{i+1} = \text{proj}_{B_\varepsilon}(\delta_i + \alpha \, \text{sign}(g_{t+1})) $. PGD \cite{Madry2018TowardsAttacks} adds random restarts to I-FG(S)M where each $\delta_0$ is sampled uniformly inside the ball $B_\varepsilon$. The relations between these attacks are illustrated in supplementary materials.

\paragraph{Ensemble Surrogate.} \citet{Liu2017DelvingAttacks} show the benefit of ensembling architectures for inter-architecture black-box attacks.
Our work leans on theirs and complements it by demonstrating that attacking models sampled with cSGLD (performing Bayesian model averaging on a unique architecture) achieves better transferability at lesser computation cost.

\paragraph{Input and Model Transformations.} Other approaches have been developed to
improve transferability of adversarial examples. These work at test time (i.e., after training, when performing the attack) and consist of transforming the model or the input. \textit{Ghost Networks} (GN) \cite{Li2018LearningNetworks} use Dropout and Skip Connection Erosion to generate on-the-fly diverse sets of surrogate models from one or more base models. \textit{Input Diversity} (DI) \cite{Xie2019} applies random transformations (random resize followed by random padding) to the input images at each attack iteration. \textit{Skip Gradient Method} (SGM) \cite{Wu2020SkipResNets} favors the gradients from skip connections rather than residual modules through a decay factor applied to the latter during the backward pass.
These techniques can naturally be combined to ours: (i) cSGLD can provide at a low computation cost a diverse set of base models to build GN;
(ii) DI applies transformations to adversarial inputs independently of the surrogate models;
(iii) SGM modifies backward passes during the attack, independently of the training method.
As our evaluation will reveal, our train-time method further improves the transferability of the above three techniques and outperforms them 87.5\% of the time. It is also compatible with other test-time approaches not considered in this paper, such as
linear backpropagation \cite{Guo2020a}, intermediate level attack \cite{Huang2019EnhancingAttack}, Nesterov accelerated gradient and scale invariance \cite{Lin2019NesterovAttacks}, and serial mini-batch ensemble attack \cite{Che2020AMemories.}.

\paragraph{Bayesian Neural Network (BNN) and adversarial examples.}
Though not our goal, past research aimed at generating adversarial examples for BNNs (we rather use Bayesian Deep Learning as a way to attack classical DNNs). \citet{Grosse2018} show that BNN uncertainty measures are vulnerable to high-confidence-low-uncertainty adversarial examples transferred from Gaussian Processes. \citet{Palacci2018ScalablePractice} show that several SG-MCMC sampling schemes are not secure against white-box attacks. \citet{Wang2018} used SGLD and Generative Adversarial Network to detect adversarial examples instead of crafting them.

The recent work of \citet{Carbone2020RobustnessAttacks} shows that in the large-data, overparametrized limit for BNN, the gradient of the posterior predictive distribution reaches zero, which makes white-box gradient-based attacks unsuccessful. Our experiments reveal opposite conclusions: our attacks on surrogates DNNs suffer more often from zero gradients than on BNN surrogates. Our setting differs from \citet{Carbone2020RobustnessAttacks} in several aspects: (i) the attacker controls the BNN surrogate and can exploit the cyclical nature of cSGLD to avoid global convergence; (ii) our cheaper SG-MCMC approximation of the posterior predictive distribution (compared to the Hamiltonian Monte Carlo \& Variational Inference used in \citet{Carbone2020RobustnessAttacks}) may produce correlated MCMC samples, which explains that the averaged gradient is not zero; (iii) we consider larger datasets (ImageNet and CIFAR-10) than them (MNIST and Fashion-MNIST), where convergence to flat loss is harder to reach.

\section{Approach}
\label{sec:approach}

\noindent\textbf{A Bayesian perspective on transferability.} We consider a classification problem with a training dataset $\mathcal{D} = \{ (x_i, y_i) \sim p(x, y) \}_{i=1}^N$ and $C$ class labels. A probabilistic classifier parametrized by $\theta$ maps $x_i$ into a predictive distribution $\hat p(y | x_i, \theta)$. A white-box adversarial perturbation of a test example $(x,y) \sim p(x, y)$ against such classifier is defined as:
$$\delta_\theta = \argmin_{\| \delta \|_p \leq \varepsilon} \hat p(y | x+\delta, \theta).$$
In practice, this optimization problem is solved by replacing the predictive distribution with a loss function (see Background section).
The \textit{transferability} phenomenon was formulated as the empirical observation that an adversarial example for one model is likely to be adversarial for another one \cite{Goodfellow2014ExplainingExamples}.
Black-box attacks can leverage this property by crafting adversarial examples using white-box attacks against a surrogate model to target an unseen model \cite{Papernot2016}.

\begin{assumption}[Threat model]
\label{assum:threat-model}
We define our threat model with the following assumptions on the targeted classifier:
\begin{enumerate}
    \item its architecture is known and so is its prediction function $\hat p(y | x, \bullet)$ \footnote{We discuss the unknown architecture case further on.}
    \item its training set $\mathcal{D}$ is known
    \item its parameters $\theta_t$ are unknown
    \item a reasonable prior on its parameters $p(\theta_t)$ is known\footnote{In practice, it corresponds to knowing the weight decay hyperparameter as used classically to train DNNs.}
    \item no oracle access (test-time feedback).
\end{enumerate}
\end{assumption}

In black-box settings, the targeted classifier is unknown. Assuming the threat model described in Assumption~\ref{assum:threat-model}, we can treat $\theta_t$ as a random variable s.t. $\theta_t \sim p(\theta | \mathcal{D})$. Then, \textit{the best transferable adversarial example minimizes the true Bayesian posterior predictive distribution} $p(y | x, \mathcal{D}) =  \E_{p(\theta | \mathcal{D})} p(y | x, \theta)$
and our black-box attack objective is:

\begin{equation}\label{eq:delta-star}
\delta^* = \argmin_{\| \delta \|_p \leq \varepsilon} \E_{\theta_t \sim p(\theta  | \mathcal{D})} \hat p(y | x+\delta, \theta_t).
\end{equation}

Transferable adversarial examples are fundamentally related to epistemic uncertainty (or knowledge uncertainty). Traditionally, it captures the ignorance about the ``true" model used to generate the data. Here it represents our ignorance about the target model. In both cases, this uncertainty decreases with an increasing amount of data. While previous research \cite{Liu2017DelvingAttacks,Li2018LearningNetworks,Xie2019} has related diversity with transferability, our work differs in that we do not seek to enrich the space of hypotheses but ``to reflect a statistical inability to distinguish the hypothesis based on limited data". This is the same conceptual difference between Ensembling and Bayesian model averaging \cite{MinkaThomasP.2002BayesianCombination}.

As previously stated, usually in adversarial learning, transferable adversarial examples are optimized against one surrogate model. This is similar to solving problem (\ref{eq:delta-star}) in a deterministic way by approximating the expectation of the predictive posterior with a ``plug-in'' estimation of the parameters, $\hat \theta_\text{MAP}$ the maximum a posteriori probability (MAP) estimate: $\delta^* \approx \delta_{\hat \theta_\text{MAP}}$. To avoid overfitting to the surrogate model, random transformations of inputs or prediction functions were developed in the literature (see related work section).

A fundamental issue is that the closed form of the posterior predictive distribution is intractable for DNNs. Our contribution lies in \textit{approximately sampling from the posterior distribution to build a surrogate in black-box adversarial attacks}. We replace the crude MAP approximation of the predictive posterior distribution with a more accurate one to generate transferable adversarial examples. Therefore, we focus on the training phase by considering the methods and the computational costs of obtaining the surrogate model, whereas most previous work searches to optimize adversarial examples crafting at the time of the attack (``test time'').

\textbf{SG-MCMC \& cSGLD.} In practice, we perform Bayesian model averaging using samples obtained from Stochastic Gradient-Markov Chain Monte Carlo (SG-MCMC).
SG-MCMC is a family of optimization techniques, inaugurated by SGLD \cite{Welling2011a}, that combines Stochastic Gradient Descent (SGD) with MCMC. By adding noise during training, it allows us to sample approximately from the posterior distribution of parameters. Our method aims to solve the following optimization problem:
\begin{equation}\label{eq:delta-star-bayesian}
\delta_{\{\theta_s\}} = \argmin_{\| \delta \|_p \leq \varepsilon} \frac{1}{S} \sum_{s=1}^S \hat p(y | x+\delta, \theta_s)
\end{equation}
where $\{\theta_s\}_{s=1}^S$ are samples of the model parameters.

We choose to apply the recently proposed \textit{cyclical Stochastic Gradient Langevin Dynamics} (cSGLD) \cite{Zhang2020CyclicalLearning}, a state-of-the-art SG-MCMC technique. cSGLD divides the training into cycles that all start from the initial learning rate value (cf. illustration in supplementary materials). Each cycle is composed of (1)~an exploration stage with larger learning rates which corresponds to the warmup period of MCMC algorithms; (2)~a sampling stage that samples parameters at regular intervals and operates with smaller learning rates and added noise. A major advantage of cSGLD is that its computation overhead compared to SGD/Adam is negligible (0.019\% flops for one epoch on PreResNet110 on CIFAR-10 and 0.015\% for ResNet50 on ImageNet).

cSGLD can be viewed as SG-MCMC with warm restarts. Starting a new cycle with a large learning rate allows the exploration of another local maximum of the loss landscape. cSGLD has the compelling advantage of sampling from both several modes of the posterior distributions on parameters and locally inside each mode. \citet{Ashukha2020PitfallsLearning} show that this technique produces one of the best results among ensemble methods considering both the quality of the ensemble and the computational cost.

\textbf{Extension to unknown architecture.} Let $\mathcal{A}  = \{a_i\}_i$ be a countable set of architectures, $p(a)$ a prior on $\mathcal{A}$,
$\theta^a $ the parameters of the architecture $a$ and $\hat p^a(y|x,\theta^a)$ its predictive distribution. Discarding hypothesis 1 of Assumption~\ref{assum:threat-model} on the knowledge of the architecture, we can consider the architecture of the target $a$ to be a random variable, and the true predictive posterior distribution can be decomposed with the law of total expectation:

\begin{equation}\label{eq:pred_posterior-interarch}
p(y | x, \mathcal{D}) = \E_{p(a)} \E_{p(\theta^a | \mathcal{D})} \hat p^a(y | x, \theta^a)
\end{equation}

If $\mathcal{A}$ is finite and small, we can approximate this quantity with a weighted average of one cSGLD predictive posterior approximation per architecture. More realistically, we sample according to $p(a)$ a finite subset $A = \{ a_i \sim p(a) \}_{i=1}^{S_A} \subset \mathcal{A}$ of architectures where the number of architectures $S_A$ is fixed by the computational budget. We sample $S$ parameters $\{ \theta^a_s \}_{s=1}^S$ for all $a \in A$. Then, our inter-architecture attack that minimizes our approximation of $p(y | x, \mathcal{D})$ becomes:

\begin{equation}\label{eq:delta-star-interarch}
\delta_A = \argmin_{\| \delta \|_p \leq \varepsilon} \frac{1}{S_A} \sum_{a \in A} \frac{1}{S} \sum_{s=1}^S \hat p^a(y | x+\delta, \theta^a_s)
\end{equation}

\textbf{Attack algorithm.} One can approximate the solution of Equations \ref{eq:delta-star-bayesian} and \ref{eq:delta-star-interarch} by applying existing adversarial attack algorithms with minor modifications.
To efficiently perform Bayesian model averaging during iterative attacks, we compute the gradient of every iteration on a single model sample per architecture. If multiple architectures are attacked, we average their gradients (see the algorithm description in supplementary materials). The cost of iterative attacks, measured as the number of backward passes, does not increase with the number of samples per architecture.

\subsection{Experimental Setup}

\noindent \textbf{Datasets.} We consider CIFAR-10 \cite{Krizhevsky2009LearningImages} and ImageNet (ILSVRC2012; \citealt{ILSVRC15}). In all cases, we train the surrogate and target models on the entire training set. For each CIFAR-10 target model, we select all the examples from the test set that are correctly predicted by it. In the case of ImageNet, we use a random subset of 5000 correctly predicted test images.

\noindent \textbf{Architectures.} Following the work of \citet{Ashukha2020PitfallsLearning}, we consider the following five architectures for CIFAR-10: PreResNet110, PreResNet164 \cite{He2016IdentityNetworks}, VGG16BN, VGG19BN \cite{Simonyan2015VeryRecognition}, and WideResNet28x10 \cite{Zagoruyko2016WideNetworks}. For ImageNet, we select five architectures with $3\times224\times244$ input size. Three classical architectures: ResNet-50 \cite{He2016}\footnote{\citet{Ashukha2020PitfallsLearning} study ResNet-50 only on ImageNet. We used their shared trained models as surrogate DNNs.}, ResNeXt-50 32x4d \cite{Xie2017} and Densenet-121 \cite{Xie2017}; and 2 mobile architectures: MNASNet 1.0 \cite{Tan2018} and EfficientNet-B0 \cite{Tan2019}.
In doing so, we cover a diverse set of architectures in terms of heterogeneity (similar and different families of architecture), computation cost, and release date.

\noindent \textbf{Target models} The target models are deterministic DNNs. In the case of CIFAR-10, they are trained using Adam optimizer for 300 epochs with step-wise learning rate decay that divides it by 10 every 75 epochs (the architecture depends on the experiments). For ImageNet, we use the pre-trained models provided by PyTorch \cite{NEURIPS2019_9015} and the pre-trained EfficientNet-B0 provided by PyTorch Image Models (\textit{timm}). The benign accuracy of all target models exceeds 83\% (CIFAR-10) and 73\% (ImageNet); see Table \ref{table-benin-acc-target} below exact values.

\noindent \textbf{Surrogate models (Deep Ensemble).} For CIFAR-10, the DNNs used to form surrogate ensembles are trained using the same process as the target models. Therefore the comparison between deterministic DNNs and cSGLD is fair since one can expect the deterministic DNNs surrogate to be ``close'' to the target. As for ImageNet, we retrieve an ensemble of 15 ResNet-50 models trained independently by \citet{Ashukha2020PitfallsLearning} using SGD with momentum during 130 epochs. For the RQ2 experiments, we train similarly one model for every 4 other ImageNet architectures.

\noindent \textbf{Surrogate models (cSGLD).} Following the work of \citet{Ashukha2020PitfallsLearning} and \citet{Zhang2020CyclicalLearning}, we train models with cSGLD on CIFAR-10 for 6 learning rate cycles (which, as our RQ4 experiments reveal, is where the transfer rate starts plateauing). cSGLD performs 5 cycles on ImageNet.
The learning rate is set with cosine annealing schedule for fast convergence. Each cycle lasts 50 epochs on CIFAR-10 and 45 on ImageNet. The last epochs of every cycle form the sampling phase: noise is added and one sample is drawn at the end of each epoch. On CIFAR-10, we obtain 5 samples per cycle (resp. 3 on ImageNet), so 30 samples in total (resp. 15). An illustration of a cSGLD cyclical learning rate schedule is in supplementary materials.
To train ResNet-50 models on ImageNet, we re-use the original cSGLD hyperparameters.

\noindent \textbf{Surrogate models (other training methods).} Additionally to Deep Ensemble cSGLD and following \citet{Ashukha2020PitfallsLearning}, we consider 2 Bayesian Deep Learning techniques (SWAG and VI) and 2 Ensemble ones (SSE and FGE). We train every technique on CIFAR-10 and cSGLD and SWAG on ImageNet. We retrieve trained Deep Ensemble, SSE, FGE and VI ImageNet models from \citet{Ashukha2020PitfallsLearning}. Technique descriptions and experimental setup of surrogates trained with SWAG, VI, FGE, or SSE are detailed below in the Bayesian and Ensemble Training Techniques section.

\noindent \textbf{Adversarial attacks.} We applied our variant of 4 gradient-based attacks as described in the approach section. The attacker's goal is misclassification (untargeted adversarial examples). We perform both $2$-norm and $\infty$-norm bounded adversarial attacks, and report means and standard deviations computed on 3 random seeds. In accordance to values commonly used in the literature \cite{Croce2020ReliableAttacks}, the maximum perturbation norm $\varepsilon$ is set respectively to $0.5$ and $\frac{4}{255}$ on CIFAR-10, and respectively to $3$ and $\frac{4}{255}$ on ImageNet. The step-size $\alpha$ is set to $\frac{\varepsilon}{10}$. We choose to perform 50 iterations such that the transferability rates plateaus for all iterative attacks (I-FGSM, MI-FGSM and PGD) on both norms and both datasets (see Figures \ref{fig:nb-iters-cifar} and \ref{fig:nb-iters-imagenet} below). PGD runs with 5 random restarts. FG(S)M aside, \textit{every iteration computes the gradient of 1 model per architecture}. Therefore, the attack computation cost and volatile memory are not multiplied by the size of the surrogate, except for FG(S)M which computes its unique gradient against all available models. cSGLD samples are attacked in random order.
The MI-FG(S)M decay factor is set to $0.9$.

\noindent \textbf{Test-time Transformations.} In the dedicated section, we consider three test-time transformations applied during attack designed for transferability (see related work section): Ghost Networks \cite{Li2018LearningNetworks}, Input Diversity \cite{Xie2019} and Skip Gradient Method \cite{Wu2020SkipResNets}. We implemented the first two in PyTorch with their original hyperparameters. To extend Input Diversity to the smaller input sizes of CIFAR-10, we keep the same maximum resize ratio of $0.9$. We reuse original implementation of the third one on ResNet50, and extend it to PreResNet110 (we set its hyperparameter via grid-search, see Figure \ref{fig:hp-tuning-sgm} below).

\noindent \textbf{Implementation.} The source code of the experiments are publicly available on GitHub\footnote{\url{https://github.com/Framartin/transferable-bnn-adv-ex}}. Our attack is built on top of the Python ART library \cite{art2018}. cSGLD, VI, SSE, and FGE models were trained thanks to the implementation of \citet{Ashukha2020PitfallsLearning} available on GitHub\footnote{\url{https://github.com/bayesgroup/pytorch-ensembles}}. All models were trained with PyTorch \cite{NEURIPS2019_9015}. We use EfficientNet-B0 from timm\footnote{\url{https://github.com/rwightman/pytorch-image-models}}. We train SWAG on ImageNet with the original implementation \cite{Maddox2019ALearning}. We use the following software versions: Python 3.8.8, Pytorch 1.7.1 (1.9.0 for Flops measurement), torchvision 0.8.2, Adversarial Robustness Toolbox 1.6.0, and timm 0.3.2.

\noindent \textbf{Flops.} We measure the training computational complexity in Flops using the PyTorch profiler. The computation overhead of one epoch with cSGLD compared to one with SGD/Adam is negligible. The main difference is the addition of noise to the weights during the sampling phase. On CIFAR-10, the overhead of 1 cSGLD epoch of PreResNet110 with added noise compared to one of a DNN trained with Adam (SGD) is 0.0187\% Flops (respectively 0.0146\% for ResNet50 on ImageNet).

\noindent \textbf{Infrastructure.} Experiments were run on Tesla V100-DGXS-32GB GPUs. The server has the following specifications: 256GB RDIMM DDR4, CUDA version 10.1, Linux (Ubuntu) operating system.

\section{Experiments}

The goal of our approach is to increase the transferability of adversarial examples by using a surrogate approximately sampled from the posterior distribution to attack a DNN.

\textbf{Setup summary}. The target models are deterministic DNNs and are never used as a surrogate. For a fair comparison between DNNs and cSGLD, we train the surrogate DNNs on CIFAR-10 using the same process as the target models. ImageNet targets are third-party pre-trained models. Each cSGLD cycle lasts 50 epochs and samples 5 models on CIFAR-10, and 45 epochs/3 models on ImageNet. We report the success rate (misclassification rates of untargeted adversarial examples) averaged over three attack runs. To craft adversarial examples, we use original test examples that are correctly predicted (all the test examples for CIFAR-10, and from a random subset of 5000 examples for ImageNet). The iterative attacks (I-FG(S)M, MI-FG(S)M, and PGD) perform 50 iterations such that the transferability rates plateaus (see supp. materials). Each attack computes the gradient of 1 model per architecture. Therefore, their computation cost and volatile memory are not multiplied by the size of the surrogate, except for FG(S)M which computes its unique gradient against all available models. The source code is publicly available\footnote{URL retracted for anonymous submission.}. The experimental setup is presented in more details in the supplementary materials.

\subsection{Intra-architecture Transferability}
\label{sec:rq1}

\begin{table}[t]
\centering
\resizebox{.95\columnwidth}{!}{
\begin{tabular}{lcl|R{4.5em}R{4.5em}}
Dataset & Attack                     & Norm & T-DEE & Flops Ratio \\
\hline
\multirow{8}{*}{CIFAR-10} & \multirow{2}{*}{I-FG(S)M} & L2 & \textgreater{}15 ±nan & \textgreater{}15 ±nan \\
        &                            & L$\infty$ & 3.76 ±0.08              & 3.76 ±0.08               \\ \cline{2-5}
        & \multirow{2}{*}{MI-FG(S)M} & L2        & 5.56 ±0.80              & 5.56 ±0.80               \\
        &                            & L$\infty$ & 2.88 ±0.03              & 2.87 ±0.03               \\ \cline{2-5}
        & \multirow{2}{*}{PGD}       & L2        & \textgreater{}15 ±nan   & \textgreater{}15 ±nan    \\
        &                            & L$\infty$ & 3.74 ±0.12              & 3.74 ±0.12               \\ \cline{2-5}
        & \multirow{2}{*}{FG(S)M}    & L2 & \textgreater{}15 ±nan & \textgreater{}15 ±nan \\
        &                            & L$\infty$ & 8.72 ±0.01             & 8.72 ±0.01               \\ \hline
\multirow{8}{*}{ImageNet} & \multirow{2}{*}{I-FG(S)M} & L2  & 4.91 ±0.11        & 2.84 ±0.06            \\
        &                            & L$\infty$            & 4.34 ±0.13        & 2.51 ±0.08               \\ \cline{2-5}
        & \multirow{2}{*}{MI-FG(S)M} & L2                   & 4.69 ±0.18        & 2.71 ±0.10               \\
        &                            & L$\infty$            & 4.38 ±0.03        & 2.53 ±0.02               \\ \cline{2-5}
        & \multirow{2}{*}{PGD}       & L2                   & 5.00 ±0.11        & 2.89 ±0.06               \\
        &                            & L$\infty$            & 4.42 ±0.16        & 2.56 ±0.09               \\ \cline{2-5}
        & \multirow{2}{*}{FG(S)M}    & L2                   & 5.81 ±0.34        & 3.35 ±0.19               \\
        &                            & L$\infty$            & 5.98 ±0.03        & 3.46 ±0.02               \\
\end{tabular}
}
\caption{Number of DNNs (T-DEE) and training computation budget (measured in flops) needed to achieve the same intra-architecture transferability as cSGLD using Deep Ensemble. Higher is better. ``\textgreater{}15'' indicates that 15 DNNs have a lower transfer rate than cSGLD for all random seeds.}
\label{tab:dee-same-arch}
\end{table}

Since SG-MCMC methods sample the weights of a given architecture, we expect our approach to work particularly well in settings where the architecture of the target model is known, but not its weights.
To demonstrate this, we compare the intra-architecture transfer success rates of cSGLD with the ones of Deep Ensemble surrogates (using 1 up to 15 independently trained DNNs).
The used architectures are PreResNet110 (CIFAR-10) and ResNet-50 (ImageNet).

Detailed results for four classical gradient-based attacks are provided as supplementary materials.
In summary, for a similar computation cost, cSGLD systematically increases the success rate of iterative attacks by 13.8 (ImageNet, MI-FG(S)M, L$\infty$) to 49.2 (CIFAR-10, I-FG(S)M, L2) percentage points, and the success rate of FG(S)M by 12.18 to 22.2. One explanation for the highest improvements is that DNN-based L2 norm attacks suffer from the vanishing gradient phenomenon on CIFAR-10, whereas cSGLD avoids it thanks to fast convergence and warm restarts (we report the proportions of vanishing gradients in supp. materials).

Inspired by the DEE metric \cite{Ashukha2020PitfallsLearning}, we define the \textit{Transferability-Deep Ensemble Equivalent (T-DEE)} metric as the number of independently trained DNNs needed to achieve the same success rate as cSGLD (computed with linear interpolation).

Table \ref{tab:dee-same-arch}  reports the T-DEE and the \emph{computing ratio}, i.e., the total number of flops to train such DNNs ensemble divided by the number of flops used to trained cSGLD. This ratio represents the computational gain factor achieved by our approach, which is different from T-DEE in the ImageNet case since 1 DNN is trained for 130 epochs while cSGLD uses 225 epochs.

In the worst case across the two datasets, an ensemble of 3 surrogate DNNs is required to beat the cSGLD surrogate, while requiring at least $2.51$ times more flops during the training phase. On CIFAR-10 and considering L$2$ attack specifically (MI-FGM aside), it even outperforms the ensemble of 15 DNNs by a significant factor (up to 30.3 percentage points). On ImageNet, the cSGLD surrogate achieves the same transfer rate as 4.38--5.98 DNNs, which corresponds to dividing the number of flops by a factor between 2.51 and 3.46.

We conclude that cSGLD meaningfully and efficiently exploits uncertainty on parameter estimation and that this uncertainty is useful to discover generic adversarial directions.
The increased transferability at constant computational cost emphasizes the benefits that one can gain from the cyclical nature of cSGLD, thanks to warm restarts, fast convergence, and the local characterization of each mode of the parameters posterior.

\subsection{Inter-architecture Transferability}

\begin{table*}[t!]
\caption{Transfer rates of I-FG(S)M attack on CIFAR-10 hold-out architectures. The $\star$ symbol indicates that 1 DNN per architecture is better than 1 cSGLD per architecture. Higher is better.}
\centering
\resizebox{1.95\columnwidth}{!}{
\begin{tabular}{ll|rrrrr|L{5em}}
Attack & Surrogate & $-$PreResNet110 & $-$PreResNet164 & $-$VGG16bn & $-$VGG19bn & $-$WideResNet & Nb epochs \\
\hline
\multirow{3}{*}{L$2$} & 1 cSGLD per arch. &  \textbf{95.56\% ±0.04} &  \textbf{95.72\% ±0.06} &  \textbf{45.96\% ±0.07} &  \textbf{42.60\% ±0.08} &  \textbf{84.04\% ±0.05}  &$4\times300$ \\
                    & 1 DNN per arch.   &  60.38\% ±1.09 &  60.93\% ±1.06 &  29.97\% ±0.48 &  27.57\% ±0.66 &  57.86\% ±0.74 & $4\times300$ \\
                    & 4 DNNs per arch.  &  77.12\% ±1.32 &  77.21\% ±1.14 &  40.89\% ±0.63 &  40.18\% ±0.76 &  77.54\% ±0.93 & $4\times1200$\\
\cline{1-8}
\multirow{3}{*}{L$\infty$}  & 1 cSGLD per arch. &  96.38\% ±0.06 &  96.51\% ±0.08 &  49.19\% ±0.06 &  45.17\% ±0.03 &  84.75\% ±0.01 & $4\times300$ \\
                            & 1 DNN per arch.   &  87.02\% ±0.04 &  88.86\% ±0.04 &  44.99\% ±0.10 &  $\star$45.55\% ±0.02 &  74.84\% ±0.03 & $4\times300$ \\
                            & 4 DNNs per arch.  &  \textbf{96.50\% ±0.01} &  \textbf{97.01\% ±0.02} &  \textbf{59.80\% ±0.01} &  \textbf{59.08\% ±0.01} &  \textbf{89.23\% ±0.04} & $4\times1200$ \\
\end{tabular}
}
\label{tab:transfer-multi-archs}
\end{table*}

\begin{table*}[t!]
\caption{Transfer rates of I-FG(S)M attack on ImageNet hold-out architectures. Higher is better.}
\centering
\resizebox{1.95\columnwidth}{!}{
\begin{tabular}{ll|rrrrr|L{5em}}
   Attack  & Surrogate &     $-$ResNet50 & $-$ResNeXt50 & $-$DenseNet121 & $-$MNASNet & $-$EfficientNetB0 & Nb epochs \\
\hline
\multirow{2}{*}{L2} & 1 cSGLD per arch. &   \textbf{ 93.28\% ±0.12} &  \textbf{90.61\% ±0.24} &  \textbf{92.25\% ±0.26} &  \textbf{95.98\% ±0.19} &     \textbf{81.88\% ±0.38} & $4\times135$ \\
     & 1 DNN per arch. &    72.99\% ±0.52 &  72.31\% ±0.44 &  64.72\% ±0.59 &  84.21\% ±0.18 &     53.99\% ±0.76 & $4\times135$ \\
\hline
\multirow{2}{*}{L$\infty$} & 1 cSGLD per arch. &    \textbf{92.21\% ±0.23} &  \textbf{89.83\% ±0.22} &  \textbf{90.86\% ±0.19} &  \textbf{95.85\% ±0.46} &     \textbf{79.40\% ±0.42} & $4\times135$ \\
     & 1 DNN per arch. &    69.65\% ±0.47 &  69.01\% ±0.70 &  61.00\% ±0.66 &  82.25\% ±0.03 &     49.71\% ±1.37 & $4\times135$ \\
\end{tabular}
}
\label{tab:transfer-multi-archs-imagenet}
\end{table*}

We now focus on black-box settings where the architecture of the target model is unknown (and not used to build the surrogate model). We consider ten architectures (five for both datasets). Following \citet{Liu2017DelvingAttacks,Xie2019,Li2018LearningNetworks,Dong2018},
we hold out one architecture to act as the target model and use the four remaining ones as surrogates. We train every architecture with cSGLD to obtain the same number of samples for each of them. We, then, apply I-FG(S)M with 1 model per surrogate architecture per iteration to keep attack cost constant (in the number of backward passes).  Given the computational cost involved in training different architectures, the training processes are limited to 135 epochs for ImageNet (for cSGLD, this means 3 cycles of 45 epochs). For every architecture, cSGLD and 1 DNN are trained for the same number of epochs.

As shown in Tables \ref{tab:transfer-multi-archs} (CIFAR-10) and \ref{tab:transfer-multi-archs-imagenet} (ImageNet), our method significantly improved transferability on all 5 hold-out architectures for both CIFAR-10 and ImageNet, except for the L$\infty$ VGG19bn target (with a difference of 0.4 percentage point in favor of 1 DNN per arch.). On CIFAR-10, the differences range from 15.0 to 35.2 percentage points for the $2$-norm constraint, and from -0.4 to 9.9 for the $\infty$-norm constraint. Our method even performs better than 4 DNNs per architecture for the L$2$ attack, despite training for 4 times fewer epochs. On ImageNet, the cSGLD surrogates improve over the single DNN counterpart by 11.8 and 29.9 percentage points of transfer rate at constant computational train and attack budget.

In the supplementary materials, we present the results for an alternative protocol where we use a single architecture as surrogate. In summary, in this setup cSGLD achieves a higher inter-architecture success rate in 38/40 cases on CIFAR-10, and 39/40 cases on ImageNet, compared to a single DNN trained for the same number of epochs. Differences range between -2.3 and 62.1 percentage points on CIFAR-10 and -0.3 and 44.8 on ImageNet.

We conclude that our method can improve transferability even when the target architecture is unknown. This tends to indicate that the adversarial directions against posterior predictive distribution are partially aligned across different architectures. In other words, given a common classification task, there might exist a part of the uncertainty in parameters' estimate that is common across architectures, and the variability of one architecture's parameters might be informative of the variability of parameters of another architecture.

\subsection{Test-time Transferability Techniques}

\begin{table*}[t!]
\caption{Transfer success rates of (M)I-FG(S)M attack improved by our approach combined with test-time transformations on ImageNet. Columns are targets. ResNet50 column is intra-architecture transferability, others are inter-architecture. Bold is best. Symbols $\star$ are DNN-based techniques better than our vanilla cSGLD surrogate, and $\dagger$ are techniques that don’t improve the corresponding vanilla surrogate. The success rate for every cSGLD-based technique is better than its counterpart with 1 DNN.}
\centering
\resizebox{1.95\columnwidth}{!}{
\begin{tabular}{ll|rrrrr}
Attack     & Surrogate   &      ResNet50 &     ResNeXt50 &   DenseNet121 &       MNASNet & EfficientNetB0 \\
\hline
\multirow{16}{*}{L$2$} & 1 DNN &  56.60\% ±0.71 &  41.09\% ±0.61 &  29.73\% ±0.30 &  28.13\% ±0.17 &   16.64\% ±0.33 \\
     & \quad+ Input Diversity &  83.15\% ±0.30 &  73.17\% ±0.80 &  61.24\% ±0.58 &  58.16\% ±0.36 &   $\star$42.10\% ±0.36 \\
     & \quad+ Skip Gradient Method &  65.64\% ±0.88 &  52.75\% ±0.42 &  38.58\% ±0.55 &  43.40\% ±0.61 &   29.11\% ±0.30 \\
     & \quad+ Ghost Networks &  78.84\% ±0.46 &  62.46\% ±0.38 &  45.76\% ±0.02 &  41.44\% ±0.58 &   25.77\% ±0.11 \\
     & \quad+ Momentum (MI-FGSM) &  $\dagger$52.53\% ±0.80 &  $\dagger$37.15\% ±0.76 &  $\dagger$26.33\% ±0.48 &  $\dagger$25.21\% ±0.42 &   $\dagger$14.74\% ±0.31 \\
     & \quad\quad+ Input Diversity &  80.81\% ±0.72 &  69.55\% ±0.83 &  56.73\% ±0.39 &  54.16\% ±0.05 &   37.07\% ±0.03 \\
     & \quad\quad+ Skip Gradient Method &  65.65\% ±0.95 &  53.25\% ±0.18 &  38.79\% ±0.62 &  44.33\% ±0.63 &   29.45\% ±0.28 \\
     & \quad\quad+ Ghost Networks &  71.50\% ±0.12 &  53.45\% ±0.65 &  37.39\% ±0.47 &  34.53\% ±0.69 &   20.29\% ±0.36 \\
     \cline{2-7}
     & cSGLD &  84.83\% ±0.55 &  74.73\% ±0.82 &  71.45\% ±0.56 &  60.14\% ±0.44 &   39.71\% ±0.20 \\
     & \quad+ Input Diversity &  \textbf{93.87\% ±0.19} &  \textbf{89.12\% ±0.24} &  \textbf{88.52\% ±0.16} &  \textbf{82.78\% ±0.28} &   \textbf{66.13\% ±0.35} \\
     & \quad+ Skip Gradient Method &  $\dagger$83.17\% ±0.85 &  $\dagger$72.79\% ±1.06 &  $\dagger$66.19\% ±0.89 &  71.71\% ±0.41 &   52.66\% ±0.31 \\
     & \quad+ Ghost Networks &  92.99\% ±0.13 &  85.69\% ±0.24 &  82.81\% ±0.42 &  72.88\% ±0.30 &   50.30\% ±0.29 \\
     & \quad+ Momentum (MI-FGSM) &  $\dagger$82.44\% ±0.19 &  $\dagger$70.93\% ±1.04 &  $\dagger$66.19\% ±0.56 &  $\dagger$55.51\% ±0.59 &   $\dagger$34.49\% ±0.59 \\
     & \quad\quad+ Input Diversity &  93.48\% ±0.23 &  87.87\% ±0.15 &  86.81\% ±0.33 &  80.37\% ±0.20 &   60.26\% ±0.02 \\
     & \quad\quad+ Skip Gradient Method &  $\dagger$82.35\% ±0.10 &  $\dagger$71.54\% ±0.58 &  $\dagger$64.50\% ±0.18 &  70.47\% ±0.22 &   50.80\% ±0.23 \\
     & \quad\quad+ Ghost Networks &  90.11\% ±0.18 &  80.35\% ±0.61 &  75.10\% ±0.67 &  64.08\% ±0.12 &   39.85\% ±0.52 \\
\hline
\multirow{16}{*}{L$\infty$} & 1 DNN &  47.81\% ±1.09 &  32.29\% ±0.64 &  23.43\% ±0.32 &  22.52\% ±0.45 &   12.77\% ±0.32 \\
     & \quad+ Input Diversity &  76.55\% ±1.01 &  62.57\% ±0.56 &  50.17\% ±0.33 &  49.31\% ±0.18 &   $\star$32.64\% ±0.09 \\
     & \quad+ Skip Gradient Method &  66.36\% ±0.50 &  51.60\% ±0.36 &  39.05\% ±0.24 &  45.60\% ±0.72 &   30.69\% ±0.03 \\
     & \quad+ Ghost Networks &  67.02\% ±0.17 &  46.74\% ±0.63 &  32.57\% ±0.17 &  31.12\% ±0.77 &   17.68\% ±0.05 \\
     & \quad+ Momentum (MI-FGSM) &  55.12\% ±0.82 &  38.47\% ±0.82 &  28.19\% ±0.14 &  27.55\% ±0.67 &   16.34\% ±0.37 \\
     & \quad\quad+ Input Diversity &  $\star$82.47\% ±0.41 &  $\star$69.69\% ±0.81 &  57.79\% ±0.57 &  $\star$55.99\% ±0.37 &   $\star$38.63\% ±0.29 \\
     & \quad\quad+ Skip Gradient Method &  68.39\% ±0.53 &  54.57\% ±0.60 &  41.48\% ±0.37 &  47.97\% ±0.41 &   $\star$33.16\% ±0.37 \\
     & \quad\quad+ Ghost Networks &  71.27\% ±0.54 &  51.46\% ±0.84 &  36.91\% ±0.48 &  34.54\% ±0.32 &   20.51\% ±0.30 \\
     \cline{2-7}
     & cSGLD &  78.71\% ±1.19 &  65.11\% ±1.45 &  61.49\% ±0.59 &  51.81\% ±1.45 &   31.11\% ±0.99 \\
     & \quad+ Input Diversity &  90.03\% ±0.10 &  82.13\% ±0.45 &  81.19\% ±0.34 &  74.48\% ±0.39 &   53.51\% ±0.39 \\
     & \quad+ Skip Gradient Method &  81.37\% ±0.72 &  69.88\% ±1.31 &  65.20\% ±0.75 &  71.68\% ±0.53 &   52.15\% ±0.32 \\
     & \quad+ Ghost Networks &  87.33\% ±0.73 &  76.00\% ±1.33 &  71.67\% ±0.97 &  61.45\% ±0.25 &   37.19\% ±0.68 \\
     & \quad+ Momentum (MI-FGSM) &  82.89\% ±0.70 &  70.42\% ±1.26 &  66.39\% ±0.74 &  56.68\% ±0.97 &   36.00\% ±1.15 \\
     & \quad\quad+ Input Diversity &  \textbf{93.97\% ±0.26} &  \textbf{87.69\% ±0.44} &  \textbf{86.78\% ±0.16} &  \textbf{81.08\% ±0.14} &   \textbf{60.87\% ±0.48} \\
     & \quad\quad+ Skip Gradient Method &  84.19\% ±0.21 &  73.14\% ±0.99 &  67.35\% ±0.26 &  74.36\% ±0.47 &   55.30\% ±0.16 \\
     & \quad\quad+ Ghost Networks &  89.53\% ±0.05 &  78.69\% ±0.19 &  73.33\% ±0.58 &  63.56\% ±0.35 &   39.79\% ±0.52 \\
\end{tabular}
}
\label{tab:transfer-test-time-techs-imagenet}
\end{table*}

Given that our approach works at training time, we check whether it can be combined with test-time techniques to further improve transferability. We apply three test-time transformations to cSGLD samples and 1 DNN obtained with the same number of epochs (300 for CIFAR-10 and 135 for ImageNet). On CIFAR-10, the surrogates are PreResNet110 (respect. ResNet50 on ImageNet), and the 5 targets are the same as in the inter-architecture experiments. Following \cite{Li2018LearningNetworks,Xie2019,Wu2020SkipResNets}, we also combine the test-time techniques with momentum.\footnote{All rows with momentum correspond to the MI-FG(S)M attack, an attack variant designed to improve transferability~\cite{Dong2018}.} As before, every attack performs a total of 50 backward passes.

Table \ref{tab:transfer-test-time-techs-imagenet} shows the results on ImageNet (detailed results on CIFAR-10 are provided in supplementary materials). We observe that our approach and the test-time techniques complement well to each other. Indeed, the best success rates are always achieved by a technique applied on cSGLD (in bold). All three techniques combined with momentum applied on cSGLD achieve a systematically higher success rate than the same technique applied on 1 DNN, with differences ranging from 10.7 to 41.7 percentages points on ImageNet and from 3.8 to 56.2 on CIFAR-10. Overall, the addition of a technique (excluding momentum alone) to our vanilla cSGLD surrogate never decrease the success rate on CIFAR-10 and only in 10\% of the averaged cases considered on ImageNet, as indicated by the $\dagger$ symbols.

Besides, our vanilla cSGLD surrogate achieves better transferability than any of the test-time techniques applied to 1 DNN in 90\% of the cases on CIFAR-10 and 93.3\% on ImageNet, using the I-FG(S)M attack. Similarly, for MI-FG(S)M, we observe 76.7\% for the former and 90\% for the latter.  This demonstrates that despite previous efforts in providing effective test-time techniques for transferability (see the related work section), \emph{improving the training of the surrogate -- in our case, through efficient approximate sampling from the posterior distribution -- yields significantly higher improvements}. Hence, while training approaches have been overlooked, canonical elements that have been related to transferability, ie. skip connections~\cite{Wu2020SkipResNets}, input~\cite{Xie2019} and model diversity~\cite{Li2018LearningNetworks}, should be put into perspective compared to the importance that the posterior distribution appears to have.

\subsection{Bayesian and Ensemble Training techniques}

In addition to cSGLD and deep ensembles, we explore the use of other training techniques to improve transferability: two other Bayesian techniques -- Variational Inference (VI) and Stochastic Weight Averaging-Gaussian (SWAG) -- and two other ensembling techniques -- Snapshot ensembles (SSE) and Fast Geometric Ensembling (FGE).
We train with every technique for an equivalent computational cost of 3 DNNs on CIFAR-10 and 2 DNNs on ImageNet (except for VI and SWAG, see discussion in supplementary materials). Due to computational limitations, we run the attack for a single random seed on 5000 images for each dataset.

Figure \ref{fig:ensemble-technique-Linf} presents the success rate of L$\infty$ I-FS(S)M attack related to the corresponding training computational cost (in flops), as we increase the number of models in each ensemble. We provide more details about every method and the results for L$2$ I-FS(S)M in supplementary materials.

On CIFAR-10, the success rate of the first 4 cycles of cSGLD increases substantially from one cycle to the next (from 76.58\% to 81.56\% for the first to the second cycle) and within a single cycle (from 81.56\% to 87.20\% between the start and the end of the second cycle). This reveals that exploring modes of the parameters posterior plays an important role to generate transferable adversarial examples and that there is some local geometric diversity of the loss landscape among local maximums. On ImageNet, transferability improves mainly by sampling from several local optima.

Interestingly, even though FGE and SWAG build an ensemble around a single local optimum, their flexibility allows capturing general adversarial directions. The FGE surrogates trained for more than 0.30 petaflops have systematically higher success rates than cSGLD and SSE on CIFAR-10.
However, the opposite is observed on ImageNet: FGE is not competitive with methods exploring several local optima (cSGLD, SSE, and Deep Ensemble). We hypothesize that modes are not as well connected on larger datasets.

The efficiency of SWAG on both datasets opens new directions to create hybrid transferable attacks based on few additional iterations over the training set. SWAG approximates the posterior distribution with a Gaussian fitted on some additional SGD epochs from a pre-trained DNN. It captures well the shape of the true posterior \cite{Maddox2019ALearning}, reinforcing our views on the strong relationship between the posterior distribution and transferability. The gap between the success rates of cSGLD/SSE and SWAG on ImageNet suggests higher geometrical discrepancies between local loss maxima on larger datasets.

VI fails to compete with Deep Ensemble on both success rate and computational efficiency for the L$\infty$ attack on CIFAR-10, but beats it on L$2$ bound and on ImageNet.

On CIFAR-10, the marginal impact beyond 6 cSGLD cycles, 17 SWAG samples, 7 SSE models, and 35 FGE models becomes noisy. We hypothesize that this is due to the limits of these methods (inherent to warm restarts), which produce correlated samples. Hence, an interesting direction is to investigate the use of multiple runs to improve intra-architecture transferability.

\begin{figure}[!ht]
    \centering
        \subfloat[CIFAR-10]{\includegraphics[width=0.4\textwidth]{figure/RQ_techniques_transfer_vs_time_Linf.pdf}}
        \qquad
        \subfloat[ImageNet]{\includegraphics[width=0.4\textwidth]{figure/RQ_techniques_imagenet_transfer_vs_time_Linf.pdf}}
    \caption{Intra-architecture L$\infty$ I-FG(S)M success rate with respect to the training computational complexity of 6 Bayesian and Ensemble methods. Every curve starts with one model and each successive point is obtained by forming an ensemble with one more model.}
    \label{fig:ensemble-technique-Linf}
\end{figure}

\section{Conclusion}

We are the first to extensively investigate training-time approaches to enhance transferability. We discover a strong connection between the posterior predictive distribution and both intra- and inter-architecture transferability. Our Bayesian deep learning surrogate is efficient and effective to craft adversarial examples transferable to deterministic neural networks. Our approach further improves existing adversarial attacks and test-time transferability techniques, as one can use it on top of them to perform Bayesian model averaging efficiently and with minor modifications. We show that our simple training-time approach improves transferability more than previous test-time techniques. We, therefore, cast an important yet overlooked direction to explain transferability and pave the way for new hybrid attacks. Overall, we provide new evidence that the Bayesian framework is a promising direction for research on adversarial examples.

\section*{Supplementary Materials}

In the supplementary materials for the paper, the following are provided:
\begin{itemize}
    \item The detailed experimental setup section
    \item The description and experimental setup of Bayesian and Ensembling training techniques
    \item Additional results, including:
    \begin{itemize}
        \item The natural accuracy of target Neural Networks;
        \item The proportions of vanishing gradients of cSGLD surrogate compared to DNN surrogate;
        \item The intra-architecture transfer success rates on cSGLD and Deep Ensemble of 1, 2, 5 and 15 DNNs surrogates;
        \item The inter-architecture transfer success rates of single architecture surrogates;
        \item The intra-architecture transfer success rate of 6 Bayesian and Ensemble training methods attacked by L$2$ I-FG(S)M;
        \item The intra-architecture transfer success rate combined with test-time transformations on CIFAR-10;
        \item The transfer rate of cSGLD with respect to the number of cycles and samples per cycle;
    \end{itemize}
    \item An illustration of the cSGLD cyclical learning rate schedule;
    \item A diagram of the relationships between gradient-based attacks;
    \item The algorithm applied to perform Bayesian Model Averaging efficiently;
    \item Details on hyper-parameters, including:
    \begin{itemize}
        \item The transfer success rate of iterative attacks with respect to the number of iterations;
        \item The tuning of the hyperparameter of the Skip Gradient Method technique to extend it to PreResNet110;
        \item The hyperparameters used to train and attack models;
    \end{itemize}
\end{itemize}

\input{4-experiments}

\begin{figure}[!htb]
\centering
\includegraphics[width=0.95\columnwidth]{figure/figure_csgld_lr_croped}
\caption{Illustration of the cSGLD cyclical learning rate schedule~(red) and the traditional decreasing learning rate schedule~(blue). Each cSGLD cycle is composed of an exploration phase~(burn-in period of MCMC algorithms --- red doted) and of a sampling phase~(red plain). Figure taken from Zhang et al. (2020).}
\label{fig:illustration-csgld-lr}
\end{figure}

\begin{figure}[!htb]
\centering
\[\begin{tikzcd}
	{\text{MI-FSGM}} \\
	\\
	{\text{I-FSGM}} && {\text{FSGM}} \\
	\\
	{\text{PGD}}
	\arrow["{\text{Momentum} = 0}"{description}, from=1-1, to=3-1]
	\arrow["\substack{\text{Nb restarts}=1 \\ \text{No random init}}"{description}, from=5-1, to=3-1]
	\arrow["\substack{\text{Nb iters}=1 \\ \text{Step-size } \alpha=\varepsilon}"{marking}, from=3-1, to=3-3]
\end{tikzcd}\]
\caption{Relationships between gradient-based attacks.}
\label{fig:relation-attacks}
\end{figure}

\begin{algorithm}[tb]
   \caption{Variant of I-FG(S)M attack to operate on a large number of models from multiple architectures}
   \label{alg:attack-variant}
\begin{algorithmic}
   \STATE {\bfseries Input:} original example $(x, y)$, $S_A$ ordered sets of model parameters $( \theta_s^1 )_{s=1}^S, \ldots, ( \theta_s^{S_A} )_{s=1}^S$, number of iterations $n_{\text{iter}}$, perturbation $p$-norm $\varepsilon$, step-size $\alpha$
   \STATE {\bfseries Output:} adversarial example $x_\text{adv}$
   \STATE Shuffle each ordered set of model samples $( \theta_s^1 )_{s=1}^S, \ldots, ( \theta_s^{S_A} )_{s=1}^S $
   \STATE $x_\text{adv} \leftarrow x$
   \FOR{$i=1$ {\bfseries to} $n_{\text{iter}}$}
   \STATE ${x_\text{adv}} \leftarrow x_\text{adv} + \frac{\alpha}{S_A} \sum_{a=1}^{S_A} \nabla \mathcal{L}(x_\text{adv} ;\, y, \theta^a_{i \bmod S}) $
   \STATE $x_\text{adv} \leftarrow \text{project}(x_\text{adv}, B_{\varepsilon}[x])$
   \STATE $x_\text{adv} \leftarrow \text{clip}(x_\text{adv})$
   \ENDFOR
\end{algorithmic}
\end{algorithm}

\subsection{Bayesian and Ensemble Training Techniques}

Following the work of \citet{Ashukha2020PitfallsLearning}, we consider the following training techniques: Deep Ensemble \cite{Lakshminarayanan2016SimpleEnsembles}, cSGLD \cite{Zhang2020CyclicalLearning}, SWAG \cite{Maddox2019ALearning}, VI, SSE \cite{Huang2017SnapshotFree}, and FGE \cite{Garipov2018LossDNNs}.

\noindent \textbf{Deep Ensemble.} Deep Ensemble \cite{Lakshminarayanan2016SimpleEnsembles} simply trains several DNNs independently with random initialization and random subsampling (mini-batch on shuffled data in practice). All DNNs have the same standard hyperparameters for training. For classification, predictions of individual DNNs are averaged. We train 15 PreResNet110, 4 PreResNet164, 4 VGG16bn, 4 VGG19bn, and 4 WideResNet28x10 DNNs on CIFAR-10. We retrieve 15 ResNet50 DNNs trained by \citet{Ashukha2020PitfallsLearning} on ImageNet, and trained on our own 1 DNN for each of the remaining studied architectures (ResNeXt50 32x4d, DenseNet121, MNASNet 1.0, and EfficientNet-B0).

\noindent \textbf{cSGLD.} We refer the reader to the approach section for a detailed description of cyclical Stochastic Gradient Langevin Dynamics. Figure \ref{fig:illustration-csgld-lr} joined as supplementary materials illustrates both the cyclical cosine annealing learning rate schedule and the separation of each cycle into an exploration phase (called the burn-in period of MCMC algorithm) and a sampling phase.

\noindent \textbf{SWAG.} Stochastic Weight Averaging-Gaussian (SWAG) \cite{Maddox2019ALearning} is a Bayesian Deep Learning method that fits a Gaussian onto SGD iterates to approximate the posterior distribution over weights. Its first moment is the SWA solution, and its second moment a diagonal plus low-rank covariance matrix. Both are estimated from SGD iterates with constant learning rate (0.001 on ImageNet and 0.01 on CIFAR-10). On ImageNet, SWAG performs 10 additional epochs to collect SGD iterates from one of the Deep Ensemble DNNs. On CIFAR-10, a regular pre-training phase of 160 epochs precedes 140 epochs to collect checkpoints. Once fitted, models are sampled from the Gaussian distribution. For every sample, batch normalization statistics are updated in a forward pass over the entire CIFAR-10 train set and over a random subset of 10\% on ImageNet. Apart from the fixed initial cost, the marginal computational cost to obtain a sample is very low. We sample a maximum of 50 models because iterative attacks perform 50 iterations of one model per iteration, and further samples would be discarded. Thus, the lines corresponding to SWAG in Figures \ref{fig:ensemble-technique-Linf} and \ref{fig:ensemble-technique-L2} are shorter than the ones of other methods. The rank of the estimated covariance matrix is 20. Batch-size is 128 on CIFAR-10, and 256 on ImageNet.

\begin{figure}[ht]
    \centering
        \subfloat[CIFAR-10]{\includegraphics[width=0.4\textwidth]{figure/RQ_techniques_transfer_vs_time_L2.pdf}}
        \qquad
        \subfloat[ImageNet]{\includegraphics[width=0.4\textwidth]{figure/RQ_techniques_imagenet_transfer_vs_time_L2.pdf}}
    \caption{Intra-architecture L$\infty$ I-FG(S)M success rate with respect to the training computational complexity of 6 Bayesian and Ensemble methods. Every curve starts with one model and each successive point is obtained by forming an ensemble with one more model.}
    \label{fig:ensemble-technique-L2}
\end{figure}

\noindent \textbf{VI.} Variational Inference (VI) approximates the true posterior distribution with a variational approximation, here a fully-factorized Gaussian distribution, and maximizes a corresponding lower bound. A Gaussian prior is chosen. Once trained, the variational approximation is used as the posterior. There is no additional sampling phase to perform Bayesian model averaging. Therefore we cannot tune the number of samples and a single VI point is plotted in Figures \ref{fig:ensemble-technique-Linf} and \ref{fig:ensemble-technique-L2}. We follow the solutions of \citet{Ashukha2020PitfallsLearning} to avoid underfitting: pre-training and annealing of $\beta$. The first moment of the Gaussian variational approximation is initially set to a DNN pre-trained similarly to Deep Ensemble (300 epochs on CIFAR-10 with initial learning rate of $10^{-4}$, and 130 epochs on ImageNet starting at $10^{-3}$). The log of its second moment is initially set to $-5$ on CIFAR-10 and $-6$ on ImageNet, and further optimized for 100 epochs (45 on ImageNet) with Adam and a learning rate of $10^{-4}$. $\beta$ is set to $10^{-5}$ on CIFAR-10 and $10^{-4}$ on ImageNet. Batch-size is 128 on CIFAR-10, and 256 on ImageNet.

\noindent \textbf{SSE.} Snapshot ensembles technique \cite{Huang2017SnapshotFree} is the foundation of cSGLD. The learning rate is cyclical with a cosine annealing schedule. Contrary to cSGLD, SSE saves a single snapshot per cycle and does not add gradient noise. The cycles are 40 epochs long on CIFAR-10, 45 on ImageNet. The maximum learning rate is 0.2, batch size is 64 on CIFAR-10, respectively 0.1 and 256 on ImageNet.

\noindent \textbf{FGE.} Fast Geometric Ensembling \cite{Garipov2018LossDNNs} is a method developed after the empirical observation of Mode Connectivity on CIFAR-10 and CIFAR-100: it's possible to find a path in the parameters space that connects two independently trained DNNs such that the models along the path have low loss and high test accuracy. In practice, it uses a cyclical triangular learning rate and collects one model during each cycle. It is quite similar to SSE, except for the learning rate schedule, the much shorter cycles (4 epochs on CIFAR-10, 2 epochs on ImageNet), and a pre-training phase. Pre-training lasts for 160 epochs on CIFAR-10. On ImageNet, FGE is initialized from one Deep Ensemble checkpoint. The learning rate varies between $5\times10^{-5}$ and $5\times10^{-3}$ on CIFAR-10 and $10^{-6}$ and $10^{-4}$ on ImageNet. Batch-size is 128 on CIFAR-10, and 256 on ImageNet.

\begin{table}[!htb]
\centering
\begin{tabular}{ll|L{5em}}
Dataset                   & Target DNN       & Benign Test Accuracy \\
\hline
\multirow{5}{*}{CIFAR-10} & PreResNet110     & 93.26 \%             \\
                          & PreResNet164     & 93.03 \%             \\
                          & VGG16bn          & 83.68 \%             \\
                          & VGG19bn          & 83.62 \%             \\
                          & WideResNet28x10  & 92.13 \%             \\
\hline
\multirow{5}{*}{ImageNet} & ResNet50        & 76.15 \%              \\
                          & ResNeXt50 32x4d & 77.62 \%              \\
                          & Densenet121     & 74.65 \%              \\
                          & MNASNet 1.0      & 73.51 \%             \\
                          & EfficientNet-B0  & 77.70 \%             \\
\end{tabular}
\caption{Top-1 accuracy on natural test examples for every target deterministic Neural Network.}
\label{table-benin-acc-target}
\end{table}

\begin{table}[!htb]
\centering
\begin{tabular}{ll|R{7em}}
Dataset                   & Surrogate & Vanishing gradients \\
\hline
\multirow{2}{*}{CIFAR-10} & cSGLD  & 10.93 \%  ±1.31      \\
                          & DNN    & 64.96 \%  ±0.56       \\
\hline
\multirow{2}{*}{ImageNet} & cSGLD  & 0.06 \%  ±0.06     \\
                          & DNN    & 0.11 \%  ±0.03    \\
\end{tabular}
\caption{Proportion of gradients with a L$2$ norm lower than $10^{-8}$, the tolerance used by the library Adversarial Robustness Toolbox in L$2$ attacks to avoid division by 0 in norm scaling. Gradients are of the models of RQ1~(PreResNet110 for CIFAR-10, ResNet-50 for ImageNet) and on 10~000 original examples of the test sets. Means and standard deviations of 15 models are reported.}
\label{tab:zero-grads}
\end{table}

\begin{table*}[!htb]
\centering
\begin{tabular}{lll|R{6.5em}R{7em}|R{5em}R{6em}}
Dataset & Attack & Surrogate &  L2 Attack &    L$\infty$ Attack &  Nb training epochs &  Nb backward passes \\
\hline
\multirow{20}{*}{CIFAR10} & \multirow{5}{*}{I-FG(S)M} & cSGLD &         \textbf{92.38\% ±0.23} &         92.74\% ±0.33 &                 300 &                  50 \\
         &        & 1 DNN &         43.17\% ±0.97 &         77.59\% ±0.01 &                 300 &                  50 \\
         &        & 2 DNNs &         52.08\% ±1.03 &         84.75\% ±0.20 &                 600 &                  50 \\
         &        & 5 DNNs &         58.74\% ±0.98 &         94.81\% ±0.17 &                1500 &                  50 \\
         &        & 15 DNNs &         62.08\% ±0.92 &         \textbf{97.83\% ±0.03} &                4500 &                  50 \\
\cline{2-7}
         & \multirow{5}{*}{MI-FG(S)M} & cSGLD &         92.29\% ±0.25 &         94.20\% ±0.14 &                 300 &                  50 \\
         &        & 1 DNN &         72.34\% ±0.23 &         80.43\% ±0.04 &                 300 &                  50 \\
         &        & 2 DNNs &         84.10\% ±0.33 &         90.70\% ±0.07 &                 600 &                  50 \\
         &        & 5 DNNs &         91.66\% ±0.26 &         97.04\% ±0.07 &                1500 &                  50 \\
         &        & 15 DNNs &         \textbf{93.87\% ±0.30} &         \textbf{98.30\% ±0.11} &                4500 &                  50 \\
\cline{2-7}
         & \multirow{5}{*}{PGD \footnotesize{(5 restarts)}} & cSGLD &         \textbf{91.65\% ±0.33} &         92.10\% ±0.25 &                 300 &                 250 \\
         &        & 1 DNN &         51.08\% ±0.10 &         77.58\% ±0.38 &                 300 &                 250 \\
         &        & 2 DNNs &         60.60\% ±0.06 &         83.67\% ±0.27 &                 600 &                 250 \\
         &        & 5 DNNs &         67.55\% ±0.21 &         94.19\% ±0.07 &                1500 &                 250 \\
         &        & 15 DNNs &         70.42\% ±0.23 &         \textbf{97.37\% ±0.06} &                4500 &                 250 \\
\cline{2-7}
         & \multirow{5}{*}{FG(S)M} & cSGLD &         \textbf{43.13\% ±0.00} &         58.85\% ±0.01 &                 300 &                  30 \\
         &        & 1 DNN &         20.92\% ±0.00 &         38.89\% ±0.01 &                 300 &                   1 \\
         &        & 2 DNNs &         23.75\% ±0.00 &         45.83\% ±0.01 &                 600 &                   2 \\
         &        & 5 DNNs &         25.60\% ±0.00 &         54.62\% ±0.01 &                1500 &                   5 \\
         &        & 15 DNNs &         26.71\% ±0.00 &         \textbf{61.81\% ±0.00} &                4500 &                  15 \\
\cline{1-7}
\multirow{20}{*}{ImageNet} & \multirow{5}{*}{I-FG(S)M} & cSGLD &  94.41\% ±0.46 &  90.77\% ±0.09 &                 225 &                  50 \\
         &        & 1 DNN &  64.95\% ±0.54 &  57.79\% ±0.17 &                 130 &                  50 \\
         &        & 2 DNNs &  80.39\% ±0.83 &  74.25\% ±0.71 &                 260 &                  50 \\
         &        & 5 DNNs &  94.53\% ±0.43 &  92.81\% ±0.45 &                 650 &                  50 \\
         &        & 15 DNNs &  \textbf{98.51\% ±0.11} &  \textbf{98.28\% ±0.16} &                1950 &                  50 \\
\cline{2-7}
         & \multirow{5}{*}{MI-FG(S)M} & cSGLD &  93.42\% ±0.73 &  93.61\% ±0.41 &                 225 &                  50 \\
         &        & 1 DNN &  61.11\% ±0.35 &  63.70\% ±0.21 &                 130 &                  50 \\
         &        & 2 DNNs &  77.93\% ±0.44 &  79.27\% ±0.76 &                 260 &                  50 \\
         &        & 5 DNNs &  94.41\% ±0.47 &  95.32\% ±0.25 &                 650 &                  50 \\
         &        & 15 DNNs &  \textbf{98.89\% ±0.13} &  \textbf{99.19\% ±0.13} &                1950 &                  50 \\
\cline{2-7}
         & \multirow{5}{*}{PGD \footnotesize{(5 restarts)}} & cSGLD &  91.81\% ±0.38 &  88.76\% ±0.24 &                 225 &                 250 \\
         &        & 1 DNN &  57.47\% ±0.52 &  53.79\% ±0.45 &                 130 &                 250 \\
         &        & 2 DNNs &  74.04\% ±0.47 &  70.90\% ±0.41 &                 260 &                 250 \\
         &        & 5 DNNs &  91.99\% ±0.41 &  91.27\% ±0.59 &                 650 &                 250 \\
         &        & 15 DNNs &  \textbf{97.83\% ±0.20} &  \textbf{97.65\% ±0.21} &                1950 &                 250 \\
\cline{2-7}
         & \multirow{5}{*}{FG(S)M} & cSGLD &  58.91\% ±0.11 &  67.17\% ±0.26 &                 225 &                  15 \\
         &        & 1 DNN &  37.37\% ±0.19 &  44.55\% ±0.72 &                 130 &                   1 \\
         &        & 2 DNNs &  46.73\% ±0.34 &  53.91\% ±0.60 &                 260 &                   2 \\
         &        & 5 DNNs &  58.17\% ±0.18 &  65.53\% ±0.10 &                 650 &                   5 \\
         &        & 15 DNNs &  \textbf{68.48\% ±0.52} &  \textbf{76.57\% ±0.62} &                1950 &                  15 \\
\end{tabular}
\caption{Intra-architecture transfer success rates of 4 attacks on PreResNet110 (CIFAR-10) and ResNet50 (ImageNet). Bold is best. Higher is better.}
\label{tab:transfer-same-arch}
\end{table*}

\begin{table*}[!htb]
\centering
\resizebox{1.95\columnwidth}{!}{
\begin{tabular}{lll|rrrrr}
           &            & Target &    PreResNet110 &  PreResNet164 &       VGG16bn &       VGG19bn &    WideResNet \\
Attack & Surrogate &  &                 &               &               &               &               \\
\hline
\multirow{15}{*}{L$2$} & \multirow{3}{*}{PreResNet110} & cSGLD &    \textbf{88.96\% ±0.02} &  \textbf{88.57\% ±0.00} &  26.18\% ±0.02 &  24.38\% ±0.00 &  \textbf{63.35\% ±0.01} \\
           &                                           & 1 DNN &    34.42\% ±0.00 &  34.39\% ±0.01 &  12.66\% ±0.01 &  12.54\% ±0.00 &  26.29\% ±0.00 \\
           &                                           & 4 DNNs &    50.50\% ±0.00 &  50.49\% ±0.00 &  \textbf{27.45\% ±0.01} &  \textbf{27.30\% ±0.00} &  46.10\% ±0.00 \\
\cline{2-8}
           & \multirow{3}{*}{PreResNet164}  & cSGLD &    \textbf{88.28\% ±0.01} &  \textbf{87.52\% ±0.01} &  25.83\% ±0.01 &  23.64\% ±0.01 &  \textbf{62.79\% ±0.01} \\
           &                                & 1 DNN &    33.89\% ±0.00 &  34.36\% ±0.01 &  11.93\% ±0.00 &  12.07\% ±0.01 &  25.95\% ±0.01 \\
           &                                & 4 DNNs &    50.36\% ±0.01 &  50.45\% ±0.00 &  \textbf{26.79\% ±0.01} &  \textbf{27.13\% ±0.00} &  45.94\% ±0.00 \\
\cline{2-8}
           & \multirow{3}{*}{VGG16bn}   & cSGLD &    \textbf{69.22\% ±0.06} &  \textbf{69.03\% ±0.03} &  43.70\% ±0.04 &  38.54\% ±0.02 &  \textbf{55.62\% ±0.07} \\
           &                            & 1 DNN &    27.22\% ±0.04 &  27.23\% ±0.05 &  29.28\% ±0.08 &  28.73\% ±0.02 &  22.22\% ±0.00 \\
           &                            & 4 DNNs &    55.14\% ±0.06 &  54.96\% ±0.04 &  \textbf{73.65\% ±0.00} &  \textbf{71.24\% ±0.04} &  44.89\% ±0.09 \\
\cline{2-8}
           & \multirow{3}{*}{VGG19bn}   & cSGLD &    \textbf{69.82\% ±0.05} &  \textbf{68.27\% ±0.07} &  44.59\% ±0.10 &  39.76\% ±0.13 &  \textbf{54.40\% ±0.08} \\
           &                            & 1 DNN &    18.09\% ±0.10 &  18.09\% ±0.06 &  $\star$44.63\% ±0.03 &  $\star$46.76\% ±0.03 &  14.38\% ±0.03 \\
           &                            & 4 DNNs &    34.30\% ±0.06 &  33.77\% ±0.01 &  \textbf{66.20\% ±0.03} &  \textbf{68.87\% ±0.05} &  27.44\% ±0.02 \\
\cline{2-8}
           & \multirow{3}{*}{WideResNet}    & cSGLD &    \textbf{82.25\% ±0.03} &  \textbf{85.06\% ±0.02} &  \textbf{26.34\% ±0.08} &  \textbf{23.81\% ±0.03} &  \textbf{69.31\% ±0.07} \\
           &                                & 1 DNN &    22.14\% ±0.01 &  23.00\% ±0.00 &   9.43\% ±0.00 &   9.54\% ±0.00 &  26.85\% ±0.00 \\
           &                                & 4 DNNs &    41.07\% ±0.00 &  41.75\% ±0.04 &  22.91\% ±0.04 &  22.65\% ±0.03 &  43.00\% ±0.01 \\
\cline{1-8}
\multirow{15}{*}{L$\infty$} & \multirow{3}{*}{PreResNet110} & cSGLD &    88.70\% ±0.00 &  88.48\% ±0.01 &  26.32\% ±0.00 &  24.27\% ±0.01 &  62.95\% ±0.01 \\
           &                                                & 1 DNN &    72.73\% ±0.00 &  74.57\% ±0.00 &  22.26\% ±0.00 &  20.98\% ±0.00 &  47.59\% ±0.01 \\
           &                                                & 4 DNNs &    \textbf{91.98\% ±0.00} &  \textbf{92.25\% ±0.00} &  \textbf{38.24\% ±0.00} &  \textbf{35.56\% ±0.00} &  \textbf{72.64\% ±0.01} \\
\cline{2-8}
           & \multirow{3}{*}{PreResNet164}  & cSGLD  &    87.99\% ±0.01 &  87.74\% ±0.00 &  26.33\% ±0.00 &  23.67\% ±0.01 &  61.83\% ±0.02 \\
           &                                & 1 DNN  &    68.97\% ±0.01 &  71.76\% ±0.00 &  20.29\% ±0.00 &  18.86\% ±0.00 &  45.07\% ±0.00 \\
           &                                & 4 DNNs &    \textbf{90.67\% ±0.00} &  \textbf{92.22\% ±0.00} &  \textbf{37.62\% ±0.00} &  \textbf{35.23\% ±0.00} &  \textbf{73.18\% ±0.00} \\
\cline{2-8}
           & \multirow{3}{*}{VGG16bn}   & cSGLD &    \textbf{66.97\% ±0.13} &  \textbf{67.48\% ±0.11} &  42.91\% ±0.05 &  37.91\% ±0.02 &  \textbf{50.52\% ±0.01} \\
           &                            & 1 DNN &    35.57\% ±0.02 &  35.89\% ±0.03 &  38.35\% ±0.00 &  35.82\% ±0.00 &  26.77\% ±0.02 \\
           &                            & 4 DNNs &    52.59\% ±0.00 &  53.12\% ±0.00 &  \textbf{70.89\% ±0.00} &  \textbf{68.53\% ±0.00} &  41.34\% ±0.00 \\
\cline{2-8}
           & \multirow{3}{*}{VGG19bn}   & cSGLD &    \textbf{67.11\% ±0.00} &  \textbf{66.55\% ±0.02} &  43.50\% ±0.01 &  38.72\% ±0.02 &  \textbf{49.69\% ±0.02} \\
           &                            & 1 DNN &    20.50\% ±0.02 &  20.97\% ±0.00 &  $\star$45.90\% ±0.02 &  $\star$48.60\% ±0.02 &  16.37\% ±0.01 \\
           &                            & 4 DNNs &    32.43\% ±0.06 &  32.25\% ±0.04 &  \textbf{63.11\% ±0.07} &  \textbf{65.64\% ±0.06} &  25.34\% ±0.02 \\
\cline{2-8}
           & \multirow{3}{*}{WideResNet}    & cSGLD &    \textbf{81.99\% ±0.01} &  \textbf{85.63\% ±0.01} &  27.04\% ±0.02 &  23.46\% ±0.01 &  68.43\% ±0.01 \\
           &                                & 1 DNN &    49.24\% ±0.16 &  52.84\% ±0.03 &  20.23\% ±0.04 &  18.53\% ±0.02 &  60.84\% ±0.09 \\
           &                                & 4 DNNs &    77.45\% ±0.01 &  79.55\% ±0.13 &  \textbf{36.33\% ±0.13} &  \textbf{33.60\% ±0.22} &  \textbf{83.24\% ±0.00} \\
\end{tabular}
}
\caption{Inter-architecture transfer success rates of I-FG(S)M of single architecture surrogate on CIFAR-10. All combinations of surrogate and targeted architectures are evaluated. Diagonals are intra-architecture. Symbols $\star$ indicate 1 DNN having higher transferability than cSGLD. 1 DNN and cSGLD have similar computation budget (300 epochs). Bold is best. Higher is better.}
\label{tab:transfer-inter-combinaison-cifar}
\end{table*}

\begin{table*}[!htb]
\centering
\resizebox{1.95\columnwidth}{!}{
\begin{tabular}{lll|rrrrr}
           &            & Target &    ResNet50 &     ResNeXt50 &   DenseNet121 &       MNASNet & EfficientNetB0 \\
Attack & Surrogate &  &                 &               &               &               &               \\
\hline
\multirow{10}{*}{L$2$} & \multirow{2}{*}{ResNet50} & cSGLD &    \textbf{84.93\% ±0.59} &  \textbf{74.70\% ±0.91} &  \textbf{71.32\% ±0.63} &  \textbf{60.09\% ±0.60} &    \textbf{39.70\% ±0.29} \\
           &                                       & 1 DNN &    56.98\% ±0.62 &  41.13\% ±0.97 &  29.81\% ±0.33 &  27.90\% ±0.43 &    16.39\% ±0.46 \\
\cline{2-8}
           & \multirow{2}{*}{ResNeXt50} & cSGLD &    \textbf{79.25\% ±0.24} &  \textbf{77.34\% ±0.39} &  \textbf{68.53\% ±0.19} &  \textbf{62.16\% ±0.19} &    \textbf{43.51\% ±0.62} \\
           &                            & 1 DNN &    37.48\% ±0.52 &  36.35\% ±0.22 &  23.77\% ±0.41 &  23.69\% ±0.21 &    14.32\% ±0.24 \\
\cline{2-8}
           & \multirow{2}{*}{DenseNet121} & cSGLD &    \textbf{63.23\% ±1.16} &  \textbf{59.89\% ±1.12} &  \textbf{73.28\% ±0.45} &  \textbf{60.84\% ±0.33} &    \textbf{40.27\% ±0.44} \\
           &                            & 1 DNN &    32.61\% ±0.29 &  32.06\% ±0.61 &  39.18\% ±0.47 &  32.01\% ±0.44 &    17.72\% ±0.49 \\
\cline{2-8}
           & \multirow{2}{*}{MNASNet} & cSGLD &     \textbf{7.81\% ±0.19} &   \textbf{5.97\% ±0.37} &   \textbf{9.81\% ±0.31} &  30.41\% ±1.45 &    \textbf{15.46\% ±0.44} \\
           &                          & 1 DNN &     7.04\% ±0.51 &   5.29\% ±0.36 &   8.41\% ±0.20 &  \textbf{32.65\% ±0.22} &    13.13\% ±0.06 \\
\cline{2-8}
           & \multirow{2}{*}{EfficientNetB0} & cSGLD &    \textbf{18.93\% ±2.17} &  \textbf{14.16\% ±1.69} &  \textbf{19.89\% ±1.21} &  \textbf{65.97\% ±3.60} &    \textbf{49.41\% ±3.64} \\
           &                                  & 1 DNN &    15.15\% ±0.30 &  13.33\% ±0.33 &  16.12\% ±0.71 &  58.73\% ±0.25 &    48.85\% ±0.56 \\
\cline{1-8}
\multirow{10}{*}{L$\infty$} & \multirow{2}{*}{ResNet50} & cSGLD &    \textbf{78.67\% ±1.19} &  \textbf{65.21\% ±1.48} &  \textbf{61.54\% ±0.83} &  \textbf{51.75\% ±1.39} &    \textbf{31.11\% ±1.13} \\
           &                                            & 1 DNN &    48.03\% ±0.94 &  32.17\% ±0.43 &  23.37\% ±0.34 &  22.60\% ±0.40 &    12.59\% ±0.21 \\
\cline{2-8}
           & \multirow{2}{*}{ResNeXt50} & cSGLD &    \textbf{71.67\% ±1.00} &  \textbf{69.33\% ±0.85} &  \textbf{59.18\% ±1.14} &  \textbf{54.75\% ±1.33} &    \textbf{35.13\% ±0.71} \\
           &                            & 1 DNN &    31.19\% ±0.42 &  28.68\% ±0.76 &  19.12\% ±0.07 &  19.53\% ±0.51 &    11.20\% ±0.33 \\
\cline{2-8}
           & \multirow{2}{*}{DenseNet121} & cSGLD &    \textbf{54.13\% ±1.70} &  \textbf{50.66\% ±1.62} &  \textbf{65.80\% ±0.66} &  \textbf{53.43\% ±1.30} &    \textbf{32.49\% ±0.36} \\
           &                              & 1 DNN &    25.49\% ±0.81 &  23.73\% ±0.59 &  30.78\% ±0.21 &  26.05\% ±0.66 &    13.41\% ±0.20 \\
\cline{2-8}
           & \multirow{2}{*}{MNASNet} & cSGLD &     \textbf{6.77\% ±0.29} &   4.72\% ±0.27 &   \textbf{8.26\% ±0.36} &  25.27\% ±1.83 &    \textbf{12.21\% ±0.84} \\
           &                          & 1 DNN &     6.52\% ±0.23 &   \textbf{5.06\% ±0.12} &   7.83\% ±0.13 &  \textbf{29.19\% ±0.05} &    11.13\% ±0.16 \\
\cline{2-8}
           & \multirow{2}{*}{EfficientNetB0} & cSGLD &    \textbf{17.81\% ±1.58} &  \textbf{13.91\% ±1.45} &  \textbf{19.71\% ±1.29} &  \textbf{63.67\% ±3.16} &    46.91\% ±3.44 \\
           &                                  & 1 DNN &    15.83\% ±0.32 &  13.51\% ±0.52 &  16.78\% ±0.38 &  60.14\% ±0.37 &    \textbf{50.16\% ±0.64} \\
\end{tabular}
}
\caption{Inter-architecture transfer success rates of I-FG(S)M of single architecture surrogate on ImageNet. All combinations of surrogate and targeted architectures are evaluated. Diagonals are intra-architecture. 1 DNN and cSGLD have similar computation budget (135 epochs). Bold is best. Higher is better.}
\label{tab:transfer-inter-combinaison-imagenet}
\end{table*}

\begin{table*}[t!]
\centering
\resizebox{1.95\columnwidth}{!}{
\begin{tabular}{ll|rrrrr}
Attack                      & Surrogate                        & PreResNet110           & PreResNet164           & VGG16bn                & VGG19bn                & WideResNet             \\
\hline
\multirow{16}{*}{L$2$}      & 1 DNN                            & 34.42\% ±0.00          & 34.39\% ±0.01          & 12.67\% ±0.00          & 12.54\% ±0.00          & 26.29\% ±0.01          \\
                            & \quad+ Input Diversity           & 59.63\% ±0.80          & 59.79\% ±0.75          & 24.37\% ±0.16          & 23.25\% ±0.12          & 46.09\% ±0.47          \\
                            & \quad+ Skip Gradient Method      & 57.00\% ±0.00          & 57.66\% ±0.04          & 20.87\% ±0.03          & 20.10\% ±0.09          & 41.80\% ±0.04          \\
                            & \quad+ Ghost Networks            & 79.22\% ±0.30          & 80.38\% ±0.16          & $\star$32.03\% ±0.25   & $\star$28.63\% ±0.17   & 56.65\% ±0.24          \\
                            & \quad+ Momentum                  & 67.12\% ±0.07          & 67.80\% ±0.00          & 20.49\% ±0.02          & 19.15\% ±0.01          & 44.11\% ±0.04          \\
                            & \quad\quad+ Input Diversity      & 81.44\% ±0.32          & 82.69\% ±0.29          & 27.64\% ±0.03          & 25.82\% ±0.42          & 57.29\% ±0.12          \\
                            & \quad\quad+ Skip Gradient Method & 73.52\% ±0.00          & 75.23\% ±0.01          & 24.52\% ±0.00          & 22.76\% ±0.00          & 49.73\% ±0.00          \\
                            & \quad\quad+ Ghost Networks       & 77.44\% ±0.28          & 79.13\% ±0.12          & $\star$28.98\% ±0.57   & 25.74\% ±0.18          & 54.06\% ±0.04          \\
                            \cline{2-7}
                            & cSGLD                            & 90.67\% ±0.39          & 89.74\% ±0.31          & 28.05\% ±0.33          & 26.12\% ±0.14          & 67.27\% ±0.89          \\
                            & \quad+ Input Diversity           & 92.45\% ±0.14          & 91.80\% ±0.14          & 33.69\% ±0.28          & 31.35\% ±0.28          & 72.41\% ±0.76          \\
                            & \quad+ Skip Gradient Method      & 92.46\% ±0.17          & 92.10\% ±0.28          & 31.96\% ±0.53          & 29.84\% ±0.34          & 71.04\% ±1.23          \\
                            & \quad+ Ghost Networks            & \textbf{92.73\% ±0.21} & \textbf{92.20\% ±0.07} & \textbf{36.17\% ±0.39} & \textbf{33.08\% ±0.32} & \textbf{74.77\% ±0.10} \\
                            & \quad+ Momentum                  & $\dagger$90.35\% ±0.37 & 89.77\% ±0.28          & $\dagger$26.89\% ±0.37 & $\dagger$25.02\% ±0.29 & $\dagger$65.98\% ±0.52 \\
                            & \quad\quad+ Input Diversity      & 92.31\% ±0.33          & 91.58\% ±0.23          & 31.92\% ±0.49          & 29.72\% ±0.46          & 70.94\% ±0.31          \\
                            & \quad\quad+ Skip Gradient Method & 92.33\% ±0.34          & 91.94\% ±0.41          & 31.95\% ±0.29          & 29.85\% ±0.28          & 70.96\% ±0.65          \\
                            & \quad\quad+ Ghost Networks       & 92.42\% ±0.16          & 91.93\% ±0.25          & 33.02\% ±0.60          & 29.77\% ±0.14          & 72.28\% ±0.53          \\
\hline
\multirow{16}{*}{L$\infty$} & 1 DNN                            & 72.73\% ±0.00          & 74.58\% ±0.01          & 22.26\% ±0.00          & 20.98\% ±0.00          & 47.59\% ±0.01          \\
                            & \quad+ Input Diversity           & 81.29\% ±0.18          & 82.77\% ±0.12          & 28.10\% ±0.22          & 26.17\% ±0.25          & 57.04\% ±0.10          \\
                            & \quad+ Skip Gradient Method      & 77.92\% ±0.00          & 79.50\% ±0.01          & 27.43\% ±0.00          & 25.31\% ±0.01          & 53.39\% ±0.00          \\
                            & \quad+ Ghost Networks            & 74.92\% ±0.08          & 77.23\% ±0.26          & $\star$29.61\% ±0.19   & 26.31\% ±0.30          & 52.93\% ±0.05          \\
                            & \quad+ Momentum                  & 76.12\% ±0.01          & 78.05\% ±0.00          & 23.77\% ±0.02          & 22.33\% ±0.01          & 50.49\% ±0.01          \\
                            & \quad\quad+ Input Diversity      & 84.66\% ±0.19          & 86.38\% ±0.12          & $\star$31.47\% ±0.05   & $\star$28.89\% ±0.31   & 61.60\% ±0.16          \\
                            & \quad\quad+ Skip Gradient Method & 79.72\% ±0.02          & 80.80\% ±0.02          & 28.75\% ±0.01          & 26.12\% ±0.00          & 55.74\% ±0.00          \\
                            & \quad\quad+ Ghost Networks       & 80.34\% ±0.34          & 82.59\% ±0.42          & $\star$34.17\% ±0.48   & $\star$29.37\% ±0.18   & 60.62\% ±0.40          \\
                            \cline{2-7}
                            & cSGLD                            & 90.98\% ±0.40          & 90.26\% ±0.35          & 29.26\% ±0.53          & 26.97\% ±0.43          & 67.18\% ±1.03          \\
                            & \quad+ Input Diversity           & 92.46\% ±0.14          & 91.62\% ±0.16          & 33.81\% ±0.25          & 30.84\% ±0.34          & 71.15\% ±0.92          \\
                            & \quad+ Skip Gradient Method      & 93.38\% ±0.50          & 92.84\% ±0.25          & 35.68\% ±0.61          & 32.43\% ±0.52          & 73.55\% ±1.08          \\
                            & \quad+ Ghost Networks            & 91.66\% ±0.40          & 91.32\% ±0.19          & 34.77\% ±0.09          & 31.01\% ±0.27          & 71.60\% ±0.40          \\
                            & \quad+ Momentum                  & 92.84\% ±0.18          & 92.18\% ±0.28          & 32.03\% ±0.49          & 28.53\% ±0.38          & 71.56\% ±0.25          \\
                            & \quad\quad+ Input Diversity      & 94.05\% ±0.31          & 93.53\% ±0.21          & 37.31\% ±0.38          & 33.23\% ±0.23          & 75.40\% ±0.25          \\
                            & \quad\quad+ Skip Gradient Method & \textbf{94.64\% ±0.26} & \textbf{94.29\% ±0.31} & \textbf{38.08\% ±0.27} & \textbf{34.28\% ±0.17} & \textbf{76.62\% ±0.50} \\
                            & \quad\quad+ Ghost Networks       & 93.76\% ±0.14          & 93.75\% ±0.13          & 38.01\% ±0.44          & 33.15\% ±0.36          & 76.23\% ±0.29
\end{tabular}
}
\caption{Transfer success rates of (M)I-FG(S)M attack improved by our approach combined with test-time transformations on CIFAR-10. Columns are targets. PreResNet110 columns are intra-architecture transferability, others are inter-architecture. Bold is best. Symbols $\star$ are DNN-based techniques better than our vanilla cSGLD surrogate, and $\dagger$ are techniques that doesn't improved the corresponding vanilla surrogate. The success rate for every cSGLD-based technique is better than its counterpart with 1 DNN.}
\label{tab:transfer-test-time-techs-cifar}
\end{table*}

\begin{table*}[ht]
\centering
\begin{tabular}{lL{6.5em}|L{5em}L{5em}L{5em}|L{6.5em}L{6em}}
                       & Hyperparameter        & \multicolumn{3}{c|}{CIFAR-10}                              & \multicolumn{2}{c}{ImageNet}           \\
Method                 &                       & cSGLD                        & DNN Surrogate & DNN Target & cSGLD              & DNN Surrogate     \\
\hline
\multirow{7}{*}{All}   & Number epochs             & 50 per cycle                 & 300           & 300        & 45 per cycle       & 130 \newline (135 for $\star$) \\
\cline{2-7}
                       & Initial learning rate & 0.5                          & 0.01          & 0.01       & 0.1                & 0.1               \\
\cline{2-7}
 &
  Learning rate schedule &
  Cosine Annealing &
  Step size decay \newline ($\times 0.1$ each 75 epochs) &
  Step size decay \newline ($\times 0.1$ each 75 epochs) &
  Cosine Annealing &
  Step size decay \newline ($\times 0.1$ each 30 epochs) \\
\cline{2-7}
                       & Optimizer             & cSGLD                        & Adam          & Adam       & cSGLD              & SGD               \\
\cline{2-7}
                       & Momentum              & 0                            & 0.9           & 0.9        & 0.9                & 0.9               \\
\cline{2-7}
                       & Weight decay          & 5e-4 \newline (3e-4 for PreResNet)  & 1e-4      & 1e-4   & 1e-4           & 1e-4          \\
\cline{2-7}
 &
  Batch-size &  64 &   128 &  128 &  256 for ResNet50, \newline 64 for others &  256 for ResNet50, \newline 64 for others \\
\hline
\multirow{4}{*}{cSGLD} & Sampling interval     & 1 sample per epoch           &        -       &    -        & 1 sample per epoch &            -       \\
\cline{2-7}
                       & Nb cycles             & 6\newline (18 for $\star\star$)                           &        -       &     -       & 5\newline (3 for $\star$, 6 for $\star\star$)      &        -           \\
\cline{2-7}
                       & Nb samples per cycle  & 5                            &       -        &      -      & 3                  &       -            \\
\cline{2-7}
                       & Nb epochs with noise  & 5                            &      -         &     -       & 3                  &      -            \\
\end{tabular}
\caption{Hyperparameters used to train cSGLD or Deep Ensemble. The $\star$ symbols refer to the inter-architecture and test-time techniques sections, and $\star\star$ to the Bayesian and Ensemble training methods section. Target DNN on ImageNet is not included because it is retrieved from PyTorch and timm pretrained models.}
\label{tab:hps-train}
\end{table*}

\begin{table*}[!ht]
\centering
\begin{tabular}{ll|ll}
Attack / Technique           & Hyperparameter             & CIFAR-10 & ImageNet \\
\hline
\multirow{2}{*}{All attacks} & Perturbation $2$-norm $\varepsilon$     & 0.5      & 3        \\
                             & Perturbation $\infty$-norm $\varepsilon$ & $\frac{4}{255}$ & $\frac{4}{255}$  \\
\hline
\multirow{2}{*}{Iterative Attacks} & Step-size $\alpha$   & $\frac{\varepsilon}{10}$  & $\frac{\varepsilon}{10}$ \\
                             & Number iterations          & 50       & 50       \\
\hline
MI-FG(S)M                    & Momentum term              & 0.9      & 0.9      \\
\hline
PGD                          & Number random restarts     & 5        & 5        \\
\hline
Ghost Network                      & Skip connection erosion random range & {[}1-0.22, 1+0.22{]} & {[}1-0.22, 1+0.22{]} \\
\hline
\multirow{2}{*}{Input Diversity}   & Minimum resize ratio                 & 90 \%                & 90 \%                \\
                             & Probability transformation & 0.5      & 0.5      \\
\hline
Skip Gradient Method         & Residual Gradient Decay $\gamma$            & 0.7 (PreResNet110)   & 0.2 (ResNet50)       \\
\end{tabular}
\caption{Hyperparameters of attacks and test-time transferability techniques.}
\label{tab:hps-attack}
\end{table*}

\clearpage

\begin{figure*}[!htb]
\centering
\includegraphics[width=0.95\textwidth]{figure/nb_iters_all_attacks_CIFAR}
\caption{Transfer success rates on CIFAR-10 of three iterative gradient-based attacks on the same architecture (PreResNet110) with respect to the number of iterations.}
\label{fig:nb-iters-cifar}
\end{figure*}

\begin{figure*}[!htb]
\centering
\includegraphics[width=0.95\textwidth]{figure/nb_iters_all_attacks_ImageNet}
\caption{Transfer success rates on ImageNet of three iterative gradient-based attacks on the same architecture (ResNet-50) with respect to the number of iterations.}
\label{fig:nb-iters-imagenet}
\end{figure*}

\begin{figure}[!htb]
\centering
\includegraphics[width=0.9\columnwidth]{figure/IFSGM_transfer_vs_nb_cycles}
\caption{Intra-architecture transfer success rate of I-FG(S)M with respect to the number of cSGLD cycles on CIFAR-10 (PreResNet110).}
\label{fig:nb-cycles}
\end{figure}

\begin{figure}[!htb]
\centering
\includegraphics[width=0.9\columnwidth]{figure/IFSGM_transfer_vs_nb_samples_per_cycle}
\caption{Intra-architecture transfer success rate of I-FG(S)M with respect to the number of cSGLD samples per cycle. We train one PreResNet110 cSGLD on CIFAR-10 for every number of cycles from 1 to 10 samples per cycle. Each additional sample per cycle increases the training cost by 1 epoch per cycle (starting at 48 epochs per cycle). A fixed number of 5 cSGLD cycles is used.}
\label{fig-nb-samples-per-cycle}
\end{figure}

\begin{figure*}[!htb]
\centering
\includegraphics[width=0.65\textwidth]{figure/HP_SGM_tuning.pdf}
\caption{Transfer success rates of the test-time transferability techniques Skip Gradient Method with varying values of its hyper-parameter $\gamma$ between $0$ and $1$ with $0.1$ steps. The surrogate is a PreResNet110 DNN trained on CIFAR-10 and evaluated on 1~independently trained DNN for every targeted architecture. The plain line represents the intra-architecture transferability, and the dotted ones the inter-architecture transferability. $\gamma = 0.7$ is selected in the rest of the paper for PreResNet110.}
\label{fig:hp-tuning-sgm}
\end{figure*}