\section{Introduction}
\vspace{-0.1in}
Over the years, the machine learning field has developed myriad techniques for training neural networks.
In image classification,
these include data augmentation, regularization, architectures, losses, pre-training schemes, and more.
Such techniques
have highlighted the ability of networks to capture
diverse features of the data:
textures/shapes~\citep{geirhos2018imagenet}, robust/non-robust features \citep{ilyas2019adversarial}, and even features that fit a random, pre-determined classifier~\citep{hoffer2018fix}.

Despite this representation-learning power, methods that yield high generalization performance seem to produce networks with little behavior diversity: models make similar predictions, with high-accuracy models rarely making mistakes that low-accuracy models predict correctly~\citep{mania2019model}.
Additionally, the quality of features learned (e.g.: for downstream tasks) seems dictated by upstream performance~\citep{kornblith2019better}.
Finally, training on subsets of the data yields low-accuracy models that don't make performant ensembles~\citep{nixon2020bootstrapped}.
This seemingly suggests that high-performing models share similar biases, regardless of training methodology.

Without behavior diversity, ensemble benefits are limited to reducing noise, since models make correlated errors~\citep{perrone1992networks, opitz1999popular}.
Without feature diversity, representations might not capture important features for downstream tasks, since feature reuse has been shown to be crucial for transfer learning~\citep{neyshabur2020being}.
Without knowing the effect of training methodology, one might conclude that low-accuracy models have no practical use, since their predictions would be dominated by high-accuracy ones.

One open question is if these findings faced unavoidable selection bias, since the highest-performing models have historically been trained with similar supervised objectives on IID datasets.
Up until recently, this hypothesis was difficult to test.
That changed with the recent success of large-scale contrastive learning, which produces competitively-high accuracy on standard generalization and robustness benchmarks~\citep{radford2021learning, jia2021scaling}.
This motivates revisiting the question:
\begin{center}
    \textit{How does training methodology affect learned representations and prediction behavior?}
\end{center}

In this paper, we conduct a systematic empirical study of 82 models, which we train or collect, across hyper-parameters, architectures, objective functions, and datasets, including the latest high performing models CLIP, ALIGN, SimCLR, BiT, ViT-G/14, and MPL. In addition to using different techniques, these new models were trained on data collected very differently, allowing us to probe the effect of both training objective, as well as pre-training data. We categorize these models based on how their training methodologies diverge from a typical, base model and show:
\begin{enumerate}
    \item Model pairs that diverge more in training methodology (reinitializations, hyper-parameters, architectures, frameworks, $\rightarrow$ dataset scales) produce increasingly uncorrelated errors.
    \item Ensemble performance increases as error correlation decreases, due to higher ensemble efficiency.
    The most typical ImageNet model (ResNet-50, 76.5\%), and its most different counterpart (ALIGN-ZS, 75.5\%)
    yield 83.4\% accuracy when ensembled, a +7\% boost.
    \item Contrastively-learned models display categorically different generalization behavior, specializing in subdomains of the data, which explains the higher ensembling efficiency. We show CLIP-S specializes in antropogenic images, whereas ResNet-50 excells in nature images.
    \item Surprisingly, we find that low-accuracy models can be useful if they are trained differently enough.
    By combining a high-accuracy model (BiT-1k, 82.9\%) with \textit{only} low-accuracy models (max individual acc. 77.4\%), we can create ensembles that yield as much as 86.7\%.
    \item Diverging training methodology yield representations that capture overlapping (but not supersetting) feature sets which, when combined, lead to increased downstream performance (91.4\% on Pascal VOC, using models with max individual accuracy 90.7\%).
\end{enumerate}
\section{Related Work}

\textbf{Diversity in Ensembles}.
It is widely understood that good ensembles are made of models that are both accurate and make independent errors~\citep{perrone1992networks, opitz1999popular, wen2020batchensemble}.
Beyond improving ensemble performance, finding diverse solutions that equally well explain the observations can help quantify model uncertainty (also known as epistemic uncertainty) -- what the model does not know because training data was not appropriate~\citep{kendall2017uncertainties, fort2019deep}.
Many works have explored ways of finding such solutions.
Boostrapping (ensembling models trained on subsets of the data) was found not to produce deep ensembles with higher accuracy than a single model trained on the entire dataset~\citep{nixon2020bootstrapped}. This emphasizes how much data deep neural networks need to achieve high performance.
Another work has examined the effect of augmentation-induced prediction diversity on adversarial robustness~\citep{liu2019deep}.
More relevant to us, \cite{wenzel2020hyperparameter} has explored the effect of random hyper-parameters, finding best ensembles when combining models that are both hyperparameter and weight-diverse, albeit still considering similar frameworks and architectures.

\textbf{Model Behavior Similarity}.
These attempts were hindered as many high-performing techniques seem to produce similar prediction behavior.
\cite{mania2019model} demonstrates, via ``dominance probabilities'', that high-accuracy models rarely make mistakes that low-accuracy models predict correctly. This indicates that, within the models studied, high-accuracy models ``dominate'' the predictions of low-accuracy ones.
\cite{recht2019imagenet} shows that out-of-distribution robustness seems correlated with in-distribution performance.
Relatedly, \cite{kornblith2019better} shows that upstream and downstream performance are very correlated. These jointly indicate that
high-accuracy models learn strictly better representations, diminishing the importance of low-accuracy solutions (even if they are diverse).
Finally, \cite{fort2019deep} shows that subspace-sampling methods for ensembling generate solutions that, while different in weight space, remain similar in function space, which gives rise to an insufficiently diverse set of predictions.

\textbf{Contrastive-Learning Models; Different Large-Scale Datasets}.
This model behavior similarity might be explained by the fact that the training techniques that yield high performance on image classification tasks have been relatively similar, mostly relying on supervised learning on ImageNet, optionally pre-training on a dataset with similar distribution.
Recently, various works have demonstrated the effectiveness of learning from large-scale data using contrastive learning~\citep{radford2021learning, jia2021scaling}.
They report impressive results on out-of-distribution benchmarks, and have been shown to have higher dominance probabilities~\cite{andreassen2021evolution}.
These represent some of the first models to deviate from standard supervised training (or finetuning) on downstream data, while still yielding competitive accuracy.
They add to the set of high-performing training techniques (which include
data augmentation~\citep{cubuk2018autoaugment, cubuk2020randaugment}, regularization~\citep{JMLR:v15:srivastava14a, szegedy2016rethinking, ghiasi2018dropblock}, architectures~\citep{tan2019efficientnet, dosovitskiy2020image, hu2018squeeze, iandola2014densenet, li2019selective, szegedy2016rethinking, simonyan2014very, sandler2018mobilenetv2}, losses~\citep{chen2020simple, radford2021learning, jia2021scaling}, pre-training schemes~\citep{kolesnikov2020big, pham2021meta}, etc), and
provide the motivation for revisiting the question of whether training methodology can yield different model behavior.
\section{Method}
\subsection{Model Categorization}
\label{sec:model_categories}
In order to evaluate the performance of learned representations as a function of training methodology, we define the following categories, which classify model pairs based on their training differences:%
\begin{enumerate}
    \item \textbf{Reinits}: identical models, just different in reinitialization.
    \item \textbf{Hyper-parameters}: models of the same architecture trained with different hyper parameters (e.g.: weight decay, learning rate, initialization algorithm, etc).
    \item \textbf{Architectures}: models with different architectures, but still trained within the same framework and dataset (e.g.: ResNet and ViT, both with ImageNet supervision).
    \item \textbf{Frameworks}: models trained with different optimization objectives, but on same dataset (e.g.: ResNet and SimCLR, respectively supervised and contrastive learning on ImageNet)
    \item \textbf{Datasets}: models trained on large-scale data (e.g.: CLIP or BiT -- trained on WIT or JFT).
\end{enumerate}

In some sense, model categories can be supersets of one another: when we change a model architecture, we may also change the hyper-parameters used to train such architecture, to make sure that they are optimal for training this new setting.
Unless stated otherwise, all ensembles are comprised of a fixed base model, and another model belonging to one of the categories above.
This way, each category is defined relative to the base model:
model pairs in a given category
will vary because Model 2 is different than the base model along that axis.
The result is that as we navigate along model categories (Reinit $\rightarrow$ ... $\rightarrow$ Dataset), we will naturally be measuring the effect of \textit{increasingly dissimilar training methodology}.

\subsection{Model Selection}

We collect representations and predictions for 82 models, across the many categories above.
We fix ResNet-50, trained with RandAugment, as our base model.
ResNet is a good candidate for a base model since it is one of the most typical ImageNet classification models, and the de-facto standard baseline for this task.
In total, we train/collect models in the categories:
1) Reinit; 2) Hyper-parameters (51): varying dropout, dropblock, learning rate, and weight decay, sometimes jointly; 3) Architectures (17): including EfficientNet, ViT, DenseNet, VGG; 4) Framework (2): including SimCLR, and models trained with distillation; and 5) Dataset (12): including CLIP, ALIGN, BiT, and more, trained on WIT, the ALIGN dataset, JFT, etc.
We additionally collect high-performing models MPL, ALIGN (L-BFGS), ViT-G/14, BiT-1k, CLIP-L, EfficientNet-B3 for some of our analyses. These are some of the latest, highest-performing models for ImageNet classification.

We found it necessary to calibrate all models using temperature scaling~\citep{roelofs2020mitigating, guo2017calibration} to maximize ensemble performance.
Finally, unless stated otherwise, we only use models in the same narrow accuracy range (74-78\% accuracy on ImageNet), which guarantees that the effects observed are indeed a function of diverging training methodology, and not of any given model being intrinsically more performant than another.
A complete list of models can be found in the Appendix.

\subsection{Ensembling}
In our paper, we use ensembling as a tool for understanding: as such, our goal is \textbf{not} to find methods to ensemble the highest accuracy models and reach state-of-the-art. Instead, we wish to use ensembling to probe whether/when training methodology yields uncorrelated (and therefore useful) predictions.

\section{Results}

In order to understand whether model similarity reported in literature varies as a function of training methodology, we evaluate error correlation by measuring the number of test-set examples where one model predicts the correct class, and the other predicts an incorrect class. We call this \textbf{Error Inconsistency}, as it complements the error consistency measure, defined in \citet{geirhos2020beyond}.

We choose error inconsistency to measure error correlation because it allows us to connect the ensemble prediction diversity (which we are interested in quantifying) directly into performance.
That is, when errors are consistent (i.e.: both models make a correct prediction, or both models make an incorrect prediction), this agreement directly translates into higher/lower accuracy after ensembling.
When errors are inconsistent (i.e.: when one model is correct and another is incorrect), the
the ensemble prediction will be determined by the models' confidences. If a model is sufficiently confident, its prediction will ``win'' the ensembling procedure. If this model's prediction was correct, we say that this prediction was ``converted'' into a correct ensemble prediction.

\subsection{As training methodology diverges, errors become more uncorrelated}

In Fig.~\ref{fig:comparing_models_in_ensemble}
we see that, as we compare increasingly different training methodologies
(``Reinit'' $\rightarrow$ ... $\rightarrow$ ``Dataset''),
error inconsistency increases -- the number of examples where \textit{only one} model makes a correct prediction. This indicates that as training methodology diverges, model predictions become dissimilar and, as a result, errors become uncorrelated.

This also represents an opportunity for these uncorrelated mistakes to be converted into a correct prediction, when we ensemble the models.
Such an opportunity will be beneficial
if the conversion rate of these examples is higher than the decrease in number of examples where both models made correct predictions.
As we will see in the following section, this indeed happens.
\begin{figure}
    \centering
    \includegraphics[height=0.3\linewidth]{figures/4_results/observed_inconsistency_marker.pdf}
    \includegraphics[height=0.3\linewidth]{figures/4_results/observed_error_inconsistency.pdf}
    \caption{
    \textbf{As training methodology diverges} (Reinit $\rightarrow$ Dataset), \textbf{errors become uncorrelated}.
    \textit{Left}: 'Observed error inconsistency' is the fraction of examples where only one model in the pair makes a correct prediction. Higher error inconsistency indicates the model errors are uncorrelated.
    \textit{Right}: As models are trained with increasingly different methodologies, their error inconsistency grows, providing an opportunity for converting these examples into correct ensemble predictions.
    }
    \label{fig:comparing_models_in_ensemble}
\end{figure}

\subsection{As uncorrelated errors increase, so does ensemble efficiency}
In order to assess whether these increased uncorrelated errors can be beneficial, we measure how models in the various categories ensemble. That is, we ensemble our base model with models of the various categories, and measure the ensemble's accuracy, relative to its member models.

Per Fig.~\ref{fig:performance_increases}, as uncorrelated errors increase, so does ensemble performance (left).
Additionally, because we restrict our models to ones within the narrow accuracy range 74-78\%, we guarantee that this relative improvement translates into absolute improvement (center), and is not due to any individual ensemble member being intrinsically better than another.
Relative to our base model ResNet-50 (76.5\% on ImageNet), the most differently-trained model ALIGN-ZeroShot (75.5\%) yields 83.4\% top-1 accuracy when ensembled, a boost of nearly 7\% accuracy.

Taken together, these results mean that combining models with diverse training methodologies can yield performance improvements that offset the decrease in examples where both models are correct. To explain this relative boost, we also measure the conversion rate of each ensemble -- the rate at which inconsistent errors are converted into correct ensemble predictions. Surprisingly, this rate also increases as training methods diverge (Fig.~\ref{fig:performance_increases}, right).
This means that different training methods not only yield more opportunities for ensemble benefits, but also more efficient ensembles.

\begin{figure}
    \centering
    \includegraphics[height=0.27\linewidth]{figures/4_results/performance_disagreement_means_single_noclipzshot_nosimclrsmall_errincon_per3.pdf}
    \includegraphics[height=0.27\linewidth]{figures/4_results/performance_disagreement_max_single_noclipzshot_nosimclrsmall_errincon_per3.pdf}
    \includegraphics[height=0.27\linewidth]{figures/4_results/conversion_disagreement_mean_single_noclipzshot_nosimclrsmall_errincon_per3.pdf}
    \caption{
    \textbf{As uncorrelated errors increase, so does ensemble performance} (left, center), \textbf{and so does ensemble efficiency} (right).
    \textit{Left}: Error inconsistency is linearly correlated with the ensemble performance improvement, relative to the mean accuracy of the models in the ensemble. Stars represent averages over models in each category.
    \textit{Center}: Because we limited our analysis to models in the same 74-78\% accuracy range, this increase in relative accuracy translates into absolute accuracy boost, with the best performing ensembles comprising of models whose training methodologies are most different.
    \textit{Right}: Surprisingly, the conversion rate -- the rate at which these examples are converted into correct predictions by the ensemble -- also increases.
    This indicates that the benefits of combining divergently-trained models go beyond increasing the number of examples that \textit{can} become correct predictions,
    to also increasing how efficiently these examples \textit{do} become correct predictions.}
    \label{fig:performance_increases}
\end{figure}

\subsection{Dissimilar training methodologies create specialized models}
To understand how diverging training setups create efficient ensembles, we measure the specialization of ensemble member models.
We first note that, for a model's prediction to ``win'' the ensembling process, it only needs to be sufficiently more confident than the other model's prediction. This means that we can measure specialization by the relative confidence of two models. We do this by measuring the angle distance $\theta$ in the confidence-confidence plot (see Fig.~\ref{fig:dissimilar_models_specialize}, top left). When $\theta$ is high, Model 1 is more confident (relative to Model 2), and vice-versa.

In Fig.~\ref{fig:dissimilar_models_specialize}, we use $\theta$ to understand the specialization of different models categories.
To simplify our analysis, we compare three different ensembles, from the categories Reinit (Model 2: ResNet-50), Architecture (EfficientNet-B3), and Dataset (CLIP-S), as they are representative of the spread of error correlation.
When we plot a histogram of examples as a function of $\theta$, we find that:
1) As observed in Fig.~\ref{fig:comparing_models_in_ensemble}, when training methods diverge, the models make mistakes in different sets of examples
2) As training methods diverge, Model 1 tends to be more confident in examples where only Model 1 is correct, and vice-versa for Model 2.
This effect is most striking when we look at the Dataset category, where Model 2 (CLIP-S) is significantly more confident than Model 1 (ResNet-50) in examples where only CLIP-S predicts correctly, and vice-versa. This is in contrast with Reinit, where models don't seem to be significantly more confident than each other at all.

These results show: as training methodology diverges, model specialize to different data subdomains.

\begin{figure}
    \centering
    \includegraphics[width=0.9\linewidth]{figures/4_results/specialization_complete.pdf}
    \caption{\textbf{Differently-trained models specialize}:
    We plot histograms of examples where at least one ensemble produced error inconsistency, as a function of specialization measure $\theta$, the angle distance in the confidence-confidence plot (upper left).
    As we saw in Fig.~\ref{fig:comparing_models_in_ensemble}, when model training setups diverge (Reinit $\rightarrow$ Arch $\rightarrow$ Dataset), the fraction of consistent errors decreases (upper center \& right), in favor of more error inconsistency (lower center \& right).
    This added error inconsistency comes with specialization of the models in an ensemble: when only Model 1 makes a correct prediction, it is often more confident (lower right), and vice-versa for Model 2 (lower center).
    Faint dotted lines indicate values of $\theta$ for which a model's top-1 prediction is likely to prevail at ensemble time.
    }
    \label{fig:dissimilar_models_specialize}
\end{figure}

\subsection{Model specialization depends on training methodology}
Next, we want to investigate what kind of data each model category specializes to.
In Fig.~\ref{fig:how_do_they_specialize}, we plot the same data as Fig.~\ref{fig:dissimilar_models_specialize}, but we divide the examples along their ImageNet class IDs. This allows us to inspect whether a given model is more/less specialized in a given class.

As before, we find that, while models in the Architecture category demonstrate higher specialization (Model 1 tends to be more confident in examples where it is correct, and vice versa for Model 2), this specialization does not seem to correlate with specific classes.
In contrast, not only do models of the Dataset category display more specialization, but this specialization seems to be correlated with groups of classes. In particular, CLIP-S seems to specialize to anthropogenic images, whereas ResNet-50 seems to specialize to nature images.
We suspect this phenomena is a result of CLIP-S being trained on a dataset that was collected independently, and much differently than the one used to train our base model, ResNet-50.

\begin{figure}
    \centering
    \includegraphics[width=0.9\linewidth]{figures/4_results/specialization_analysis.pdf}
    \caption{\textbf{Specialization type depends on training setup}:
    When models differ in dataset (right plot), not only is specialization highest (see Fig.~\ref{fig:dissimilar_models_specialize}), but also this specialization happens in different classes -- CLIP is better at anthropogenic classes (cids 500-900) than ResNet-50, which is better at nature classes (cids 0-300; Right detail).
    When models differ in their architecture (Center plot), they are more specialized than reinitializations (Left plot), but such specialization does not correlate with specific classes.}
    \label{fig:how_do_they_specialize}
\end{figure}

\subsection{Different enough low-accuracy models can improve accuracy}
\label{sec:low_acc_helps}

In classic machine learning, a common way to create high-performing ensembles is to create specialized models with bootstrapping, where models are trained on subsets of the data and subsequently ensembled. In deep learning, bootstrapping has not been found to work well, since training on IID subsets of the data (with similar methodology) does not seem to yield ensembles that perform better than an individual model trained on the entire dataset \citep{nixon2020bootstrapped}. This seems to indicate that low-accuracy models would have little practical benefit for performance.

In order to investigate this, and
encouraged by the finding that differently-trained models can be experts in subdomains of the data, we ask whether low-accuracy models can be beneficial. After all, some of the models in the Dataset category were trained on much larger-scale datasets (JFT, WIT), which are collected independently.

To do this, we first combine our base model,
ResNet-50 (76.2\%), with a low-accuracy model trained very differently, CLIP-S-ZeroShot (63.3\%, ``Dataset'' category), and observe a performance boost (77.7\%).
To push this idea further, we
combine a high-accuracy model (BiT-1k, 82.85\%) with \textit{only} low-accuracy models (max individual accuracy 77.44\%), by greedily selecting ensemble members that maximize accuracy. With this procedure, we can create ensembles that yield as much as 86.66\%.
In Fig.~\ref{fig:best_next}, we repeat this procedure for 3 additional high-accuracy models and find that low-accuracy models can consistently improve the accuracy of high-accuracy models. Importantly, the low-accuracy models need to be different enough in their training methodology: for a given high-accuracy model, the most beneficial low-accuracy model to ensemble is one trained with a different loss and/or dataset, often both. See table in Fig.~\ref{fig:best_next} for details.

\begin{figure}
    \centering
    \includegraphics[width=0.49\linewidth]{figures/4_results/best_next.pdf}
    \includegraphics[width=0.49\linewidth]{figures/4_results/best_next_table.pdf}
    \caption{\textbf{Low-accuracy models can benefit high-accuracy ensembles}:
    \textit{Left}: Starting with 4 high-accuracy models (colored markers; CLIP-L, ALIGN, BiT-1k, EfficientNet-B3), we greedily select the best low accuracy models (each with max individual accuracy 77.9\%, indicated in the legend) to ensemble, and plot the ensemble's accuracy. Colors indicate which high-accuracy model the ensemble begins with, shapes indicate which models are added. By adding only low-accuracy models to a high-accuracy one, we are able to create ensembles that reach as high as 86.7\%. This shows that low-accuracy models can be made useful, if they are diverse enough. \textit{Right}: In each case, the best first model to add is one that complements the base model's training methodology.}
    \label{fig:best_next}
    \vspace{-0.17in}
\end{figure}

This result calls into question the idea that the information learned by high accuracy models is a strict superset of those learned by low accuracy models, which we explore further in sections below.

\subsection{Specialization comes from overlapping (but not supersetting) representations}

In order to explain why differently-trained models specialize, we posit that training methodology changes the features learned by individual models. In this view, models trained differently would have an overlapping (but not supersetting) set of features which, when combined, provide more information for classification than any individual model's representation could.

We test this directly by concatenating the features of our base model (ResNet-50) with those of the models above (ResNet-50, EfficientNet-B0 and CLIP-S), and linearly evaluating these combined features on ImageNet classification.
More specifically, we first randomly select features from the base model and each other model at inversely proportional rates. For example, we concatenate 25\% of ResNet-50 features with 75\% of CLIP-S features, yielding a final embedding that is at most the same dimensionality of the ResNet-50 features. This guarantees that any performance boost is not due to a higher number of dimensions being available, but by the quality of the features represented.

In Fig.~\ref{fig:divese_features} (center), we see that the best performance is obtained when features are combined. Additionally, we find that combining features yields higher performance as training methodology diverges, with the best performing combination being ResNet-50 + CLIP-S.
Concurrent work similarly finds that different training objectives yield different final-layer feature embeddings~\citep{grigg2021selfsupervised}.
This seems to confirm the idea that the methods used to train networks can generate diverse features, which capture information that neither embedding alone has captured.

To push this further, we ask how efficiently these representations capture important features of the data.
To test this, we first compute the covariance of each dimension of embeddings from both models (e.g.: ResNet-50 and CLIP-S). This gives us a ranking of highest to lowest covariance dimensions, which we use as a measure of diversity. We then select features in order of most diverse to least diverse, and linearly evaluate them on ImageNet classification.

Fig.~\ref{fig:divese_features} (right) shows how divergently-trained models yield more diverse (and therefore more compressible) features.
We tested multiples of 256 features, and found that Reinit models needed 2304 features on average to achieve 76\% accuracy. In constrast, Hyperparameter models required 2057, Architecture 1392, and Dataset only required 512 features.

\begin{figure}
    \centering
    \includegraphics[width=0.9\linewidth]{figures/4_results/compression_analysis.pdf}
    \caption{\textbf{Specialization comes from overlapping (not supersetting) representations}. When we combine the representations of two models at different rates (Left, 25/75 using 25\% of the ResNet embedding and 75\% of the other model's embedding), we find that: 1) despite being smaller number of dimensions, the concatenated embeddings (Center, e.g.: 50/50) yield higher LBFGS accuracy than the complete embedding of any of the two models (100/0 or 0/100). This indicates that each model has learned overlapping, but not supersetting, features, which when combined maximize performance. 2) when we combine embeddings of models that are more differently-trained (Center, ``Reinit'' $\rightarrow$ ``Dataset''), we find bigger gains, indicating that divergent training induces the learning of different features.
    When we compress the concatenated representations using a diversity heuristic, we find that differently-trained embeddings are also more efficiently compressible, reaching higher accuracies with fewer dimensions (Right, see text for details).}
    \label{fig:divese_features}
    \vspace{-0.17in}
\end{figure}

\subsection{Downstream task performance}

With the knowledge that diverse training leads to diverse feature embeddings, we ask whether these can transfer better to downstream tasks.
In order to maximize performance, we pick three of the highest accuracy models on imagenet, that are also representative of very diverse training setups.
Meta-Pseudo Labels (MPL) was the previous SOTA on imagenet, reporting 90.24\% top-1 accuracy~\citep{pham2021meta}. It is trained with a
teacher network that generate pseudo labels on unlabeled data to teach a student network, whose performance on the labeled set is then used to adapt the teacher. It is trained on the JFT-300M dataset.
ViT-G/14 is the current SOTA, with 90.45\% top-1 accuracy~\citep{zhai2021scaling}, when measured with EMA~\citep{polyak1992acceleration}.
We obtained our image embeddings without the use of EMA, so we find effective accuracy of 88.93\%
It is trained supervisedly, on JFT-3B, a larger version of the JFT-300M dataset.
Finally, CLIP-S is a constrastive learning model, yielding 75.7\% top-1 accuracy (with L-BFGS linear evaluation of its features on ImageNet)~\citep{radford2021learning}.
It is trained with constrastive learning, on the WIT dataset. Despite being a lower accuracy model, as we will see, it is a useful model.

In order to test the downstream generalization ability of these models' learned representations, we linearly evaluate them on Pascal VOC~\citep{everingham2010pascal} using the same evaluation method as \citet{kornblith2019better}.
Pascal VOC is a multi-label image classification benchmark, with 20 diverse classes ranging from vehicles, animals, household objects and people. This diversity of scenes make it an interesting downstream task to study. Performance on this benchmark is measured by 11-point mAP. We measure it according to the definition in  \citet{everingham2010pascal}.

In Fig.\ref{fig:voc_results} we find that, surprisingly, the highest performing model on ImageNet (MPL) is not the best performing model on VOC. We also see that the contrastive model CLIP-S deviates from the linear trend described in \cite{kornblith2019better}. Indeed, when compared with the other models, CLIP-S is yields the most prediction diversity on ImageNet.
We additionally test how the combined features perform by concatenating them pairwise and performing linear evaluation on top of these feature combinations -- known as Stacked Generalization~\citep{wolpert1992stacked}. %
The highest performing model combinations (on VOC) are ones which combine differently trained models.
Further, the best combinations do not even include MPL, which indicates that diversity can be better indicator of downstream performance than any single models' accuracy.

We posit that the reason diversely-trained model combinations yield highest performance on this downstream task is due to the diversity of features learned, which provide higher coverage of features captured that explain the data. This allows for better feature reuse, which is crucial for transfer learning~\citep{neyshabur2020being}.

\begin{figure}
    \centering
    \includegraphics[width=0.9\linewidth]{figures/4_results/voc_results_relative_tight.pdf}
    \vspace{-0.1in}
    \caption{\textbf{Diverse training methodologies yield the best downstream performance}.
    \textit{Left}: The highest-accuracy ImageNet model (MPL) is not the best-performing on Pascal VOC.
    \textit{Center}: CLIP-S is the model with the most prediction diversity, among the models analyzed.
    \textit{Right}: Combining models that are most diverse seems to yield the biggest boost when performing stacked generalization on a downstream task (i.e.: training a linear layer on top of 2 models' concatenated embeddings).}
    \label{fig:voc_results}
    \vspace{-0.17in}
\end{figure}

\vspace{-0.1in}

\section{Conclusion}
\vspace{-0.1in}
We have shown that diverse training methodologies, in particular, training with diverse optimization objectives on different large-scale datasets, can produce models that generate uncorrelated errors. These models ensemble more efficiently, attaining higher ensembling accuracy, since their different training setups allows them to specialize to different subdomains of the data. Due to this specialization, different-enough models can be useful for achieving high accuracies, even if they display low accuracies individually. Finally, we have also shown that they learn overlapping (but not supersetting) features, and that combining their embeddings can boost downstream performance.

The importance of behavior diversity has been highlighted in different fields.
In signal processing, sensor fusion -- the combination of signals from multiple sensors into a more accurate estimation -- relies on these sensors making independent errors.
In deep reinforcement learning, it has been hypothesized that maximizing the coverage of possible behaviors of an agent may help it acquire the skills that are useful~\citep{eysenbach2018diversity}.
In image classification, the model is our agent, its predictions the behavior, and ensembling our sensor fusion.

Our work demonstrates that this diversity of features is possible, and highlights a key question: \textit{why didn't SGD learn it?} Perhaps there exists an objective that can produce a single best/supersetting representation but, as we've shown, none of our existing training methodologies have found it.

If we look closely, our results may also provide clues for why this is the case.
To find models with categorically different generalization behavior, we had to train them with objectives that do not directly maximize the classification accuracy we are ultimately interested in.
In our collective search for novelty, we have stumbled upon the decade-old finding that
some problems are best solved by methods that ignore the objective,
since “the objective function does not necessarily reward the stepping stones in the search space that ultimately lead to the objective''~\citep{lehman2008exploiting}.

Together, our results provide encouragement for machine learning practitioners to continue creating methodologies to train high-performing models, including new frameworks \& datasets, in order to expand the types of features our models learn, behaviors they exhibit, and novel ways to train them.

\subsubsection*{Acknowledgments}
We would like to thank the following people for their early feedback, thoughtful discussions, and help over the course of this project:
Becca Roelofs, Ben Poole, Simon Kornblith, Niki Parmar, Ben Caine, Keren Gu, Ethan Dyer, Anders Andreassen, Katherine Heller, Chiyuan Zhang, Jon Shlens, Tyler Scott, Gamaleldin Elsayed, Rosanne Liu, Samy Bengio, Hieu Pham, Thomas Unterthiner, Chao Jia, Mostafa Dehghani, Neil Houlsby, Lucas Beyer, Josip Djolonga, Rodolphe Jenatton, Benham Neyshabur, Alec Radford, and Sam Altman.
\section{Appendix}

\subsection{Model Categorization}
In some sense, model categories are supersets of one another: when we change a model architecture, we also change the hyper-parameters used to train such architecture, to make sure that they are optimal for training this new setting.
In a similar fashion, when we change the training framework (supervised to contrastive), not only do we change hyper parameters, but the architecture also changes to best suit the new learning framework. Finally, when we change dataset scales (e.g.: ImageNet $\rightarrow$ WIT), we use the framework, architecture, and hyper parameters that allow the best performance possible on the new dataset.

\subsection{Model List Per Category}
In Table~\ref{tab:main_models}, we list all models we used in our main analysis, Figs.~\ref{fig:comparing_models_in_ensemble},~\ref{fig:performance_increases},~\ref{fig:dissimilar_models_specialize},~\ref{fig:how_do_they_specialize},~\ref{fig:divese_features}, along with their training methodologies, calibration temperatures, individual ImageNet accuracy, error inconsistency (relative to our base model ResNet-50), and ensemble accuracy (with our base model ResNet-50). We selected these models by controlling for their individual accuracy, which helps guarantee our analysis concerns training methodology, not any individual model being inherently better.

In table~\ref{tab:ha_models}, we list the high-accuracy models used in Figs.~\ref{fig:best_next} and ~\ref{fig:voc_results}. These models are higher accuracy, and provide a stronger base for ensembling.

Table~\ref{tab:la_models} lists low accuracy models, but which are trained with very different methodologies (relative to the typical model). They are therefore still useful (as we show), and were used in Sec.~\ref{sec:low_acc_helps} and Fig.~\ref{fig:best_next}.

Finally, Table~\ref{tab:other_models} lists other models that we trained and analyzed, but which did not reach our target accuracy range 74-78\%, so were not included in the main paper analysis.

\begin{table}[]
    \centering
    \tiny{
    \centerline{
    \begin{tabular}{l|l|l|l|l|l|l|l|l}
        Model & Citation & Method & \linebreakcell{Prediction\\Head Type} & \linebreakcell{Calibration\\Temp} & \linebreakcell{ImageNet\\Acc} & \linebreakcell{Error\\Inconsistency\\(w/ ResNet-50)} & Category & \linebreakcell{Ensemble Acc\\(w/ ResNet-50)}\\
        \hline
        ResNet-50 & \cite{he2016deep} & Sup. / IN & Trained & 1.10 & 76.20\% & 0.0850 & REINIT & 77.31\% \\
        \hline
        ResNet-101x0.5 &  & Sup. / IN & Trained & 1.00 & 74.27\% & 0.0972 & HPARAM & 76.73\% \\
        ResNet-50 (Dropout 0.1) & \cite{JMLR:v15:srivastava14a} & Sup. / IN & Trained & 1.10 & 76.90\% & 0.0853 & HPARAM & 77.23\% \\
        ResNet-50 (Dropout 0.2) &  & Sup. / IN & Trained & 1.10 & 75.85\% & 0.0883 & HPARAM & 77.27\% \\
        ResNet-50 (Dropout 0.3) &  & Sup. / IN & Trained & 1.10 & 75.43\% & 0.0910 & HPARAM & 77.12\% \\
        ResNet-50 (Dropout 0.4) &  & Sup. / IN & Trained & 1.10 & 75.37\% & 0.0928 & HPARAM & 77.12\% \\
        ResNet-50 (Dropout 0.5) &  & Sup. / IN & Trained & 1.10 & 75.03\% & 0.0965 & HPARAM & 77.06\% \\
        ResNet-50 (Dropout 0.6) &  & Sup. / IN & Trained & 1.10 & 74.72\% & 0.1006 & HPARAM & 77.01\% \\
        ResNet-50 (Dropout 0.7) &  & Sup. / IN & Trained & 1.10 & 74.48\% & 0.1058 & HPARAM & 76.89\% \\
        ResNet-50 (Label Smoothing 0.1) & \cite{szegedy2016rethinking} & Sup. / IN & Trained & 0.90 & 76.22\% & 0.0877 & HPARAM & 77.30\% \\
        ResNet-50 (Label Smoothing 0.2) &  & Sup. / IN & Trained & 0.80 & 76.24\% & 0.0911 & HPARAM & 77.44\% \\
        ResNet-50 (Label Smoothing 0.3) &  & Sup. / IN & Trained & 0.80 & 75.74\% & 0.0969 & HPARAM & 77.26\% \\
        ResNet-50 (Label Smoothing 0.4) &  & Sup. / IN & Trained & 0.70 & 75.70\% & 0.0996 & HPARAM & 77.31\% \\
        ResNet-50 (Label Smoothing 0.5) &  & Sup. / IN & Trained & 0.70 & 75.51\% & 0.1030 & HPARAM & 77.26\% \\
        ResNet-50 (Label Smoothing 0.6) &  & Sup. / IN & Trained & 0.70 & 75.03\% & 0.1063 & HPARAM & 77.00\% \\
        ResNet-50 (Label Smoothing 0.7) &  & Sup. / IN & Trained & 0.70 & 74.54\% & 0.1071 & HPARAM & 76.80\% \\
        ResNet-50 (Label Smoothing 0.8) &  & Sup. / IN & Trained & 0.80 & 74.28\% & 0.1130 & HPARAM & 76.98\% \\
        ResNet-50 (DropBlock 34, 0.9) & \cite{ghiasi2018dropblock} & Sup. / IN & Trained & 1.00 & 74.98\% & 0.0891 & HPARAM & 76.84\% \\
        ResNet-50 (DropBlock 1234, 0.9) &  & Sup. / IN & Trained & 1.00 & 74.87\% & 0.0895 & HPARAM & 76.72\% \\
        ResNet-50 (Learning Rate 0.05) &  & Sup. / IN & Trained & 1.10 & 75.56\% & 0.0871 & HPARAM & 77.10\% \\
        ResNet-50 (Learning Rate 0.2) &  & Sup. / IN & Trained & 1.10 & 76.07\% & 0.0858 & HPARAM & 77.35\% \\
        ResNet-50 (Weight Decay 0.00001) & \cite{krogh1992simple} & Sup. / IN & Trained & 1.20 & 74.13\% & 0.1027 & HPARAM & 76.70\% \\
        ResNet-50 (Weight Decay 0.00005) &  & Sup. / IN & Trained & 1.10 & 75.87\% & 0.0866 & HPARAM & 77.23\% \\
        ResNet-50 (Weight Decay 0.0002) &  & Sup. / IN & Trained & 1.10 & 75.79\% & 0.0865 & HPARAM & 77.16\% \\
        ResNet-50 (LR 0.05, WD 0.0002) &  & Sup. / IN & Trained & 1.10 & 75.58\% & 0.0856 & HPARAM & 77.08\% \\
        ResNet-50 (LR 0.2, WD 0.00005) &  & Sup. / IN & Trained & 1.10 & 76.36\% & 0.0864 & HPARAM & 77.49\% \\
        ResNet-50 (LR 1.0, WD 0.00001) &  & Sup. / IN & Trained & 1.20 & 75.71\% & 0.1011 & HPARAM & 77.53\% \\
        \hline
        EfficientNet-B0 & \cite{tan2019efficientnet} & Sup. / IN & Trained & 0.90 & 76.88\% & 0.1136 & ARCH & 78.65\% \\
        SK-ResNet-34 & \cite{li2019selective} & Sup. / IN & Trained & 0.90 & 76.91\% & 0.1099 & ARCH & 78.52\% \\
        MobileNet-V2 & \cite{sandler2018mobilenetv2} & Sup. / IN & Trained & 0.90 & 77.29\% & 0.1033 & ARCH & 78.48\% \\
        VGG19 BatchNorm & \cite{simonyan2014very} & Sup. / IN & Trained & 1.10 & 74.22\% & 0.1232 & ARCH & 77.45\% \\
        Legacy-SEResNet-34 & \cite{hu2018squeeze} & Sup. / IN & Trained & 1.10 & 74.81\% & 0.1206 & ARCH & 77.70\% \\
        Legacy-SEResNet-50 & \cite{hu2018squeeze} & Sup. / IN & Trained & 0.90 & 77.64\% & 0.1144 & ARCH & 79.11\% \\
        SEResNeXt-26d & \cite{hu2018squeeze} & Sup. / IN & Trained & 0.90 & 77.60\% & 0.1187 & ARCH & 79.25\% \\
        DenseNet-Blur-121d & \cite{iandola2014densenet} & Sup. / IN & Trained & 0.90 & 76.58\% & 0.1082 & ARCH & 78.22\% \\
        DenseNet-121 & \cite{iandola2014densenet} & Sup. / IN & Trained & 0.80 & 75.57\% & 0.1076 & ARCH & 77.70\% \\
        Inception-V3 & \cite{szegedy2016rethinking} & Sup. / IN & Trained & 1.00 & 77.44\% & 0.1270 & ARCH & 79.49\% \\
        TF-Inception-V3 & \cite{szegedy2016rethinking} & Sup. / IN & Trained & 1.00 & 77.86\% & 0.1259 & ARCH & 79.65\% \\
        Adv Inception-V3 & \cite{kurakin2018adversarial} & Sup. / IN & Trained & 1.00 & 77.58\% & 0.1252 & ARCH & 79.41\% \\
        HRNet-W18-Small & \cite{wang2020deep} & Sup. / IN & Trained & 1.00 & 75.13\% & 0.1143 & ARCH & 77.78\% \\
        DPN-68 & \cite{chen2017dual} & Sup. / IN & Trained & 1.20 & 76.31\% & 0.1141 & ARCH & 78.44\% \\
        DLA-60 & \cite{yu2018deep} & Sup. / IN & Trained & 1.10 & 77.02\% & 0.1064 & ARCH & 78.35\% \\
        ESE-VoVNet & \cite{lee2020centermask} & Sup. / IN & Trained & 0.80 & 76.80\% & 0.1159 & ARCH & 78.62\% \\
        ViT-B/16 & \cite{dosovitskiy2020image} & Sup. / IN & Trained & 1.70 & 74.55\% & 0.1472 & ARCH & 78.85\% \\
        \hline
        SimCLR & \cite{chen2020simple} & Cont. / IN & Fine Tuned & 1.00 & 75.60\% & 0.1277 & FWORK & 78.35\% \\
        ViT-DeiT-Tiny & \cite{touvron2021training} & Sup. / IN & Distilled & 1.00 & 74.50\% & 0.1247 & FWORK & 77.93\% \\
        \hline
        ViT-S/16 & \cite{dosovitskiy2020image} & Sup. / IN-21k & Trained & 0.90 & 77.86\% & 0.1249 & DATASET & 79.73\% \\
        ALIGN-ZS & \cite{jia2021scaling} & Cont. / ALIGN & Zero Shot & 0.30 & 75.50\% & 0.2295 & DATASET & 83.42\% \\
        CLIP-S & \cite{radford2021learning} & Cont. / WIT & L-BFGS & 0.90 & 75.67\% & 0.1687 & DATASET & 80.42\% \\
        CLIP-L-ZS & \cite{radford2021learning} & Cont. / WIT & Zero Shot & 1.10 & 76.57\% & 0.2140 & DATASET & 83.22\% \\

    \end{tabular}
    }
    }
    \caption{\textbf{Models Used}. These models are all within the 74-78\% accuracy range, of which we base our main results.
    "Sup." indicates supervised learning, "Cont." constrastive learning. "IN" indicates ImageNet. }
    \label{tab:main_models}
\end{table}

\begin{table}[]
    \centering
    \tiny{
    \centerline{
    \begin{tabular}{l|l|l|l|l|l|l|l|l}
        Model & Citation & Method & \linebreakcell{Prediction\\Head Type} & \linebreakcell{Calibration\\Temp} & \linebreakcell{ImageNet\\Acc} & \linebreakcell{Error\\Inconsistency\\(w/ ResNet-50)} & Category & \linebreakcell{Ensemble Acc\\(w/ ResNet-50)}\\
        \hline
        EfficientNet-B3 & \cite{tan2019efficientnet} & Sup. / IN & Trained & 0.90 & 80.89\% & 0.1147 & ARCH & 80.98\% \\
        ALIGN & \cite{jia2021scaling} & Constrastive / ALIGN Dataset & L-BFGS & 1.00 & 84.71\% & 0.1684 & DATASET & 85.50\% \\
        ViT-H/14 & \cite{dosovitskiy2020image} & Sup. / JFT & Trained & 1.10 & 88.31\% & 0.1646 & DATASET & 87.48\% \\
        ViT-G/14 & \cite{zhai2021scaling} & Sup. / JFT & Trained & 1.40 & 88.93\% & 0.1801 & DATASET & 88.40\% \\
        MPL & \cite{pham2021meta} & Pseudo-Label / JFT & Trained & 0.90 & 90.24\% & 0.1741 & DATASET & 88.97\% \\
        CLIP-L & \cite{radford2021learning} & Contrastive / WIT & L-BFGS & 1.00 & 85.04\% & 0.1638 & DATASET & 85.36\% \\
        BiT-1k & \cite{kolesnikov2020big} & Sup. / JFT & Fine Tuned & 1.20 & 82.85\% & 0.1593 & DATASET & 84.28\% \\
    \end{tabular}
    }
    }
    \caption{\textbf{High Accuracy Models}. These models are all above the 80\% accuracy.
    "Sup." indicates supervised learning, "Cont." constrastive learning. "IN" indicates ImageNet. }
    \label{tab:ha_models}
\end{table}

\begin{table}[]
    \centering
    \tiny{
    \centerline{
    \begin{tabular}{l|l|l|l|l|l|l|l|l}
        Model & Citation & Method & \linebreakcell{Prediction\\Head Type} & \linebreakcell{Calibration\\Temp} & \linebreakcell{ImageNet\\Acc} & \linebreakcell{Error\\Inconsistency\\(w/ ResNet-50)} & Category & \linebreakcell{Ensemble Acc\\(w/ ResNet-50)}\\
        \hline
        CLIP-S-ZS & \cite{radford2021learning} & Cont. / WIT & Zero Shot & 0.01 & 63.25\% & 0.2675 & DATASET & 78.01\% \\
        BiT-JFT & \cite{kolesnikov2020big} & Sup. / JFT & Class Mapping & 0.02 & 65.63\% & 0.2744 & DATASET & 77.93\% \\
    \end{tabular}
    }
    }
    \caption{\textbf{Low Accuracy Models}. These models are all below the 74\% target accuracy, but they are trained with very different methodologies (relative to typical models), so we include them in a few analyses, as indicated in main text.
    "Sup." indicates supervised learning, "Cont." constrastive learning. "Class Mapping" indicates that the model's original JFT class vector was used, and a mapping of JFT-to-ImageNet classes was used to obtain the final prediction logits.}
    \label{tab:la_models}
\end{table}

\begin{table}[]
    \centering
    \tiny{
    \centerline{
    \begin{tabular}{l|l|l|l|l|l|l|l|l}
        Model & Method & \linebreakcell{Prediction\\Head Type} & \linebreakcell{ImageNet\\Acc} & Category\\
        \hline
        ResNet-101 & Sup. / IN & Trained & 78.42\% & HPARAM \\
        ResNet-50x2 & Sup. / IN & Trained & 78.70\% & HPARAM \\
        ResNet-50 (Dropout 0.8) & Sup. / IN & Trained & 73.68\% & HPARAM \\
        ResNet-50 (Dropout 0.9) & Sup. / IN & Trained & 72.48\% & HPARAM \\
        ResNet-50 (Label Smoothing 0.9) & Sup. / IN & Trained & 72.63\% & HPARAM \\
        ResNet-50 (DropBlock 34, 0.1) & Sup. / IN & Trained & 29.88\% & HPARAM \\
        ResNet-50 (DropBlock 34, 0.2) & Sup. / IN & Trained & 50.17\% & HPARAM \\
        ResNet-50 (DropBlock 34, 0.3) & Sup. / IN & Trained & 58.01\% & HPARAM \\
        ResNet-50 (DropBlock 34, 0.4) & Sup. / IN & Trained & 62.50\% & HPARAM \\
        ResNet-50 (DropBlock 34, 0.5) & Sup. / IN & Trained & 66.19\% & HPARAM \\
        ResNet-50 (DropBlock 34, 0.6) & Sup. / IN & Trained & 68.73\% & HPARAM \\
        ResNet-50 (DropBlock 34, 0.7) & Sup. / IN & Trained & 70.95\% & HPARAM \\
        ResNet-50 (DropBlock 34, 0.8) & Sup. / IN & Trained & 73.33\% & HPARAM \\
        ResNet-50 (DropBlock 1234, 0.1) & Sup. / IN & Trained & 27.19\% & HPARAM \\
        ResNet-50 (DropBlock 1234, 0.2) & Sup. / IN & Trained & 47.27\% & HPARAM \\
        ResNet-50 (DropBlock 1234, 0.3) & Sup. / IN & Trained & 55.59\% & HPARAM \\
        ResNet-50 (DropBlock 1234, 0.4) & Sup. / IN & Trained & 60.63\% & HPARAM \\
        ResNet-50 (DropBlock 1234, 0.5) & Sup. / IN & Trained & 64.57\% & HPARAM \\
        ResNet-50 (DropBlock 1234, 0.6) & Sup. / IN & Trained & 67.68\% & HPARAM \\
        ResNet-50 (DropBlock 1234, 0.7) & Sup. / IN & Trained & 70.35\% & HPARAM \\
        ResNet-50 (DropBlock 1234, 0.8) & Sup. / IN & Trained & 72.73\% & HPARAM \\
        ResNet-50 (Learning Rate 0.01) & Sup. / IN & Trained & 69.88\% & HPARAM \\
        ResNet-50 (Learning Rate 1.0) & Sup. / IN & Trained & 73.53\% & HPARAM \\
        ResNet-50 (Weight Decay 0.001) & Sup. / IN & Trained & 72.06\% & HPARAM \\
        ResNet-50 (LR 0.01, WD 0.001) & Sup. / IN & Trained & 73.90\% & HPARAM \\
    \end{tabular}
    }
    }
    \caption{\textbf{Other Models}. These hyper-parameter sweeps were trained, but did not fit our target 74-78\% accuracy range, so we did not use them in our main analyses.
    "Sup." indicates supervised learning, "IN" indicates ImageNet.}
    \label{tab:other_models}
\end{table}

\subsection{Additional Error Analysis}

In Fig.~\ref{fig:comparing_models_in_ensemble_complete}, we reproduce Fig.~\ref{fig:comparing_models_in_ensemble} (left), and additionally show how the number of examples where both models predict correct (center), as well as examples where both models predict incorrectly (right), both decrease. The added prediction diversity that comes with ensembling models trained differently will only be beneficial if the ensemble can efficiently convert the examples on the left, to compensate for the decreased number of examples in the center.

\begin{figure}[b]
    \centering
    \includegraphics[width=0.3\linewidth]{figures/4_results/div_boxplot_moremodels_noclipzshot_nosimclrsmall.pdf}
    \includegraphics[width=0.3\linewidth]{figures/4_results/ac_boxplot_moremodels_noclipzshot_nosimclrsmall.pdf}
    \includegraphics[width=0.3\linewidth]{figures/4_results/ai_boxplot_moremodels_noclipzshot_nosimclrsmall.pdf}
    \caption{
    \textbf{As training methodology diverges, errors become uncorrelated}.
    The number of test examples where only one model predicts correctly increases (Left, same as "Error Inconsistency" in Fig.~\ref{fig:comparing_models_in_ensemble}).
    Similarly, the number of examples where both models predict correctly (Center) and neither model predicts correctly (Right) decreases.
    For the most diverse training methodology ("Dataset", purple), we find more examples with inconsistent errors (left) than examples where both models predict correctly (right).
    This means that for an ensemble to perform better than another, it needs to resolve these inconsistent errors efficiently, to compensate for the decrease in the examples in the center plot.}
    \label{fig:comparing_models_in_ensemble_complete}
\end{figure}


\section{Introduction}

The modeling and simulation of natural phenomena using differential equations is an important tool for science. However, to represent physical structures, that is, the domain to be studied in a computational environment, the use of simpler data organization structures, such as matrices and vectors, tend not to be able to realistically represent the object of study. Thus, different techniques have been used for the representation of domains, such as structured and unstructured meshes, generalized or hybrid meshes, among other techniques, that have advantages and disadvantages in terms of flexibility and ability to represent objects under study (THOMPSON et al., 1977; THOMPSON et al., 1985; THOMPSONet al., 1998; MALISKA, 2004; CIRILO et al. 2006; KOOMULLIL et al, 2008;  LAIPING et al., 2013; SAITA et al., 2017; BELINELLI et al, 2020).

In this work, we looked for a simplified way to build a structured mesh for a complex domain, which was possible through a change in the coordinate system. This process allowed to convert a complex domain into a set of easily manipulated data (MALISKA, 2004). Mathematically, any geometry, described in Cartesian system, can be transformed into a generalized system, allowing better adaptation in computational modeling (CIRILO et al. 2006; MALISKA, 2004; ROMEIRO et al., 2011; PARDO et al., 2012; SAITA et al., 2017; ROMEIRO et al., 2017; CIRILO et al., 2018).

In this context, this work describes the creation and implementation of a two-dimensional mesh generator using transformation metrics in generalized coordinates. This procedure generates a domain mapped for mathematical manipulations, which describes any physical object, from a finite set of points. The mesh generator is encoded in \textit{Python} and uses the libraries \textit{numpy} and \textit{matplotlib} for manipulations and operations on matrices and graphical plots, respectively (CAI et al., 2005; SCIPY-NUMPY, 2020; SCIPY-MATPLOTLIB,2020).

The work is structured as shown below. Initially, transformation metrics are described in generalized coordinate theory. Then the mesh generator is developed. Finally, some examples of meshes obtained by the generator are presented.

\section{Generalized Coordinate}

For a computational methodology to be applied to a physical problem, it is necessary to discretize the domain of the problem, that is, to build a computational mesh that can represent the studied geometry and, thus, obtain the values of interest.

The discretization of the physical domain can be carried out according to a structured or unstructured mesh. Unstructured meshes are more adaptable than structured meshes, especially in problems with complex geometries (MALISKA, 2004; FORTUNA, 2012). However, the major disadvantage of unstructured meshes is the difficulty of ordering the elements, which implies a variation in the size of the diagonals of the coefficient matrix and the additional cost of using memory, making it difficult to apply numerical methods to obtain the solution of linear systems. Therefore, the domain discretization in this work will be performed using structured meshes.

As for the coordinate system, in general, the problem domain is discretized according to the Cartesian coordinate system because it is simpler. However, for problems with complex geometry, it is convenient to adopt another coordinate system, due to the fact that the Cartesian coordinate system leads to a poor adaptation of the border, since the physical domain does not always coincide with the domain of the Cartesian mesh. To solve the problem, the generalized coordinate system will be used. In generalized coordinates, the computational mesh coincides with the geometry of the problem and computational treatment becomes more appropriate. Other reasons that justify the use of generalized coordinates in the discretization of computational meshes are the simplicity in programming computational codes to solve complex problems and the ease in developing generic methodologies.

In the following, some important concepts about generalized coordinate theory will be presented

\subsection{Transformation Metrics}

The Cartesian system $(x, y)$ is called the physical domain and the generalized system $(\xi, \eta)$ is called the transformed domain or computational domain.

The transformation from a non-trivial geometry described in a Cartesian coordinate system to a generalized coordinate system involves transformation metrics, or mathematical relations, that can accurately describe the transformed data.

The mapping of irregular or regular geometries written in Cartesian coordinates $(x, y)$, Fig. 1a, is performed numerically for regular geometries written in generalized coordinate system $(\xi, \eta)$, Fig. 1b.

\begin{figure}[!ht]
\begin{center}
\caption{Coordinate systems}
\subfigure[Cartesian coordinate]{\includegraphics[height=4.4cm,width=5.4cm]{coordenadas_cartesianas.png}}
   \label{fig2a}\qquad \qquad
\subfigure[Generalized coordinate]{\includegraphics[height=4.4cm,width=5.4cm]{coordenadas_generalizadas.png}}
    \label{fig2b}
\end{center}
\center{{\bf{Source:}} The Authors}
\label{fig2}
\end{figure}

Since the transformed domain is regular, for convenience, unitary normalization of elementary volumes is assumed, that is, $\Delta\xi=\Delta\eta=1$. In this way, even if in the physical plane the coordinated lines assume arbitrary spacing, in the computational plane the dimensions are fixed to the unit.

To obtain the $(\xi, \eta)$ system, the following generating equations are used
\begin{eqnarray}
	\label{eq1}
		\xi = \xi(x,y), \\

	\label{eq1a}
		\eta = \eta(x,y).
\end{eqnarray}

The transformation metrics, based on differentials of the Eqs. (\ref{eq1}) and (\ref{eq1a}), are
\begin{eqnarray}
	\label{eq2}
		&d\xi = \xi_{x}dx + \xi_{y}dy, \\
	\label{eq2a}
		&d\eta = \eta_{x}dx + \eta_{y}dy,
\end{eqnarray}

\noindent
or, in the matrix form
\begin{equation}
	\left(
		\begin{array}{c}
			d \xi\\
			d \eta
		\end{array}
	\right)
	=
	\left(
		\begin{array}{cc}
			\xi_{x} 	& \xi_{y}\\
			\eta_{x}	 & \eta_{y}
		\end{array}
	\right)
	\left(
		\begin{array}{c}
			d x\\
			d y
		\end{array}
	\right),
	\label{eq3}
\end{equation}

\noindent
which can also be written as
\begin{equation}
	d^{t} = Ad^{f},
	\label{eq4}
\end{equation}

\noindent
where $\xi_{x}$, $\eta_{x}$, $\xi_{y}$, and $\eta_{y}$ denoting partial derivatives, $d^{t}$ and $d^{f}$ represent, respectively, the differentials in the transformed and physical domain, while $A$ is the transformation matrix between the domains.

Starting from the assumption that it is possible to find a representation in a Cartesian coordinate system for a model described in generalized coordinates, then we admit the existence of the inverse of the equations Eqs. (\ref{eq1}) and (\ref{eq1a}), so
\begin{eqnarray}
	\label{eq9}
		&x = x(\xi,\eta), \\
	\label{eq9a}
		&y = y(\xi,\eta),
\end{eqnarray}

\noindent
and from differentials of Eqs. (\ref{eq9}) and (\ref{eq9a}) we have
\begin{equation}
	\left(
		\begin{array}{c}
			d x\\
			d y
		\end{array}
	\right)
	=
	\left(
		\begin{array}{cc}
			 x_{\xi} 	& x_{\eta}\\
			 y_{\xi} 	& y_{\eta}
		\end{array}
	\right)
	\left(
		\begin{array}{c}
			d \xi\\
			d \eta
		\end{array}
	\right),
	\label{eq5}
\end{equation}

\noindent
which can also be written as

\begin{equation}
	d^{f} = Bd^{t},
	\label{eq6}
\end{equation}

\noindent
where $x_{\xi}$, $x_{\eta}$, $y_{\xi}$, and $y_{\eta}$ denoting partial derivatives and $B$ is the transformation matrix between the physical and transformed domains.

Replacing Eq. (\ref{eq4}) in Eq. (\ref{eq6}), we get  $d^{f} = BAd^{f}$, so that $BA = I$, or equivalent $A = B^{- 1}$. Thus, the $A$ matrix becomes
\begin{equation}
	\left(
		\begin{array}{cc}
			\xi_{x} 	& \xi_{y}\\
			\eta_{x}	 & \eta_{y}
		\end{array}
	\right)
	=
	\left(
		\begin{array}{cc}
			\frac{y_{\eta}}{x_{\xi}y_{\eta} - x_{\eta}y_{\xi}} & \frac{-x_{\eta}}{x_{\xi}y_{\eta} - x_{\eta}y_{\xi}}\\
			\frac{-y_{\xi}}{x_{\xi}y_{\eta} - x_{\eta}y_{\xi}} & \frac{x_{\xi}}{x_{\xi}y_{\eta} - x_{\eta}y_{\xi}}
		\end{array}
	\right),
	\label{eq7}
\end{equation}

\noindent
where $J = (x_{\xi}y_{\eta} - x_{\eta}y_{\xi})^{-1}$ is called Jacobian of transformation (MALISKA, 2004).

\subsection{Two-dimensional Mesh Generation}

In this work, we opted for the use of elliptical partial differential equations (EPDE) as a method of generating two-dimensional meshes, since their solutions do not generate null Jacobian, and the lines $\xi$ or $\eta$ never intersect (THOMPSON, 1985; MALISKA, 2004). Thus, the governing equations for the generation of two-dimensional meshes in a domain, for example the domain illustrated in Fig. 2a, are
\begin{eqnarray}
	\label{eq13}
		&\nabla^{2}\xi = P(\xi,\eta), \\
	\label{eq13a}
		&\nabla^{2}\eta=Q(\xi,\eta),
\end{eqnarray}

\noindent
whose boundary conditions of the type \textit{Dirichlet} are expressed by
$\xi = \xi_ {1}$ in $\Gamma_ {1}$ (left border),
$\xi = \xi_{N}$ in $\Gamma_{3}$ (right border);
$\eta = \eta_{1}$ in $\Gamma_{4}$ (lower border) and
$\eta = \eta_{M}$ in $\Gamma_{2}$ (upper border).
In particular, in Fig. 2a were used
$N = 10$ and $M = 5$ , where $N$ is the number of lines from
$\xi$ and $M$ is the number of lines from $\eta$.

\begin{figure}[!ht]
\begin{center}
\caption{Example of mesh in generalized coordinates}
\subfigure[Predefined points]{\includegraphics[height=4.4cm,width=5.4cm]{bordo_dominio_cut.png}}
   \label{fig4a}\qquad \qquad
\subfigure[Mesh in generalized coordinates]{\includegraphics[height=4.4cm,width=5.4cm]{refinamento_dominio_cut.png}}
    \label{fig4b}
\end{center}
\center{{\bf{Source:}} The Authors}
\label{fig4}
\end{figure}

Solving the equations (\ref{eq13}) and (\ref{eq13a}), in relation to the $(x, y)$ coordinates, using the transformation metrics of the $(\xi, \eta)$ system, then the coordinated lines can be generated in the directions $\xi$ and $\eta$, inside the computational mesh, Fig. 2b, as in any other geometry, through the equations
\begin{eqnarray}
	\label{eq25}
	\alpha x_{\xi\xi} + \gamma x_{\eta\eta} - 2\beta x_{\xi\eta} + \frac{1}{J^{2}}(Px_{\xi} + Qx_{\eta}) = 0, \\
	\label{eq26}
	\alpha y_{\xi\xi} + \gamma y_{\eta\eta} - 2\beta y_{\xi\eta} + \frac{1}{J^{2}}(Py_{\xi} + Qy_{\eta}) = 0,
\end{eqnarray}

\noindent
where $x$ and $y$ are the Cartesian coordinates of the physical domain, $\xi$ and $\eta$ are the generalized coordinates of the computational domain, $P$ and $Q$ are the source functions, $J$ is the Jacobian of transformation, and
\begin{eqnarray}
	\label{eq26a}
		\alpha = x^{2}_{\eta} + y^{2}_{\eta} , \\
	\label{eq25b}
		\beta = x_{\xi}x_{\eta} + y_{\xi}y_{\eta}, \\
	\label{eq26b}
		\gamma = x^{2}_{\xi} + y^{2}_{\xi}.
\end{eqnarray}

The numerical solution of elliptical EDPs (\ref{eq25})
and (\ref{eq26}), subject to initial and boundary conditions, provide the lines $\xi$ and $\eta$, which generate the computational mesh (DE BORTOLI, 2000, MALISKA, 2004; CIRILO et al., 2006; FORTUNA, 2012). For convenience, Eqs. (\ref{eq25}) and (\ref{eq26}) can be written using a generic $\phi$ variable as
\begin{equation}
	\alpha\phi_{\xi\xi} + \gamma\phi_{\eta\eta} - \alpha\beta\phi_{\xi\eta} + \frac{1}{J^{2}}(P\phi_{\xi} + Q\phi_{\eta}) = 0.
	\label{eq27}
\end{equation}

To approximate the derivatives in Eq. (\ref{eq27}), the finite difference method is used. The mesh nodes are labeled by the cardinal points $P$ (center), $E$ (east), $W$ (west), $N$ (north), $S$ (south), $NW$ (northwest ), $SW$ (southwest), $SE$ (southeast) and $NE$ (northeast), as in Fig. \ref{fig5}.

\begin{figure}[!ht]
\begin{center}
\caption{Index labeling}
  \includegraphics[height=4.4cm,width=5.4cm]{rotulagem_indices.png}
	\label{fig5}
\end{center}
\center{{\bf{Source:}} The Authors}
\label{fig4}
\end{figure}

Approximating the derivative terms of Eq. (\ Ref {eq27}) by means of central differences, around $ P $, we obtain
\begin{eqnarray}
		\alpha(\frac{\phi_{E} - 2\phi_{P} + \phi_{W}}{\Delta\xi^{2}}) + \gamma(\frac{\phi_{N} - 2\phi_{P} + \phi_{S}}{\Delta\eta^{2}}) - 2\beta(\frac{\phi_{NE} - \phi_{NW} + \phi_{SW} - \phi_{SE}}{4\Delta\xi\Delta\eta}) + \nonumber\\
		\frac{1}{J^{2}}(P\frac{\phi_{E} - \phi_{W}}{2\Delta\xi} + Q\frac{\phi_{N} - \phi_{S}}{2\Delta\eta}) = 0,
\label{eq28}
\end{eqnarray}

\noindent
and regrouping the terms
\begin{eqnarray}
		-(2\alpha - 2\gamma)\phi_{P} + (\alpha + \frac{P}{2J^{2}})\phi_{E} + (\alpha - \frac{P}{2J^{2}})\phi_{W} + (\gamma + \frac{Q}{2J^{2}})\phi_{N} + \nonumber\\
		(\gamma - \frac{Q}{2J^{2}})\phi_{S} - \frac{\beta}{2}\phi_{NE} + \frac{\beta}{2}\phi_{NW} - \frac{\beta}{2}\phi_{SW} + \frac{\beta}{2}\phi_{SE} = 0.
\label{eq29}
\end{eqnarray}

Thus, the numerical solution of the two-dimensional mesh generation equations, Eqs. (\ref{eq25}) and (\ref{eq26}), written according to the generic $\phi$ variable,
Eq. (\ref{eq29}), is given by
\begin{eqnarray}
		\phi_{P} = \frac{1}{A_{P}}(A_{E}\phi_{E} + A_{W}\phi_{W} + A_{N}\phi_{N} + A_{S}\phi_{S} + A_{NE}\phi_{NE} + A_{SE}\phi_{SE} + \nonumber\\
	\label{eq30}
		A_{NW}\phi_{NW} + A_{SW}\phi_{SW}),
\end{eqnarray}

\noindent
where
\begin{eqnarray}
	\begin{array}{lclcl}
		A_{P} = 2\alpha + 2\gamma, &\quad & A_{N} = \gamma + \frac{Q}{2J^{2}}, &\quad & A_{SE} = \frac{\beta}{2}, \\ \\
		A_{E} = \alpha + \frac{P}{2J^{2}}, &\quad & A_{S} = \gamma - \frac{Q}{2J^{2}}, &\quad & A_{NW} = \frac{\beta}{2}, \\ \\
		A_{W} = \alpha - \frac{P}{2J^{2}}, &\quad & A_{NE} = -\frac{\beta}{2}, &\quad & A_{SW} = -\frac{\beta}{2},
	\end{array}
\end{eqnarray}

\noindent
with derivatives present in $J$ approximated by central differences.

Next, the mesh generator in generalized coordinates is presented.

\section{Mesh Generator in Generalized Coordinate}

For the creation of an automated procedure for mapping a geometry in Cartesian coordinates, through a generalized coordinate system, we chose to use the programming language
Python.

The reason for this choice is due to the fact that Python is a general, free, open, and multiplatform language, which can allow the creation of extensions of the developed application, such as, for example, a graphical interface module, among others.

Regarding performance, it is necessary to use extra libraries, specific to numerical computing applications, to achieve satisfactory results at run time. Because it is a scripting language, that is, interpreted, the performance of an application written in Python tends to be inferior when compared to a compiled language, as is the case of the Fortran language. In Python, the tools available for matrix manipulation and linear algebra are limited and not optimized for the type of application that was intended to be developed. However, because the adopted language is modular, it is possible to use routines and libraries in compiled languages, in order to optimize more complex operations, such as those used for domain transformations (CAI et al., 2005).

For the creation of the mesh generator, the following libraries were used:
\begin{itemize}
	\item Numpy - library dedicated to matrix and algebraic operations in general (SCIPY-NUMPY, 2020);
  \item Matplotlib - library used to generate graphs and manipulate data in graphical form (SCIPY-MATPLOTLIB, 2020).
\end{itemize}

The additional modules employed are also free for use.

\subsection{Description of the Algorithm Developed}

For the definition and creation of a geometry mapped in generalized coordinates, it starts from a initial set of points that describe the border, that is, the physical contour. In this work, the Parametric Linear Spline method was chosen to interpolate the set of border points under study. Parametric Linear Spline equations (BURDEN et al., 2015) are
\begin{equation}
	\label{eq33a}
		s^{x}_{i}(t) = x_{i-1}\frac{t_{i} - t}{t_{i} - t_{i-1}} + x_{i}\frac{t - t_{i-1}}{t_{i} - t_{i-1}},
\end{equation}
\begin{equation}
	\label{eq33b}
		s^{y}_{i}(t) = y_{i-1}\frac{t_{i} - t}{t_{i} - t_{i-1}} + y_{i}\frac{t - t_{i-1}}{t_{i} - t_{i-1}},
\end{equation}
$\forall t \in [t_{i-1},t_{i}]$, where $s$ interpolating spline curve and $t$ is the interpolated variable.

Having obtained the interpolating lines of the border,
Eqs. (\ref{eq33a}) and (\ref{eq33b}), the number of partitions desired for each border must be defined.

Then, points of the Splines curves are selected, which define the new border of the approximate geometry. To do this, an approach is used to position the points by weighted average of the components of the border, ie
\begin{equation}
	\label{eq34a}
		x_{i j} = p^{\Gamma_{1} \Gamma_{3}}_{x} (x_{0 j} + i \Delta^{x \xi}_{j-1}) + p^{\Gamma_{2} \Gamma_{4}}_{x} (x_{i 0} + j \Delta^{x \eta}_{i-1}),
\end{equation}
\begin{equation}
	\label{eq34b}
		y_{i j} = p^{\Gamma_{1} \Gamma_{3}}_{y} (y_{0 j} + i \Delta^{y \xi}_{j-1}) + p^{\Gamma_{2} \Gamma_{4}}_{y} (y_{i 0} + j \Delta^{y \eta}_{i-1}),
\end{equation}

\vspace{0.3cm}

\noindent
where $\Delta^{x \xi}_{j} = \frac{x_{\xi j} - x_{0 j}}{\xi}$, $\Delta^{x \eta}_{i} = \frac{x_{i \eta} - x_{i 0}}{\eta}$, $\Delta^{y \xi}_{j} = \frac{y_{\xi j} - y_{0 j}}{\xi}$, $\Delta^{y \eta}_{i} = \frac{y_{i \eta} - y_{i 0}}{\eta}$, $i = 1,...,\xi$ e $j = 1,...,\eta$.

\vspace{0.3cm}
\noindent
The values of $p^{\Gamma_{1} \Gamma_{3}}_{x}$, $p^{\Gamma_{2} \Gamma_{4}}_{x}$, $p^{\Gamma_{1} \Gamma_{3}}_{y}$, and $p^{\Gamma_{2} \Gamma_{4}}_{y}$ indicate the weights on the left/right borders in $x$, top/bottom in $x$, left/right in $y$ and top/bottom in $y$ , respectively. The percentage weights in Eqs. (\ref{eq34a}) and (\ref{eq34b}) can better adjust the distribution of points in the domain, avoiding concentrations of points. In this way, the arrangement of the internal points is influenced by the edges.

Finally, on the set of ordered pairs, the resolution of the equation (\ref{eq30}) is applied.
\vspace {0.3cm}

In summary, the implemented algorithm can be illustrated in this way:

\noindent
Step 1. Reading the input data: Read the number of partitions from the physical and transformed planes, the weights for calculating the weighted average and the border points.

\noindent
Step 2. Border interpolation: Define the border using the Parametric Linear Spline method and the border points previously read.

\noindent
Step 3. Calculate the points of the Spline lines, using the weighted average formulas, and define the new border points.

\noindent
Step 4. Application of transformation metrics: On the generated data, apply the transformation metrics to obtain the interior mesh curves.

\vspace {0.3cm}
The next two sections are intended to exemplify and test the algorithm. In the first applications, monoblock meshes are built (3 applications). Following a multiblock mesh is obtained. The multiblock technique consists of dividing the physical domain into parts and, on each of the parts, applying the generation of meshes in generalized coordinates. Such a procedure allows to better capture the complexity of the geometry compared to the strategy employed on a single block (CIRILO, 2006; PARDO et al., 2012, SAITA et al., 2017).

\subsection{Monoblock Mesh Generation}
\label{sec3.2}

As a first application of the mesh generation algorithm, the set of input data presented in Table \ref{tab1} was used. The weighted values for $\Gamma_{j}$, $j =$ 1 and 3 are omitted, since they are known for complementarity, that is, $p^{\Gamma_{1} \Gamma_{3}}_{x} + p^{\Gamma_{2} \Gamma_{4}}_{x} = 1$ e $p^{\Gamma_{1} \Gamma_{3}}_{y} + p^{\Gamma_{2} \Gamma_{4}}_{y} = 1$.

\begin{table}[H]
	\begin{center}
		\caption[caption]{Input data of a Rectangular Geometry with Cutout}
		\begin{tabular}{c|c|c|c} \hline
			Number of partitions ($d^{f}$) & \multicolumn{2}{c|}{Number of partitions ($d^{t}$)} & Weighted average \\
			Input data & Refinement 1 & Refinement 2 & Weights ($p^{\Gamma_{2} \Gamma_{4}}$) \\ \hline
			$x \quad \quad y$ & $\xi \quad \quad \eta$ & $\xi \quad \quad \eta$ & $x \quad \quad y$ \\
			$10 \quad \quad 4$ & $15 \quad \quad 6$ & $30 \quad \quad 12$ & $0.9 \quad \quad 1.0$ \\ \hline
		\end{tabular}
		\label{tab1}
	\end{center}
\end{table}

Given the set of predefined points, Fig. 4a, the number of partitions and the weights at the borders,
Table \ref{tab1}, then the border geometry is obtained by Parametric Spline Linear interpolation. In sequence, the algorithm calculates the new border points, using the given refinement. In this way, the algorithm starts to solve the transformation metrics, finally generating the mesh in generalized coordinates. Figs. 4b and 4c present the meshes in generalized coordinates, according to the refinements described in Table \ref{tab1}.

\begin{figure}[!ht]
\begin{center}
\caption{Rectangular Geometry with Cutout}
\subfigure[Predefined points]{\includegraphics[height=4.4cm,width=5.4cm]{bordo1.png}}
   \label{fig6a}\qquad \qquad
\subfigure[Refinement 1]{\includegraphics[height=4.4cm,width=5.4cm]{refinamento1_1.png}}
    \label{fig6b}\qquad \qquad
\subfigure[Refinement 2]{\includegraphics[height=4.4cm,width=5.4cm]{refinamento2_1.png}}
    \label{fig6c}
\end{center}
\center{{\bf{Source:}} The Authors}
\label{fig6}
\end{figure}

As a second application, a mesh in a doubly-connected domain was generated. The set of input data is shown in Table \ref{tab3}.

\begin{table}[H]
	\centering
	\caption[caption]{Input data of a Doubly-Connected Geometry}
	\begin{tabular}{c|c|c|c} \hline
		Number of partitions ($d^{f}$) & \multicolumn{2}{c|}{Number of partitions ($d^{t}$)} & Weighted average \\
		Input data & Refinement 1 & Refinement 2 & Weights ($p^{\Gamma_{1} \Gamma_{3}}$) \\ \hline
		$x \quad \quad y$ & $\xi \quad \quad \eta$ & $\xi \quad \quad \eta$ & $x \quad \quad y$ \\
		$20 \quad \quad 4$ & $30 \quad \quad 6$ & $40 \quad \quad 8$ & $0.05 \quad \quad 0.0$ \\ \hline
	\end{tabular}
	\label{tab3}
\end{table}

In this application, the left/right border corresponds to the line segment that joins the two circles, while the upper/lower border corresponds to the two circles, as shown in Fig 5a. From the set of points given in Fig. 5a and the input data in Table \ref{tab3}, the algorithm constructs the meshes shown in Figs. 5b and 5c, according to the refinements described in Table \ref{tab3}.

\begin{figure}[!ht]
\begin{center}
\caption{Doubly-Connected Geometry}
\subfigure[Predefined points]{\includegraphics[height=4.4cm,width=5.4cm]{bordo3.png}}
   \label{fig8a}\qquad \qquad
\subfigure[Refinement 1]{\includegraphics[height=4.4cm,width=5.4cm]{refinamento1_3.png}}
    \label{fig8b}\qquad \qquad
\subfigure[Refinement 2]{\includegraphics[height=4.4cm,width=5.4cm]{refinamento2_3.png}}
    \label{fig8c}
\end{center}
\center{{\bf{Source:}} The Authors}
\label{fig8}
\end{figure}

The latest generation of monoblock mesh is shaped like a bottle, as illustrated in Fig. 6. The set of input data is explained in Table \ref{tab2}.

\begin{table}[H]
	\centering
	\caption[caption]{Input data of a Bottle Geometry}
	\begin{tabular}{c|c|c|c} \hline
		Number of partitions ($d^{f}$) & \multicolumn{2}{c|}{Number of partitions ($d^{t}$)} & Weighted average \\
		Input data & Refinement 1 & Refinement 2 & Weights ($p^{\Gamma_{1} \Gamma_{3}}$) \\ \hline
		$x \quad \quad y$ & $\xi \quad \quad \eta$ & $\xi \quad \quad \eta$ & $x \quad \quad y$ \\
		$4 \quad \quad 10$ & $6 \quad \quad 15$ & $12 \quad \quad 30$ & $1.0 \quad \quad 0.5$ \\ \hline
	\end{tabular}
	\label{tab2}
\end{table}

Using the set of predefined points, Fig. 6a, and the input data in Table \ref{tab2}, the meshes in generalized coordinates are generated, as shown in Fig. 6b and Fig. 6c, according to the refinements described in Table \ref{tab2}.

\begin{figure}[!ht]
\begin{center}
\caption{Bottle Geometry}
\subfigure[Predefined points]{\includegraphics[height=4.4cm,width=5.4cm]{bordo2.png}}
   \label{fig7a}\qquad \qquad
\subfigure[Refinement 1]{\includegraphics[height=4.4cm,width=5.4cm]{refinamento1_2.png}}
    \label{fig7b}\qquad \qquad
\subfigure[Refinement 2]{\includegraphics[height=4.4cm,width=5.4cm]{refinamento2_2.png}}
    \label{fig7c}
\end{center}
\center{{\bf{Source:}} The Authors}
\label{fig7}
\end{figure}

automated mesh generator
In the applications presented for monoblock meshes, the automated mesh generator precisely built the geometry in the physical domain.

\subsection {Multiblock Mesh Generation}

In order to show a more complex application, points from the contour of Lake Igapó I, located in Londrina, Paraná, Brazil, were collected. The points were obtained from satellite images offered by the Google Earth software (GOOGLE, 2020). The generated mesh has the same scale as the physical domain.

Unlike previous applications, in addition to generating the monoblock meshes (now called monobloc sub-meshes), in this application the sub-meshes must be connected. The multiblock mesh algorithm uses the monoblock mesh generator as a subroutine of the multiblock algorithm.
\vspace {0.3cm}

In summary, the multiblock algorithm can be illustrated in this way:

\noindent
Step 1. Loop condition: If there is a monoblock input file being passed as an argument, perform step 2. Otherwise, perform step 4.

\noindent
Step 2. Monoblock mesh generation: Use the data present in the input file to generate the monoblock mesh.

\noindent
Step 3. Plot the monoblock mesh: Use the results of the monoblock mesh generator algorithm to plot the monoblock mesh, and return to step 1.

\noindent
Step 4. Plot the multiblock mesh: Show all monoblock meshes generated in a single multiblock mesh.
\vspace {0.3cm}

For the geometry of Lago Igapo, it is considered that it consists of three blocks, Lake Igapó I (main block) and its two tributaries (two other blocks). The set of predefined points are given in Fig. 7a. The input data are presented in Table \ref{tab4}. As for the weighted average, the percentage weights ($p^{\Gamma_{1} \Gamma_{3}}$ e $p^{\Gamma_{2} \Gamma_{4}}$), in $ x $ and $ y $, were all chosen as $ 0.5 $.

\begin{table}[H]
	\centering
	\caption[caption]{Input data of Igapó I Lake}
	\begin{tabular}{c|c|c|c} \hline
		Monoblock & Number of partitions ($d^{f}$) & \multicolumn{2}{c}{Number of partitions ($d^{t}$)} \\
		Name & Input data & Refinement 1 & Refinement 2 \\ \hline
		Principal & $x \quad \quad y$ & $\xi \quad \quad \eta$ & $\xi \quad \quad \eta$ \\
		          & $6 \quad \quad 70$ & $6 \quad \quad 70$ & $12 \quad \quad 140$ \\ \hline
    Tributary 1 & $x \quad \quad y$ & $\xi \quad \quad \eta$ & $\xi \quad \quad \eta$ \\
                	& $10 \quad \quad 2$ & $10 \quad \quad 2$ & $20 \quad \quad 4$ \\ \hline
	  Tributary 2 & $x \quad \quad y$ & $\xi \quad \quad \eta$ & $\xi \quad \quad \eta$ \\
      	        	& $8 \quad \quad 5$ & $8 \quad \quad 5$ & $28 \quad \quad 10$ \\ \hline
		\end{tabular}
	\label{tab4}
\end{table}

After executing the multiblock mesh algorithm, the meshes were obtained in generalized coordinates as shown in Figs. 7b and 7c, according to the refinements of Table \ref{tab4}.

\begin{figure}[!ht]
\begin{center}
\caption{Igapó I Lake Geometry}
\subfigure[Predefined points]{\includegraphics[height=4.4cm,width=5.4cm]{bordo4_cut.png}}
   \label{fig9a}\qquad \qquad
\subfigure[Refinement 1]{\includegraphics[height=4.4cm,width=5.4cm]{refinamento1_4_cut.png}}
    \label{fig9b}\qquad \qquad
\subfigure[Refinement 2]{\includegraphics[height=4.4cm,width=5.4cm]{refinamento2_4_cut.png}}
    \label{fig9c}
\end{center}
\center{{\bf{Source:}} The Authors}
\label{fig9}
\end{figure}

Again, the automated mesh generator accurately represented the physical domain of Igapó I lake, located in Londrina, Paraná, Brazil.

\section{Final Considerations}

In this work, using the generalized coordinate method, a mesh generator in Python was created and implemented. This automated mesh generator allowed the reproduction of complex geometries with several mesh refinements, as shown by the applications presented.

Specifically, as original contributions to this work, we highlight two aspects:

1- Parameterized Linear Spline. The use of the Parameterized Linear Spline method to obtain points at the border of the physical domain, given predefined points, allows the construction of the internal mesh in generalized coordinate, with the desired refinement, in a simple and fast way.

2- Python language. The Python programming language and the libraries used make it possible to optimize the algorithm, to facilitate the visualization of the meshes, and the portability of the application in different software architectures, such as Linux and Windows.

\vspace{0.9cm}

\noindent \textbf{\large{Referências}}

\vspace{0.5cm}

\noindent BELINELLI, E.O.; NATTI, P.L.; ROMEIRO, N.M.L.; CIRILO, E.R.; FANTIN, L.H.; OLIVEIRA, K.B.; CANTERI, M.G.; NATTI, E.R.T. {\it Geração de malha para descrever a dispersão da ferrugem da soja no Paraná}. In: Júlio Cesar Ribeiro. (Org.). Ciências Exatas e da Terra: Conhecimentos Estratégicos para o Desenvolvimento do País. 1ed.: Atena Editora, 2020, v. , p. 225-239.

\noindent BURDEN, R.L., FAIRES, J.D., BURDEN, A.M. \textit{Numerical Analysis}. Boston: Cengage Learning, 2015.

\noindent CAI, X., LANGTANGEN, H. P., MOE, H. On the performance of the Python programming language for serial and parallel scientific computations. \textit{Scientific Programming}, vol. 13, n. 1, pp. 31-56, 2005.

\noindent CIRILO, E.R.; DE BORTOLI, A.L. Cubic splines for trachea and bronchial tubes grid generation. {\it Semina: Exact and Technological Sciences}, v. 27, p.147-155, 2006.

\noindent CIRILO, E. R. ; BARBA, A. N. D. ; NATTI, P. L. ; ROMEIRO, N. M. L. . A numerical model based on the curvilinear coordinate system for the MAC method simplified. SEMINA. CIÊNCIAS EXATAS E TECNOLÓGICAS (ONLINE), v. 39, p. 87, 2018.

\noindent DE BORTOLI, A. L. \textit{Introdução à dinâmica de fluidos computacional}. Porto Alegre: Editora da UFRG, 2000.

\noindent FORTUNA, A.O. \textit{Computational techniques for fluid dynamics: Basic concepts and applications}. S\~ao Paulo: EDUSP, 2012.

\noindent GOOGLE. \textit{Google Earth}. Disponível em: <\underline{\url{https://www.google.com/earth/}}>. Acessed in oct. 2020.

\noindent KOOMULLIL, R.; SONI B.; SINGH R. A comprehensive generalized mesh system for CFD applications. \textit{Mathematics and Computers in Simulation}, vol. 78, pp. 605-617, 2008.

\noindent LAIPING, Z.; ZHONG, Z.; XINGHUA, C., XITHAMES, H. A 3D hybrid grid generation technique and a multigrid/parallel algorithm based on anisotropic agglomeration approach. \textit{Chinese Journal of Aeronautics}, vol. 26, n.1, pp. 47-62, 2013.

\noindent MALISKA, C. R. \textit{Transferência de calor e mecânica dos fluidos computacional}. São Paulo: LTC, 2004.

\noindent PARDO, S.R.; NATTI, P.L.; ROMEIRO, N.M.L.; CIRILO, E.R. A transport modeling of the carbon-nitrogen cycle at Igap\'o I Lake-Londrina, Paran\'a State, Brazil. {\it Acta Scientiarum. Technology}, v. 34, p. 217-226, 2012.

\noindent ROMEIRO, N.L.M.; CASTRO, R.G.S.; CIRILO, E.R.; NATTI, P.L. Local calibration of coliforms parameters of water quality problem at Igap\'o I Lake - Londrina, Paran\'a, Brazil. {\it Ecological Modelling}, v. 222, p. 1888-1896, 2011.

\noindent ROMEIRO, N.L.M.; MANGILI, F.B.; COSTANZI, R.N.; CIRILO, E.R.; NATTI, P.L. Numerical simulation of BOD5 dynamics in Igap\'o I lake, Londrina, Paran\'a, Brazil: Experimental measurement and mathematical modeling. {\it Semina: Exact and Technological Sciences}, v. 38, p. 50-58, 2017.

\noindent SAITA, T.M.; NATTI, P.L.; CIRILO, E.R.; ROMEIRO, N.L.M.; CANDEZANO, M.A.C.; ACUNA, R.A.B.; MORENO, L.C.G. Simula\c{c}\~ao num\'erica da din\^amica de coliformes fecais no lago Luruaco, Col\^ombia. {\it Trends in Applied and Computational Mathematics}, v. 18, p. 435-447, 2017.

\noindent SCIPY-NUMPY. In <\underline{\url{http://www.numpy.org/}}>. Acessed in oct. 2020.

\noindent SCIPY-MATPLOTLIB. In <\underline{\url{http://matplotlib.org/}}>. Acessed in oct. 2020.

\noindent THOMPSON, J. F.; THAMES, F. C.; MASTIN, C. W. TOMCAT - A Code for Numerical Generation of Boundary Fitted Curvilinear Coordinate Systems on Fields Containing Any Number of Arbitrary Two-Dimensional Bodies. \textit{Journal of Computational Physics}, vol. 24, pp. 274-302, 1977.

\noindent THOMPSON, J. F.; WARSI, Z. U. A.; MASTIN, C. W. \textit{Numerical grid generation: foundations and applications}. New York: Elsevier Science Publishing, 1985.

\noindent THOMPSON, J.F.; SONI, B.K.; WEATHERILL, N.P. \textit{Handbook of Grid Generation}. Florida: CRC Press, 1998.

\end{document}