\section{Introduction}

Deep learning has achieved strong results on a plethora of challenging tasks. However, performing well on a highly specific dataset is usually insufficient to satisfactorily solve real-world problems~\cite{tan2018survey,zhuang2019comprehensive}. This has lead to a particular interest in consistently learning more structured representations with useful properties to help with a variety of downstream tasks~\cite{bengio2013representation,tschannen2018recent,1703.07718,tschannen2018recent,radhakrishnan2018memorization}. Here, deep learning provides a flexible paradigm to train complex architectures based on autoencoders~\cite{ballard1987modular,lange2010deep} which are typically latent variable models, thus allowing us to embed powerful inductive biases into the models to further structure the representations.
However, it is still largely an open question as to what kinds of structure in a representation are the most effective for generative modeling and how to learn such structures without supervision \cite{locatello2018challenging,khrulkov2021disentangled,shu2019weakly,chen2020deep,nie2020semi}. One direction that may contribute to an answer is causal modeling, as it focuses on the underlying (causal) mechanisms that generate the observations, instead of relying on (possibly spurious) correlations \cite{Pearl2009,PetJanSch17,1911.10500,louizos2017causal,mitrovic2020representation,shen2020disentangled}.

With the versatility of deep learning on one hand, and the conceptual insights of causality on the other, our contributions herein include:

\begin{itemize}
    \item We propose an architecture called the \textit{Structural Autoencoder} (SAE), where the {\em structural decoder} emulates a general acyclic structural causal model to learn a hierarchical representation that can separate and order the underlying factors of variation in the data.

    \item We provide a sampling method that can be used for any autoencoder-based generative models which does not use an explicit regularization objective and instead relies on independence in the latent space.

    \item We investigate how well the encoder and decoder are able to extrapolate upon seeing novel samples.

\end{itemize}

We release our code at \url{*anonymized*}.

\subsection{Related Work}

The most popular autoencoder based method is the Variational Autoencoder (VAE)~\cite{KinWel13} and the closely related $\beta$VAE~\cite{higgins2017beta}. These methods focus on matching the distribution in the latent space to a known prior distribution by regularizing the reconstruction training objective~\cite{locatello2020sober,zhou2020evaluating}. Although this structure is convenient for generative modeling and even tends to disentangle the latent space to some extent, it comes at the cost of somewhat blurry images due to posterior collapse and holes in the latent space
~\cite{locatello2018challenging,higgins2017beta,burgess2018understanding,kim2018disentangling,stuhmer2020independent}.

To mitigate the double-edged nature of VAEs, less aggressive regularization techniques have been proposed such as the Wasserstein Autoencoder (WAE), which focuses on the aggregate posterior~\cite{tolstikhin2017wasserstein}. Unfortunately, WAEs generally fail to produced a particularly meaningful or disentangled latent space~\cite{rubenstein2018latent}, unless weak supervision is available~\cite{han2021disentangled}.

A more structured alternative is the Variational Ladder autoencoder (VLAE)~\cite{zhao2017learning} which separates the latent space into separate chunks each of which is processed at different levels of the encoder and decoder (called "rungs").
Our proposed architecture takes inspiration from the VLAEs but crucially we do not use the variational regularization, and instead use a sampling method based on hybridization~\cite{besserve2018counterfactuals}. We also infuse the information from the latent variables using Str-Tfm layers (see section~\ref{sec:strc}) which are based on the Ada-IN layers from Style-GANs~\cite{karras2019style}, which significantly improves on the ladder rungs.

\section{Methods\label{sec:method}}

\subsection{Causal Representation Learning} \label{sec:causal}

Graphical causal modeling builds on random variables
\begin{equation}\label{eq:SA}
S_i := f_i (\B{PA}_i, U_i),   ~~~~ (i=1,\dots,n),
\end{equation}
connected by a directed acyclic graph (DAG) whose edges represent direct causation. Each $S_i$ is computed using a function $f_i$ depending on its parents $\B{PA}_i$ in the graph and on an {\em unexplained} noise variable $U_i$, ensuring that the ``mechanism'' \eq{eq:SA} can represent any conditional distribution $p(S_i|\B{PA}_i)$. The noises $U_1,\dots,U_n$ are assumed to be jointly independent.
The DAG along with the mechanisms \eq{eq:SA} is referred to as a Structural Causal Model (SCM) \cite{Pearl2009}. Any joint distribution of the $S_i$ can be expressed as an SCM using suitable $f_i$ and $U_i$. However, the SCM also
represents {\em interventions} (e.g., fixing values of $U_i$ or $S_i$), and it makes it explicit how statistical dependences between the $S_i$ are {\em generated} by mechanisms \eq{eq:SA}.

Real-world observations, however, are often not structured into meaningful causal variables and mechanisms to begin with.
E.g., images are high-dimensional, and it is hard to learn objects and their causal relationships from data \citep{LopNisChiSchBot17}.
One may thus attempt to try and learn a {\em representation} consisting of causal variables or disentangled ``factors'' which are statistically independent \cite{higgins2017beta}. However, in an SCM it is not the $S_i$ that should be statistically independent, but the $U_i$. For this reason, our representations will comprise the $U_i$ as latent variables, driving the causal mechanisms in a way that can structurally implement \eq{eq:SA}.
This embeds an SCM into a larger model whose inputs and outputs may be high-dimensional and unstructured (e.g., images) \cite{suter2018robustly,1911.10500}.

Given (high-dimensional) $X=(X_1,\dots,X_d)$, we first use an {\bf encoder} $f_{enc}:\R^d \to \R^n$ taking $X$ to a latent representation $U=(U_1,\dots,U_n)$. Next, we apply the {\bf SCM}, with the $U_i$ feeding into subsequent computation layers according to a causal ordering (see Supplement), i.e., the root node(s) in the DAG only depend on ``their'' noise variables, while later ones depend on their noise and potentially also those of their parents, and so on. The depth in the network thus corresponds to a causal ordering (cf. Figure~\ref{fig:branch} below).
Finally, we apply a {\bf decoder} $f_{dec}$ taking us back to $\R^d$.
If the causal graph were known, the topology of the network implementing the SCM could be fixed accordingly. Below, we assume that we do not know it, and thus our decoder will learn the composition of SCM and decoder, with the SCM effectively becoming an unidentified part thereof.

By choosing the decoder topology, we will ensure that different parts of the noise vector should feed into separate subsequent components/mechanisms (using connections skipping layers). In view of the above considerations, this means that in principle, any SCM can be learned and embedded in this architecture. In analogy to structural causal models, we will below refer to this as a {\em structural decoder.}

The above shows that an autoencoder with independent latent variables can be viewed as the composition of an (anticausal) encoder that recognizes or reconstructs causal drivers in the world, and a (causal) generator that maps the low dimensional latent representation (of the noises driving the causal model) back to the high dimensional world.
In practice, without some form of supervision or side information, the learned SCM is not guaranteed to match the true SCM.

\subsection{Structural Decoders} \label{sec:strc}

\begin{figure}

  \begin{center}
    \includegraphics[width=0.5\columnwidth]{figures/branch_structure.pdf}
  \end{center}
    \caption{\label{fig:branch} The Structural Decoder reconstructs (or generates) an image from a latent vector $q$  by first splitting $q$ into $K$ segments each of which transforms the image features at a different layer of the model using a Str-Tfm layer (green box where all pixels $\mathbf{x}_j$ are transformed with the same $\mathbf{z}_s$ and $\mathbf{z}_b$).}
    \vspace{-1mm}
\end{figure}

Our {\em structural decoder} splits the latent vector into $K$ $D_k$-dimensional separate segments for a total latent space dimensionality of $D=K*D_k$. Each segment is applied to its own \emph{Structural-Transform} (Str-Tfm) layer (as seen in Figure~\ref{fig:branch}) to produce a scale $\mathbf{z}_s$ and bias $\mathbf{z}_b$ which are then used to transform the features much like in Ada-IN~\cite{karras2019style} except without the normalization.

Each Str-Tfm layer thus acts like an $f_i$ in \eq{eq:SA} by integrating the information from a latent variable $U_i$ to transform the features $\B{PA}_i$ from earlier layers producing a hierarchical structure in the latent space. Since the decoder uses significantly more layers for the first few latent dimensions than the last few, more nonlinear information is naturally encoded in the first few segments. Meanwhile, the low-level linear features are only processed by the relatively shallow last few layers of the decoder.

\subsection{Hybrid Sampling}\label{sec:hybrid}

\begin{figure}[b]
  \begin{center}
    \includegraphics[width=\columnwidth]{figures/hybrid_sampling.pdf}
  \end{center}
    \caption{\label{fig:hybrid} In hybrid sampling, each component of latent vectors (colored boxes) are uniformly randomly resampled to produce novel latent samples. In general, each latent dimension is resampled independently, however, since the SAE models, each segment (which may include multiple latent dimensions) is hybridized.
    }

\end{figure}

For generative modeling, it is necessary to sample novel latent vectors that are transformed into (synthetic) observations using the decoder. Usually, this is done by regularizing the training objective so the posterior matches some simple prior (e.g. the standard normal).
However, in practice, regularization techniques can fail to match the prior perfectly and actually exacerbate the information bottleneck, leading to blurry samples from holes in the learned latent distribution and unused latent dimensions due to posterior collapse~\cite{dai2019diagnosing,stuhmer2020independent,lucas2019understanding,hoffman2016elbo}.
Instead of trying to match some prior distribution in the latent space, we suggest an alternative sampling method that eliminates the need for any regularization of the loss. Inspired by~\cite{besserve2018counterfactuals}, we refer to it as {\em hybrid} sampling: the model stores a finite set of $N$ ($=128$) latent vectors, selected uniformly at random from the training set. To generate a new latent vector, a value for each of the $D$ ($=12$)  dimensions is selected independently from the $N$ stored latent vectors uniformly at random (see figure~\ref{fig:hybrid}). This allows the model to generate $N^D$ latent vectors.

Hybrid sampling implicitly relies on the latent dimensions being independent since resampling the marginal of the posterior independently breaks any existing correlations, however, this is consistent with the causal perspective of the latent variables as noises driving an SCM. Independence between latent variables also aligns with the objective to disentangle the latent factors of variation. Note that hybrid sampling is directly applicable to any learned representation as it does not affect training at all, however the fidelity of generated samples will diminish if there are strong correlations between latent dimensions, so maximizing the independence between latent variables becomes the implicit goal.
Since the structural decoders already split the latent space into separate segments, each full segment is hybridized together, instead of independently resampling dimensions that contribute to the same segment, which would destroy any learned structure or dependence within a segment.

\section{Experiments}
\label{sec:exp}

We train the proposed architectures and baselines on four different datasets: 3D-Shapes~\cite{burgess20183d}, the three variants ("toy", "sim", and "real") of the MPI3D Disentanglement dataset~\cite{gondal2019transfer}, as well as Celeb-A~\cite{liu2015faceattributes}\footnote{\label{cele}The CelebA dataset is available for non-commercial research purposes (no license provided). It does show images of individuals, and permission may not have been obtained; however, our understanding is that usage of these images is permitted since the individuals are celebrities. We will remove those experiments from the paper if this is considered inappropriate.} and the Robot Finger Dataset (RFD)~\cite{dittadi2020transfer}.

After training our models on a standard 70-10-20 (train-val-test) split of the datasets we evaluate the quality of the reconstructions based on the reconstruction loss (using binary cross entropy loss) and the Fréchet Inception Distance (FID)~\cite{heusel2017gans} as in~\cite{williams2020hierarchical}. The FID is able to capture higher level visual features and can be used to directly compare the reconstructed and generated sample quality, while the binary cross entropy is a purely pixelwise comparison.

Next we compare the performance of the hybrid sampling method, which can be applied to all of the models, to the prior based sampling, which instead only makes sense for the models that use regularization (VLAE, VAE, $\beta$VAE, and WAE).
Finally we take a closer look at the learned representations to understand how the model architecture affects the induces certain structure and possibly disentanglement.

\subsection{Models}

All models use the same CNN backbone for both the encoder and decoder with the same number of convolution layers and filters, each of which is followed by group normalization and a MISH nonlinearity~\cite{misra2019mish}, where the encoder uses max-pooling to down-sample the extracted features and the decoder uses bilinear up-sampling to up-sample features (see the appendix for details). For 3D-Shapes and the MPI3D datasets, the encoder and decoder have 12 convolution layers each with 64 filters, the latent space has 12 dimensions, and the models are trained for 100k iterations, while for CelebA each the encoder/decoder has 16 convolution layers each with twice as many filters, the latent space is 32 dimensional, and the models are trained for 200k iterations. For the RFD dataset, the settings are the same as for CelebA, except that the number of channels per convolution layer was doubled again to 256. All training and evaluation was done using V100 GPUs, where training on the smaller datasets took about 2 GPU hours per model, and around 16-20 GPU hours for CelebA and RFD.

We compare four kinds of autoencoder architectures. The first type is our Structural Autoencoders (SAE) which use a conventional encoder and a structural decoder with 2, 3, 4, 6, or 12 Str-Tfm layers placed evenly between convolution layers (and 16 for CelebA). Since the total number of dimensions in the latent space is constant, the number of dimensions per segment is 6, 4, 3, 2, 1 for the SAE-2, SAE-3, SAE-4, SAE-6, and SAE-12 respectively. The simplest "baseline" architecture uses the traditional "hourglass" architecture, in addition to a variety of different regularization methods: (1) unregularized autoencoders ("AE"), (2) Wasserstein-autoencoders ("WAE")~\cite{tolstikhin2017wasserstein}, (3) VAEs~\cite{kingma2013auto}, and a $\beta$VAE~\cite{higgins2017beta} (with the experimentally determined best setting of $\beta=2$). The next more structured baseline architecture is called "AdaAE-12" (and referred to as "Adaptive" as it is identical to the SAE-12 model except that instead of splitting the latent vector into 12 segments with one dimension used per Str-Tfm layer, each Str-Tfm layer receives the entire latent vector, so the architecture is effectively the same as using Ada-IN layers~\cite{karras2019style}.

The final type of architecture we investigate is the Variational Ladder Autoencoder~\cite{zhao2017learning} which also uses a structured architecture to learn a hierarchical representation, but unlike our Structural Autoencoders, VLAEs also use the variational regularization and use an encoder architecture that roughly mirrors the decoder. Another subtle, but significant difference between the SAE and VLAE architecture is that the ladder rungs in the VLAE decoder concatenate the features from the latent space segment of the corresponding rung in the encoder, instead of using the segment to scale and shift the convolution features directly. Just like for the SAE models, we include variants of the VLAEs with 2, 3, 4, 6, and 12 rungs.

\subsection{Extrapolation} \label{sec:extrapolation}

While the encoder and decoder are generally optimized jointly, the tight coupling  combined with the inherent asymmetry in the respective learning tasks begs the question which of the two is the weaker link when it comes to generalization. To this end, we investigate to what extent the encoder vs the decoder can extrapolate to novel observations.

First, both the encoder and decoder are trained jointly on a subset of 3D-Shapes where only three distinct shapes exist (instead of four, as the ball is missing) for 80k iterations. Then either the encoder only, the decoder only, or both are trained for another 20k iterations on the full 3D-Shapes training dataset. The reconstruction error with for samples not seen during any part of training is compared for each of the variants and each of the architectures to identify whether updating the encoder or decoder has a more significant impact on performance and how well the encoder extrapolates compared to the decoder.

\section{Results}
\label{sec:results}

In terms of reconstruction quality, the structural autoencoder architecture consistently outperforms the baselines (seen in figure~\ref{fig:perf}). As expected, unregularized methods like the SAE, AdaAE, and AE tend to have significantly better reconstruction quality. However, the structured architectures like the SAE and VLAE also improve performance somewhat.

\begin{figure*}
  \centering
  \begin{subfigure}{0.33\linewidth}
  \centering
    \includegraphics[width=\linewidth]{figures/3ds_rec-fid.pdf}
    \vspace{-5mm}
    \caption{\label{fig:perf-3ds} 3D-Shapes}
    \vspace{-1mm}
  \end{subfigure}%
  \begin{subfigure}{0.33\linewidth}
  \centering
    \includegraphics[width=\linewidth]{figures/toy_rec-fid.pdf}
    \vspace{-5mm}
    \caption{MPI3D Toy}
    \vspace{-1mm}
  \end{subfigure}%
  \begin{subfigure}{0.33\linewidth}
  \centering
    \includegraphics[width=\linewidth]{figures/real_rec-fid.pdf}
    \vspace{-5mm}
    \caption{MPI3D Real}
    \vspace{-1mm}
  \end{subfigure}%

  \begin{subfigure}{0.33\linewidth}
  \centering
    \includegraphics[width=\linewidth]{figures/celeb_rec-fid.pdf}
    \vspace{-5mm}
    \caption{Celeb-A}
    \vspace{-1mm}
  \end{subfigure}%
  \begin{subfigure}{0.33\linewidth}
  \centering
    \includegraphics[width=\linewidth]{figures/RFD_rec-fid.pdf}
    \vspace{-5mm}
    \caption{RFD}
    \vspace{-1mm}
  \end{subfigure}%
  \caption{Results on the reconstruction quality for all models and datasets. The "Baseline" models correspond to traditional "hourglass" CNN architectures, while the "Structural" models use our novel architectures to further structure the learned representation.
  }
  \label{fig:perf}
\end{figure*}

\begin{figure*}
  \centering
  \begin{subfigure}[b]{0.33\linewidth}
  \centering
    \includegraphics[width=\linewidth]{figures/3ds_gen_tall.pdf}

    \caption{\label{fig:perf-3ds} 3D-Shapes}

  \end{subfigure}%
  \begin{subfigure}[b]{0.33\linewidth}
  \centering
    \includegraphics[width=\linewidth]{figures/toy_gen_tall.pdf}

    \caption{MPI3D Toy}

  \end{subfigure}%
  \begin{subfigure}[b]{0.33\linewidth}
  \centering
    \includegraphics[width=\linewidth]{figures/real_gen_tall.pdf}

    \caption{MPI3D Real}

  \end{subfigure}%

  \begin{subfigure}[b]{0.33\linewidth}
  \centering
    \includegraphics[width=\linewidth]{figures/celeb_gen_tall.pdf}

    \caption{Celeb-A}

  \end{subfigure}%
  \begin{subfigure}[b]{0.33\linewidth}
  \centering
    \includegraphics[width=\linewidth]{figures/RFD_gen_tall.pdf}

    \caption{RFD}

  \end{subfigure}%

  \caption{Comparison of the generative models using different sampling methods. Note that our SAE models perform well without having to regularize the latent space towards a prior. In fact, even with the conventional "hourglass" architecture (in orange), the hybrid sampling method generates relatively high quality samples.
  }
  \label{fig:gen}
\end{figure*}

In comparing the SAE models to the AdaAE architecture, we see that there can be a slight penalty in reconstruction quality incurred from splitting the latent vector into segments. However this is more than made up for in the quality of the generated samples (shown in figure~\ref{fig:gen}), where the SAE models perform significantly better than the baselines. Even the regularized models such as VLAE and VAEs usually generate higher quality samples using the hybrid sampling than when sampling from the prior they were trained to match. This can qualitatively be observed from figure~\ref{fig:celeb-gen}. Surprisingly, the AdaAE architecture actually outperforms all other models on CelebA using hybrid sampling. This may be explained by the severe information bottleneck experienced when embedding CelebA into only 32 dimensions. Since the AdaAE model produces the best reconstructions, the generated samples have a higher fidelity even though any learned structure between latent dimensions is disregarded by the hybrid sampling.

In general, if we consider the distribution of the latent variables (i.e., the push-forward of the data distribution into the latent space), then sampling from a simple factorized prior can introduce at least two types of errors: (1) errors due to not taking into account statistical dependences among latent variables, and (2) errors due to sampling from "holes" in the latent distribution if the prior does not match it everywhere. Whenever (2) is the dominating source of error, hybrid sampling is effective.

\subsection{Hierarchical Structure}\label{sec:hier}

To get a rough idea of how the representations learned using the structural decoders differ from more conventional architectures, figure~\ref{fig:trav} shows the one dimensional latent traversals (i.e., each row corresponds to the decoder outputs when incrementally increasing the corresponding latent dimension at a time from the min to the max value observed). The traversals illustrate the hierarchical structure in the representation learned by the SAE models: the information encoded to the first few latent variables can be more nonlinear with respect to the output (pixel) space, as the decoder has more layers to process that information, while the more linear information can be embedded in the last few latent dimensions. This aligns nicely with the ordering of placing more "high-level" information (such as object shape or viewpoint) first, followed by the "low-level" information (such as color). This means the structural decoder architecture biases the representation to separate and order the information necessary for reconstruction (and generation) in a meaningful way, and thereby tends to disentangle the underlying factors of variation better.

Figure~\ref{fig:metrics} evaluates how disentangled the representations are for five different random seeds (used to initialize the network parameters) using common metrics. Most noteworthy is that the SAE-12 model consistently achieves very high disentanglement scores. This shows, empirically, that the SAE architecture promotes independence between latent variables (especially SAE-12). We may explain this as a consequence of splitting up the latent dimensions so that each variable has a unique parameterization in the decoder, making different latent variables less likely to be processed in the same way.
Unsurprisingly, as you split the latent space into fewer pieces like in SAE-6 or SAE-4, each segment gets more entangled, decreasing the overall score.

\begin{figure*}
  \centering
  \begin{subfigure}{0.45\linewidth}
  \centering
    \includegraphics[width=0.99\linewidth]{figures/celeb-interventions.pdf}

    \caption{\label{fig:celeb-samples} Here we use hybridized chunks of latent vector to show how different aspects of the resulting generated image can be affected using our SAE-16 architecture and hybrid sampling technique. For each row the corresponding quarter of latent dimensions (8/32) are hybridized (see section~\ref{sec:hybrid}) while the remaining 3/4 are fixed. This shows how the SAE architecture is able to order partially disentangled factors of variation from high-level (more nonlinear, like facial expressions and features) to low-level (such as color/lighting) without any additional regularization or supervision.

    }
  \end{subfigure}%
  \hspace{1em}
  \begin{subfigure}{0.45\linewidth}
  \centering
    \includegraphics[width=0.99\linewidth]{figures/celeba_samples_vae.pdf}

    \caption{\label{fig:celeb-gen} Samples generated using hybrid and prior-based sampling using several models trained on CelebA. Note that the hybrid sampling tends to produce relatively high quality samples both for our proposed SAE and AdaAE architectures as well as baselines.}

  \end{subfigure}%
  \caption{CelebA Controllable Generation and Sampling Comparison }
\end{figure*}

\begin{figure}[tb]
  \centering
  \begin{subfigure}[b]{0.25\linewidth}
    \centering
    \includegraphics[width=0.9\linewidth]{figures/3ds_S12_traversals17.pdf}
    \caption{\label{fig:trav-s12}SAE-12}
  \end{subfigure}%
  \begin{subfigure}[b]{0.25\linewidth}
    \centering
    \includegraphics[width=0.9\linewidth]{figures/3ds_S3_traversals17.pdf}
    \caption{\label{fig:trav-s3}SAE-3}
  \end{subfigure}%
  \begin{subfigure}[b]{0.25\linewidth}
    \centering
    \includegraphics[width=0.9\linewidth]{figures/3ds_VAE_traversals17.pdf}
    \caption{\label{fig:trav-vae}VAE}
  \end{subfigure}%
  \begin{subfigure}[b]{0.25\linewidth}
  \centering
    \includegraphics[width=0.9\linewidth]{figures/3ds_S12n_traversals17.pdf}
    \caption{\label{fig:trav-s12n}AdaAE-12}
  \end{subfigure}%
  \caption{\label{fig:trav} Latent traversals of several models trained on 3D-Shapes, in the original order. Note the ordering of the information in the structural decoder models (SAE-12 and SAE-3) where higher level, nonlinear features (like shape and orientation) are encoded in the first few dimensions, which are located deeper in the network.
  }
\end{figure}

\begin{figure}[ht]
\begin{minipage}{0.6\linewidth}
\centering
\begin{tabular}{ccccccccc}
\hline
 Model             & DCI-D       & IRS         & MIG                 & ModExp             \\
\hline
 SAE-12      & {\bf 0.999} & {\bf 0.855} & 0.586              & 0.968        \\
 SAE-6             & 0.815       & 0.712       & 0.133              & {\bf 0.969}       \\
 SAE-4            & 0.693       & 0.560       & 0.218              & 0.918             \\
 \hline
 VLAE-12         & 0.829       & 0.676       & {\bf 0.662}  & 0.929             \\
 VLAE-6          & 0.785       & 0.689       & 0.326              & 0.929              \\
 VLAE-4           & 0.690       & 0.544       & 0.282              & 0.900              \\
 \hline
 AdaAE-12         & 0.272       & 0.477       & 0.046              & 0.862              \\
 AE               & 0.326       & 0.610       & 0.093             & 0.880              \\
 VAE               & 0.441       & 0.712       & 0.252              & 0.904              \\
 $\beta$VAE        & 0.196       & 0.640       & 0.107              & 0.834              \\
 WAE               & 0.205       & 0.640       & 0.057             & 0.958              \\
\hline
\end{tabular}
\end{minipage}
\begin{minipage}{0.4\linewidth}
\centering
\includegraphics[width=\textwidth]{figures/seeds-metrics-main-est-narrow.pdf}
\end{minipage}

  \caption{\label{fig:metrics} Disentanglement scores for 3D-Shapes. The DCI-D metric corresponds to the DCI-disentanglement score~\cite{eastwood2018framework}, IRS is the Interventional Robustness Score~\cite{suter2018interventional}, MIG is the Mutual Information Gap~\cite{chen2018isolating}, and ModExp refers to the Modularity Explicitness score~\cite{ridgeway2018learning} (for all these metrics higher is better). The figure on the right shows how the scores vary across five models with different random seeds as "x"s (and the resulting mean and standard deviation as lines). Note that not only does the SAE architecture consistently achieve high disentanglement scores, but its performance is also less sensitive to the random seed.}
\end{figure}

For a real world demonstration of how well SAE models are able to order information in the latent space, figure~\ref{fig:celeb-samples} shows generated CelebA samples when varying only a quarter of the latent vector at a time. The labels on the left are empirically chosen by the authors to describe roughly what kind of semantic information that quarter of the latent space appears to contain. Although the the inductive biases are not strong enough to fully disentangle the factors of variation into individual latent dimensions. The hierarchical representation encourages a diffuse kind of disentanglement where information pertaining to higher-level features tend to be encoded in the first few dimensions while lower level factors of variation show up towards the last few dimensions.

SAE models achieve this structured disentanglement using the Str-Tfm layers as opposed to the standard Ada-In layers. Each Str-Tfm layer only has access to a small segment of the latent space, and that segment is not directly seen by any other part of the decoder. It thus has to learn to use the information in the corresponding segment to transform the pixels in the output image. In contrast, the Ada-In layers used by the AdaAE allows information from anywhere in the latent vector to leak into any part of the decoder. Consequently, the AdaAE does not disentangle the representation at all (although it achieves impressive results for reconstruction nonetheless).

\subsection{Extrapolation}

Perhaps unsurprisingly, none of models were particularly adept at zero-shot extrapolation to observations that were not in the initial training data. As seen in figure~\ref{fig:extrapolation} on the right comparing the first and second columns, without any update, the reconstructed images filter out the novel information in the sample (in this case, ball shape), and instead reconstruct a similar sample seen during training (a cylinder).

If only the encoder is updated on some observations with the additional shape while the decoder is frozen, then the reconstruction performance increases somewhat, but some deformations and artifacts become visible in the reconstruction. This suggests the frozen decoder struggles to adequately extrapolate, even when the encoder extends the representation to include the ball.

In contrast, when only the decoder is updated, the reconstructions qualitatively look much more similar to the original observations. This also suggests that although the encoder can be expected to generally lose out any information it has not been trained to encode into the latent space due to the bottleneck, as long as the changes to the observations are mild, the representation may still extrapolate provided the decoder can learn to reconstruct any novel features.

\begin{figure}[h!]
\begin{minipage}{0.6\linewidth}
\centering
\begin{tabular}{l|llll}
\hline
 Model   &   Neither &   Encoder &   Decoder &   Both \\
\hline
 SAE-12   &       13.21 &       7.7  &       0.42 &        {\bf 0.34} \\
 LVAE-12  &       18.37 &       7.69 &       1.55 &        0.62 \\
 VAE    &       12.97 &       8.78 &       0.44 &        0.46 \\
 $\beta$VAE    &       15.49 &       8.31 &       1.35 &        0.52 \\
 AE      &       11.81 &       {\bf 7.31} &       0.38 &        0.35 \\
 WAE     &       {\bf 11.68} &       7.87 &       {\bf 0.37} &        0.35 \\
\hline
\end{tabular}

\end{minipage}
\begin{minipage}{0.4\linewidth}
\vspace{-3mm}
\centering

\includegraphics[width=\textwidth]{figures/sh_extrap_example.pdf}
\end{minipage}

  \caption{\label{fig:extrapolation} The table shows the average reconstruction error (MSE x1000) on novel observations (example shown on the right) after updating either the encoder, decoder, both, or neither in the extrapolation setting (see section~\ref{sec:extrapolation}). Note that all models perform significantly better when updating the decoder than the encoder, and reach a reconstruction quality that is almost indistinguishable from the model when updating both the encoder and decoder. Furthermore, note that the SAE-12 generally outperforms the all variational baselines, suggesting the aggressive regularization of VAEs makes updating the representation more difficult.}
\end{figure}

\section{Conclusion}
\label{sec:conclusion}

While VAEs provide a principled approach to generative modeling with autoencoders, in practice, the regularization sharpens the information bottleneck by penalizing any dependence of the posterior on the observation~\cite{hoffman2016elbo}, resulting in a trade-off between reconstruction quality and matching the prior. While this can encourage a more compact and even disentangled representation, it also tends to result in blurry generated samples and relatively poor reconstructions.

This motivated us to look for an alternate sampling method that does not require aggressive regularization as VAEs. Our hybrid sampling technique relies on independence between latent variables instead of expecting the learned posterior to match a (relatively unstructured) prior. This effectively unifies the goals of achieving a disentangled and samplable representation.

To that end, we propose the structural autoencoder architecture inspired by structural causal models, which orders information in the latent space, while also, as shown by our experiments, encourages independence. Notably, it does so without additional loss terms or regularization. The SAE architecture produces high quality reconstructions and generated samples, improving extrapolation, as well as achieving a significant degree of disentanglement across a variety of datasets.

The results obtained thus suggest that in order to enforce a desired structure on the representations, implicit architectural bias might be a valid alternative to explicit assumptions on the latent space distribution, whose accuracy is often questionable outside of the synthetic data setting.

While it is encouraging how far one can get with a suitable architectural bias, future work should assay whether the learned models can be structured further by more explicit forms of causal training. For instance, we could explicitly encourage, during training, that hybridization produce realistic samples, or that across domain shifts or actions/interventions external to the network, only a sparse set of the latent factors change (cf.\ the Sparse Mechanism Shift Hypothesis, \cite{scholkopf2021toward}).

\section*{Acknowledgements}

This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039B, and by the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Felix Leeb.

\bibliographystyle{unsrtnat}