\section{Introduction}

\iffalse

In this paper, we study the problem of compressing an image into a series of abstract compact interpretable, conception-oriented high-level representations from visual data.
Different from traditional compression methods, the proposed conceptual compression concentrates on highly compact visual information instead of simply reducing signal-level data redundancy.
The concept of conceptual compression was first introduced in~\cite{gregor2016towards}, the progressively detailed representations are less interpretable, analyzable or editable, which limits its capability for downstream applications such as image processing and machine vision tasks.

Intuitively, different global conceptual aspects of visual components (\textit{e.g.}, structure, texture, semantics) compose the meaningful images.
However, the underlying implication of the conceptual representation remains ambiguous, which lacks feasibility in explicitly preserving concerned information.

\fi

The human visual system (HVS)~\cite{kruger2012deep} perceives visual contents by processing and integrating manifold information into abstract high-level concepts (\textit{e.g.}, structure, texture, semantics), which form the basis for subsequent cognitive process~\cite{zhang2020connecting}.
From the perspective of machine vision, high-level visual concepts also play a more important role in the practical applications than signal-level pixels.
Existing compression methods, including traditional block-based (\textit{e.g.},~JPEG~\cite{pennebaker1992jpeg} and HEVC~\cite{sullivan2012overview}) and deep learning based methods~\cite{balle2019end,minnen2018joint}, mainly focus on the modeling and removal of signal-level redundancy, while the potential and versatility of accomplishing compression task through decomposing visual data into compact conceptual components still lack further exploration.
Following the insight of Marr~\cite{marr1982computational} and Guo~\textit{et al.}~\cite{guo2007primal}, visual objects usually appear as structures and textures.
As two primal visual components, structure and texture not only play a deterministic role in visual content synthesis, but also are critical visual feature descriptors in various analysis-based tasks.
In particular, due to the lack of explicit modeling for structure, existing compression methods often introduce severe distortions in decoded images under low bit-rate scenarios, such as ringing/blocking artifacts and blurring edges, which not only degrade human perception, but also hamper the performance of visual analysis tasks.
Moreover, the encoded bitstreams are less correlated to visual concepts, resulting in the difficulty of utilizing the encoded information for subsequent manipulation tasks, such as image synthesis, shape modification and content recreation.

Deep generative models, such as variational auto-encoders (VAE)~\cite{kingma2014auto} and generative adversarial networks (GANs)~\cite{goodfellow2014generative}, have offered a new approach for conceptualizing images with compact latent representations, where the decoding process is implemented in a generative fashion.
However, existing researches on conceptual compression~\cite{gregor2016towards}\cite{santurkar2018generative} often attempt to capture image contents with a single latent vector, with different conceptual components entangled together. In consequence, the encoded conceptual representation is less interpretable and editable, limiting
its potential towards downstream image processing and machine vision tasks, such as target-guided content manipulation~\cite{wang2018high}. Apparently, it is desirable to develop visual components disentangled representations for a clearer understanding and more flexible control of image contents.

In this work, we propose a novel conceptual compression framework. In this framework, images are encoded into compact structure and texture representations and decoded in a deep synthesis fashion to achieve better visual reconstruction quality, flexible content manipulation and potential support for various analysis tasks.
More specifically, we propose to process images by a dual-layered model consisting of two complementary visual features: 1) structure layer represented by edges since edges depict key structure information of images, and 2) texture layer which represents non-structural content including texture, color, luminance.
As shown in Fig.~\ref{fig:framework}, at the encoder side, the structural maps are extracted first and then compressed to bitstream.
Without complex texture, structural maps are more sparse and compression-friendly.
Meanwhile, the highly compact deep texture representations are extracted from input images with the variational auto-encoder in the form of low-dimensional latent variables and then scaled, quantized and entropy coded, leading to a bits-saving stream.
At the decoder side, a hierachical fusion GAN (HF-GAN) is proposed to integrate texture layer and structure layer to synthesize images after reconstructing texture representations and structural maps.
Apart from the image compression task, the texture pattern and synthesis paradigm are well jointly learned due to the fact deep generative models tend to capture unique statistical features.
Thus, compared to encoding image into both a raw data layer and a specific feature layer separately~\cite{akbari2019dsslic,ma2018joint},
the proposed framework realizes the unification of visual features and basis data identity in the cross-modality sense, satisfying the demand of machine and human vision with one unified conceptual bitstream.

Our methods are evaluated upon diverse images of fashion items~\cite{yu2014fine,zhu2016generative}, faces~\cite{karras2018progressive} and natural scenes, achieving significant perceptual gain against traditional block-based and deep learning-based end-to-end image compression frameworks.
Furthermore, the versatility of the proposed conceptual coding scheme are verified upon a wide range of image processing tasks, including content manipulation, texture synthesis and face detection.
\begin{figure*}[t]
    \centering
    \vspace{-6mm}
    \includegraphics[width=1.0\linewidth]{./Image/Framework.pdf}
    \vspace{-6mm}
    \caption{\textbf{Overview of proposed conceptual compression framework.} The framework consists of a structure layer and a texture layer. On the encoder side, structural maps and texture representations are extracted from input images using edge detector and texture extractor (VAE) respectively. Both two layers are compressed into bitstreams and reconstructed with specific codecs individually. A proposed generator integrates structure and texture to synthesize target images on the decoder side.}
    \label{fig:framework}
    \vspace{-6mm}
\end{figure*}

Our contributions can be summarized as follows:
\begin{itemize}
    \item We propose a novel compression framework which encodes by abstracting visual data into compact structure and texture representations and decodes by deep synthesis processing to produce cross-modal unification of visual features and basis data.

    \item We propose to realize conceptual compression via extracting sparse structural maps and deep texture representations, and an HF-GAN is proposed as decoder to reconstruct images of high visual quality from texture and structure.

    \item The superiority of the proposed framework in image compression and vision tasks is justified and thoroughly analyzed via extensive experiments.

\end{itemize}

The rest of the paper is organized as follows. The related work and technologies are introduced in Section \ref{related work}. The proposed framework and networks are presented in Section \ref{proposed method}. Section \ref{experients} shows the detailed experimental results and analyses, and Section~\ref{conclusion} concludes the paper and discuss directions for future research.

\section{Related works}
\label{related work}
\subsection{Traditional Image Compression}
Traditional image compression technologies have played a fundamental role in visual communication and image processing, bringing up a series of crafted image codecs, such as JPEG~\cite{pennebaker1992jpeg}, JPEG2000~\cite{rabbani2002jpeg2000} and H.265/HEVC-based BPG~\cite{bellard2015bpg}.
JPEG is the most popular and widely used compression standard, which integrates well-known technologies including block-partition, discrete cosine transform (DCT), quantization  and entropy coding.
JPEG2000~\cite{rabbani2002jpeg2000} applies discrete wavelet transform (DWT) technology instead of DCT, achieving higher compression ratio and allowing various editing or processing applications.
After decades of development, a series of block-based hybrid prediction/transform coding standards (\textit{e.g.}, MPEG-4 AVC/H.264~\cite{wiegand2003overview}, AVS~\cite{gao2014overview} and HEVC~\cite{sullivan2012overview}), are built to significantly improve the coding efficiency by reducing pixel-level redundancy.
Different from JPEG, HEVC utilizes more intra prediction modes from neighboring reconstructed blocks in spatial domain to remove redundancy.
\begin{figure*}[t]
    \centering
    \vspace{-6mm}
    \includegraphics[width=0.95\linewidth]{./Image/Generator.pdf}
    \vspace{-6mm}
    \caption{\textbf{Architecture of hierachical fusion generator networks.} Generator consists of fully connected layers (FC), Residual blocks (Resblock), RGB transformation module (To RGB), upsampling and cumulatively summing module. Structural maps are resized and concatenated to feature maps as input for corresponding Resblock.}
    \label{fig:networks}
    \vspace{-6mm}
\end{figure*}
\subsection{Deep Learning-based Image Compression}
Recent years have witnessed a surge of interest in learning-based image compression, which benefits from large-scale data, effective network architectures, end-to-end optimization, and other advanced techniques such as generative models and unsupervised learning.
Typical deep image compression systems usually adopt an end-to-end framework~\cite{ma2019image} to model the pixel distribution and jointly optimize rate-distortion performance with reconstruction tasks, outperforming popular block-based codecs, such as JPEG, JPEG2000 and HEVC.
Pixel probability modeling~\cite{salimans2017pixelcnn++} and auto-encoder~\cite{balle2019end,minnen2018joint} are two main approaches in deep learning based image coding schemes, as pointed in~\cite{liu2020deep}.
Meanwhile, multiple deep structures, such as CNN~\cite{balle2019end}, RNN~\cite{toderici2015variable} and GAN~\cite{santurkar2018generative,agustsson2019generative}, are utilized to explore efficient compression architectures.
Despite the constant improvement of compression performance, most deep learning-based end-to-end compression algorithms mainly aim to model signal-level correlations, while the visual concepts are less explored.

In addition to rate-distortion optimization, subsequent analysis tasks, such as image retrieval~\cite{zhang2017deep} and semantic analysis~\cite{luo2018deepsic,torfason2018towards}, have also been incorporated into learning-based compression frameworks to encourage analysis-friendly signal representation.
However, extra task-related networks are adopted in their frameworks for joint image analysis and compression, potentially limiting the application to specific tasks.
\subsection{Conceptual Compression}
Conceptual compression~\cite{gregor2016towards, chang2019layered} aims to encode images into compact, high-level interpretable representations for reconstruction, allowing a more efficient and analysis-friendly compression architecture.
Gregor \textit{et al.}~\cite{gregor2016towards} introduce convolutional Deep Recurrent Attentive Writer (DRAW), which extends VAE by using RNNs as encoder and decoder, to transform an image into a series of increasingly detailed representations.
However, the models in~\cite{gregor2016towards} are only verified on datasets of small resolutions.
Hu \textit{et al.}~\cite{hu2020towards} compress images into compact structure and color representations with generative models, where the representations are characterized with edge maps and reference sample pixels near edges specifically.
Despite that the structure representation can be well analyzed, the support for vision tasks, such as image synthesis and modification, are limited by pixel-level color representations.
The semantic guidance is used to assist in reconstructing images from compact compressed visual data in~\cite{agustsson2019generative,akbari2019dsslic}, where the main data stream is still signal-oriented though.

\section{Conceptual image compression framework}
\label{proposed method}
Our goal is to explore the potential of performing compression via decomposing visual data into compact visual components and develop a set of feasible examplar scheme.
In particular, we propose to compress images by extracting the compact representations of two primal complementary visual features, structure and texture.
In the proposed framework, the images ${I}$ are represented with compact texture representations $I^t$ and sparse structural maps $I^s$,~\textit{i.e.}, $I \to I^s + I^t$.
As shown in Fig.~\ref{fig:framework}, images are decomposed into texture layer and structure layer and compressed separately on the encoder side.
We obtain the sparse structure maps $I^s$ with the edges extractor $\mathit{Enc^s}$ (Section~\ref{structure layer compression}) and low-dimensional texture representations $I^t$ with variational texture auto-encoder $\mathit{{Enc}^t}$ (Section~\ref{texture layer compression}), \textit{i.e.}, $I^s=Enc^s(I)$, $I^t=Enc^t(I)$.
Both texture representations and sparse structural maps are further compressed to bitstreams for transmission using existing standard codec.
On the decoder side, a hierarchical fusion generator $Gen$ is designed to reconstruct the high quality images $\hat{I}$ from decoded components, texture $\hat{I}^t$ and structure $\hat{I}^s$,~\textit{i.e.}, $\hat{I}=Gen(\hat{I}^t,\hat{I}^s)$ (Section~\ref{image synthesis}).
To achieve reconstruction of high visual quality and fidelity, we take advantage of both adversarial training and multi-perspective distortion measurement to supervise the reconstruction process.
In particular, we also propose a latent regression loss to better learn the paradigm of texture extraction and synthesis (Section~\ref{loss objectives}).
\subsection{Structure Layer Compression}
\label{structure layer compression}
Considering that edges are one of the most sparse and abstract image representations and can depict the key structure information of images, edge maps are extracted as the structure layer representation via widely used edge detection methods, such as holistically-nested edge detection (HED)~\cite{xie2015holistically} and Canny edge detection~\cite{ding2001canny}.
On account of the sparsity and binarization of structural maps, we employ the Lanczos downsampling algorithm to further reduce data volume of structure layer with 4-scale.
Furthermore, screen content coding~\cite{6783726} is adopted to compress the structure maps into bitstreams since this framework has strong capability of compressing images with abundant sharp edges.

At the decoder side, the process is reversed to recover the structural maps before synthesizing images.
To regain the structural maps of original resolution, we upsample the decoded low resolution structure maps using one of the state-of-the-art super-resolution methods, deep back-projection networks (DBPN)~\cite{haris2018deep}.
The DBPN model is optimized with the mean square error (MSE) in~\cite{haris2018deep} which is not sensitive to the fluctuation of sparse binary data, leading to the significant distortion in reconstructed edges.
To improve the restoration performance on the datasets which are characterized by sparse and binary edges, the binary cross entropy (BCE) is employed to replace the MSE loss for the super resolution model training,
\begin{equation}
\begin{split}
\mathcal{L}_{BCE}=&-\sum_u \sum_v [I_{(u,v)}^s\log (\hat{I}^s_{(u,v)})\\&+(1-I_{(u,v)}^s)\log(1-\hat{I}^s_{(u,v)})],
\end{split}
\end{equation}
where $I_{(u,v)}^s$ represents the true pixel value of structural map $I^s$ at position $(u,v)$, while $\hat{I}^s_{(u,v)}$ represents the predicted value of upsampled structural map $\hat{I}^s$ at position $(u,v)$.
\subsection{Texture Layer Compression}
\label{texture layer compression}
As shown in Fig.~\ref{fig:framework}, the compact texture representations are extracted with a neural network-based encoder and integrated with the reconstructed structural maps to synthesize target images by an elaborate generator.
The encoder $Enc^t$ and generator $Gen$ are jointly trained in an end-to-end manner.
In this section, we will introduce the process of texture extraction and compression.

The image texture extractor is designed based on variational auto-encoder for conceptual level information extraction~\cite{gregor2016towards}.
In particular, the encoder $Enc^t$ is constructed with several residual blocks~\cite{he2016deep} and convolutional layers to model the input image $I$ into a posterior multivariate Gaussian distribution $q_{\phi}(I^t|I)$.
Then the representation $I^t$ is generated by sampling from the posterior distribution $q_{\phi}(I^t|I)$ using re-parameterization~\cite{kingma2014auto} method, \textit{i.e.}, $I^t \sim q_{\phi}(I^t|I)$.

Subsequently, the extracted texture discrete representation $I^t$ is further compressed through scalar quantization and entropy coding. We follow the quantization method defined in HEVC~\cite{sullivan2012overview}, where the quantization step $Q_{step}$ is determined by the quantization parameter ($QP$) and we take the factor $s=2^{10}$ empirically:
\begin{equation}
    Q_{step}=\frac{2^\frac{QP-4}{6}}{s}=2^{\frac{QP-4}{6}-10}.
    \label{quantization}
\end{equation}
The quantization is performed by,
\begin{equation}
    I^{qt}=floor\left(\frac{I^t}{Q_{step}}\right),
    \label{floor}
\end{equation}
where representation $I^t$ is quantized to $I^{qt}$ and $floor(\cdot)$ represents the truncation operation for designated binary digit precision.
Finally the quantized texture representations are encoded with Arithmetic codec~\cite{witten1987arithmetic} for transmission.
The decoded texture representations for image synthesis can be denoted by $\hat{I}^t$, where $\hat{I}^t=I^{qt}*Q_{step}$.
\begin{figure}[t]
    \centering
    \vspace{-4mm}
    \includegraphics[width=1.0\linewidth]{./Image/Resblock.pdf}
    \vspace{-8mm}
    \caption{\textbf{Basic fusion residual block in generator.} Resblock input ($h \times w \times (n+1)$) consists of feature maps ($h \times w \times n$) and a structural map ($h \times w \times 1$). Texture representation ($d \times 1$) is transformed into two groups of affined parameters: $\alpha$ ($(n+1)\times 1$) and $\beta$ ($(n+1) \times 1$). Structural maps and texture representations are fused through parameterized normalization in each subblock.}
    \label{fig:resblock}
    \vspace{-5mm}
\end{figure}

\subsection{Image Synthesis}
\label{image synthesis}
Motivated by the strong capability of the instance normalization in style transfer~\cite{karras2020analyzing}, we adopt the adaptive instance normalization~\cite{huang2017arbitrary} to integrate texture layer and structure layer.
Specifically, a group of modulation parameters, including mean and variance, are learned from texture representations.
The feature statistics of the feature maps are transferred with learned modulation parameters during adaptive affine transformations, where feature maps contain both structure information and previously generated texture information in forward propagation as shown in Fig.~\ref{fig:resblock}.
The particular channel-wise modulation parameters are learned corresponding to each intermediate feature map.
In this manner, the structure and texture are fused and integrated gradually through convolutional operation and affine transformation.
Additionally, inspired by impressive generation capability of progressive growing GAN~\cite{karras2018progressive}, we devise a generator which progressively increases the resolution of synthesized feature maps as shown in Fig.~\ref{fig:networks}.
The generator consists of a pile of residual blocks as basic units and takes advantage of skip connections and hierarchical fusion.
Each residual block includes three fully convolutional layers followed by adaptive instance normalization operation. The structural maps are concatenated to the feature maps as new input for each residual block.
Finally, the target images are obtained by upsampling and summing the contributions of RGB outputs corresponding to different resolutions as~\cite{karras2020analyzing}.
The formulation of the hierarchical fusion process is presented as follows.

Given the input image $I \subseteq \mathbb{R}^{H\times W \times 3}$, we can obtain the structure map $I^s \subseteq \mathbb{R}^{H\times W \times 3}$ and the texture representation $I^t \subseteq \mathbb{R}^{d \times 1} $, where $H,W$ represent the size of input image and $d$ denotes the dimension of texture representation.
Let the initial block of the generator function $\mathcal{G}_0$ be defined as $\mathcal{G}_0:I^s \mapsto A_0$ such that $A_0 \subseteq \mathbb{R}^{2\times 2 \times c_0}$ and $c_0$ denotes the number of feature channels.
We define the intermediate output of the $i$-th residual block as $A_i$, and $i \in \mathbb{N}$. The structure map $I^s$ is resized to $I^s_i$ which has the same size as $A_i$. Thus, the input of each residual block is defined as:
\begin{equation}
    \bar{A}_i=[A_i;I^s_i],
\end{equation}
where [;] is a channel-wise concatenation operation. Let $\mathcal{G}_i$ be a generic function which acts as the basic generator block: $\mathcal{G}_i: \bar{A}_{i-1} \mapsto A_i$, where $A_i \in \mathbb{R}^{2^{i+1}\times 2^{i+1}\times c_i}$ and $c_i$ is the number of channels in the intermediate activations of the $i$-th generator block. Then $A_k$ can be obtained by the general formula composed with a sequence of $\mathcal{G}$ functions, where $k \in \mathbb{N}$:
\begin{equation}
\begin{aligned}
    A_k = & \mathcal{G}_k(\cdot,I^s_k) \circ \mathcal{G}_{k-1}(\cdot,I^s_{k-1}) \circ \cdots \mathcal{G}_i(\cdot,I^s_i)\\
    & \circ \cdots \mathcal{G}_1(\cdot,I^s_1) \circ \mathcal{G}_0(I^s).
\end{aligned}
\end{equation}

We employ skip connections and multi-scale fusion to improve reconstruction quality of the generator. $up^{(2)}$ is defined as the upsampling function by 2-scale and $conv^{(3\times 3)}$ simply serves as a $3 \times3 $ convolution function which converts the output feature map $A_i$ of each block into RGB images: $conv^{(3\times 3)}_i:A_i \mapsto B_i$, where $B_i\subseteq \mathbb{R}^{2^{i+1}\times 2^{i+1}\times 3}$. Hence, the RGB output of initial convolutional layer $x_0$ is:
\begin{equation}
    x_0= conv^{(3\times 3)}_0(A_0)=conv^{(3\times 3)}_0(\mathcal{G}_0(I^s)).
\end{equation}
The RGB output of the $i$-th intermediate block can be computed with the recurrence formula as following:
\begin{equation}
    x_i=conv^{(3\times 3)}_i(A_i)+up^{(2)}(x_{i-1}), i\geq1.
\end{equation}
Therefore, the final synthesized image from the proposed generator consisting of $7$ blocks is $x_7$.

\begin{figure*}[t]
    \centering
    \vspace{-11mm}

    \includegraphics[width=0.9\linewidth]{./Image/Loss_v2.pdf}
    \vspace{-4mm}
    \caption{\textbf{Loss functions of end-to-end VAE-HFGAN.} During training, various losses are adopted: for adversarial training, a discriminator is adopted to discriminate real labels and fake labels; to reduce the distortion of reconstructed images, we apply self reconstruction $L_1$ loss, structure similarity loss $SSIM$ and perceptual $L_1$ loss which measure the distance between features encoded from \textit{VGG} networks; to align texture representations with a prior Gaussian distribution, a $KL$ loss is incorporated; to enforce the connection between paired texture representations and images, the latent regression $L_1$ loss is employed to measure the difference between texture representations extracted from original image and reconstructed image. }
    \label{fig:loss}
    \vspace{-5mm}
\end{figure*}
In the basic residual block of the generator shown in Fig.~\ref{fig:resblock}, there are three convolutional layers whose scale and bias are modulated by three groups of parameters learned from texture representations by three fully connected layers respectively.
The feature maps at the site of $j$-th convolutional layer of $i$-th basic block is denoted by $A_{ij}$. Receieving the texture representation $I^t$, a fully connected function $f_{ij}:I^t \mapsto {\alpha_{ij},\beta_{ij}}$ first maps $I^t$ into affine parameters, where $\alpha_{ij},\beta_{ij} \subseteq \mathbb{R}^{1\times 1 \times c_i}$. Thus, we adaptively normalize the previous activations $A_{ij}$ with the learnable texture affine parameters:
\begin{equation}
    AdaIN(A_{ij},I^t)=\alpha_{ij} \frac{A_{ij}-\mu(A_{ij})}{\sigma(A_{ij})}+\beta_{ij},
\end{equation}
where the normalized feature maps are scaled with $\alpha_{ij}$, and shifted with $\beta_{ij}$ in the channel-wise manner, and $\mu(A_{ij})$ and $\sigma(A_{ij})$ are the means and standard deviations of the activations in $A_{ij}$.
By transferring the feature statistics from texture representations and concatenating the structural maps with intermediate generated content, structure and texture are effectively integrated to reconstruct images of which texture are progressively enriched.
Moreover, during the training of reconstruction task, the mapping and synthesis paradigm between deep latent space and spatially-aware texture content are jointly well learned and stored as deep prior in the designed generative model, allowing the flexible examplar-based image content manipulation in the compressed domain.

\subsection{Loss Objectives}
\label{loss objectives}
The proposed texture encoder and the image generator are jointly trained in an end-to-end manner with a multi-scale discriminator.
The image compression and reconstruction tasks are mainly optimized on three type of losses as shown in Fig.~\ref{fig:loss}: \textit{\textbf{reconstruction loss}} which aims to improve reconstruction visual quality and fidelity, \textit{\textbf{prior divergence}} which provides a prior distribution instruction for deep texture representation, and the proposed \textit{\textbf{latent regression loss}} which constraints the mapping between deep latent space and synthsis texture content.

Compared to typical learned lossy compression methods~\cite{balle2019end,minnen2018joint} which only optimize image reconstruction quality with pixel-wise similarity metric, we introduce diverse distortion metrics to supervise reconstruction task from low level pixel-wise loss to high level perceptual loss and adversarial loss. The~\textbf{reconstruction loss} is introduced as follows:
\begin{itemize}
    \item Self-reconstruction loss: the pixel-wise loss $\mathcal{L}{^{rec}_{1}}$ is imposed to force the visual components to complete reconstruct the original images:
    \begin{equation}
    \mathcal{L}{^{rec}_{1}} = \mathbb{E}_{I\sim p(I), {I^t} \sim p(I^t)}\left\|{I-Gen(\hat{I}^t,\hat{I}^s))} \right \|_1.
    \end{equation}
    \item Structure similarity loss: the SSIM~\cite{wang2004image} loss is incorporated to supervise the optimization process of improving structural fidelity:
    \begin{equation}
    \mathcal{L}_{SSIM}=\mathbf{SSIM}(I,Gen(\hat{I}^t, \hat{I}^s)).
    \end{equation}
    \item Perceptual loss: we use the perceptual loss $\mathcal{L}_{vgg}$ in~\cite{johnson2016perceptual} to encourage the perceptual fidelity through deep feature matching from pre-trained VGG-16 netoworks~\cite{simonyan2014very}.

    \item Adversarial loss: we apply the discriminator $Dis$ and follow the variant of conditional adversarial training scheme~\cite{mao2017least} to encourage the visual realism of reconstructed images:
    \begin{multline}
         \mathcal{L}{^{GAN}_{G}} = \frac{1}{2}\mathbb{E}_{I\sim p(I)}||1-Dis(I,I^s)||_2 \\ +
         \frac{1}{2}\mathbb{E}_{I\sim p(I), I^t \sim p(I^t))}||Dis(Gen(\hat{I}^t, \hat{I}^{s}),I^s)||_2,
    \end{multline}
    \begin{multline}
        \mathcal{L}^{GAN}_{D} = -\mathbb{E}_{I\sim p(I), I^t \sim p(I^t))} \\ || Dis(Gen(\hat{I}^t, \hat{I}^{s}),I^s)||_2.
    \end{multline}
\end{itemize}

Additionally, to obtain meaningful texture representations by stochastic sampling and benefit entropy coding process, we introduce the \textbf{prior divergence} to enforce the distribution of extracted texture representation to be close to the prior Gaussian distribution,~\textit{i.e.} $p_{\theta}(I^t)\sim\mathcal{N}(0,\mathbf{I})$ and $\mathbf{I}$ denotes the identity matrix:
\begin{equation}
\mathcal{L}_{KL} = \mathbb{E}_{I\sim p(I)}[\mathcal{D}_{KL}(q_{\phi}(I^t|I)\|p_{\theta}(I^t))].
\end{equation}

Furthermore, we propose a \textbf{latent regression loss} to further constrain the bidirectional mapping between the learned texture representation ${I^t}$ and the synthesis texture content $\hat{I}$.
More specifically, the texture encoder $Enc^t$ is utilized to analyze and extract the texture representation $\hat{I}^t_{est}$ of the generated image $\hat{I}$.
Given the fact that the reconstructed images $\hat{I}$ are trained to maintain texture fidelity, the texture representations which are extracted from original images and reconstructed images should be encouraged to be consistent,~\textit{i.e.}~$I^t = Enc^t(I) \approx \hat{I}^t_{est} = Enc^t(\hat{I})$.
Thus, the latent regression loss $\mathcal{L}{^{latent}_1}$ is proposed to minimize the $L_1$ distance between two texture representations as follows,
\begin{equation}
    \mathcal{L}{^{latent}_1}=E{_{I^t \sim p(I^t),\hat{I}^t_{est} \sim p({\hat{I}^t_{est})}}} \|{I^t-\hat{I}^t_{est}} \|_1.
\label{con:latent}
\end{equation}
The unique association between specific representation and corresponding texture are further reinforced by the latent regression loss, which not only helps learn a continuous latent space for texture expression in the data-driven manner, but also benefits the learning of synthesis paradigm.

Above all, the final training objective is shown as following:
\begin{multline}
\label{loss_G}

    \mathcal{L}_{G,D,E}= \lambda_{GAN} \mathcal{L}{^{GAN}_{G}}+\lambda_{ rec}\mathcal{L}_{1}^{rec}+
    \lambda_{vgg}\mathcal{L}_{vgg}
    \\+\lambda_{SSIM}\mathcal{L}_{SSIM}+ +\lambda_{KL}\mathcal{L}{_{KL}} +\lambda_{ latent}\mathcal{L}_{1}^{latent},
\end{multline}
where the hyper-parameters $\lambda_{name}$ weights each loss respectively.
\begin{figure*}[t]
    \centering
    \vspace{-6mm}
    \includegraphics[width=0.9\linewidth]{./Image/qualitative_results.pdf}
    \caption{\textbf{Qualitative comparisons with baselines under similar low bit-rate.} The \textbf{\textit{bpp}} of each image is reported under corresponding image. The results show that the proposed method can reconstruct high visual quality images and outperform other methods in regard to visual fidelity and aesthetic sensibility.}
    \label{fig:qualitative}
    \vspace{-6mm}
\end{figure*}
\section{Experiments}
\label{experients}
\subsection{Implementation details}
We implement the proposed model\footnote{For reproducible research, the source codes of our method will be made public when this paper is accepted.} using PyTorch and trained on two NVIDIA Tesla V100 GPUs.
For training, we use the Adam optimizer with exponential decay rates $(\beta_1,\beta_2)=(0.5,0.999)$.
The batch size is set to 16 and the learning rate is set to 0.0002.
The training procedure follows the Least Squares GANs (LSGANs)~\cite{mao2017least}.
We adopt the following hyper-parameters in all experiments for the training: $\lambda_{GAN}=1.0, \lambda_{rec}=10.0, \lambda_{SSIM}=0.25, \lambda_{vgg}=0.2, \lambda_{latent}=1.0, \lambda_{KL}=0.01$.
As for the size of texture representations, we find that the capacity and capability of texture representations are growing as size increases. Thus, factored by texture complexity of specific datasets, the best sizes vary for different datasets.
In our experiments, the dimension of texture representations is empirically set to $d = 64$ across all datasets for comparison.
\subsection{Datasets}
We conduct experiments on several datasets including edges2shoes~\cite{yu2014fine}, edges2handbags~\cite{zhu2016generative}, CelebA-HQ~\cite{karras2018progressive}, and the multiple seasons dataset. All images are resized to $256\times 256$ in the experiments.

\textbf{Edges2shoes and edges2handbags.}
We combine these two datasets for training due to their high content similarity.
It contains $188,392$ paired training images and $400$ images for testing.
The images of edges are utilized for providing structural information.

\textbf{CelebA-HQ.} It includes $30000$ high-quality face images~\cite{karras2018progressive}.
We split $29800$ as training set and $200$ as testing set.
For structural maps, we employ facial landmark detection to obtain contours in the facial region, and the Canny edge detector~\cite{ding2001canny} to obtain structural edges in the background region.

\textbf{Multiple seasons dataset. }
We collect $8000$ images from the Yosemite dataset~\cite{zhu2017unpaired} and the alps seasons dataset~\cite{anoosheh2018combogan}.
The dataset contains four seasons of images, and $400$ images are split for testing.
We use the Canny edge detector incorporated with the Gaussian blur algorithm to acquire the corresponding structural maps.

\begin{figure*}[t]
    \centering

    \includegraphics[width=1.0\linewidth]{./Image/User_study.pdf}
    \vspace{-6mm}
    \caption{ \textbf{Fidelity and aesthetics preference results.} A user study is conducted, in which participants are asked to choose results which better match original images (Fidelity) and have better perceptual quality (Aesthetics) through pairwise comparisons. The numbers in histogram indicate the percentage of preference for corresponding comparison pair. The results show that the proposed method obtains the best preference ratio in each comparison pair for both fidelity and aesthetics.}
    \label{fig:user_study}
    \vspace{-4mm}
\end{figure*}
\begin{table*}[t]
    \centering
    \caption{Quantitative results on edges2shoes$\&$handbags, CelebA-HQ and multiple seasons datasets. $\downarrow$ means lower score is better.}
    \begin{tabular}{lcccccc}
    \toprule
          Datasets & \multicolumn{2}{c}{Edges2shoes$\&$handbags} & \multicolumn{2}{c}{CelebA-HQ} & \multicolumn{2}{c}{Multiple seasons} \\
         \hline
         & LPIPS$\downarrow$ & DISTS$\downarrow$ & LPIPS$\downarrow$ & DISTS$\downarrow$ & LPIPS$\downarrow$ & DISTS$\downarrow$ \\
         \hline
         JPEG & 0.274 & 0.326 & 0.492 & 0.400 & 0.517 & 0.392 \\
         JPEG2000 & 0.309 & 0.265 & 0.289 & 0.241 & 0.532 & 0.331 \\
         BPG & \textbf{0.110} & 0.169 & 0.180 & 0.187 & 0.371 & 0.261 \\
         Minnen \textit{et al.}~\cite{minnen2018joint} & 0.121 & 0.199 & 0.179 & 0.177 & 0.363 & 0.265 \\

         Proposed method & 0.118 & \textbf{0.163} & \textbf{0.164} & \textbf{0.149} & \textbf{0.354} & \textbf{0.215} \\
         \bottomrule
    \end{tabular}
    \label{tab:quatitative_results}
    \vspace{-4mm}
\end{table*}
\subsection{Compression Performance Comparison}
In this subsection, we conduct extensive qualitative and quantitative experiments to compare the compression performance with traditional and learning-based approaches.
To compress structure and texture representation into bitstreams with high efficiency, we adopt different strategies for structure layer and texture layer compression.
In particular, the reference software of screen content coding~\cite{6783726} with $QP = 35$ is employed to compress the structural maps.
Moreover, texture representations are quantized with $QP=51$ (Eq.~\ref{quantization}) and truncated with $16$ bits range (Eq.~\ref{floor}).
Then, the quantized results are encoded with lossless Arithmetic codec~\cite{witten1987arithmetic}.
Note that quantization loss does not cause visible degradation of reconstruction results according to experimental results.

\subsubsection{Baselines}
We compare the proposed method with traditional compression standards and learning-based compression methods on the test datasets.
For traditional compression framework, we choose widely-used compression standards JEPG and JPEG2000, and current state-of-the-art HEVC-based image compression codec BPG for comparison.
For deep learning-based image compression systems adopting end-to-end optimization, the method in Minnen \textit{et al.}~\cite{minnen2018joint} is used for comparison.
The specific settings are detailed as follows:
\begin{itemize}
    \item \textbf{JPEG}: we use JPEG Encoder of Matlab with quality factor $QF = 1$, which obtains maximum compression ratio of JPEG.
    \item \textbf{JPEG2000}: the images are compressed by OpenJPEG platform with quality parameter aligned with the compression ratio of proposed method, ranging from $200$ to $500$.
    \item \textbf{BPG}: we use standard bpg codec\footnote{https://bellard.org/bpg} with $QP$ ranging from $40$ to $51$ for compression ratio alignment.
    \item \textbf{Minnen \textit{et al.}~\cite{minnen2018joint}}: The deep compression network is trained exactly following the procedure in~\cite{minnen2018joint} with hyper prior. We set $\lambda=0.0025$ for obtaining average 0.16 bits per pixel (bpp), and $\lambda=0.005$ for average 0.2 bpp.

\end{itemize}

\subsubsection{Qualitative Evaluations}
For the evaluation of visual reconstruction quality, we mainly focus on the global structural fidelity and aesthetic sensibility.
Making comparisons at the same bitrate is difficult since most compression methods cannot generate a specified bitrate.
As such, we select comparison images with sizes that are as close as possible to our encoding bpp.
We present qualitative comparisons in Fig.~\ref{fig:qualitative}.
Overall, the results demonstrate that our method can produce reconstructed images of high visual quality under extreme low compression bitrate and outperform the baselines in regard to visual fidelity and aesthetic sensibility.

In particular, the decoded images of JPEG and JPEG 2000 show serious structure and color distortion and blocking artifacts under extreme low bit-rate scenarios resulting from the limitation of block-wise processing.
In the mean time, BPG and Minnen \textit{et al.}~\cite{minnen2018joint} provide similar visual quality and blurring texture.
The results of BPG show more distortion and degradation in visual quality than which of ours, \textit{e.g.}, the face skin looks unnatural and partial key edges are missing in the second row.
Meanwhile, the structure and texture are over smoothed and blurred in Minnen \textit{et al.}~\cite{minnen2018joint}, hence losing significant structural details and lessening sense of reality compared to our method, \textit{e.g.}, the teeth in the third row of Fig.~\ref{fig:qualitative}.
Above all, compared to other compression frameworks, our proposed conceptual compression shows significant advantages in synthesizing sharp, realistic images while preserving high visual fidelity.
\begin{figure*}[t]
    \centering
    \vspace{-2mm}
    \includegraphics[width=1.0\linewidth]{./Image/texture_transfer.pdf}
    \caption{\textbf{Texture synthesis results.} With all accessible images serving as a texture source library, the proposed method is able to synthesize different texture by replacing the texture representation with ones encoded from images of desired texture styles, and generate creative and attractive images.}
    \label{fig:texture_transfer}
    \vspace{-4mm}
\end{figure*}
\subsubsection{Quantitative Evaluations}
Due to the ultimate arbiter of lossy compression remaining human evaluation, we choose LPIPS~\cite{zhang2018unreasonable}, DISTS~\cite{ding2020image} and user preference as perceptual quality metric to evaluate the fidelity quantitatively, which highly correlate with human quality judgements instead of only assessing signal fidelity.
Lower score of LPIPS and DISTS indicates higher visual fidelity of reconstructed images.
The average results of LPIPS and DISTS on each dataset from different comparison methods are shown in Table~\ref{tab:quatitative_results}.
It should be noted that the bitrate of comparison images is strictly close or higher than ours.
It can be clearly seen that the proposed method almost excels all other comparison methods on DISTS and LPIPS in all scenes under similar bit-rate (except for one score of LPIPS on edges2shoes and edges2handbags testset).

A user study of pairwise comparison is conducted to evaluate fidelity and aesthetics.
Given image pairs, users are asked to choose the one matching original image better (fidelity) in the first part of the survey, and the one of better visual quality (aesthetic) in the second part.
16 cases from test data of three datasets are selected to show for comparison.
A total of 46 subjects participate in the user study and a total of 2944 selections are tallied.
The preference ratio in comparison between proposed method and each compared method is calculated for evaluation.
User preference results are shown in Fig.~\ref{fig:user_study}.
The proposed conceptual compression method obtains the best preference ratio in each comparison pair for both fidelity and aesthetics.
The quantitative results fully verify the superiority of our method in obtaining better visual quality and maintaining higher structure and texture fidelity.
\subsection{Application in Image Manipulation}
In our conceptual compression framework, images are explicitly disentangled to structure domain and texture domain, and reconstructed through integrating texture and structure with generator.
The structure and texture representations act as both raw frame data and inter-operable and manipulable visual features.
During the data-driven training, a continuous bidirectional mapping between deep latent representations and spatially-aware texture is well learned, allowing generator to synthesize corresponding content from any representation in texture latent space.
Benefited from the disentanglement of visual components and learned synthesis paradigm, the generator is able to progressively render the texture following the instruction of any given structural maps.
Thus, besides high efficiency compression, the proposed framework can be also applied in image manipulation tasks via editing structure and texture representations.

In particular, we can synthesize new texture in the way of changing the texture representations while constraining specific structural maps as image content.
On the other hand, we can also modify the structural maps while maintaining the image texture to satisfy particular needs for image manipulation.
The powerful capability of manipulating images flexibly in compression domain demonstrates the versatility of our conceptual compression framework in supporting both compression tasks and efficient content manipulation in the compressed domain compared to normal compression methods.
\subsubsection{Texture Synthesis}
The results of the texture synthesis results are shown in Fig.~\ref{fig:texture_transfer}.
Attributed to analyzing and compressing images into compact disentangled structure and texture representations, the proposed method is able to process images in two independent layer before decoding.
Due to the unique association between specific representation and global texture of image instance, the implication of texture representation can be confirmed and reflected in the source images and synthesized images.
Thus, with all accessible images serving as a texture source library, the proposed method can synthesize different texture by replacing the texture representation with ones encoded from images of desired texture styles, and construct creative and attractive images.
The texture synthesis results in Fig.~\ref{fig:texture_transfer} show that our model has strong capacity of effectively capturing image global texture distribution, integrating representation layers without relying on encoder, and synthesizing pleasing target images, which suggest a great potential of proposed method in support of joint image compression and vision tasks.
\begin{figure}[t]
    \centering
    \vspace{-2mm}
    \includegraphics[width=1.0\linewidth]{./Image/shape_transfer.pdf}
    \caption{\textbf{Image synthesis results via modifying structural maps.} With the structural maps edited in optional ways, \textit{e.g.}, user interactive edition, image morphing algorithms, the texture layer can be flexibly rendered to fit the layout and shape of updated structural maps.}
    \label{fig:shape_transfer}
    \vspace{-4mm}
\end{figure}
\subsubsection{Structure Modification}
We also demonstrate example results of applying the proposed method to structure modification in Fig.~\ref{fig:shape_transfer}.
Under the layered framework, the shape of images can be modified through editing the structural maps in optional ways, \textit{e.g.}, user interactive edition, image morphing algorithms.
The texture layer can be flexibly rendered fitting the layout and shape of updated structural maps and the results show that original texture can perfectly fit new shapes in synthesized images.
Structure modification is very useful in visual communication and image-based retrieval applications.
Moreover, operating machine vision tasks in the compressed domain is shown to be more efficient than operating it after decoding under low bit-rate scenarios~\cite{ma2018joint}.
Our experimental results also prove the potential advantage of the proposed method in performing high efficiency coding and machine vision tasks jointly.
\subsection{Advantage in Image Analysis}
To further verify the advantage of our framework in image analysis task, we also perform the facial landmark detection task on the decoded images.
Facial landmark detection~\cite{kazemi2014one} is carried out on the original images in CelebA-HQ testset and the detection results are served as ground truth.
For comparison, we perform landmark detection on the decoded images from JPEG and the proposed method respectively.
Then the normalized root mean squared error (NRMSE) between the detection results on compressed images and original images is calculated as quantitative metric.
Fig.~\ref{fig:RMSE} illustrates the cumulative error distribution of our method and JPEG with quality factor $\left \{ 2,3,4 \right \}$, where about $97\%$ of the test images reconstructed by proposed method have minor errors less than $0.4$.
It should be noted that the average bit-rate of our method and JPEG under $QF=1$ are $0.099$ bpp and $0.237$ bpp respectively.
Our method can achieve $58.1\%$ bits saving and $56.5\%$ accuracy improving of landmark detection task compared to JPEG under $QF=1$.
Above all, the face landmark detection results demonstrate robustness and accuracy of the proposed image compression method in image analysis tasks.

In essence, in the proposed implementation scheme, the structural maps already contain key facial edges which act like dense landmarks, allowing straightforward content analysis without decoding.
Moreover, the concrete form of the structure representation can be further adjusted towards specific vision applications, leading to a unified framework for connecting machine vision and human perception through inter-operable conceptual representations.
\begin{figure}[t]
    \centering
    \vspace{-4mm}
    \includegraphics[width=1.0\linewidth]{./Image/RMSE.png}
    \caption{\textbf{Cumulative error distribution of facial landmark detection with JPEG under quality factor ${2,3,4}$ and proposed method.} It can be seen from the figure that the decoded test images from the proposed method ($97\%$), JPEG with quality factor 4 ($88\%$), JPEG with quality factor 3 ($65\%$), JPEG with quality factor 2 ($49\%$) achieve errors of less than 0.4 respectively.}

    \label{fig:RMSE}
    \vspace{-4mm}
\end{figure}
\section{Conclusions and Future Works}
\label{conclusion}
This paper proposes a novel conceptual compression framework for efficient, interpretable and versatile visual data representation, leading to high efficiency compression, better visual reconstruction quality, increased flexibility in content manipulation and potential support for various vision tasks.
In particular, the proposed framework decomposes the image contents into structural and textural layers, and performs compression in two layers respectively.
To reconstruct the original image from the compressed layered features, a hierarchical fusion GAN is proposed to integrate texture layer and structure layer in a disentangled fashion.
Qualitative and qualitative results show that the proposed method can reconstruct high visual quality images with better structural fidelity and aesthetic sensibility.
The advantages of our conceptual compression framework are also verified in compression, content manipulation and  analysis tasks through extensive experiments.

Despite the current achievements, the conceptual compression will further benefit from the enhanced rate-distortion optimization schemes and better layered decomposition of image contents. Furthermore, extending the proposed conceptual image compression framework to video domain is another promising direction, where effective spatio-temporal representation learning plays a crucial part in a wide range of video analysis tasks, such as action recognition, tracking, summarization, and editing.
\iffalse
\begin{itemize}
    \item \textit{Joint Rate-Distortion Optimization.} In this work, the trade-off between reconstruction quality and bitrate isn't explicitly optimized during training. Joint rate-distortion optimization is promising to further broaden the application of the proposed method.
    \item \textit{Separable Conceptual Compression for Video Compression.} Although training deep models on video data is more difficult than image data due to temporal correlation, it is meaning and necessary to extend separable conceptual compression to video compression for various applications.
    \item \textit{Multiple Layered Combination.} This paper implements the structure-texture separable conceptual compression as an example to prove the feasibility of separable conceptual compression. Multiple layered combination can play a significant role in the future study.
\end{itemize}
\fi
\bibliographystyle{IEEEbib}


\section{Introduction}

\IEEEPARstart{V}{isual} inspection using imaging technologies is an essential part of
material quality control.
For example, solar modules are subject to regular inspection by \acl{el}
imaging~\cite{jahn2018review} for material defect detection.
\changed{To inspect every single cell of a solar module, the entire module needs to be located in the \ac{el} image first, to allow for a subsequent segmentation into single cells~(see~\cref{fig:motivation})\chmark{ch001}}. In addition, removal of perspective and lens distortion simplifies further
analysis~\cite{deitsch2018segmentation}.
Modern CMOS sensors allow \ac{el} imaging at high frame rates, which is
beneficial for drone-based in-field inspection.
However, due to practical requirements on the signal-to-noise ratio and the
sensor integration time, their spatial resolution for \ac{el} imaging is limited,
challenging the detection of small material defects.
In this respect, the use of \ac{sr} techniques for software-based
resolution enhancement is promising.

\input{8eec2f53d49941d7aed78e331312b4ec.tex}

\input{b0779e5953294e699eb62d5078e79ab2.tex}

In this work, we propose to capture a sequence of \ac{lr} images using fast CMOS sensors and to subsequently apply \ac{mfsr} in order cell images at higher resolution. Prior to the reconstruction of a \ac{hr} image, the sequence of \ac{lr} images needs to be registered. Traditionally, one of the frames is used as a reference frame that all others are registered to. However, this approach does not compensate for perspective or lens distortion. We propose to integrate the rectification and undistortion steps into the motion estimation procedure of a \ac{mfsr} approach by performing registration w.\,r.\,t\onedot a virtual reference of the module.

The contributions of our work are as follows:
\begin{enumerate}
    \item We fuse rectification and undistortion with \ac{sr} which leads to a uniform spacing of pixels in the module plane, regardless of perspective or lens distortion.
    \item We conduct an extensive user study showing that \ac{mfsr} simplifies manual inspection of solar cells by human experts and that our proposed registration outperforms the state of the art.
    \item We apply automated crack detection and segmentation on the resulting images and show that \ac{mfsr} enhances automated inspection of solar cells.
\end{enumerate}

\section{Related Work}
\label{sec:RelatedWork}

Visual inspection of solar cells is an active research area. In previous studies, it has been shown that, for example, micro cracks result in power loss of the module, especially after simulated aging~\cite{kontges2010quantifying}. Hence, reliable detection of these and other defect types is of particular interest. Only recently, \ac{dl}-based methods are applied to automatically detect defects on solar cells using \ac{el} imaging~\cite{deitsch2019automatic}. Many of these works focus on the detection~\cite{akram2019cnn} and segmentation~\cite{rahman2020defects} of cracks. However, a reliable detection of small cracks is challenging and requires a sufficient image resolution. For this reason, we focus on the automated detection and segmentation of cracks and show that super-resolution is beneficial for this task.

~\\\noindent
\textbf{Observation model for super-resolution.}
Given a sequence of \ac{lr} images captured from a scene of interest, \ac{sr} can be modelled by the observation model:
\begin{equation}\label{eq:observation-model}
	\bm{g}_i = \bm{W}_i \bm{f} + \bm{n}_i\quad \text{for}\quad 1 \leq i \leq N\,,
\end{equation}
where $\bm{f}$ denotes the desired \ac{hr} image, $\bm{g}_i$ is the $i$-th frame of the \ac{lr} sequence, and $\bm{n}_i$ is additive noise. The system matrix $\bm{W}_i$ describes the imaging process including motion between the \ac{hr} and \ac{lr} domain, the \acl{psf} of the optical system, and sampling on the sensor array. \ac{sr} algorithms aim at inverting this model w.\,r.\,t\onedot $\bm{f}$.

~\\\noindent
\textbf{Single-frame super-resolution.}
\changed{Many mainstream methods use single images ($N = 1$) to compute a super-resolved image~\chmark{ch002}}. Here, especially deep neural networks~\cite{dong2014learning, Kim2016, Lim2017,zhang2020deep} and generative adversarial learning~\cite{wang2018esrgan,rakotonirina2020esrganplus} advanced the state of the art in \ac{sfsr}. They greatly benefit from large natural image databases, enabling supervised learning of mappings from \ac{lr} to the \ac{hr} domain. However, without fine-tuning to the target application, their performance can drastically drop. This is particularly severe in imaging applications such as ours, where comprehensive training datasets are still unavailable and image properties like noise or blur models can considerably deviate from those of natural images~\cite{kohler2017benchmarking}. \changed{To overcome this issue, zero-short \ac{sfsr} methods have been developed~\cite{shocher2018zero,soh2020meta}. These methods exploit recurring information within a single image to train a convolutional neural network at test time. Only recently, it has been shown that combining classification and \ac{sfsr} into a single network in terms of multi-task learning helps to improve the performance of \ac{sfsr} and classification at the same time~\cite{schirrmacher2020sr}.} Still, existing \ac{sfsr} methods mostly target to satisfy quality perceptions of humans and might even hallucinate \ac{hr} additional details. This is caused by the fact that they only use the information available in a single image and incorporate prior knowledge to reconstruct \ac{hr} the image. In contrast, visual inspection requires authentic data for quantitative and objective analyses, like solar cell defect detection. This prohibits the simple use of existing pre-trained models.

~\\\noindent
\textbf{Image-based multi-frame super-resolution.}
\ac{mfsr} aims at the inversion of \cref{eq:observation-model} using multiple frames ($N > 1$) assuming sub-pixel inter-frame motion, to fuse them into an \ac{hr} image. This can be done by non-uniform interpolation~\cite{alam2000infrared,batz2016multi}, iterative reconstruction to invert \cref{eq:observation-model}~\changed{\cite{elad1997restoration, Liu2014, kohler2016robust,liu2018robust,honda2019multi,borsoi2020robust}}, or deep learning~\changed{\cite{kappeler2016video,liao2015video,wang2019edvr,isobe2020video,li2020mucan,xu2020deformable,pan2020deep}} to directly predict the \ac{hr} image.

During the imaging process, crucial details are lost in \ac{lr} frames, which is mainly caused by the sampling process on the sensor. In principle, \ac{mfsr} is able to reconstruct crucial \ac{hr} details on solar cells from aliasing artifacts in the \ac{lr} images~\cite{kohler2018phd}.
However, such approaches are entirely image-based.
Specifically, inter-frame motion is directly obtained on the image plane by
classical image-to-image registration or learned motion estimation modules.
We show that the robustness of these image-based methods for \acl{el} imaging of
solar cells is strongly limited by the accuracy of motion estimation.
In contrast, we propose to incorporate 3D geometry into the underlying model to
increase robustness of \ac{mfsr}.

~\\\noindent
\textbf{3D geometry-based multi-frame super-resolution.}
In a closely related work to ours, Park and Lee~\cite{park2017joint} have
coupled the estimation of the 3D camera pose, depth, and \ac{hr} data from
\ac{lr} frames via joint optimization based on a projective model.
Similar to this approach, we consider 3D camera motion in a static scene.
Additionally, we exploit the fact that imaged solar cells are planar and
distortion-free objects in space.
Therefore, a purely projective model is not sufficient and we propose to include a
pinhole camera model along with radial distortion, such that the overall model
complies to the imaging process.
This allows a joint rectification and \ac{sr} in \acl{el} imaging of solar cells.
We also show how to seamlessly integrate our general-purpose model in
regularized reconstruction based \ac{mfsr}.

\section{Methodology}\label{sec:method}

An overview of our method is shown in~\cref{fig:overview}. It consists of two steps: First, \ac{lr} images are
registered~(\cref{subsec:registration}). As a result, we know the motion $m_i$ from the reference to every low resolution frame $\bm{g}_i$. As opposed to previous works on \ac{sr}, this reference is not an arbitrary low resolution frame. We perform registration w.\,r.\,t\onedot to an image of the module that is free of lens and perspective distortion and find that this increases registration stability. Hence, $m_i$ describes the deformation of $\bm{f}$ by perspective projection and lens distortion (see~\cref{fig:motionmod}). To simplify notation, we define $m$ such that it includes the downsampling as well. Hence, $m$ is not a pure motion model. In a second step, this virtual module image is reconstructed from the \ac{lr} images using the estimated motion~(\cref{subsec:reconstruction}).

\subsection{Registration}\label{subsec:registration}

\input{f3d56bb2ff5f4ba8aef7754428f88d8f.tex}

\input{93abd320714c4d04aaba5839e3572acc.tex}

In the general case, $m$ may be any motion and downsampling model that relates two corresponding points $\lc_{i,k}$ and $\bm{y}_k$ in either view. Hence, it holds that
\begin{equation}\label{eq:motionmod}
\lc_{i,k} = m_i(\bm{y}_k)\,,
\end{equation}
where $\lc_{i,k}$ is the \ac{lr} correspondence to $\bm{y}_k$ and $k$ denotes the \ac{hr} pixel index.We assume that the virtual reference of solar modules lies in the $x$-$y$-plane of the reference coordinate system, which we refer to as the module coordinate system. A 2D\xspace point in the module coordinate system in homogeneous coordinates $\tilde{\mc} = \left[\tilde{y}_1,\,\tilde{y}_2,\,s\right]$ with scaling factor $s$ and its image $\lch_i = \left[\tilde{u}_i,\,\tilde{v}_i,\,s_i\right]$ in multiple views $1 \leq i \leq N$ are related by the non-linear model~\cite{hartley2003multiple,fryer1986lens}:
\begin{equation}\label{eq:pinhole-homo-dist}
    m_i \coloneqq \lch_i \simeq \bm{K} d(\bm{H}_i\tilde{\mc},\, \bm{p})\,.
\end{equation}
Here, $\bm{H} \in \mathbb{R}^{3\times3}$ encodes the camera pose, $\bm{K}$ are the intrinsic camera parameters, and $d(\cdot,\, \bm{p})$ is a distortion model with parameters $\bm{p}$. We assume that $\bm{H}$ changes between frames, whereas $\bm{K}$ and $\bm{p}$ are constant. For simplicity, we resort to radial distortion with a single coefficient. We further assume that the center of distortion is located at the origin.
Let $\tilde{\mc}' = \bm{H}\tilde{\mc} = \left[\tilde{y}_1',\,\tilde{y}_2',\,s'\right]$, $y_1' = \frac{\tilde{y}_1'}{s'}$ and $y_2' = \frac{\tilde{y}_2'}{s'}$. Then, $d$ follows as
\begin{equation}\label{eq:distortion}
    d(\tilde{\mc}',\,\bm{p}) = \left(
        \frac{r_d}{r_u}y_1',\,
        \frac{r_d}{r_u}y_2'
    \right)^\top,
\end{equation}
where $r_u$ is the distance to the center of distortion, $\kappa$ is the distortion coefficient and $r_d$ is given by~\cite{brown1971close}
\begin{equation}\label{eq:brown}
    r_d = r_u + \kappa r_u^3\,.
\end{equation}

For registration, we use the approach by \etal{Hoffmann} to find corresponding points between \ac{lr} frames~\cite{hoffmann2019fast} and Zhang's method to initially estimate the intrinsic and extrinsic camera parameters~\cite{zhang2000flexible}. Then, registered images are obtained by warping each \ac{lr} frame into the common module coordinate system. To achieve sub-pixel  registrations in the presence of noisy point correspondences, we refine the initial parameters by non-linear least-squares minimization:
\begin{equation}
    \hat{\bm{H}}_i = \argmin_{\bm{H}_i} \lVert \bm{g}_i - \tau(\bm{f},m_i) \rVert_2\,.
\end{equation}
Here $\tau(\bm{f},m_i)$ denotes inverse warping of $\bm{f}$ according to the motion model $m_i$. To improve stability, we perform registration in a multi-scale scheme~\cite{modersitzki2004numerical}.

\subsection{Joint super-resolution and rectification}\label{subsec:reconstruction}

By the observation model in \cref{eq:observation-model}, the \ac{hr} image is transformed to obtain a \ac{lr} view and we can relate any point in the module coordinate frame $\bm{y}_k$ with its image $\lc_{i,k}$ in the $i$-th \ac{lr} frame by~\cref{eq:motionmod,eq:pinhole-homo-dist} (cf\onedot~\cref{fig:motionmod}). However, for smooth results, it is preferable to implement~\cref{eq:observation-model} by inverse warping. Therefore, we need to relate $\lc_{i,l}$ with its image in $\bm{y}$, where $l$ is the \ac{lr} pixel index. Hence, the inverse of $m$ is required. Using the motion model in \cref{eq:pinhole-homo-dist}, we need to invert $\bm{K}$, $\bm{H}_i$ and $\bm{p}$. Since $\bm{K}$ and $\bm{H}_i$ are square matrices and have full rank, they are invertible right away. According to~\cref{eq:brown}, distorted and undistorted radii are related by a polynomial of degree 3, which has an analytic inverse by the formula of Cardano. The solution depends on the discriminant $D$. For $D > 0$, the undistorted radius $r_u$ is given by
\begin{gather*}
    r_u = \frac{1}{6\kappa} \left( \sqrt[3]{-4q + 4\sqrt{q^2 + 4p^3}} + \sqrt[3]{-4q - 4\sqrt{q^2 + 4p^3}} \right)\,,
\end{gather*}
for $D < 0$ it is
\begin{gather*}
    r_u = 6\kappa \sqrt{-p} \cos\left( \frac{1}{3}\arccos\left( \frac{-q}{2\sqrt{-p^3}} \right) + \frac{4\pi}{3} \right)
\end{gather*}
and for $D=0$ it holds that $r_u = r_d$. Note that we substituted $p = 3\kappa$ and $q = -27 \kappa^2 r_d$ and that the discriminant is $D = q^2 + 4p^3$. A detailed derivation of this result is given in~\cref{sec:inverse-distortion}.

Inter-frame motion can be expressed in various forms and we use motion vector fields as a general model. The motion vector for the $l$'th pixel of the $i$'th \ac{lr} frame is computed as $\bm{v}_{i,l} = m^{-1}(\lc_{i,l})-\lc_{i,l}$. Using the derived motion fields, a known camera \acl{psf}, and the desired \ac{sr} magnification factor, we parameterize the system matrices $\bm{W}_i$, $i = 1, \ldots, N$ of the observation model in \cref{eq:observation-model} similar to~\cite{kohler2016robust}. Then, $\bm{W}_i$ relates the \ac{hr}, undistorded and rectified image $\bm{f}$ of the virtual module to the \ac{lr} frame $\bm{g}_i$. Given $N$ frames, we aim at reconstructing $\bm{f}$ as \acl{map} estimate:
\begin{equation}
    \hat{\hrimg} = \argmin_{\bm{f}} \sum_{i = 1}^N|| \bm{g}_i - \bm{W}_i \bm{f} ||_2^2 + \lambda R(\bm{f})\quad,
\end{equation}
where $R(\bm{f})$ denotes a regularization term with regularization weight
$\lambda \geq 0$ to induce prior knowledge on the desired solution.
In this paper, we use the weighted bilateral total
variation~\cite{kohler2016robust} for sparse and detail-preserving
regularization.

\section{Experiments and Results}\label{sec:experiments}

This section reports experimental results on five solar module datasets. An example image for every sequence in shown in~\cref{fig:dataset-examples}. The first sequence (\textsc{D1}\xspace) exhibits strong noise artifacts while the second sequence (\textsc{D2}\xspace) is characterized by heavy lens distortion and streak artifacts. The third sequence (\textsc{D3}\xspace) shows substantial motion blur. The last two sequences are not corrupted by any particular kind of artifacts. The sequences have been recorded with four different CMOS cameras at a spatial resolution of \SI{640 x 512}{px}. Here, \textsc{D1}\xspace, \textsc{D2}\xspace and \textsc{D3}\xspace use different cameras, while \textsc{D4}\xspace and \textsc{D5}\xspace share the same camera.

Furthermore, we have \ac{hr} images available for the solar module \textsc{D2-5}\xspace, which will be used as reference for the crack detection in~\cref{subsec:segmentation}. For \textsc{D1}\xspace, there is no such data available, since these images have been captured by a drone on-site.

\subsection{Reference methods and parametrization}\label{subsec:reference-methods}

\input{1a44595e35f64d6fa800fe79d9addb68.tex}

We evaluated multiple \ac{sfsr} and \ac{mfsr} methods. For \ac{sfsr}, we use
bicubic upsampling as a baseline and also compare against
\mbox{\textsc{SRCNN}}\xspace~\cite{dong2014learning} and the recent \mbox{\textsc{ESRGAN}}\xspace~\cite{wang2018esrgan}. For
\ac{mfsr}, we exploit $N = 20$ \ac{lr} frames and compare our proposed approach
against EDVR\xspace~\cite{wang2019edvr}. Due to the lack of comprehensive solar cell datasets to train \acl{dl} methods for our application, we use the pretrained models provided by the authors for \mbox{\textsc{SRCNN}}\xspace, \mbox{\textsc{ESRGAN}}\xspace, and EDVR\xspace. \changed{In contrast, our approach is completely unsupervised and does not require any training\chmark{ch012}.}

\input{553c41c5f40f4a00b0d4ef69c1d47380.tex}

\input{c8ffb5ec6d9147fab18f96b8c1b3c1bf.tex}

\input{3ef49756be2b48959e8b3e0eb358f7ab.tex}

A qualitative comparison of the results is given in
\cref{fig:resultimgs}. The \ac{dl}-based \ac{sfsr} methods (for example \mbox{\textsc{ESRGAN}}\xspace) tend to only sharpen the result and may generate high frequency artifacts as well. Overall, they fail to reconstruct additional details. In contrast, \ac{dl}-based \ac{mfsr} (EDVR\xspace) only performs well on \textsc{D3}\xspace and \textsc{D4}\xspace. Further investigation shows that EDVR\xspace breaks down completely with stronger noise or artifacts, which is in agreement with the finding of another study in a different context~\cite{schirrmacher2019adaptive}. \changed{A detailed comparison of EDVR\xspace and our method is given in~\cref{fig:resultimgs-detail}\chmark{ch008}.} In contrast, the proposed method (\mbox{\textsc{Ours-P}}\xspace) performs well on most datasets, giving smooth results and showing many details without enhancing noise or artifacts. On~\textsc{D1}\xspace, \mbox{\textsc{Ours-P}}\xspace does not perform well. We found that this is caused by strong motion blur in the \ac{lr} images that lead to a larger error in the initial parameter estimation using~\cite{hoffmann2019fast} and~\cite{zhang2000flexible}. In addition, shift-variant motion blur is currently not covered by the reconstruction approach.

\changed{\subsection{Ablation study}\label{subsec:ablation-study}

\chmark{ch010}For an ablation study concerning the impact
of motion estimation and rectification, we replace our registration scheme
by standard in-plane homography estimation between planar solar cells using the image alignment toolbox~\cite{iat2013}. Then, we determine motion vectors for regularized reconstructions from the homographies. In the following, we denote this method as \mbox{\textsc{Ours-H}}\xspace. The impact of our proposed motion estimation can be qualitatively seen from~\cref{fig:resultimgs}. It turns out that the homography-based approach (\mbox{\textsc{Ours-H}}\xspace) gives a relatively blurry result, compared to \mbox{\textsc{Ours-P}}\xspace. This is mostly caused by unstable registration.}

\subsection{User study}\label{subsec:user-study}

\input{4c97cc2de1934089bdb167c2a7ff94f7.tex}

As opposed to many traditional \ac{sr} benchmarks, where \ac{lr} images are obtained by downsampling, we are working with low resolution cameras, such that there are no exact ground truth \ac{hr} images available. On the upside, this corresponds to a realistic application case. On the downside, we cannot report traditional metrics like PSNR or SSIM that require a pixel-perfect \ac{hr} reference image. To this end, we conduct a user study. Here, we assess, if \ac{sr} is suitable for manual inspection of defects by human experts. For the user study, we include all reference methods and our registration in combination with the regularized reconstruction. Further, we include $20$ randomly selected cells from every sequence and compare all $6$ methods using $3\times$ as well as $4\times$ magnification.

In summary, the user study comprises 3000 pairwise comparisons. It was performed by seven experts in \acl{el} imaging. They were presented a pair through a web interface and were asked to decide, which of the samples is better suited for detecting defects. \changed{During the study, the arrangement of methods (left or right) and the order of samples was randomized\chmark{ch004}.} Similar to the benchmark in~\cite{kohler2017benchmarking}, we use Bradley-Terry (BT) scores~\cite{david1963method} derived from pair-wise comparisons to compute a ranking of the methods. Using the scores $p_k$ and $p_l$, the probability that method $k$ is preferred over method $l$ is computed as $P(k>l) = \exp(p_k)/(\exp(p_k) + \exp(p_l))$.

The result is presented in~\cref{fig:bt-scores-all}. On average, the \ac{dl} approaches do not perform well. In contrast, \mbox{\textsc{Ours-H}}\xspace shows moderate performance, but is limited by instable registration. Our proposed method clearly outperforms the other methods. Using the BT scores, we find that \mbox{\textsc{Ours-P}}\xspace is preferred over \mbox{\textsc{Ours-H}}\xspace with a probability of $P=0.57$ ($3\times$) or $P=0.61$ ($4\times$). Regarding the magnification factor, it turns out that the difference in perceived performance is increased for $4\times$ magnification. This is explained by the negative impact of instable registration for the \mbox{\textsc{Ours-H}}\xspace approach, which is amplified with the increased magnification (see~\cref{fig:compare-magn}).

\changed{Finally, we evaluate the inter-observer variance to determine the amount of agreement between users. We compute the ranking of methods per user and Kendall's coefficient of concordance $W \in [-1, 1]$ to obtain the inter-observer variance of method rankings, where $W=1$ denotes total agreement between users, $W=0$ denotes no agreement and $W=-1$ states that user rankings are the exact inverse of each other. We find that the value of $W$ depends on the magnification. For $3\times$ magnification, the coefficient computes as $W_{3\times} = 0.33$, whereas for $4\times$ magnification the agreement is stronger and amounts to $W_{4\times}=0.44$. The same effect has been reported by \etal{K\"ohler} in a similar study~\cite{kohler2017benchmarking}. We conclude that, with higher magnification factors, different methods are easier to distinguish for human observers, which leads to a higher overall consensus\chmark{ch005}.}

\subsection{Automated crack segmentation and detection}\label{subsec:segmentation}

\input{162bb1c35c534aadaf428ec8c9f4f373.tex}

We also investigate if the proposed method improves automated inspection. For this analysis, we include all cell images from \textsc{D2-5}\xspace, since \ac{hr} measurements are available for this data. The results are summarized by~\cref{tab:results} and \cref{fig:resultimgs-seg}.

For crack segmentation, we apply the weakly supervised approach proposed by \etal{Mayr}~\cite{mayr2019weakly} to the results with $3\times$ and $4\times$ magnification as well as to the \ac{hr} images. We use the segmentation result obtained on the \ac{hr} images as ground truth. Obviously, this automatically generated ground truth is not perfect. However, the bias is constant for every method. Hence, the relative performance of the methods remains unaffected.

\etal{Mayr} use a fully convolutional network trained for classification of cracks. The segmentation is then derived from the class activation maps. As a result, the resolution of segmentation maps is reduced by a factor of $16$, compared to the original image. This is beneficial to our task, since a registration of \ac{hr} images that are used as pseudo ground-truth with the super-resolved cell images is challenging and the registration mismatch is hidden by the subsampling.

We report the pixel-wise metrics with respect to the automated segmentation obtained from the \ac{hr} images. It turns out that EDVR\xspace results in a recall that is comparable to our method. However, it generates many false positives, as indicated by the low precision and average precision (AP) in~\cref{tab:results}. On the other hand, \mbox{\textsc{Ours-H}}\xspace results in a high precision but a small recall. Overall, our method performs $\sim2\times$ better than EDVR\xspace and $\sim3\times$ better than \mbox{\textsc{Bicubic}}\xspace regarding the average precision and $F_1$ score.

\input{8be8742d198f44e0a8158c7a2d708dd9.tex}

In addition to the pixel-wise metrics, we compare the detection of crack instances too. For many applications, it is more interesting to know the number of cracks, rather than having a perfect segmentation. Furthermore, a segmentation that is only one pixel off results in a significant drop in the quantitative segmentation performance, since cracks are usually thin.

We perform connected component analysis on the coarse segmentation masks, assuming that a single connected component corresponds to a crack instance. Then, we compute the fraction of crack instances for the \ac{hr} segmentation masks, where at least one pixel of the corresponding area on the \ac{sr} image is classified as a crack. This is the recall of the crack instance detection. Inversely, the fraction of detected crack instances on the \ac{sr} image that overlaps with a crack instance detected on the \ac{hr} image by at least a single pixel, is the precision of crack instance detection. We find that our method outperforms the other methods with respect to recall and $F_1$ score. \mbox{\textsc{Ours-H}}\xspace performs a little better regarding the precision, but fails in the recall of crack instances. Furthermore, EDVR\xspace has a recall that is close to ours, but has a much lower precision. Regarding the $F_1$ score, which combines recall and precision into a single score, our method performs $\sim35\%$ better than the best reference method.

\changed{Finally, we analyze the correlation between the results obtained from the user study to the quantitative results. Spearman's rank correlation coefficient $\rho \in [-1, 1]$ is widely used to quantify the amount of correlation between rankings of ordinal variables. As opposed to, for example, Pearson's correlation, it is not limited to linear associations. The results are summarized by~\cref{tab:results}. We find that, apart from the precision score, the association between the metrics and the user study rankings is weaker for the higher magnification. This complies to a previous study, that reported a weaker correlation for reference metrics at higher magnifications, as opposed to reference-less metrics~\cite{kohler2017benchmarking}. Overall, the experiment reveals a fair correlation of the user study results to the quantitative results for $3\times$ magnification, while they do not correlate well for $4\times$ magnification. This shows that, for higher magnifications, qualitative analysis of the results cannot substitute the qualitative analysis and vice versa\chmark{ch006}.}

\changed{\subsection{Computational performance}\label{subsec:time}

\chmark{ch009}From an application perspective, not only quality of results, but also runtime matters. To this end, we conduct an evaluation of the computational performance. The experiments have been conducted using a workstation equipped with an Intel i7-7820X CPU, \SI{64}{\giga\byte} or RAM and a Nvidia Titan Xp GPU using a magnification of $4\times$ with an initial resolution of $512^2$~pixels. Note that the results were averaged over \textsc{D1-5}\xspace. In~\cref{fig:computation-time}, we summarize the computational performance and compare it to the quantitative segmentation metrics. In summary, we find that, in our experiments, \ac{sfsr} methods are much faster than \ac{mfsr} and might be suitable for online image processing. However, we have shown that they are not well suited for the task at hand, since \ac{hr} details are not authentic. In contrast, \ac{mfsr} methods require about \SIrange{30}{50}{\second} of processing time, with the proposed method being faster than EDVR\xspace. As a result, \ac{mfsr} needs to run offline. This is not prohibitive for many applications: For example, given a scenario, where data is collected on-site and later analyzed to identify and possibly replace faulty modules, this is an acceptable amount of computation time.}

\section{Conclusion}\label{sec:conclusion}

This paper presents a new method to increase the spatial image resolution of
solar modules.
It is based on parametric registration and can be combined with various
reconstruction approaches.
As opposed to non-parametric registration, such as optical flow, this ensures
that the registration result is plausible given the imaging process.
In contrast to parametric registration based on plain homography estimation, it accounts for lens distortion, too.

We present a user study showing that our approach outperforms state of the art single-frame and \acl{mfsr} methods. Furthermore, we combine the proposed method with automatic crack detection and segmentation and show that~\ac{sr} is not only beneficial for manual inspection by human experts, but also for automated inspection. In particular, we show that the proposed method is better suited for subsequent automated inspection than state of the art methods. \changed{Furthermore, we investigate the methods in terms of computational performance and find that the proposed method is computationally fast, compared to other state of the art \ac{mfsr} methods.}

In future works, we aim to integrate the proposed motion model with \ac{sr} to form a single optimization problem, as shown for the purely projective case in~\cite{park2017joint}. This would also allow for joint estimation of motion. Furthermore, an interesting opportunity is to combine the conventional and \ac{dl}-based methods in terms of known operator learning~\cite{maier2019learning} to join the robustness of conventional methods with the power of learning individual steps of the algorithm from the data\chmark{ch007}.

\vspace{.2cm}
\noindent\textbf{Acknowledgements}
\noindent We gratefully acknowledge funding of the Federal Ministry for Economic Affairs and Energy (BMWi: Grant No. 0324286, iPV4.0). In addition, we would like to thank the user study participants: J.~Hepp, M.~Mayr, A.~Vetter, J.~Denz, T.~Winkler and C.~Buerhop. Further, we would like to thank Ircam GmbH and Greateyes GmbH for providing the cameras and images and Rauschert Heinersdorf-Pressig GmbH for access to their photovoltaic plants.

{\small
\bibliographystyle{IEEEbib}