\section{Introduction}
Recent development in the field of deep learning has led to a rising interest in Facial Attribute Editing (FAE) ~\cite{DBLP:conf/cvpr/ChoiCKH0C18,DBLP:conf/aaai/Yin0L19,DBLP:journals/corr/abs-1908-07191,DBLP:journals/corr/abs-1907-11922,DBLP:journals/corr/abs-1905-01920,DBLP:journals/corr/abs-1908-08989,DBLP:journals/corr/abs-1904-06624,DBLP:conf/cvpr/0018DXLDZW19,DBLP:journals/corr/PerarnauWRA16}, \textit{e}.\textit{g}., manipulating structural, emotional, semantic attributes (\textit{e}.\textit{g}., gender, hair color, age etc.) of a given facial image.
These functionalities are highly demanded in emerging mobile applications.
However, most of recent facial editing approaches~\cite{DBLP:conf/cvpr/0018DXLDZW19,DBLP:journals/corr/abs-1907-11922,DBLP:journals/corr/abs-1908-07191} are based on deep image-to-image translation model, which can ONLY deal with low resolution (LR) facial images due to excessive deep model (\textit{e}.\textit{g}.,  GAN~\cite{DBLP:conf/cvpr/XuZHZGH018,DBLP:conf/cvpr/0018DXLDZW19}) size.
Concretely, as the image size increases, the memory consumption of deep models also increases dramatically, even surpasses the limitation of device.
Such limitation forbids many promising applications on mobile platforms (\textit{e}.\textit{g}., ZAO~\cite{ZAO} processing images in the cloud service).
Although upgrading hardware can alleviate these issues, it is a non-cheap solution and still hard to process higher resolution images (\textit{e}.\textit{g}., 4K).
In this work, we aim at developing a memory-efficient deep framework to effectively handle high resolution facial attribute editing on resource-constrained devices.

There are two major challenges of deep model based HR facial attribute editing:
1.~\emph{Constrained Computational and Memory Resource.}
In some mobile scenarios
(\textit{e}.\textit{g}., smartphone, AR/VR glasses) with only limited computational and memory resources, it is infeasible to use popular image editing models~\cite{DBLP:journals/corr/abs-1907-11922,DBLP:journals/corr/abs-1905-01920}  which require sophisticated networks.
To address this issue, methods based on model pruning and operator simplifying~\cite{DBLP:conf/cvpr/ZhaoNZZZT19,DBLP:journals/corr/HowardZCKWWAA17,DBLP:journals/corr/abs-1801-04381,DBLP:conf/cvpr/ZhangZLS18,DBLP:conf/eccv/MaZZS18} have been proposed to reduce  the inference computational complexity.
However, the metric-based way to reduce the model size will do harm to model perceptual representation ability, the output facial image quality is usually largely sacrificed.
2.~\emph{Inefficient Multi-scale Features Fusion.}
In order to achieve high-level semantic manipulation while maintaining local details during image generation, multi-scale features fusion is widely adopted.
It is a common practice to utilize skip connection, \textit{e}.\textit{g}.,  U-Net~\cite{DBLP:conf/miccai/RonnebergerFB15}.
However, fixed schemes, such as skip connection, usually result in infeasible or even self-contradicting fusion output (\textit{e}.\textit{g}., during style transfer, content is well-preserved but failed to change the image style), and flexible schemes such as GRU~\cite{DBLP:journals/corr/ChungGCB14} lead to additional computational burden (\textit{e}.\textit{g}., applying GRU-based unit~\cite{DBLP:conf/cvpr/0018DXLDZW19} directly for multi-scale features fusion can achieve excellent fusion effects, but will increase network parameters by more than four times).

In this work, a novel image translation framework for high resolution facial attribute editing, dubbed \emph{CooGAN}, is proposed to explicitly address above issues.
It adopts a divide-and-combine strategy to break the \textbf{intractable} whole HR image generation task down to several sub-tasks for reducing memory cost greatly.
More concretely, the pipeline of our framework consists of a series of local HR patch generation sub-tasks and a global LR image generation sub-task. To handle these two types of sub-tasks, our framework is also composed of two paths, \textit{i}.\textit{e}., local network path and global network path.
Namely, the local sub-task is to generate an HR patch with edited attributes and fine-grained details. And a global sub-task is to generate an LR whole facial snapshot with structural coordinates to guide the local workers properly recognize the correct patch semantics.
As only tiny size patch (\textit{e}.\textit{g}., $64\times 64$) generation sub-task is involved in the pipeline, the proposed framework avoids processing large size feature maps. As a result, this framework is very light-weighted and suited for resource constrained scenarios.
In addition, a local-to-global consistency objective is proposed to enforce the cooperation of sub modules, which guarantees between-patch contextual consistency and appearance smoothness in fine-grained scale.

Moreover, we design a variant of SRU~\cite{DBLP:conf/emnlp/LeiZWDA18},~\emph{Light Selective Transfer Unit (LSTU)}, for multi-scale features fusion.
The GRU-based STU~\cite{DBLP:conf/cvpr/0018DXLDZW19} has similar functions, but needs two states (one state obtained from encoder, another from the higher level hidden state) to inference the selected skip feature.
As a result, it has to face a heavy computing burden and is not friendly to GPU acceleration.
Unlike STU, our SRU-based LSTU just need a single hidden state to get the gating signal, which greatly reduces the complexity of the unit.
Actually, the LSTU has only half as many parameters as STU and almost the same multi-scale features fusion effect, achieving a good balance between model efficiency and output image fidelity.
Under this design, the framework is able to selectively and efficiently transfer the shallow semantics from the encoder to decoder, enabling more effective multi-scale features fusion with constrained memory consumption.

We extensively experiment the proposed framework in terms of both qualitative and
quantitative evaluations on facial attribute editing tasks.
It is demonstrated that our framework can process $768\times 768$ images well with less than $1GB$ GPU memory
rather than over $6.3GB$ (the previous state-of-the-art~\cite{DBLP:conf/cvpr/0018DXLDZW19} level).
Furthermore, the proposed model has the capability to achieve \emph{$84.8\%$} facial attribute editing accuracy on CelebA-HQ.

\section{Related Works}
{\bf Image-to-image translation.} Image-to-image translation aims at learning cross-domain mapping in supervised or unsupervised settings~\cite{DBLP:conf/cvpr/0018DXLDZW19}.
In the early stage,  methods like pix2pix use image-conditional GANs to train paired images in the supervised manner.
Later, unpaired image-to-image translation frameworks, which work in the unsupervised manner, are also proposed in~\cite{DBLP:conf/iccv/ZhuPIE17,DBLP:conf/nips/LiuBK17}.
For instance, CycleGAN~\cite{DBLP:conf/iccv/ZhuPIE17} learns the mapping between one image distribution and another and introduces a cycle consistency loss to preserve key features during translation.
However, all those mentioned frameworks can only perform coarse-grain translation from one image distribution to another. They cannot modify specific attributes of an image, which is more important in real world applications. To address such problem, StarGAN~\cite{DBLP:conf/cvpr/ChoiCKH0C18} is proposed to learn multiple domain translation using only a single model.

Facial attribute editing is a typical task of image-to-image translation that focuses on editing specific attributes within image. Early methods~\cite{DBLP:conf/cvpr/Kemelmacher-ShlizermanSS14,DBLP:journals/tog/ThiesZNVST15,DBLP:conf/cvpr/ChenLSLTSYJ18} are proposed to learn only a single attribute editing model like expression transfer~\cite{DBLP:journals/tog/ThiesZNVST15}, age progression~\cite{DBLP:conf/cvpr/Kemelmacher-ShlizermanSS14}, and so on.
To achieve arbitrary attribute editing, IcGAN~\cite{DBLP:journals/corr/PerarnauWRA16} uses a conditional GAN to process encoded features.
To strengthen the relations between high-feature space and attributes, Lample G. et al.~\cite{DBLP:conf/nips/LampleZUBDR17} applies adversarial constraints.
StarGAN~\cite{DBLP:conf/cvpr/ChoiCKH0C18} and AttGAN~\cite{DBLP:journals/tip/HeZKSC19} use target attribute vector input and achieve great success.
Based on them, the STGAN~\cite{DBLP:conf/cvpr/0018DXLDZW19} is further proposed for simultaneously enhancing the attribute manipulation ability and image quality.
Especially, Ran Yi et al.~\cite{DBLP:conf/cvpr/YiLLR19} propose a hierarchical facial editing method to modify the face via facial features patches.
In our work, we analyze the limitation of StarGAN, AttGAN,~\cite{DBLP:conf/cvpr/YiLLR19} and STGAN and then further develop a patch-based model to make facial attribute editing suitable for higher resolution images.

\section{Methodology}
\subsection{Overview}
The proposed CooGAN framework for conditional facial generation presents two innovative modules, the global module and the local module. The global module is designed to generate LR translated facial image, and the local module aims at generating HR facial image patches and stitching them together. A cooperation mechanism is introduced to make these two modules work together, so that the global module provides the local module with a global-to-local spatial consistency constraint.
In addition, to guarantee the performance and edit-ability of the generated images, we propose a well-designed unit, LSTU, to filter the features from latent space and infuse them with detail information inside the naive skip connection.

\subsection{The Cascaded Global-to-Local Face Translation Architecture}
The \emph{CooGAN} consists of two interdependent generation modules.
We depict the framework architecture in Fig.~\ref{framework}.

{\bf Global module.}
The global module is designed for generating the translated snapshot $X'_t$ which carries the whole image spatial coordinate information.
As is shown in Fig.~\ref{framework}, this module contains two components: the global-aware generator $\mathcal{G}_g$ and the global-aware discriminator $\mathcal{D}_g$.
$\mathcal{G}_g$ has a conventional U-Net~\cite{DBLP:conf/miccai/RonnebergerFB15} structure strengthening by LSTU. It takes in $X'_i$, the down-sampled image of $X_i$, and generates the LR snapshot $X'_t$.
Following the ACGAN-like~\cite{DBLP:conf/icml/OdenaOS17} fashion, the discriminator $\mathcal{D}_g$ has two headers sharing the same extracted features.
Note that the main purpose of global module is to guarantee the global semantic consistency of the final result.

\begin{figure}[t]
\centering
\includegraphics[width=1\linewidth]{figures/Framework.pdf}
\caption{An overview of our framework. Our framework has two modules--the global image translation module on the left side and the local patch refinement module on the right side. Either module contains one generator and one discriminator, as indicated in the graph. The two modules cooperate with each other through the cooperation mechanism in the middle. All specific operations are elaborated in the graph.}
\label{framework}
\end{figure}

{\bf Local module.}
The local module is designed for processing HR image patches under the guidance of LR translated patches with limited computational resources.
As is shown in Fig.~\ref{framework}, the local module contains two components: the local-aware generator $\mathcal{G}_l$ and the local-aware discriminator $\mathcal{D}_l$.
We concatenate the original HR image patch and up-sampled snapshot patch, \textit{i}.\textit{e}., $[x_{i}(m,n), x_{t}(m,n)]$, where $(m,n)$ represents the coordinate of the patch in the whole image.
Then the local-aware generator $\mathcal{G}_l$ processes the concatenated data sequentially to generate translated HR patches. The generated patches can be stitched together \emph{without overlap} to synthesize the final output $X_o$.
To avoid the inconsistency between generated patches, the local-aware discriminator $\mathcal{D}_l$ is introduced to make the final stitched output smooth and seamless.
Note that the discriminator is just introduced temporarily for training our model, so it will not add any memory cost during the inference stage.

{\bf Cooperation mechanism.} In the previous sections, we discuss the functions of the local module and the global module.
The global module requires more computational resources while the size of image increases, which seriously limits its application for generating HR images.
The local module decomposes the full-size image into patches and translates the patches into HR ones, which is suitable for processing HR image without adding any memory cost.
But the local module alone cannot correctly perceive the patch semantics and is inapplicable for editing local attributes, as can be seen in Fig.~\ref{localalone}.
To this end, we introduce a valid cooperation mechanism to encourage the two generators to cooperate well for achieving satisfying generation performance.
As shown in the middle of Fig.~\ref{framework}, the global snapshot $X'_t$ is up-sampled to the same size as HR image $X_i$. Then, we decompose $X'_t$ and $X_i$ into a series of patches and concatenate the patches with the same coordinate.
The global patch carries the global spatial coordinate information while the HR image contains detailed textural information.
By this way, we combine the global spatial information and detailed textural information together.
Namely, the global spatial information is used to make the generated local patch smoother and globally consistent, and the detailed textural information is used to preserve the quality of the generation.

\subsection{Objective Function}
\label{Objective Function}
Our framework contains two sets of adversarial training in global and local module, respectively.
For the sake of clarity, we first describe the inference stage of either module.
In the global module, we get the snapshot $X'_t$ by the generator $\mathcal{G}_g$ and then feed $X'_t$ into the discriminator $\mathcal{D}_g$.
Following ~\cite{DBLP:conf/cvpr/0018DXLDZW19}, we use $A_{d}$ to facilitate the adversarial training, where $A_{d}$ denotes the difference attribute vector between the target $A_{t}$ and the source $A_{i}$.
The training process can be defined as:
\begin{equation}
    X'_t = \mathcal{G}_g\left(X'_i,A_{d}\right).
\end{equation}

In the local module, $x_{i(i,j)}$ denotes a patch of HR image $X_i$ and $x_{t(i,j)}$ denotes a patch of up-sampled snapshot image $X_t$. We use $[ \cdot, \cdot]$ to represent the channel-wise patch concatenation,
\begin{equation}
    x_{o(i,j)} = \mathcal{G}_l\left(\left[x_{i(i,j)},x_{t(i,j)}\right],A_{d}\right).
\end{equation}

{\bf Image reconstruction loss.} We apply the reconstruction loss in our framework to improve image quality and avoid as many editing miscues as possible. This loss is achieved in a self-supervision manner, \textit{i}.\textit{e}., we feed the framework with zero condition input, which can be depicted as:
\begin{equation}
    \mathcal{L}_{rec} = ||x -\mathcal{G}(x,0)||_1.
\end{equation}

{\bf Adversarial loss.} For FAE task, there is no ground-truth for training.
Adversarial loss is usually applied due to its domain-to-domain translation nature.
To alleviate the instability of adversarial loss in training process, gradient penalty~\cite{DBLP:journals/corr/GulrajaniAADC17} is introduced:

\begin{equation}
    \begin{split}
    \begin{aligned}
      \max \limits_{\mathcal{D}_{adv}} \mathcal{L}_{\mathcal{D}_{adv}} = \mathbb{E}_x\big[\mathcal{D}_{adv}(x)\big] - \mathbb{E}_{\hat{y} }\big[\mathcal{D}_{adv}({\hat{y}})\big] &+\qquad \\
        \lambda \mathbb{E}_{\hat{x}}\big[(&\|\nabla_{\hat{x}}\mathcal{D}_{adv}({\hat{x}})\|_2-1)^2\big],
    \end{aligned}
    \end{split}
\end{equation}
\begin{equation}
    \max \limits_{\mathcal{G}} \mathcal{L}_{\mathcal{G}_{adv}} = \mathbb{E}_{x,A_{d}}\big[\mathcal{D}_{adv}(\mathcal{G}(x,A_{d}))\big],\quad\quad\quad\qquad\quad\quad\qquad\qquad
\end{equation}
where $\mathcal{D}_{adv}$ denotes the adversarial header of the discriminator. $x$ is the input real image, $\hat{y}$ is the generated image and $\hat{x}$ is the sampled point along straight lines between the real image distribution and generated image distribution.

{\bf Attribute editing loss.} As discussed above, attribute editing has no ground truth. Following the ideology of ACGAN~\cite{DBLP:conf/icml/OdenaOS17}, we add a classification header sharing features with adversarial header. Such a design effectively realizes adversarial training along with attribute learning. The attribute editing loss below supervises the two modules of our framework individually,
\begin{equation}
    \begin{split}

        \mathcal{L}_{\mathcal{D}_{a}} =
        -\sum_{i=1}^{N_c}
        \left[a_t^{(i)}\log{\mathcal{D}_{a}^{(i)}(x)}+(1-a_t^{(i)})\log{(1-\mathcal{D}_{a}^{(i)}(x))} \right],

    \end{split}
\end{equation}
where $a^{(i)}_t$ represents the $i$-th target attribute and $\mathcal{D}_a$ represents the classification header.
We use the combination of the above three types of loss to train our model.

\begin{table}[b]
\tiny
\setlength{\tabcolsep}{5mm}
\begin{center}
\footnotesize
\label{parameters}
\begin{tabular}{cccccc}
\hline
SC type&SC$0$&SC$1$&SC$2$&SC$3$&SC$4$\\
\hline
PSNR&$20.2$&$24.29$&$28.0$&$29.9$&$34.8$\\
SSIM&$0.643$&$0.777$&$0.877$&$0.916$&$0.989$\\
Average Accuracy &$89.5\%$&$82.4\%$&$75.1\%$&$69.1\%$&$63.9\%$\\
\hline
\end{tabular}
\caption{PSNR/SSIM performance and average attribute editing accuracy of models with different Skip Connection (SC) numbers, SC$i$ represents model with $i$ skip connections.}
\label{PSNR}
\end{center}
\end{table}

\subsection{Light Selective Transfer Unit}
As mentioned above, we proposed a high-resolution facial image editing framework which can process HR images with limited computational resources.
In order to further improve the generation performance, we improve the manner of multi-scale features fusion.

The most popular method for multi-scale features fusion in image-to-image translation is the \emph{Skip Connection} structure. It helps the network balance the contradiction between the pursuit of larger receptive field and loss of more details.
One of its classic deployments is U-Net.
However, there is a fatal drawback of the original skip connection: it will degrade the function of deeper parts and further damage the effectiveness of condition injection.
From Table~\ref{PSNR}, it is obvious that PSNR increases but attribute editing accuracy decreases when the skip connection number multiplies. A detailed graph showing the editing accuracy of each specific attribute is given in suppl.
STGAN~\cite{DBLP:conf/cvpr/0018DXLDZW19} tries to alleviate the problem with the STU, a variant of GRU~\cite{DBLP:conf/emnlp/ChoMGBBSB14,DBLP:journals/corr/ChungGCB14}, which uses the latent feature to control the information transfer in the skip connection through the unit.
This feature carries the conditional information added through the concatenation.
Unfortunately, such a unit omits the spatial and temporal complexity and it is a time-consuming process for the underlying feature to bubble up from the bottleneck to drive the STU of each layer.

To explicitly address the mentioned problem, we present our framework to employ \emph{Light Selective Transfer Unit (LSTU)} to efficiently and selectively transfer the encoder feature.
LSTU is an SRU-based unit with totally different information flow.
Compare to STU, our LSTU discards the dependence on the two states when calculating the gating signal, which greatly reduces our parameters but the unit is still efficient.
The detailed structure of LSTU is shown in Fig.~\ref{lstu}.
Without loss of generality, we choose the $l$-th LSTU as an analysis example.
The $l$-th layer feature coming from the encoder side denotes as $x^l$.
$h^{l+1}$ denotes the feature in the adjacent deeper layer.
It contains the filtered latent state information of that layer .
$h^{l+1}$ is firstly concatenated with attribute difference $A_d$ to obtain up-sampled hidden state $\hat{h}^{l+1}$.
Then $\hat{h}^{l+1}$ is used to independently calculate the masks $f^l,r^l$ for the forget-gate and reset-gate.
$\mathbf{W_T}$, $\mathbf{W_{1\times 1}}$, $\mathbf{W_f}$ and $\mathbf{W_r}$ represent parameter matrix of transpose convolution, linear transform, forget gate and update gate.
The further process is similar to SRU.
The equation of gates is shown on the right side of Fig.~\ref{lstu}.

\begin{figure}[t]
\begin{center}
\includegraphics[width=0.9\linewidth]{figures/LSTU.pdf}
\end{center}
  \caption{The structure of proposed LSTU. The design of LSTU is inspired by SRU, which makes LSTU more lightweight than STU and suitable for GPU parallel acceleration. The right side shows the mathematical expression of the inference process of our LSTU. LR is short for LeakyRelu.}
  \label{lstu}
\end{figure}

\section{Experiments}
In this section, we take the following steps to evaluate the performance of proposed CooGAN:
firstly, we verify the effectiveness of the LSTU in multi-scale features fusion.
Then, we discuss the facial attribute editing performance and the memory usage of our overall framework.
\subsection{Implement Details}
{\bf Dataset.} We train our model on the CelebA~\cite{liu2015faceattributes} dataset (for the global module) and CelebA-HQ dataset (for the local module). CelebA is a collection of more than 200,000 human facial images and labels of 40 attributes for each of the images. CelebA-HQ is an artificial high resolution dataset translated from CelebA utilizing the method from~\cite{DBLP:conf/iclr/KarrasALL18}.
Considering the distinguishability of attributes, in the experiments we mainly use 13 attributes including \emph{Bald, Bangs, Black Hair, Blond Hair, Brown Hair, Bushy Eyebrows, Eyeglasses, Male, Mouth Slightly Open, Mustache, No Beard, Pale Skin and Young}.

{\bf Optimizer.} The optimizer used in our training process is Adam~\cite{DBLP:journals/corr/KingmaB14} with hyper-parameter $\beta=(0.5,0.99)$ and learning rate $0.0002$.
And we apply the learning rate decay strategy with the decay rate 0.1 and decay epoch 100.

\begin{table}[b]
\begin{center}
\footnotesize
\setlength{\tabcolsep}{4.05mm}
\begin{tabular}{ccccc}
\hline
Method&StarGAN&AttGAN&STGAN&Ours\\
\hline
PSNR/SSIM&$22.86/0.848$&$24.1/0.821$&$31.6/0.934$&${\bf 32.1/0.938}$\\
\hline
\end{tabular}
\caption{Reconstruction PSNR and SSIM (CelebA $128\times 128$) of different facial attribute editing frameworks.}
\label{diffframeworkpsnr}
\footnotesize
\end{center}
\end{table}

{\bf Quantitative metric.} We design and train an attribute-classifier to quantify attribute editing effects in form of attribute classes accuracy.
It takes form of the classifying part in our discriminator and achieves a $96.0\%$ average accuracy on CelebA.
And for the texture sharpness of reconstruction results, Peak Signal to Noise Ratio(PSNR) and Structural Similarity(SSIM) are used. They are two common metrics to measure the generated image fidelity.
In addition, we use the floating-point operations per second (FLOPS) and model parameter size to evaluate the memory efficiency of networks.

\subsection{Experimental analysis of Framework.}
{\bf Experiment setup.}
Our framework consists of two modules: global module and local module.
In the following experiments, the input image size of the global module is set as $256 \times 256$
and the input patch size of local module is $128\times 128$. The final high resolution image size is $768\times 768$.
We use the two most effective previous methods (AttGAN, STGAN) and our global module as comparison objects.
Since these models are not originally designed for HR image processing, we make proper adjustments of increasing their layers (AttGAN-HR: 7 layers, STGAN-HR: 7 layers, LSTU-net-HR: 7 layers) to expand their perceptual fields. We expand the models based on their official codes and train them with their original loss functions. Those adjusted approaches can be denoted as: AttGAN-HR (AttGAN), STGAN-HR (STGAN) and LSTU-net-HR(global module).
The complete model structure setting of our framework, AttGAN-HR and STGAN-HR can be found in suppl.
Models are evaluated in the CelebA-HQ dataset.

\begin{figure}[t]
\begin{center}
\includegraphics[width=0.6\linewidth]{figures/HR-diffframework.pdf}
\end{center}
  \caption{Our CooGAN against other approaches in attribute translation accuracy. Our framework transcend others in most attributes.}
\label{acc1_hr}
\end{figure}

\begin{table}[b]
\begin{center}
\setlength{\tabcolsep}{6.6mm}
\begin{tabular}{ccccc}
\hline
Method&AttGAN-HR&STGAN-HR&Ours\\
\hline
GPU Memory&$3692 $MB&$6306 $MB&${\bf 985} $MB\\
\hline
Inference time&$0.0414 $s&$0.0502 $s&${\bf 0.0645} $s\\
\hline
\end{tabular}
\caption{GPU memory consumption and average inference time (200 images), measured on a 2080ti, for different framework with batch size 1, input images at the $768\times 768$ pixels.}
\label{gpuconsumption}
\end{center}
\end{table}

\begin{figure}[!t]
\begin{center}
\includegraphics[width=1\linewidth]{figures/ResultsHR.pdf}
\end{center}
\caption{High resolution ($768\times 768$) facial attribute editing results of AttGAN-HR, STGAN-HR, LSTU-net-HR and our framework. The textural details can be observed clearly through zooming in. More HR editing results can be found in suppl.}
\label{results}
\end{figure}

{\bf Quantitative evaluation.} We report the attribute editing accuracy of our framework and other methods in Fig.~\ref{acc1_hr}. Our framework can achieve $84.8\%$ in classification accuracy, 31.3 in PSNR and 0.923 in SSIM.
In terms of memory usage, the largest memory consumption occurs in the global module for the reason that the local module only processes image patches with considerably small sizes. As is shown in Table~\ref{gpuconsumption}, while processing $768\times 768$ images the maximum consumption of our framework is below all compared models, even StarGAN. This is on the grounds that the peak consumption in our framework happens when the global module processes $256\times 256$ images, while other models process $768\times 768$ images directly. Besides, our framework reduces large memory consumption at the cost of only a little inference time increase, as there is \emph{no overlap} when our framework processes patches, i.e., our advantage. Therefore, the prominent image processing efficiency of our framework is verified.

{\bf Qualitative analysis.}
The visual effect comparison are shown in Fig.~\ref{results}.
In general, our CooGAN can effectively generate high-resolution images with smooth patch boundaries.  As is shown, images generated by our CooGAN possess more prominent characteristics as well as better textural quality. For local attributes like bald and mustache, our framework performs much more accurate and significant modification. And in global attributes like gender and age, our CooGAN translates the image more thoroughly and successfully. It is worth mentioning that HR translated images have better effect in some attributes like age and gender. This is because these attribute can not be properly manifested in LR images, \textit{e}.\textit{g}., the wrinkles of aging are obvious only in high resolution.

\begin{figure}[t]
\begin{center}
\includegraphics[width=0.8\linewidth]{figures/ablationstudy.pdf}
\end{center}
\caption{$768\times 768$ results generated by our framework without and with global module.}
\label{localalone}
\end{figure}
\begin{figure}[t]
\begin{center}
\includegraphics[width=1\linewidth]{figures/LR_results.pdf}
\end{center}
\caption{Low resolution ($128\times 128$) facial attribute editing results of StarGAN, AttGAN, STGAN and our LSTU-Net. }
\label{lrresults}
\end{figure}

\begin{table}[b]
\scriptsize
\begin{center}
\setlength{\tabcolsep}{2.5mm}
\begin{tabular}{cccccccc}
\hline
Method&\emph{Bald}&\emph{Bangs}&\emph{Black}&\emph{Blond}&\emph{Brown}&\emph{Eyebrows}&\emph{Glasses}\\
\hline
AttGAN-HR& $25.7\%$&   $66.2\%$&   $61.2\% $&   $89.4\%$&   $47.9\%$&   $51.7\%$&   $86.7\%$\\
STGAN-HR&  $38.8\%$&   $50.5\%$&   $78.2\%$&   $92.8\%$&   $53.5\%$&   $65.4\%$&   $71.5\%$\\
Ours&$\bf61.5\%$&$\bf93.3\%$&$\bf90.6\%$&$\bf94.9\%$&$\bf73.6\%$&$\bf84.9\%$&$\bf94.8\%$\\
\hline
\end{tabular}
\setlength{\tabcolsep}{2.3mm}
\begin{tabular}{cccccccc}
\hline
Method&\emph{Male}&\emph{Mouth}&\emph{Mustache}&\emph{No Beard}&\emph{Pale}&\emph{Young}&\emph{Smooth}\\
\hline
AttGAN-HR&     $36.8\%$&   $ 83.2\%$&   $29.8\%$&   $ 51.4\%$&   $34.5\%$&   $41.4\%$&   $99.8\%$\\
STGAN-HR&$51.3\%$&   $67.7\%$&   $49.1\%$&   $67.8\%$&   $61.4\%$&   $49.9\%$&   $99.9\%$\\
Ours&$\bf87.9\%$&$\bf91.7\%$&$\bf61.9\%$&$\bf93.8\%$&$\bf98.1\%$&$\bf64.7\%$&$\bf91.5\%$\\
\hline
\end{tabular}
\caption{User study results of our framework, STGAN-HR and AttGAN-HR.}
\label{userstudy}
\end{center}
\end{table}

{\bf User study.} To further validate the subjective performance of our framework, it is important to conduct survey on a crowdsourcing platform to acquire people's general opinions of our HR edited images and compare our CooGAN with the modified HR versions of previous state-of-the-art models including STGAN-HR and AttGAN-HR. In the survey, 50 questions of 14 attributes/quality are involved, \textit{i}.\textit{e}., 13 previous mentioned attributes and the smoothness of the image. And in each question, a randomly selected image and its edited versions by the three models are presented in random sequences. And people will determine whether an edited image has correct and realistic attributes and is generally smooth and consistent. The user study results are given in Table~\ref{userstudy}.
In comparison, our CooGAN shows its excellent performance in attributes editing.
Because our framework generates image by patches, it is slightly inferior to the other two methods in terms of smoothness.

{\bf Ablation study for Cooperative mechanism.} In this part, our focus is to study the importance and effectiveness of global module in the framework. We train our framework with and without the global module. The results show that without global module the attribute editing will be impractical, as is shown in Fig.~\ref{localalone}. On the one hand, without the spatial information given by global snapshot, the local module can not well recognize local attributes like bangs regions and mouth. On the other hand, the shrinkage of perceptive field causes loss of global semantics, rendering certain features (like gender characteristics), difficult for the framework to discern. Thus, we confirm the irreplaceable role of global module in our framework.

\subsection{Ablation study for LSTU}
LSTU is a vital component of our framework.
We will verify the feasibility and effectiveness of LSTU in this section.

{\bf Experiment setup.}
Our LSTU is a lightweight and equally effective replacement unit for STU~\cite{DBLP:conf/cvpr/0018DXLDZW19} (previous state-of-the-art multi-scale features fusion unit proposed in STGAN).
To validate its effect and efficiency, we replicate the network structure of STGAN and replace its STU with our LSTU. We name this new network LSTU-Net to be distinguished from the original STGAN.
Besides, we also compare LSTU-Net with StarGAN and AttGAN since they are the most relative and effective facial attribute editing methods that can be found.
However, these compared models are not designed for HR image generation tasks. They are mostly purposed to process $128\times 128$ images.
To be fair, all models are tested on $128\times 128$ CelebA dataset.

In addition, the official codes and pretrained models of those three compared models are used in our experiments for the sake of impartiality.

\begin{figure}[t]
\begin{center}
\includegraphics[width=0.6\linewidth]{figures/LR-diffframework.pdf}
\end{center}
  \caption{Our LSTU-Net against other approaches in attribute translation accuracy.}
\label{acc1_LSTUnet}
\end{figure}

\begin{table}[t]
\begin{center}
\footnotesize
\setlength{\tabcolsep}{3.5mm}
\label{parameters2}
\begin{tabular}{ccccccc}
\hline
{channel}  & \multicolumn{2}{c}{16\ in-channel} &  \multicolumn{2}{c}{32\ in-channel}& \multicolumn{2}{c}{64\ in-channel} \\
\cline{1-7}
{metric}& param & FLOPS & param & FLOPS & param & FLOPS \\
\hline
{STU}  & 1.63M & 5.06G & 6.40M & 19.99G & 25.37M & 79.50G \\
\hline
{LSTU}  & 0.81M & 2.02G & 3.20M & 7.85G & 12.68M & 30.80G \\
\hline
\end{tabular}
\caption{The total number of parameters and FLOPS in the skip connections within a five-layer network with different input channels for the LSTU and STU. G denotes GFLOPS.}
\label{complexity}
\end{center}
\end{table}

{\bf Quantitative evaluation.}
The attribute editing accuracy and image fidelity comparisons are shown in Fig.~\ref{acc1_LSTUnet} and Table~\ref{diffframeworkpsnr}. For attribute accuracy, our LSTU-Net outperforms StarGAN, AttGAN and ties with STGAN.
This circumstance remains the same for image fidelity manifested through PSNR and SSIM. These results show that LSTU has the same level of effect as STU, whether in attribute editing accuracy or image fidelity.
The complexity and computational cost comparisons of different units are demonstrated in Table~\ref{complexity}. Compared to STU, our LSTU reduces parameter size by half and FLOPS by over 60 $\%$.
These comparisons prove the effectiveness and efficiency of implementing LSTU.

{\bf Qualitative analysis.} The visual effect comparison of $128\times 128$ images is given in Fig.~\ref{lrresults}. The editing result of our LSTU is better than StarGAN, AttnGAN, and achieves similar visual effect with STGAN. In almost all attributes, our model not only performs thorough and accurate translation, but also suffer the least from textural artifacts. In contrast, the results of StarGAN have distinct color difference due to the lack of skip connection. AttGAN's results contain fake artificial textures and half-changed features in attributes like bald and hair color. Such inconsistency is caused by the direct use of skip connection. STGAN is more successful than AttGAN and StarGAN. And its edited features and ours are equally effective.
These comparison results present the state-of-the-art level attribute editing ability and tremendous efficiency advantage of our LSTU. More results generated by the LSTU-Net are given in the suppl.

\section{Conclusion}
In this paper, we study the problem of high-resolution facial attribute editing in resource-constrained conditions. By proposing the patch-based local-global framework \emph{CooGAN} along with the multi-scale features fusion method LSTU, we attained the high-resolution facial attribute editing with constrained GPU memory. We use an up-to-down approach for patch processing to preserve global consistency and most importantly, lower the computational resource requirement. And the LSTU retains the attribute variety in skip connection with fewer parameters and lower computation cost. Experiments on facial attribute editing exhibit the superior performance of our framework in quality, attribute accuracy and efficiency to state-of-arts in the scope of facial attribute editing with constrained computational resources. Theoretically, our framework has no image size limitation owing to its patch processing method. We believe it has promising prospect not only in resource-constrained situations, but also in extremely high-resolution image processing tasks.

\section{Acknowledge}
This work was supported by National Science Foundation of China (61976137, U1611461, U19B2035) and STCSM(18DZ1112300).

\clearpage
\bibliographystyle{splncs04}


\section{Introduction}
\label{sec:intro}

Lots of progress has been made in automatic speech recognition fields in recent years. Many new neural networks such as Deep Neural Network (DNN), Convolution Neural Network (CNN), Recurrent Neural Network (RNN), and Self-attention Neural Network (SAN) \cite{NIPS2017_7181} have greatly boosted the acoustic model performance for hybrid ASR models. However, the hybrid ASR model has a long pipeline that has to be built independently. Recent advances in end-to-end(E2E) models \cite{chan2016listen, prabhavalkar2017comparison, battenberg2017exploring, sak2017recurrent} reveal the E2E model has great potential to replace the traditional hybrid model in the industry area. The E2E ASR system directly transduces input utterance to a sequence of readable tokens. Among the E2E approaches(CTC \cite{graves2006connectionist}, LAS \cite{chan2016listen}, RNN-T \cite{graves2012sequence}), RNN-T has a few advances. First, it has great streaming nature compared to LAS models. Second, it has internal language models compared to CTC models, which learn the acoustic and language features jointly. Last but not least, RNN-T can recognize very long utterance stably \cite{chiu2019comparison}. Recently, Google has successfully deployed RNN-T on their mobile devices for streaming ASR \cite{he2019streaming}, which greatly impacts both the academic and industrial areas.

However, training RNN-T from scratch is still a challenging issue and is actively studied in recent years \cite{irie2019choice,chai2019investigation,sainath2020streaming,mohamed2019transformers,huang2020conv}. Rao et al. revealed that RNN-T system converges better with an encoder network pre-trained with CTC and predictor network pre-trained with language model \cite{rao2017exploring}. Tian et al. improved RNN-T performance by adding alignment constraints to RNN-T's loss function \cite{tian2020synchronous}. Hu et al. showed RNN-T's encoder network with CE pre-train performs better than other pre-train methods \cite{hu2020exploring}. These systems still have a long pipeline due to the need for pre-training. Most recently, advanced networks such as Contextnet\cite{han2020contextnet} and Conformer\cite{gulati2020conformer} have been introduced to RNN-T systems, which achieve SOTA performance on Librispeech benchmark. However, the detail of training is not yet well explained. We further exploited the possibility of leveraging these networks in the RNN-T system without the requirement for pre-training.

\label{sec:rnnt}
\begin{figure}
\centering
\centerline{\includegraphics[width=0.7\columnwidth]{final_rnnt.png}}
\caption{Our RNN-T baseline system consists of a masked conformer encoder network, a transformer-XL predictor, and a proposed normalized jointer network. }
\label{fig1}
\end{figure}

\begin{figure}
\centering
\centerline{\includegraphics[width=0.8\columnwidth]{jointer.png}}
\caption{The validation loss curve with our proposed normalized jointer network.}
\label{fig2}
\end{figure}

In this work, we observed a huge gradient variance during RNN-T training and suspect this prevents RNN-T from fast convergence. We analyze the cause of the huge variance in RNN-T training. We find the issue arises during the back-propagation from the jointer to predictor and encoder networks. For the encoder network, the gradient variance is amplified for \(U\) (transcription length) times. While the gradient variance for predictor is amplified for \(T\) (acoustic length) times. This makes the predictor and encoder hard to be optimized. To address the issue, we propose the \textit{normalized jointer network} which applies normalization for gradients at both encoder and predictor by a factor of \(U\) and \(T\) respectively. The experiment shows that the approach improved the performance and accelerated the convergence of the training procedure. Furthermore, we propose to use a masked Conformer as the and encoder network and the Transformer-XL \cite{dai2019transformer} as the predictor network to further improve the performance of the RNN-T system.

With all the proposed improvements, we achieve SOTA in the Chinese AIShell-1 dataset. In the AISHELL-1 dataset,  we get CER 6.15\% and 5.37\% for streaming and non-streaming recognition respectively. We also compared our improved RNN-T system with an internal well-trained hybrid system based on 30000-hour-industry data. The improved RNN-T system outperforms the hybrid system even without external language models.

The contribution of this work is threefold. First, we observe the huge gradient variance issue and propose a normalized jointer network that addresses the issue. Second, we further exploit the potential RNN-T system with a proposed masked conformer encoder network and Transformer-XL predictor network. Third, the experiment result outperforms the SOTA of AISHELL-1 with the proposed RNN-T network.

\section{RNN-T System Description}
Figure \ref{fig1} shows the diagram of the RNN-T model. The encoder network is analogous to the encoder networks in LAS \cite{chan2016listen}, or the acoustic model in hybrid system. Usually the input to encoder network is mel-fbank features \(X=[x_1,x_2,...,x_t,...,x_T]\), the encoder network converts the mel-fbank features to the high-level representations \(H^{enc}=[h_1^{enc},h_2^{enc},...,h_t^{enc},...,h_T^{enc}]\). The encoder network can be modeled by LSTM, CNN, DFSMN \cite{zhang2018deep} or transformer networks \cite{NIPS2017_7181}. Lots of works have done to improve the performance of ASR system by modifying the encoder network \cite{han2020contextnet, gulati2020conformer}. The predictor network plays the role of language model for RNN-T system. The non-blank tokens \(Y=[y_1,y_2,...,y_u,...,y_U]\) are encoded by predictor to high-level representations \(h^{pre}=[h_1^{pre},h_2^{pre},...,h_u^{pre},...,h_U^{pre}]\). The predictor networks are normally LSTM, causal CNN or causal transformer networks \cite{NIPS2017_7181}. The jointer network combines the high representation from encoder and predictor network as follows
\begin{equation}
\label{eq1}
    z(t,u)=FC(tanh(h_t^{enc} + h_u^{pre}))
\end{equation}
where FC(.) is the fully connected network. The posterior  \(p(t,u)\) can be obtained by applying softmax function to \(z(t,u)\)
\begin{equation}
\label{eq2}
    p(t,u)=softmax(z(t,u))
\end{equation}

and the RNN-T loss is defined as
\begin{equation}
\label{eq3}
    L=\sum_{(t,u):t+u=T+U}{\alpha(t,u)\cdot\beta(t,u)}
\end{equation}
where
\[\alpha(t,u)=\alpha(t-1,u)\cdot p^{blk}(t,u)+\alpha(t,u-1)\cdot p^{y_{u-1}}(t,u-1)\]
\[\beta(t,u)=\beta(t+1,u)\cdot p^{blk}(t,u)+\beta(t,u+1)\cdot p^{y_u}(t,u)\]
Since \(\alpha(t,u)\) and \(\beta(t,u)\) can be effectively computed by forward-backward algorithm \cite{graves2012sequence}, the whole system can be trained in an end to end manner.

\section{Improvement to RNN-T System}
In this section, three improvements to our RNN-T system are detail explained.

\subsection{Masked Conformer Encoder}
\label{ssec:masked_conformer}
Our RNN-T baseline adopts conformer \cite{gulati2020conformer} network as the encoder. The conformer network adds convolution to SAN's  \cite{NIPS2017_7181} self-attention part. We modify the conformer network by adding mask to self-attention part. The self-attention output at index \(i\) can be calculated as Eq.\ref{eq4}. For more details of self-attention mechanism, please refer to \cite{NIPS2017_7181}.
\begin{equation}
\label{eq4}
out_i = \sum_{l} {\left( \cfrac {\exp (\cfrac {Q_i \cdot K_l} {\sqrt {d_k}} ) \cdot M_{il} } { \sum_{j}  {\exp ( \cfrac {Q_i \cdot K_j}  {\sqrt {d_k}} )} \cdot M_{ij}}  \cdot V_l  \right)}
\end{equation}
This mask mechanism introduced in conformer has two advantages. First, adding mask to self-attention in conformer helps the convergence, especially when the training utterance is very long. We observed that the masked conformer always converges faster and are more stable than the original conformer networks. Second, with the mask introduced in self-attention, it is quite easy to change a non-streaming RNN-T system to a streaming fashion by masking out the right context of self attention part.

\subsection{Transformer-XL Predictor}
\label{ssec:transformerxl_predictor}
The RNN-T predictor networks play the role of the neural language model and usually adopts LSTM or native causal transformer network. However, as far as we can see, transformer-XL \cite{dai2019transformer} performs the best in the language model domain. The transformer-XL contains a segment-level recurrence mechanism, which maintains an extra-long context. Meanwhile, transformer-XL proposed a novel positional encoding scheme that adapts the sinusoid formulation in the relative positional embedding. It helps the model generalize to a longer length during evaluation. So we choose transformer-XL as our RNN-T's predictor networks and find it works better than the LSTM or transformer predictor networks. As the training corpus size increases, the RNN-T system with the transformer-XL predictor network will results in much better performance than LSTM or native transformer predictor network.

\subsection{Normalized Jointer}
\label{ssec:normalized_jointer}
The jointer network composes the hidden feature from encoder and predictor network in Eq.\ref{eq1}. In the forward progress computation, the jointer network representation \(z(t,u)\) has the following relationship with \(h_t^{enc}\) and \(h_u^{pre}\)
\begin{equation}
\label{eq5}
    z(t,:) = FC(tanh((h_t^{enc}+h_{:}^{pre}))
\end{equation}
\begin{equation}
\label{eq6}
    z(:,u) = FC(tanh((h_{:}^{pre}+h_u^{pre}))
\end{equation}
In the backward progress, gradient of \(dh_t^{enc}\) and \(dh_u^{pre}\) will have the following relationship with jointer network's gradient \(dz(t,u)\)
\begin{equation}
\label{eq7}
    dh_t^{enc}=\sum_{u=1}^{u=U}{dz(t,u)}
\end{equation}
\begin{equation}
\label{eq8}
    dh_u^{pre}=\sum_{t=1}^{t=T}{dz(t,u)}
\end{equation}

Let the gradient of \(dz(t,u)\) be a random variable with mean \(\mu\) and variance \(\sigma\). Then the variance of \(dh_t^{enc}\) is \(U\cdot\sigma\), and the variance of \(dh_u^{pre}\) is \(T\cdot\sigma\).  The huge variance of the gradient to RNN-T's encoder and predictor network will cause the unhealthy optimization of parameters, especially when the training batch utterances have different acoustic length (T) and different transcript length (U) in a mini-batch. To overcome this problem, we apply a simple method by dividing the \(dh_t^{enc}\) with \(U\) and \(dh_u^{pre}\) with \(T\). As we can see in Eq.\ref{eq9} and Eq.\ref{eq10},

\begin{equation}
\label{eq9}
    \hat{dh_t^{enc}}=\frac{dh_t^{enc}}{U}
\end{equation}
\begin{equation}
\label{eq10}
    \hat{dh_u^{pre}}=\frac{dh_u^{pre}}{T}
\end{equation}

with the above modification to the gradient from RNN-T's jointer network to encoder and predictor network, the gradient norm of RNN-T's training becomes more stable. And the validation loss decreases faster and lower as illustrated in Figure\ref{fig2}.

\section{Experiments}
In this section, we conduct two mandarin speech recognition tasks, named AISHELL-1 task and 30000-hour-industry task. In the AISHELL-1 task, we carry out our experiments on the 170-hour AISHELL-1 dataset \cite{bu2017aishell}. In the 30000-hour-industry task, we collect 30,000 hours of audio data from various acoustic domains and content domains.   The acoustic domain contains utterances collected from fields such as \textit{near field record}, \textit{far field record}, and \textit{telephone record}. The content are from domains such as \textit{news}, \textit{meeting}, \textit{recreation}, \textit{sports}, \textit{tourism}, \textit{game}, \textit{literature}, \textit{education}. We adopt raw waveform as our input and compute the 80-dimensional energy-based log-mel filter-banks (FBK) on-the-fly. The filter-banks are computed on a window of 25ms and 10ms shift. All our encoder networks have VGG blocks with two convolutions and down-sampling the FBK acoustic feature to a frame rate of 40ms. All our experiments are carried with Pytorch and our RNN-T system is trained from scratch without any auxiliary loss or pre-train methods, which greatly simplify the pipeline of this system.

\begin{table}[t]
\small
  \centering
  \label{table1}
  \begin{tabular}{ccccc}
    \toprule
    \multicolumn{1}{c}{\textbf{ID}} &
    \multicolumn{1}{c}{\textbf{Model}} &
    \multicolumn{1}{c}{\textbf{Latency}} &
    \multicolumn{1}{c}{\textbf{Param.}}  &
    \multicolumn{1}{c}{\textbf{Testset}}\\
    \midrule
    -&Path-Aware SA-T \cite{tian2019self} &inf&  - & 9.30  \\
    \midrule
    -&SAN-M \cite{gao2020san} &  inf&43M & 6.46\\
    \midrule
    B0&baseline& inf&61M & 6.35\\
    \midrule
    E1&+Transformer-XL& inf & 46M &  6.18\\
    \midrule
    E2&+Masked Conformer & inf& 46M &  6.09\\
    \midrule
    E3& + Normalized Jointer&inf & 46M &  \textbf{5.91}\\
    \midrule
    E4&+ Large &  inf &110M&  \textbf{5.37}\\
    \midrule
    \midrule
    -&Sync-Transformer \cite{tian2020synchronous} & 400ms & - &  8.91\\
    \midrule
    -&SCAMA \cite{zhang2020streaming} & 600ms & 43M & 7.39\\
    \midrule
    E5& Streaming Base & 400ms & 46M &  \textbf{6.83}\\
    \midrule
    E6& Streaming Large & 400ms & 110M & \textbf{6.15}\\
    \midrule
  \end{tabular}
  \caption{Experiment results on AISHELL-1 task.}
\end{table}

\subsection{AISHELL-1 Task}
We first evaluate the performance of our RNN-T system on the AISHELL-1 benchmark. We build our baseline RNN-T system with 12 layers of default conformer encoder \cite{gulati2020conformer}, and set multi-head number as 4, dimension as 256. The fully connected hidden size is 2048. The predictor network is a 1 layer LSTM with cell size 2048. The jointer network has a dimension of 768, while the output vocabulary is 4234. With this configuration, the baseline RNN-T system has 61M parameters (\textit{B0}).  We use Adam \cite{kingma2014adam} as the optimizer with \(\beta_1=0.9\), \(\beta_2=0.999\). The learning rate is set to rise linearly from 1e-7 to 5e-4 in the first 15,000 steps, then exponentially decay it in the rest of our experiments. Our RNN-T system is trained on the AISHELL-1 task with 128 utterances per mini-batch, the model is stopped after 150,000 iterations. The dropout is set to 0.1 in all the experiments. The SpecAugment \cite{park2019specaugment} is adopt as default. All the results are reported without any external language model.

As shown in Table 1, our baseline RNN-T system (\textit{B0}) gets CER 6.35\% on the AISHELL-1 testset. It outperforms the previous state-of-the-art benchmark achieved by SAN-M \cite{gao2020san}. After replacing the LSTM predictor network in \textit{B0} with the transformer-XL predictor, the CER decreases to 6.18\% with this modification. The parameters of our RNN-T system are reduced from 61M to 46M (\textit{E1}). Then, we improved our RNN-T system by replacing the default conformer encoder network with the proposed masked conformer encoder network, where each layer has a mask with 40 left contexts and 40 right contexts. With masked conformer encoder network, the CER drops to 6.09\% on testset (\textit{E2}). In \textit{E3}, we adopt our proposed normalized jointer network to \textit{E2} and observe a signification CER reduction from 6.09\% to 5.91\%. Figure \ref{fig2} presents our proposed jointer network helps the RNN-T system converging faster and better. In \textit{E4}, we further increase our RNN-T's model complexity by set the multi-head to 8, and dimension to 512. With this bigger RNN-T model (110M parameters), we get CER 5.37\% on the AISHELL-1 testset. It further improves the state-of-the-art benchmark.

For the streaming RNN-T system, we simply set the right context of mask in the first 10 encoder layers to 1, and 0 in the remaining layers. In this setting, the streaming RNN-T System has a 400ms latency. We denote it \textit{Streaming Base}. This system gets CER 6.83\% on the AISHELL-1 testset, which again outperforms the previous state-of-the-art benchmark on AISHELL-1 (\textit{E5}). We further conduct our streaming RNN-T experiment with our 110M parameters system, denoted as \textit{Streaming Large}. It gets CER 6.15\% on the AISHELL-1 testset, which again improves the streaming benchmark on AISHELL-1 (\textit{E6}).

\subsection{30000-hour-industry Task}
In this task, we carry out our experiments on our internal 30,000 hours dataset. There are 12 testsets in this task. Let the averaged CER of the 8 near field tests be \textit{Testset Near}, the averaged CER of the 4 far-field test be \textit{Testset Far}. We compare three types of systems, including \textit{DFSMN-CTC}, \textit{Streaming Base} and \textit{Streaming Large}. For the \textit{DFSMN-CTC} hybrid system, we train DFSMN-CTC systems with 50 DFSMN-layers \cite{zhang2018acoustic}. The language model(LM) used in our hybrid system is a 5-gram language model. For the RNN-T system, we train the system in an end-to-end manner with a vocabulary size of 13,000, which has 7000 Chinese chars and 6000 English BPE subwords \cite{sennrich2015neural}.

As the results shown in Table 2, the \textit{StreamingBase} can get comparable results to our well-trained hybrid baseline system both on \textit{Testset Near} and \textit{Testset Far} with CER 6.80\% and 14.13\%, respectively. Note that our \textit{Streaming Base} has only 46M parameters and does not need any external language models. It has less footprint to deploy this system on devices comparing to the hybrid system. Our \textit{Streaming Large} further improves the performance and gets CER 6.14\% and 12.70\%d on \textit{Testset Near} and  \textit{Testset Far}, respectively. With the same model complexity, our RNN-T system outperforms our hybrid system by relative 9.0\% and 9.2\% improvement on \textit{Testset Near} and \textit{Testset Far}, respectively.

\begin{table}[t]
  \centering
  \small
  \label{table2}
  \begin{tabular}{cccc}
    \toprule
    \multicolumn{1}{c}{\textbf{Model}} &
    \multicolumn{1}{c}{\textbf{Latency}}  &
    \multicolumn{1}{c}{\textbf{Param.}}  &
    \multicolumn{1}{c}{\textbf{Test Near/Far}}\\
    \midrule
    DFSMN-CTC  & 400ms & 110M & 6.75/13.98  \\
    \midrule
   Streaming Base & 400ms & 46M &  6.80/14.13  \\
    \midrule
    Streaming Large & 400ms & 110M & 6.14/12.70  \\
    \midrule
  \end{tabular}
  \caption{Experiment results on 30000-hour-industry task.}
\end{table}

\section{Conclusion}
In this paper, we observed the issue of huge gradient variance during RNN-T training. To address the issue, we propose a normalized jointer network that performs stable convergence during training. We also improved our RNN-T system with a masked conformer encoder network and a transformer-XL predictor network. On the AISHELL-1 dataset, our proposed RNN-T system achieves CER of 6.15\% and 5.37\% for streaming and non-streaming speech recognition without external language models. We further compared our RNN-T system to an internal well-trained hybrid system on a 30000-hour-industry dataset and get 9\% relative improvement.

\bibliographystyle{IEEEbib}