\section{Introduction}
Scene text detection in the wild becomes a hot research topic of computer vision for its various application in document image analysis, scene understanding, and autonomous driving, etc. Recently, scene text detection community focuses on arbitrary shape text, and many profound approaches have been proposed.

\begin{figure}[ht]
  \centering

  \subfloat[Text rectification and representation based on TPS transform.]{\includegraphics[width=\linewidth]{figures/f1a_1.pdf}}\hspace{3pt}

    \subfloat[Cheby]{\includegraphics[width=1.03in]{figures/ray_fit.jpg}}\hspace{2pt}
  \subfloat[Fourier]{\includegraphics[width=1.03in]{figures/fce_fit.jpg}}\hspace{2pt}
  \subfloat[TPS]{\includegraphics[width=1.03in]{figures/tps_fit.jpg}}

  \caption{Text shape representation methods. (a) shows that our proposed TPS representation can be seen as the reverse process of text rectification. Bottom is the text shape fitting results of (b) Cheby  \cite{Wang2020textray}, (c) Fourier  \cite{zhu2021fourier} and (d) our proposed TPS. The red lines are the fitting curves and the green lines are the ground truth. The dimensions of the parameters are 44, 22 and 22 respectively. The Cheby fails on highly-curve shapes and both of Cheby and Fourier miss partial corner pixels on extreme aspect ratio cases.}
  \label{fig:intro}
\end{figure}

Arbitrary shape text detection methods can be classified into two types: segmentation based and regression based. Segmentation based  \cite{wang2019PSENet,liao2020db} methods represent text regions with pixel-level classification masks that are not constrained by the shape of the text, but they have drawbacks such as computationally intensive post-processing and the lack of noise resistance. The regression based methods regress the text boundaries directly, making the prediction process much simpler. For horizontal and multi-oriented straight text, regressing the quadrilateral is sufficient to represent the text shape  \cite{zhou2017east,liao2018textboxes++}. However, complex representations must be designed for arbitrary shape texts. Apart from directly increasing the number of points to represent text contours  \cite{wang2019ATRR,dai2021progressive}, some methods apply parametric curves to fit the text boundaries  \cite{Liu2020ABCNet, Wang2020textray, zhu2021fourier},  resulting in tighter and smoother contour curves.

In our opinion, a good representation for arbitrary shape text should: (1) be compact and complete enough so that the presence of background pixels and the missing of text pixels are as less as possible; (2) be integral and not easy to be confused with other instances; (3) be effective in both detection and recognition. Although there are various representation methods for text, none of them can satisfy all the requirements.
As is shown in Fig. \ref{fig:intro}, TextRay \cite{Wang2020textray} is not compact for the highly-curved text, FCE \cite{zhu2021fourier} will miss partial corner pixels of the long text, and both of them can not beneficial for subsequent recognition directly. The Bezier curve representation \cite{Liu2020ABCNet} meets the first and the third requirements, but it takes two curves to represent the text's upper and lower boundaries separately which is not integral, thus may be confused with other nearby instances.

To solve the problems that previous representations encountered, we propose a better arbitrary shape text representation: Thin-Plate-Spline (TPS) representation. TPS transformation \cite{bookstein1989principal} is typically applied in scene text recognition for rectification  \cite{shi2016robust, shi2018aster, zhan2019esir,yang2019symmetry, shang2020character}, where the irregular text region is rectified to the horizontal regular region so that the classical simple methods like CRNN  \cite{crnn} can recognize it well. Though TPS is effective in scene text recognition, it has not been applied in scene text detection to the best of our knowledge. If we think the usage of TPS reversely, as shown in Fig. \ref{fig:intro} (a), it can be a novel and simple representation for detection.
While rectifying text in recognition, the TPS parameters are solved based on the corresponding control points, with which the source shape can be transformed into the target rectangular shape. Considering the reverse process of rectification, if we get the TPS parameters, every points on the target rectangle can be transformed to the points of source arbitrary shape. Since the target rectangle can be fixed, the TPS parameters can be taken as a good text representation and meet all the three criteria for better representation. As the TPS representation takes the rectangle as the basic shape of text, it is adaptive to the characteristics of large aspect ratios and right angles in the corners, treats the text as an integral shape, and can be applied to the recognition straightforwardly, as shown in Fig. \ref{fig:intro}.

The standard approach for calculating TPS parameters is to use the corresponding control points on the source and target shapes. However, even if we have text boundary annotations, it is not easy to obtain control points from them, because any rule that selects a small number of control points is not appropriate for all shapes. That means that trying to use control points as a representation of shapes is not reliable,
so we choose to predict the TPS parameters. Without control points, the ground truth of the TPS parameters is not available, so we propose the boundary set loss and shape alignment loss as supervisions. The text's boundary points are inferred from the TPS parameters, and the boundary set loss is calculated between the predicted boundary set and the ground truth boundary set. Furthermore, we apply the TPS parameters to rectify the text ground truth mask to a horizontal rectangle, the shape alignment loss is obtained by the difference between the rectified rectangle and the fiducial rectangle. With these two supervisions, the prediction of TPS parameters can be optimized robustly.
\begin{figure*}[t]
  \centering
  \includegraphics[width=0.8\linewidth]{figures/f2.pdf}
  \caption{The architecture of the proposed TPSNet. The multi-level feature maps are extracted from image by Backbone and FPN. Then the detection head  $3\times3$ convolution to classify the text region and text center, and regresses the TPS parameter maps. The text region and text center prediction will be used as score in NMS. The TPS parameters are transformed into text shape in TPS Decoder, where the fiducial shape has been pre-defined. With the text shape,
 not only can text boundaries be obtained, but irregular text can also be rectified.}
 \label{fig:tpsnet}
\end{figure*}

The contributions of this work are summarized as follows:
\begin{itemize}
\item An exquisite representation - TPS representation is first proposed for arbitrary shape text detection to our best knowledge. The inspiration is from the extensively used TPS transformation in scene text recognition and a sophisticated reverse thinking. It is compact, complete, integral and can be reused in recognition directly.

\item To address the ambiguity of the boundary annotation and improve the supervision of the text shape, we design the boundary set loss and the shape alignment loss to ensure that the network can converge effectively and correctly.

\item TPSNet equipped with TPS representation, boundary set loss and shape alignment loss is presented, and is evaluated on two arbitrary shape text detection benchmarks. The performance is superior to previous counterparts.

\end{itemize}

\section{Related Works }

\subsection{Scene Text Representation}
As a special kind of object, the scene text needs to be represented appropriately for accurate detection. Segmentation mask is the common representation and has been widely used \cite{deng2018pixellink, wang2019PSENet, lyu2018masktextspotter,liu2019arbitrarily, liao2020db,ye2020textfusenet, xiao2020SD}. The mask can naturally represent arbitrary shape text, but it has the limitation of confusing different text instances. Most of segmentation-based methods try to address this problem by linking pixels belong to the same text  \cite{deng2018pixellink, liu2019CSE,liu2018MCN}  or distinguishing pixels of different texts  \cite{tian2019SAE, wang2019PAN}. Some methods convert the binary mask to continuous masks to represent more information  \cite{xue2019msr, xu2019textfield, zhu2021textmountain}. Although it is easy to represent arbitrary-shape text with segmentation mask, it suffers from the lack of noise resistance and computationally intensive post-processing. Some methods represent the text with a set of text components \cite{tian2016CTPN, shi2017SegLink, tang2019seglink++, zhang2020DRRG}, which also belongs to segmentation-based methods but the units are text blocks rather than pixels.

Regressing the geometry of the text shape and position is another kind of representation. For the horizontal and multi-oriented straight text, a rectangle  \cite{zhang2016multi, zhou2017east, liao2017textboxes,he2017deep,liu2017deep, he2021most, wang2018ITN} or quadrilateral \cite{liao2018rotation, liao2018textboxes++, xue2018border, lyu2018corner} is sufficient.
When it comes to curve texts, the representation becomes complicated. TextSnake  \cite{long2018textsnake} regresses the radius along the text center line, which is similar to segmentation-based methods.  \cite{wang2019ATRR} employs LSTM to predict varying number of boundary points for different texts, while PCR  \cite{dai2021progressive} progressively refines the contour points iteratively.
Directly representing contours with points is inefficient; too many points are redundant, while too few points can not describe complicated shapes, making text contour unsmooth. To address this issue, parameter curves have been applied for better representation. ABCNet  \cite{Liu2020ABCNet} formulates the long sides of the text with two Bezier curves to get compact border fitting, but the text shape is not regressed integrally, causing confusing with adjacent texts. TextRay  \cite{Wang2020textray} converts the coordinates of boundary points from Cartesian  coordinate system to polar one, and then employs the Chebyshev polynomials to approximate the boundary. Since text instances usually have large aspect ratios, the distribution of sampling points are not homogeneous on the boundary
, making it hard for TextRay to fit the long and highly curved texts.
FCENet \cite{zhu2021fourier} adopts the trigonometric series or fourier transform to fit the text boundaries more compactly and simply. However, although the fourier curve is an excellent fitter, it struggles to fit text at right angle corners with relatively fewer parameters, resulting in incomplete characters.

Our proposed TPS representation is fundamentally different from previous representation. TPS parameters represent the text by transforming a rectangular shape to the text shape, taking the text as in integral, which is able to represent arbitrary shape and naturally adapt to the corners and large aspect ratios of the text.

\subsection{Text Rectification }
Rectification is a pre-processing module for irregular text recognition
\cite{shi2016robust,liu2018charnet,zhan2019esir, yang2019symmetry, shi2019aster, luo2019moran,yunze2020progressive,gong2021unattached,lin2021stan,zhang2021spin}.
The rectification methods are mainly based on the affine transformation \cite{liu2016star, liu2018charnet,lin2021stan} or TPS transformation \cite{shi2016robust, shi2018aster, zhan2019esir, shang2020character}.   \cite{luo2019moran, zhang2021spin} predict offset maps to shift characters to the horizontal line. Most of them optimize the transformation parameters learning in weak supervision manners, while the character position labels help the network predict better transformation  \cite{yang2019symmetry}. The strategy of iteration is also applied for complex shape rectification \cite{zhan2019esir, dai2021progressive}.

Since the rectification requires the geometry and localization information, it will be more effective if this process is conducted by text detection model.
TextSnake \cite{long2018textsnake} exploits local geometries to sketch the structure of the text instance and transforms the predictable curved text instances into canonical form, but the transformation based on local geometries is discontinuous and causes distortion.
ABCNet \cite{Liu2020ABCNet} utilizes the Beizer curve to formulate the text shape and obtains the text region with BeizerAlign, which meanwhile rectifies the curve text.
\section{Methodology}
In this section, we first introduce the TPS representation for arbitrary shape text. Then the boundary set loss and shape alignment loss are presented in detail. Finally, the TPSNet based on the representation and loss functions is described.
\subsection{Thin-Plate-Spline Representation}

Unlike generic objects, text is a special kind of object. If a word or a text line is taken as an instance, its shape should be a rectangle in a standard case such as in plain documents. In scene images, texts emerge in various shapes containing bends and twists, but these shapes are still deformed from the basic rectangle one, mostly retaining the characteristics of right angle corners and large aspect ratios. From the perspective of deformation, we try to establish a mapping between arbitrary text shapes to regular rectangles, thus enabling arbitrary shape text representations.

TPS has been widely used as the non-rigid transformation model in image alignment and shape matching. We apply TPS as the basic model to implement the deformation from an arbitrary shape to a regular rectangle, which requires to obtain the correspondence of each point from the rectangle to arbitrary shape. We formulate the target shape as the rectangle $A$ and the source text shape as $S$, and the rectangle $A$ is also called fiducial shape. According to TPS  \cite{bookstein1989principal}, the corresponding point of $(x,y) \in A$ on $S$ can be calculated by
\begin{equation}
\begin{array}{l}
    x' = \Phi_x(x,y) = c_x + a_{1_x} x + a_{2_x} y + \sum_{i=1}^{k}w_{i_x} r(d_i) \\
    y' = \Phi_y(x,y) = c_y + a_{1_y} x + a_{2_y} y + \sum_{i=1}^{k}w_{i_y} r(d_i) \\
\end{array}
\label{equ:1}
\end{equation}
where $r$ is the radial basis function
\begin{equation}
    r(d) = \left\{
    \begin{array}{l}
    0, if\ d = 0\\
    d^2ln d, otherwise
    \end{array}
    \right.
\end{equation}

$d_i=||(x,y)-(x_i,y_i)||_2$ is the distance from $(x,y)$ to $(x_i,y_i)$. $\{(x_1,y_1), (x_2,y_2),..., (x_k,y_k)\}$ are the fixed points on the fiducial shape A, called fiducial points, and $k$ is the number of fiducial points. Given fiducial points, the basis function is defined as
\begin{equation}
    \mathbf{\phi}(x,y) = \left [\begin{array}{c}
    1 \quad x \quad y \quad r(d_1) \quad ... \quad r(d_k)
   \end{array} \right] ^ \mathrm{ T }
\end{equation}
then the TPS transform function is determined by the parameters \begin{equation}
    \mathbf{T} = \left [
    \begin{array}{cccccc}
         c_x & a_{1_x} & a_{2_x}  & w_{1_x} & ... &  w_{k_x}\\
         c_y & a_{1_y} & a_{2_y} &  w_{1_y} & ... &  w_{k_y}
    \end{array}
    \right ]
\end{equation} with the shape of $2\times(k+3)$. Any fiducial point set can define a basis function, so it is specified as equidistant points on the boundary of the rectangle $A$ for simplicity; the number of fiducial points $k$ is 8, and the dimension of TPS parameters $\mathbf{T}$ is 22, which is sufficient for the deformation of arbitrary texts. The scale and aspect ratio of fiducial shape $A$ have an great impact on the results, and we will conduct experiments to find the best setting.

With the TPS transform function,

the grids on $A$ can be  transformed into the corresponding points on $S$, where the text boundaries are naturally obtained,
as shown in Fig. \ref{fig:tpsnet}. Note that, the grids on $A$ are  predefined, so the $\phi(x,y)$ is also calculated in advance. The TPS parameters $\mathbf{T}$ can be decoded quickly to the text shape with a matrix multiplication $(x',y')^T =\textbf{T}\phi(x,y)  $.

We emphasize that, unlike previous rectification methods, we do not attempt to predict control points on the text shape $S$ because the locations of control points on arbitrary shapes lack strict definition, and we directly predict the TPS parameters with the neural network instead.
\begin{table}[t]
\setlength\tabcolsep{3pt}

\centering
\begin{tabularx}{\linewidth}{YYYYY}
\toprule
Method    & TIOU      & Integrate & Corner & Rectify \\ \midrule
Cheby   & 91.4 (44) & \checkmark      & $\times$    & $\times$    \\
Fourier   & 90.6 (22) & \checkmark      & $\times$    & $\times$    \\
Beizer    & 96.8 (16) & $\times$      & \checkmark   & \checkmark    \\
TPS(Ours) & 96.3 (22) & \checkmark      & \checkmark   & \checkmark    \\ \bottomrule
\end{tabularx}
\caption{Comparison of text shape representation. TIOU \cite{liu2019tightness} means the tightness-IOU between the fitted shapes and ground truth, and the number in the brackets are the dimensions of the parameters. Integrate, Corner and Rectify means whether the representation is integral, able to keep the corner and rectify irregular text for recognition.}

\label{tab:fittingmeasure}
\end{table}
To verify the fitting ability of TPS, we directly predict the TPS parameters by taking equidistant sampling points on the annotated text boundaries of shape $S$ as control points (This strategy will not be used for network training in experimental evaluation). The text boundary can be derived with equation (\ref{equ:1}), and then evaluated the
Tightness-IOU  \cite{liu2019tightness} comparing with the ground truth annotations, where the value reveals the compactness and the completeness of the fitting.
The fitting results of four typical representations are demonstrated on Table \ref{tab:fittingmeasure}, and the visualization of the shape fitting is shown in the Fig.\ref{fig:intro}. The Cheby fails on highly-curve shapes, and both of Cheby and Fourier fail on extreme aspect ratio cases and miss the text corner. The Beizer  \cite{Liu2020ABCNet} can reach the highest fitting performance, but it uses two separate curves to represent the text shape rather than take it as a whole.
It is apparent that our proposed TPS representation is most appropriate for arbitrary shape text shape with the characteristics of large aspect ratios and right angle corners.

\subsection{Shape Losses}
\subsubsection{Boundary Set Loss}

As mentioned by  \cite{Wang2020textray}, compared to directly minimize the distances between TPS parameters $\mathbf{T}$ and its ground truth in parameter space, designing the loss function in geometry space is more efficient. In other words, the TPS parameters should be decoded into the shape first, and the loss can be calculated as the distance of boundary point pairs between the decoded shape and its ground truth.

This distance can be simply defined as the distance of matching point pairs, which aligns every predicted points to the ground truth points. Nevertheless, due to the variety of the shape and ambiguity of the annotation, it is hard to define the target position. Additionally, some researches declare that fixed matching between regressed points and annotations is not optimal for network optimization  \cite{Yang2020DenseRR, wei2020point}.

Based on the consideration above, we propose the boundary set loss as shown in Fig.\ref{fig:loss}. We first decode the TPS parameters into text shape, and naturally get the predicted boundary points. Then we take each boundary as a point set, the distance from predicted point set and ground truth point set is estimated by Chamfer distance  \cite{fan2017point}. The boundary set loss is formulated as
\begin{equation}
\small
\begin{aligned}
    L_{boundary} =-\sum_{B\in \{l,r,b,t\}} ( \frac{1}{2|B|} \sum_{i\in B} \min_{j,j\in B_{gt}} ||(x_i,y_i)-(x'_j,y'_j)||_2\\
    +\frac{1}{2|B|} \sum_{j\in B_{gt}} \min_{i,i\in B} ||(x_i,y_i)-(x'_j,y'_j)||_2 )
\end{aligned}
\end{equation}

where $(x_i,y_i)$ is the point on predicted boundary $B$, and  $(x'_j,y'_j)$ is the point on the ground-truth boundary $B_{gt}$; $l,r,b$ and $t$ denote the left, right, bottom and top boundaries.

 The ground truth boundary points are sampled equidistantly and densely from the sparse annotation points.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.9\linewidth]{figures/f3_loss_1.pdf}
  \caption{ Illustrations of the boundary set loss and shape alignment loss. Only the top boundary is demonstrated here.}
  \label{fig:loss}
\end{figure}

\subsubsection{Shape Alignment Loss}

To further supervise the shape regression, we propose shape alignment loss, which utilizes the interpolation capability of TPS representation. IOU loss \cite{yu2016unitbox} is popular in object detection and its essence is to calculate the global match of the shape. However, The IOU loss can only be easily applied for regular shape regression, and for arbitrary shape, there is not convenient or differentiable algorithm to calculate the areas. TPS representation provides an alternative to solve this problem. As the text rectification process, we apply the TPS parameters to rectify the text ground truth masks. For every points on the text shape grids, we use the bilinear interpolation to get value on text ground truth mask to get the aligned mask. If the TPS parameters are totally right, there will be a perfect rectification and the aligned text mask should be rectangular, so we set this retangular mask as the target mask. The difference between the aligned and target mask is calculated by MSE loss, as shown in Fig.\ref{fig:loss}. The shape alignment loss is formulated as
\begin{equation}
    L_{shape} =- \frac{1}{|m_{gt}|}\sum_{pixels}  L2(M_p,M_t)
\end{equation}

where $M_p$ is the shape alignment mask, $M_t$ is the default target mask and $m_{gt}$ is the original text ground truth mask. The L2 distance between two masks is divided by the area of the original text mask for balance because the text mask has been cropped and resized to the same scale and irregular curve shape will cover less points getting a smaller area than regular shape.
As the $M_p$ is generated by bilinear interpolation, which is differentiable, and the loss can be propagated to the TPS parameters and contribute to the optimization. To get more gradients at the mask boder, we smooth the text ground truth mask with average pooling.

The prediction of TPS parameter are optimized by boundary set loss and shape alignment loss together, and the regression loss $L_{reg}$ are given by:
\begin{equation}
    L_{reg} = L_{boundary} + \lambda L_{shape}
\end{equation}
where $\lambda$ is the weight parameter.
\subsection{TPSNet}
Equipped with TPS representation, we propose our one-stage network TPSNet for arbitrary shape text detection.

Following previous regression-based text detection network  \cite{zhou2017east, Wang2020textray, zhu2021fourier}, we adopt a compact one-stage architecture, which consists of a backbone, a detection head, and a TPS decoder, as shown in Fig. \ref{fig:tpsnet}. The backbone followed by Feature Pyramid Network (FPN) is applied to extract multi-scale features. The multi-level feature maps are passed into the detection head, which consists of a classification branch and a regression branch. The classification branch predict the per-pixel masks of text region and text center, and these two masks are multiplied as the confidence of the detection at every position. In the regression branch, we directly predict the TPS parameters on each feature bin, and the confidence from the classification branch is used for Non-Maximum Suppression (NMS) to remove duplicated predictions.

In the TPS decoder, the predicted TPS parameters $\mathbf{T}$ are decoded to text shape with pre-defined fiducial shape and $\phi(x,y)$. The decoder process is defined as Equation (\ref{equ:1}). From the text shape, the text boundary can be obtained directly, and it is also convenient to rectify the text by sampling on the input image.

The optimization objectives of the classification branch and regression branch respectively are $L_{cls}$ and $L_{reg}$, and the whole TPSNet is optimized by:
\begin{equation}
    L = L_{cls} + L_{reg}
\end{equation}
The classification loss consists of the Text Region loss $L_{TR}$ and Text Center Region loss $L_{TCR}$:
\begin{equation}
    L_{cls} = L_{TR} + L_{TCR}
\end{equation}
Both of $L_{TR}$ and $L_{TCR}$ are cross entropy loss. To solve the sample imbalance problem, OHEM is adopted for $L_{TR}$ with the ratio between negative and positive samples being 3 : 1.

\section{Experiments}
In this section, we evaluate our proposed TPSNet by CTW1500 and TotalText datasets to validate its effectiveness. We first conduct some ablation studies to demonstrate the advantages of proposed designs and the setting of hyper-parameters. Then we compare the detection performance of our model with previous state-of-the-art methods. Finally, the rectified text instance is recognized to show the representation's contribution to subsequent recognition.
\subsection{Datasets}
\noindent \textbf{SUCT-CTW1500.}
CTW1500 \cite{yuan2019ctw} is a dataset for curved text. It contains 1,000 training images and 500 test images. Text is represented by polygons with 14 points at text-line level.\\
\textbf{TotalText.}
TotalText  \cite{ch2017total} includes curved, horizontal, and multi-oriented text. It consists of 1, 255 training images and 300 test images. All text annotations are word-level. Similar to CTW1500, text areas are annotated with polygons.\\
\textbf{ICDAR2015.}
ICDAR2015\cite{karatzas2015icdar2015} is a multi-oriented text detection dataset only for English, which includes 1000 training images and 500 testing images. The text regions are annotated with quadrilaterals.\\
\textbf{SynthText.}
SynthText \cite{gupta2016synthetic} is a synthetically generated dataset composed of 800000 synthetic images. We use the dataset to pre-train our model.
\subsection{Implementation Details}
We implement our TPSNet based on MMOCR  \cite{mmocr2021} with Pytorch  \cite{paszke2019pytorch} library. The backbone is ResNet50 pretrained on ImageNet with DCN in stage 2, 3 and 4, followed by FPN. Feature maps of P3, P4 and P5 are used in classification and regression branch, where 4 $3\times3$ convolutional layers are applied for the text region and text center region classification and the TPS regression.
Text instances are assigned into different feature maps according to its scale ratio (instance scale/image scale), and the ranges are [0, 0.25], [0.2, 0.65] and [0.55, 1.0] for P3, P4 and P5 respectively.

The training images are resized to $800\times800$, and data augmentaion strategies are applied, including ColorJitter, RandomCrop, RandomRotate and RandomFlip. The training batch size is set to 8. Stochastic gradient descent (SGD) is adopted as optimizer with the weight decay of 0.001 and the momentum of 0.9. The initialized learning rate is 0.001, which is reduced by $0.9\times$ every 150 epochs, and the number of total training epochs is 1000. The weight of shape alignment loss is 0 before the 500th epoch and raises to 1.0 after that. Our TPSNet is directly trained on corresponding training set and not pretrained on any other dataset.

In the test stage, the long side of the test image is set to 1080, 1280 for CTW1500 and TotalText, while the short side is resized to keep the original aspect ratio.  All experiments are conducted on a single NVIDIA RTX3090 GPU.

\begin{table}[t]
\centering
\begin{tabularx}{1\linewidth}{@{}llcccX@{}}
\toprule
Target                          & Supervision              & R & P & F \\ \midrule
Control Points                  & $L1(P_{control})$  & 79.8     & 87.4      & 83.4      \\ \midrule
\multirow{4}{*}{TPS Parameters} & $L1({\mathbf{T}})$      & 78.4   & 87.0      & 82.4      \\
                                & $L1(P_{boundary})$ & 80.6   & 86.2      & 83.3      \\
                                & $L_{boundary}$          & 82.0     & 87.3      & 84.4      \\
                                & $L_{boundary}$+$L_{shape} $           & \textbf{83.3}   & \textbf{89.1}      & \textbf{86.1}      \\ \bottomrule
\end{tabularx}
\caption{Ablation study about regression target and supervision on CTW1500. L1 means the Smoothed-L1 loss; $P_{control}$ and $P_{boundary}$ means control points and boundary points; $\mathbf{T}$ is the TPS parameters.}.
\label{ablaloss}
\end{table}

\subsection{Ablation Study}
The ablation study is conducted on CTW1500, and all experiments are conducted under the same training and test settings without pretraining.

\subsubsection{Regression Target and Supervision} To verify the advantage of our design, we set different regression targets and supervisions. The alternative regression target is the control point, although it is not reliable as above discussion; the corresponding regression is smooth L1 of the coordinates of control points. For TPS parameters prediction, the available regression supervisions include:  1) L1 distance between predicted TPS parameters and its ground truth solved from equidistant control points, 2) distance of the boundary points that employ the simple point-to-point matching strategy and 3) the proposed boundary set loss and shape alignment loss. The evaluation results are shown in Table \ref{ablaloss}.

The boundary set loss can improve 2\% on F-measure compared to direct calculating loss for TPS parameters, because it gets rid of the reliance on the unreliable ground truth of TPS parameters , reducing noise due to the ambiguity of the annotations and boundary points sampling strategy. The shape alignment loss further optimizes the TPS parameters by the global shape constraints, thus it brings 1.3\% improvement on F-measure. Indeed, the control points are equivalent to the TPS Parameters since the latter can be solved directly from the former. If the proposed loss functions are applied for controlling points prediction, it might reach the similar performance with TPS parameters, but the process to get text shape is complicated: from control points to TPS parameters, and then from TPS parameters to text shape. Predicting the TPS parameters is a simple but efficient choice.
\begin{table*}[ht]
\centering
\begin{tabularx}{\linewidth}{@{}lYYYYYYYYYYX@{}}
\toprule
\multirow{2}{*}{Methods} & \multirow{2}{*}{Ext} & \multicolumn{3}{c}{CTW1500} & \multicolumn{3}{c}{TotalText} & \multicolumn{3}{c}{ICDAR2015} \\ \cmidrule(l){3-11}
             &      & R    & P    & F    & R    & P    & F    & R    & P    & F    \\ \midrule
TextSnake    & \checkmark & \textbf{85.3} & 67.9 & 75.6 & 74.5 & 82.7 & 78.4 & 80.4 & 84.9 & 82.6 \\
PSENet       & \checkmark & 79.7 & 84.8 & 82.2 & 78.0 & 84.0 & 80.9 & 84.5 & 86.9 & 85.7 \\
CRAFT        & \checkmark & 81.1 & 86.0 & 83.5 & 79.9 & 87.6 & 83.6 & 84.3 & 89.8 & 86.9 \\
SAE          & \checkmark & 77.8 & 82.7 & 80.1 & 77.8 & 82.7 & 80.1 & 85.0 & 88.3 & 86.6 \\
MSR          & \checkmark & 78.3 & 85.0 & 81.5 & 74.8 & 83.8 & 79.0 & 78.4 & 86.6 & 82.3 \\
PAN          & \checkmark & 81.2 & 86.4 & 83.7 & 81.0 & 89.3 & 85.0 & 81.9 & 84.0 & 82.9 \\
SAST         & \checkmark & 77.1 & 85.3 & 81.0 & 76.9 & 83.8 & 80.2 & 87.1 & 86.7 & 86.9 \\
TextField    & \checkmark & 79.8 & 83.0 & 81.4 & 79.9 & 81.2 & 80.6 & 83.9 & 84.3 & 84.1 \\
DB           & \checkmark & 80.2 & 86.9 & 83.4 & 82.5 & 87.1 & 84.7 & 83.2 & \textbf{91.8} & 87.3 \\
DRRGN        & \checkmark & 83.0 & 85.9 & 84.5 & \textbf{84.9} & 86.5 & 85.7 & 84.7 & 88.5 & 86.6 \\
CRNet        & \checkmark & 80.9 & 87.0 & 83.8 & 82.5 & 85.8 & 84.1 & 84.5 & 88.3 & 86.4 \\ \midrule
SPCNet       & \checkmark & $-$ & $-$ & $-$ & 82.8 & 83.0 & 82.9 & 85.8 & 88.7 & 87.2 \\
LOMO         & \checkmark & 69.6 & \textbf{89.2} & 78.4 & 75.7 & 88.6 & 81.6 & 83.5 & 91.3 & 87.2 \\
CSE          & $\times$ & 76.0 & 81.1 & 78.4 & 79.1 & 81.4 & 80.2 & $-$ & $-$ & $-$ \\
ContourNet   & $\times$ & 84.1 & 83.7 & 83.9 & 83.9 & 86.9 & 85.4 & 86.1 & 87.6 & 86.9 \\
Mask-TTD     & $\times$ & 79.0 & 79.7 & 79.4 & 74.5 & 79.1 & 76.7 & 87.6 & 86.6 & 87.1 \\ \midrule
ATRR         & $\times$ & 80.2 & 80.1 & 80.1 & 76.2 & 80.9 & 78.5 & 86.0 & 89.2 & 87.6 \\
Boundary     & \checkmark & $-$ & $-$ & $-$ & 83.5 & 85.2 & 84.3 & \textbf{88.1} & 82.2 & 85.0 \\
ABCNet       & \checkmark & 78.5 & 84.4 & 81.4 & 81.3 & 87.9 & 84.5 & $-$ & $-$ & $-$ \\
TextRay      & $\times$ & 80.4 & 82.8 & 81.6 & 77.9 & 83.5 & 80.6 & $-$ & $-$ & $-$ \\
FCENet       & $\times$ & 83.4 & 87.6 & 85.5 & 82.5 & 89.3 & 85.8 & 82.6 & 90.1 & 86.2 \\
PCR          & \checkmark & 82.3 & 87.2 & 84.7 & 82.0 & 88.5 & 85.2 & $-$ & $-$ & $-$ \\ \midrule
TPSNet(ours) & $\times$ & 83.3 & 89.1 & 86.1 & 84.4 & 89.3 & 86.8 & 85.1 & 90.5 & 87.7 \\
TPSNet(ours) & \checkmark & 85.1 & 87.7 & \textbf{86.4} & 84.6 & \textbf{90.8} & \textbf{87.6} & 86.8 & 90.7 & \textbf{88.7} \\ \bottomrule
\end{tabularx}
\caption{Comparison with previous methods on CTW1500, TotalText and ICDAR2015 test sets. ‘Ext’ means using the external dataset to pretrain the model.}
\label{table:sota}
\end{table*}

\begin{table}[h]
\centering
\begin{tabularx}{1\linewidth}{@{}cp{1.2cm}<{\centering}p{1.2cm}<{\centering}p{1.2cm}<{\centering}p{1.2cm}<{\centering}X@{}}
\toprule
aspect   ratio       & scale & R    & P    & F      \\ \midrule
1:1                  & 1     & 80.3 & 87.7 & 83.8   \\ \midrule
1:2                  & 1     & 79.9     & 87.4     &  83.4      \\ \midrule
\multirow{4}{*}{1:4} & 0.4   &  81.6    &   89.4   &   85.3     \\
                     & 1     &\textbf{83.3}   & \textbf{89.1}      & \textbf{86.1}   \\
                     & 2     &   80.8   &  88.6    &   84.5     \\
                     & 4     & -    & -    & Failed \\ \bottomrule
\end{tabularx}
\caption{Ablation study about the fiducial shape on CTW1500.}
\label{ablafiducial}
\end{table}

\subsubsection{The Fiducial Shape Setting}

The fiducial shape decides the basis function and have a great impact on the performance. To find the best setting of fiducial shape, we set different aspect ratios and scales for ablation as shown in Table \ref{ablafiducial}. As expected, larger aspect ratio is more suitable for text shape representation. Although the TPS parameters based on the fiducial shape of small aspect ratio can also fit the text boundary, the inner points are uneven,
causing the shape alignment loss can not work anymore. The scale of fiducial shape mainly affects the optimization of the network. Since the function basis have $d^2 ln d$ term, larger scale generate larger $d$ and will cause larger gradients, making the loss unstable and even explode when the scale is 4. According to the ablation, the aspect ratio is set to 1:4 and the scale is set to 1, which means the width of fiducial rectangle is 1 and the height is 0.25.

\begin{figure*}[!ht]
  \centering
  \captionsetup[subfigure]{labelformat=empty}
  \includegraphics[width=1.5in]{figures/beizer_1.jpg}\hspace{1.5pt}
  \includegraphics[width=1.5in]{figures/ray_1.jpg}\hspace{1.5pt}
  \includegraphics[width=1.5in]{figures/fce_1.jpg}\hspace{1.5pt}
  \includegraphics[width=1.5in]{figures/ours_1.jpg}\vspace{1.5pt}\\
  \subfloat[ABCNet]{\includegraphics[width=1.5in]{figures/beizer_2.jpg}}\hspace{1.5pt}
  \subfloat[TextRay]{\includegraphics[width=1.5in]{figures/ray_2.jpg}}\hspace{1.5pt}
  \subfloat[FCENet]{\includegraphics[width=1.5in]{figures/fce_2.jpg}}\hspace{1.5pt}
  \subfloat[Ours]{\includegraphics[width=1.5in]{figures/ours_2.jpg}}\vspace{-6pt}\\

 \caption{Qualitative comparisons with ABCNet  \cite{Liu2020ABCNet},  TextRay~ \cite{Wang2020textray} and FCENet  \cite{zhu2021fourier} on selected challenging samples in CTW1500 .}
 \label{fig:quali}
\end{figure*}
\subsection{Comparisons with Previous Methods}
We evaluate our TPSNet on benchmark datasets and compare with previous methods as shown in Table \ref{table:sota}. Previous methods are divided into three categories: segmentaion-based, regression-based and hybrid-based methods that use both segmentation and regression for detection. Obviously, the segmentation-based methods rely on external dataset to pretrain the model. As a regression-based  method, our TPSNet can significantly outperforms previous methods on curved scene text detection datasets CTW1500 and TotalText, and for straight scene text in ICDAR2015, it can also achieve the best performance. In addition, the TPSNet is easy to train as it can get comparable performance without extra data for pretraining. FCENet is the most competitive method with TPSNet, but it can not rectify the detected text region directly, and as shown in Fig. \ref{fig:intro}\&\ref{fig:quali}, it misses partial corner pixels which will lead to poor recognition accuracy.

Qualitative comparison is shown in Fig. \ref{fig:quali}. ABCNet\cite{Liu2020ABCNet} is easy to be confused by adjacent instances since it predicts two separate curves rather than the whole shape together. TextRay\cite{Wang2020textray} fails in highly-curved or large aspect ratio cases and FCENet\cite{zhu2021fourier} prefers to missing the corners of long text, which is not conducive to the subsequent recognition. By comparison, our proposed TPSNet obtained the most compact and complete detection.

\subsection{Rectification for Recognition}
To verify that the TPS representation can provide robust rectification for recognition, we conduct the recognition experiment with the following settings: we apply TPSNet to detect texts on TotalText test images, and then we match the detection results to the ground truth with IOU threshold at 0.5; all the matched detection results take the corresponding text labels as the ground truth, where only latin alphabet and numbers are preserved, and the text regions are rectified by our TPS parameters or directly cropped by bounding boxes (no rectification). We use ASTER\cite{shi2018aster} as the recognition model, and the open source trained model is applied. We switch off and on the rectification module of ASTER to recognize the bounding-box text and the text rectified by TPSNet. The recognition accuracy are shown in Table \ref{tab:recog} and the rectified texts shown in Fig. 5.

The rectification from TPSNet improves the recognition accuracy by 13.8\%, superior to the rectification in ASTER.
\begin{table}[h]
  \begin{minipage}[]{0.4\linewidth}
      \centering
      \includegraphics[width = 0.95\linewidth]{figures/rectify.pdf}
    \label{fig:recog}
      \figcaption{Rectification results from ASTER (left) and ours (right).}
  \end{minipage}
  \hspace{5pt}
  \noindent \begin{minipage}[]{0.4\linewidth}
    \centering
    \begin{tabular}{@{}ccc@{}}
\toprule
\multicolumn{2}{c}{Rectification} & \multirow{2}{*}{Accuracy} \\ \cmidrule(r){1-2}
Aster           & TPSNet          &                           \\ \midrule
                &               & 60.3                      \\
 \checkmark               &               & 71.4                     \\
               & \checkmark              & 74.1                     \\ \bottomrule
\end{tabular}
\tabcaption{Recognition performance with different rectifications.}
\label{tab:recog}
  \end{minipage}
\end{table}
\section{Conclusion}
In this paper, we have proposed a novel TPS representation for arbitrary shape text, which is the first application of TPS transformation in scene text detection. The TPS representation is compact, complete, integral, and reusable for subsequent recognition. Equipped  with  TPS  representation, we implement the TPSNet, and design two shape losses for network training. Experimental results on CTW1500 and TotalText show the effectiveness. Since rectification directly is the most advantage of TPS representation, we will extend TPSNet to end-to-end text spotting in our future work.

\newpage
{\small
\bibliographystyle{ieee_fullname}


\section{Introduction}
The natural image matting is an important task in computer vision, which serves as a prerequisite for a broad set of applications, including image or video editing, compositing and film post-production~\cite{DBLP:conf/cvpr/TangAOGA19,DBLP:conf/cvpr/SenguptaJCSK20,DBLP:journals/corr/abs-2012-07810,tang2020xinggan,tang2020local,tang2019multi,tang2021attentiongan,tang2020dual,tang2020bipartite}. Following~\cite{DBLP:conf/siggraph/PorterD84}, the natural image $I$ is defined as a convex combination of foreground image $F$ and background image $B$ at each pixel $i$ as:
\begin{equation}
I_i = \alpha_iF_i + (1-\alpha_i)B_i, \alpha_i \in [0,1],
\end{equation}
where $\alpha_i$ is the alpha value at pixel $i$ that denotes the opacity of the foreground object. Mathematically, image matting requires expressing pixel colors in the transition regions
\begin{wrapfigure}{r}{0.5\linewidth}
    \centering
    \includegraphics[width=\linewidth]{./intro.pdf}
    \caption{Comparison with SOTA methods. Best viewed by zooming in. Smaller SAD and FLOPs mean better performance and less computational cost. }
    \label{fig:intro}

\end{wrapfigure}
from foreground to background as a convex combination of their underlying foreground and background colors. Since neither the foreground and background colors nor the opacities are known, estimating the opacity values is a highly ill-posed problem. To alleviate the difficulty of this problem, typically a trimap is provided in addition to the original image~\cite{DBLP:conf/cvpr/AksoyAP17}. The trimap can be provided by user or generated from the network. The trimap is a rough segmentation of the input image into foreground($\alpha_i=1$), background($\alpha_i=0$), and regions with unknown opacity($\alpha_i\in(0,1)$) which like other segmentation tasks~\cite{DBLP:conf/iccv/LiSLWH19,DBLP:conf/aaai/LiSG19,DBLP:conf/ijcai/0061STSS19,DBLP:conf/mm/LiSWL19,DBLP:journals/mta/LiSXWY20,DBLP:conf/accv/Tang020,DBLP:conf/pcm/LiSHX17,Tang_2021_ICCV}. From then on, many traditional methods~\cite{DBLP:conf/cvpr/ChuangCSS01,DBLP:journals/tog/SunJTS04,DBLP:conf/cvpr/LevinLW06,DBLP:conf/cvpr/WangC07,DBLP:conf/cvpr/HeRRTS11,DBLP:conf/cvpr/ShahrianRPC13,DBLP:conf/cvpr/AksoyAP17} are proposed for matting task. These methods leverage color features to predict alpha mattes, which will produce obvious artifacts when foreground and background share similar colors.

Deep learning has recently demonstrated success in many computer vision applications. Recent researches use deep neural networks to improve the performance of natural image matting. Compared to traditional methods, deep-learning matting methods~\cite{DBLP:conf/eccv/ChoTK16,DBLP:conf/cvpr/XuPCH17,DBLP:conf/cvpr/TangAOGA19,DBLP:conf/iccv/CaiZFHLLLWS19,DBLP:conf/iccv/Hou019,DBLP:conf/iccv/0003DSX19,DBLP:conf/aaai/LiL20a} have achieved remarkable performance and further advance the matting field.  However, the widespread application of image matting still faces two major challenges: first, most methods rely on user-provided trimaps as assistant input, which is not easy to use for novice users without expertise in digital matting; second, their heavy model sizes make them resource-hungry and hardly applicable on low-power devices with limited storage/computational capability.

To address first problem,~\cite{DBLP:conf/cvpr/ZhangGFRHBX19,DBLP:conf/cvpr/QiaoLYZXZW20} design trimap-free natural image matting models.~\cite{DBLP:conf/cvpr/ZhangGFRHBX19} investigated semantic segmentation variant for foreground and background weight map fusion to obtain alpha values.~\cite{DBLP:conf/cvpr/QiaoLYZXZW20} explores advanced semantics and appearance cues synthetically to achieve high-quality alpha values. However, to capture sufﬁcient semantics, these two methods utilizate large backbones (DenseNet-201 and ResNeXt101), which make it difficult to apply these methods to some devices with limited storage/computational capability. To address second problem, ~\cite{DBLP:journals/spl/YoonPC20} designs a light-weighted natural image matting model with the help of knowledge distillation. However, this method also needs extra trimaps, which is not user-friendliness. Hence, designing a trimap-free natural image matting method with a light-weighted model is barely explored.

Without the user-provided trimap, there are two basic tasks for automatic natural image matting: identify the foreground, background and unknown regions in the image; predict the precise alpha values of unknown regions. The first one is a classic classiﬁcation task while the second task is typical regression task. As discussed in ~\cite{DBLP:conf/cvpr/ZhangGFRHBX19}, it is hard to handle the two tasks simultaneously with a single structure, we thus propose a two-stage end-to-end deep learning framework for trimap-free light-weighted matting task. The first stage is designed to capture sufﬁcient semantics to identify the foreground, background, and unknown regions. Then under the guidance of the first stage, the second stage can give the precise alpha values of unknown regions. Although such two-stage network design has appeared before (e.g. \cite{DBLP:conf/mm/ZhuCWLZT17}, \cite{ DBLP:conf/crv/HuC19}, \cite{DBLP:conf/mm/ChenGXZYG18}), it is an intutitive idea. The most important is that \cite{DBLP:conf/mm/ZhuCWLZT17} designs redundant structures besides low-resolution input ($128\times 128$), and the strong portrait prior limit their application prospects. Then the core problem lies in how to accomplish these two tasks with light-weighted network structures.

We approach this core problem from two perspectives: a light-weighted backbone and efficient feature aggregation/enhancement strategies of each stage. To be specific, we first build a light-weighted basic block with some popular efficient convolutional operations. Then we stack these basic blocks to construct light-weighted U-Net like backbones for two stages. The first stage is segmentation network (SN), and the second stage is matting refine network (MRN). In the first stage, we design a cross-level fusion module (CFM) to directly fuse the high-level semantic feature with low-level feature, which can improve the discriminability of feature representation. In the second stage, we design a novel efficient non-local attention module (ENA) to model the relevance between different pixels in encoder/decoder feature and image feature extracted from octave block (OCBlock), which can help regress high-quality alpha values. Moreover, in matting task, modeling the relevance of different pixels in long-range is not as important as modeling the relevance of pixels in short-range. So the proposed ENA first calculates long-range relevance with sampling strategy, then directly calculates short-range relevance. This design meets the principle of a light-weighted model. As can be seen in Fig.\ref{fig:intro}, compared with LateFusion~\cite{DBLP:conf/cvpr/ZhangGFRHBX19}, our proposed method can predict a better result with accurate semantics and alpha value.

Our major contributions can be summarized as:
\begin{itemize}

\item We make one of the earliest effort on designing a light-weighted trimap-free natural image matting method, which expands matting to much wider application scenarios.

\item We construct two U-Net like backbone with light-weighted basic blocks for the proposed two-stage framework and propose two corresponding efficient feature aggregation/enhancement strategies CFM and ENA for two stages to better identify the unknown regions and give precise alpha values.

\item We perform extensive experiments to demonstrate the proposed method refreshes the SOTA performance on Composition-1k and Distinctions-646 testing sets with ~1\% parameters (344k) of SOTA large models.
\end{itemize}

\begin{figure}[t]
    \centering
    \includegraphics[width=\textwidth]{./pipeline.pdf}
    \caption{The framework of the our trimap-free light-weighted natural image matting model.}

    \label{fig:pipeline}
\end{figure}

\section{Method}
In this section, we ﬁrst describe the overall architecture of the proposed trimap-free light-weighted natural image matting network, then introduce our proposed light-weighted backbone, SN and MRN. The architecture of the proposed approach is illustrated in Fig.\ref{fig:pipeline}.

\subsection{Network Overview}
SN has a simple U-Net like Encoder-Decoder architecture, and the backbone is formed by stacking our proposed OCBlock. Similar to ResNet-50 \cite{DBLP:conf/cvpr/HeZRS16}, the proposed backbone has four levels, named as En1-1, En1-2, En1-3, En1-4. The number of OCBlock in each level is 3, 4, 6, 4. Because En1-1 is too close to the input and its resolution is too high, to save computational cost, we only use the last three levels features for the following process. Cross-level fusion module (CFM) is added between En1-4 and En1-2 to improve the discriminability of feature representations. SN has 3 decoders, and each decoder is stacked by 1 OCBlock. Decoder fuses the processed features from the encoders and the upsampled features from the previous decoder level in a bottom-up manner. The output of each decoder is defined as De1-4, De1-3, and De1-2. Finally, the network generates the 3-channel classification logits $T$ by a $3\times3$ convolution.

MRN has a basic Encoder-Decoder architecture, and the input of MRN is predicted logits $T$ and an image. The backbone is also stacked by our proposed OCBlock, and all encoder levels contain 2 OCBlocks, respectively. To accurately regress alpha values in unknown regions, we stick two efficient non-local attention (ENA) module to the encoder and decoder symmetrically in our stem. Moreover, image features extracted from OCBlocks contain more detailed texture information, which can help regress alpha values. Thus, we utilize the image features extracted from OCBlocks to guide the information flow on Encoder/Decoder features. MRN has 4 decoders, and each decoder is stacked by 1 OCBlock. Decoder fuses the features in each encoder and the upsampled features from the previous level in a bottom-up manner. Finally, together with the unknown and foreground maps predicted by SN, MRN can regress a 1-channel result $\alpha$.

\subsection{Light-weighted backbone}
Recently, some researchers have designed various light-weighted backbones~\cite{DBLP:journals/corr/HowardZCKWWAA17,DBLP:conf/cvpr/Chollet17,DBLP:conf/cvpr/ZhangZLS18,DBLP:conf/icassp/LiSTH19,DBLP:conf/icassp/TangLWXD21} with special convolution operations. A classic example is MobileNet which constructs a light-weighted backbone by using the depthwise convolution. However, these light-weighted models are designed for classification, it is not appropriate to directly use them for matting task.
Unlike classification task, matting needs multi-scale information to capture sufﬁcient semantics for pixel-level prediction. Some researchers try to perform extra multi-scale feature extraction after backbone, like atrous spatial pyramid pooling (ASPP), global convolutional network (GCN)~\cite{DBLP:conf/cvpr/PengZYLS17}, which need more computational cost. Inspire by MobileNet, we want to use depthwise convolution to build our lightweight backbone. But only using depthwise convolution cannot capture sufficient multi-scale information. To address this problem, by leveraging the OctConv \cite{DBLP:conf/iccv/Chen0XYKRYF19}, we design our
lightweight OCBlock, and stack OCBlocks to construct a light-weighted backbone. To be specific, we replace a depthwise convolution with a vanilla OctConv. The vanilla OctConv \cite{DBLP:conf/iccv/Chen0XYKRYF19} is presented to conduct the convolution operation on features across high/low scales, as shown in Fig.\ref{fig:pipeline}(a). In OctConv, features from different scales are fused to generate output. The proposed OCBlock is shown in Fig.\ref{fig:pipeline}(b), which combines the advantages of depthwise convolutions and OctConv to achieve a better weight/performance balance.

\vspace{-0.3cm}
\subsection{Segmentation Network}
SN plays the role of capturing sufﬁcient semantics and classifying the pixels into unknown, foreground, and background regions with light-weighted backbone. Thus, learning discriminant feature representations with less computational cost is essential. To achieve this goal,  we first utilizate our proposed light-weighted backbone to obtain multi-scale feature representations. Moreover, we utilize CFM module to leverage the advantages of features at high-level to learn more discriminant feature representations. \\
\textbf{Cross-Level Fusion Module.} High-level feature from En1-4 contains much semantic information, which can make the network better distinguish the appearance difference between unknown, foreground, and background regions. However, our proposed SN is a U-like architecture, when information is progressively returned from the high-level to the low-level, the high-level semantic information is gradually diluted. Thus, we directly fuse low-level feature En1-2 with high-level feature En1-4, to propagate the high-level semantic information to the low-level. As different channels of feature in high-level response to different semantics, it is unwise to treat all channels without distinction. To alleviate the interference of the irrelevant semantic information, we use channel-wise attention on En1-4 to assign larger weights to channels which are more important for matting task. The operations in channel-wise attention are with low time complexity. Therefore, it demands negligible computational cost. \\
\vspace{-0.7cm}
\subsection{Matting Refine Network}
MRN aims at regressing the alpha values of pixels in unknown regions. The MRN takes the concatenation of 3-channel images and the 3-channel segmentation results from SN as 6-channel input. For easy remembering, we named unknown regions as $U$. We design an efficient non-local attention module which can help better regress the alpha values with less computational cost. \\
\textbf{Efficient Non-Local Attention.} Modeling the relevance of different pixels can help regress the accurate alpha value. Because it can help the network effectively group pixels with similar texture, then pay more attention to these pixels. Inspired by~\cite{DBLP:conf/cvpr/0004GGH18}, we use non-local attention to achieve this goal. Common non-local attention can be defined as:
\begin{equation}
    A=Sim(Q,K)V,
\end{equation}
where $Q,K$ and $V$ mean $query$, $key$ and $value$ feature, and $Sim$ is the similarity between query and key feature.

According to traditional methods, pixels with almost identical texture should have similar opacity. Hence, pixels that share similar texture information should have similar opacity features. However, as the network layers deepen, the feature representations of each pixel will be blurred, which will affect the regression of alpha value. Therefore, we use detailed image features to guide the information flow on Encoder/Decoder features, which improves the regression accuracy of alpha values. In our proposed work, the $query$ feature is the feature from Encoder/Decoder, named as $X_Q$. The $key/value$ feature is the feature extracted from the input image, named as $X_K$ and $X_V$.

\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{./att.pdf}
    \caption{The illustration of the proposed ENA. The query feature is obtained from the multiplication of encoder/decoder features and unknown region. Image is passed through OCBlocks to obtain the key and value feature. Our proposed ENA is a cross-level non-local attention, but for the sake of simplicity, we have only shown the long-range and short-range attention operations on query feature $X_Q$. In this figure, we set $k=4$ and $n=16$.}
    \label{fig:att}
    \vspace{-0.3cm}
\end{figure}

Because MRN only cares about regressing the alpha values in unknown region, we multiply $X_Q$ with $U$ to make $query$ feature more focused on the unknown region. It can be formulated as:
\begin{equation}
    X_Q = X_Q * U.
\end{equation}

Directly using conventional non-local attention in our work is time-consuming, which is not in line with the design principle of light-weighted models. Moreover, motivated by~\cite{DBLP:conf/cvpr/ChenLT12}, modeling the relevance of different pixels in long-range is not as important as modeling the relevance of pixels in short-range, and sometimes will lead to some noise because of irrelevant matches. Thus, our proposed ENA is the combination of long-range attention with sampling strategy and short-range attention. The main point of the long-range attention is to apply the non-local attention to the subsets of positions that have long spatial interval distances, which can not only help reduce the probability of irrelevant matches between long-range pixels, but also save computational cost. The details of long-range attention can be seen in Fig.\ref{fig:att}(a). The input feature $X\in\{X_Q,X_K,X_V\}$ are first divided into $k$ partitions, then sampling these pixels in different partitions at intervals $\sqrt{k}$ into new subsets. In our work, we use permute operation on behalf of sampling. So each subset contains $\frac{n}{k}$ pixels,
where $n$ represents the size of input features. Thus the new subsets are defined as: $X_L=[X^1,X^2,\cdots,X^k]$, where $X^k\in\{X_Q^k,X_K^k,X_V^k\}$, and the shape of $X^k$ is $\mathbb{R}^{\frac{n}{k}\times C}$. $C$ means the channel numbers. Then we apply the non-local attention on $X^k$ independently:
\begin{equation}
    A_G^k=Sim(X_Q^k, X_K^k)X_V^k,
\end{equation}
which is shown in Fig.\ref{fig:att}(b). Finally, we merge all the $A_G^k$ to get the output $A_G$. For short-range attention, we permute the $A_G$ to group the originally nearby positions together, which is the $query$ of short-range attention. So the input of short-range attention is $Z\in\{A_G,X_K,X_V\}$. Similarly, We divide the into $\frac{n}{k}$ partitions and each partition contains $k$ neighboring positions, defined as: $Z_S=[Z^1,Z^2,\cdots,Z^\frac{n}{k}]$, where $Z^\frac{n}{k}\in\{A_G^\frac{n}{k},X_K^\frac{n}{k},X_V^\frac{n}{k}\}$, and the shape of $Z^\frac{n}{k}$ is $\mathbb{R}^{k\times C}$. Then we apply the non-local attention on each $Z^\frac{n}{k}$ independently. Accordingly, we can get $A_L^\frac{n}{k}$ and merge all of them to get the output $A_L$. Finally, we directly fuse the $A_L$ with Encoder/Decoder features to get the refined features. In our proposed model, we set $k=16$. \textbf{We analyze the computation/memory cost of the common non-local attention mechanism and our ENA in supplementary materials Section 3.} \\
\textbf{MRN Output.} Together with unknown maps, MRN will output the refined alpha values $\alpha_{r}$ in unknown regions. The final alpha matte result $\alpha$ can be written as:
\begin{equation}
    \alpha=F+\alpha_{r}.
\end{equation}

\vspace{-0.3cm}
\subsection{Loss Function}
To supervise our proposed network, the loss function contains two parts: $L_{SN}$ and $L_{MRN}$.
Let $\alpha^{gt}$ be the groundtruth of final prediction $\alpha$.
\\
\quad The prediction of SN is 3-channel classification logits $T$. Following~\cite{DBLP:conf/iccv/CaiZFHLLLWS19}, we generate its groundtruth $T^{gt}$ according to $\alpha^{gt}$, which is defined as:
\begin{equation}
	T^{gt}(x,y) =
	\begin{cases}
	0, \  &{\alpha^{gt}(x,y) = 0} \\
	1, \  &{ \alpha^{gt}(x,y) = 1} \\
	2, \  &{ 0 < \alpha^{gt}(x,y) < 1}
  \end{cases}
\end{equation}
where $(x,y)$ stands for each pixel location on the image. The value $\{0, 1, 2\}$ means the pixel is in background, foreground or unknown region. We use Softmax cross-entropy loss to supervise SN, which is defined as:
\begin{equation}
\begin{aligned}
L_{SN} = \sum_l^{l=3} CE(T_l,T_{gt}).
\end{aligned}
\end{equation}
As show in Fig.\ref{fig:pipeline}, we use multi-level supervision as an auxiliary loss to facilitate suﬃcient training. $T_l$ means the prediction result in $l-$level.
\\
\quad To supervise MRN, following \cite{DBLP:conf/cvpr/ZhangGFRHBX19}, we adopt the weighted $L_1$ loss and gradient loss $L_g$. The weighted $L_1$ loss is defined as:
\begin{equation}
    L_1 = \sum_p w_p \cdot |\alpha_p - \alpha_p^{gt}|,
\end{equation}
where $\alpha_p$ means predicted alpha value at pixel $p$. The weight $w_p$ is set to 1 when $0<\alpha_p<1$ and set to 0.1 when $\alpha_p=1$ or $\alpha_p=0$. This operation can make the network pay attention to unknown region. The gradient loss $L_g$ is used to remove the over-blurred alpha matte, which is defined as:
\begin{equation}
    L_g =  | \nabla_x(\alpha) - \nabla_x(\alpha^{gt})| + | \nabla_y(\alpha) - \nabla_y(\alpha^{gt})|.
\end{equation}
\\
\quad Thus $L_{MRN}$ can be written as: $L_{MRN} = L_1 + L_g.$
Note that all parts of our proposed network are trained jointly, so the overall loss function is given as:
\begin{equation}
    L = L_{SN} + L_{MRN}.
\end{equation}
\vspace{-0.6cm}
\section{Experiments}
\subsection{Dataset and Evaluation Metrics}

\textbf{Dataset}. We evaluate our proposed network on the public Adobe Composition-1k \cite{DBLP:conf/cvpr/XuPCH17}. The training set consists of 431 foreground objects with the corresponding ground truth alpha mattes. Each foreground image is combined with 100 background images from MS COCO dataset \cite{DBLP:conf/eccv/LinMBHPRDZ14} to composite the input images. For the test set, the Composition-1k contains 50 foreground images as well as the corresponding alpha mattes, and 1000 background images from PASCAL VOC2012 dataset \cite{DBLP:journals/ijcv/EveringhamGWWZ10}. The training and test sets were synthesized through the algorithm provided by \cite{DBLP:conf/cvpr/XuPCH17}.

For the data augmentation, following \cite{DBLP:conf/aaai/LiL20a}, the foreground object and the alpha image will be resized to 640 $\times$ 640 images with a probability of 0.25 to make the network see the whole foreground image instead of a cropped snippet. Then, a random affine transformation is used. Subsequently, we randomly crop one 512 $\times$ 512 patch from each foreground. All of the patches are centered on an unknown region according to the alpha images. The foreground images are then converted to HSV space, and different jitters are imposed to the hue, saturation, and value. During testing, the resolution of input image is $512\times512$. \\
\textbf{Evaluation metrics}. There are four metrics used in the evaluations: SAD (sum of absolution difference), MSE (mean square error), gradient, and connectivity defined in \cite{DBLP:conf/cvpr/XuPCH17}. The lower values of the metrics, the better the predicted alpha matte is. The details of gradient and connectivity metrics can be found in \cite{DBLP:conf/cvpr/RhemannRWGKR09}, they are used to reflect the visual quality of the alpha matte when observed by a human. \\
\textbf{Implementation Details}. We use Pytorch to implement our model. The whole network is trained in an end-to-end manner. For loss optimization, we use the Adam optimizer with $\beta_1=0.5$ and $\beta_2=0.999$. The learning rate is initialized to $4\times10^{-4}$. Warmup and cosine decay \cite{DBLP:conf/iclr/LoshchilovH17,DBLP:conf/cvpr/HeZ0ZXL19} are applied to the learning rate. The network is trained for 70 epochs with a batch size of 12. We set the weight for $L_{SN}$ to 1, for $L_g$ to 0.5, and for $L_1$ to 5. If not specially mentioned, we use this as the default settings. Our model can be trained from scratch without other datasets.

\begin{figure}[t]
    \centering
    \includegraphics[width=\textwidth]{./res.pdf}
    \caption{The visual comparisons on the Composition-1k testing dataset. Zoom in for the best view.}
    \label{fig:res}
\end{figure}

\subsection{Evaluation on Composition-1k Test Set}

Here we compare our network with 5 traditional matting methods: Shared Matting \cite{DBLP:journals/cgf/GastalO10}, Learning Based \cite{DBLP:conf/iccv/ZhengK09}, Global Matting \cite{DBLP:conf/cvpr/HeRRTS11}, ClosedForm \cite{DBLP:conf/cvpr/LevinLW06}, KNN Matting \cite{DBLP:conf/cvpr/ChenLT12}, and 7 deep learning based methods: DCNN \cite{DBLP:conf/eccv/ChoTK16}, DIM \cite{DBLP:conf/cvpr/XuPCH17}, AlphaGAN \cite{DBLP:conf/bmvc/LutzAS18},
IndexNet \cite{DBLP:conf/iccv/0003DSX19}, Late Fusion \cite{DBLP:conf/cvpr/ZhangGFRHBX19}, HAttMatting \cite{DBLP:conf/cvpr/QiaoLYZXZW20}, GCA \cite{DBLP:conf/aaai/LiL20a}. LateFusion, HAttMatting and our proposed network are trimap-free matting methods. For other methods, we feed RGB images and trimaps produced by 25 pixels random dilation refer to \cite{DBLP:conf/cvpr/XuPCH17}. We compute the metrics at $800\times800$ for all methods. The testing settings in our work are following two trimap-free methods ~\cite{DBLP:conf/cvpr/ZhangGFRHBX19,DBLP:conf/cvpr/QiaoLYZXZW20}. The qualitative results are illustrated in Fig.\ref{fig:res} and quantitative comparisons are reported in Tab.\ref{tab:total}. Because we cannot obtain the executable code or results of HAttMatting, we do not show their qualitative results and directly cite quantitative results from their original paper.

\begin{wraptable}{r}{0.55\linewidth}
\centering
\caption{Quantitative and complexity comparison on the Composition-1k testing dataset. 'DIM-Trimap-less' denotes the results of the DIM method without trimap as input during training. The underline indicates the best of the trimap-based methods. The bolded numbers represent the best performace in trimap-free methods.}
\resizebox{\linewidth}{25mm}{
\begin{tabular}{||lcccccc||}
\hline
\multicolumn{1}{||c}{\multirow{2}{*}{Model}} & \multicolumn{2}{c}{Complexity} & \multicolumn{4}{c||}{Composition-1k} \\
\multicolumn{1}{||c}{} & \#PARM. & FLOPs & SAD & MSE & Grad & Conn \\ \hline  \hline
Shared Matting & - & - & 115.20 & 0.031 & 139.88 & 121.35 \\
Learning Based & - & - & 100.51 & 0.027 & 107.83 & 104.74 \\
Global Matting & - & - & 121.46 & 0.034 & 125.11 & 133.23 \\
Closed-form & - & - & 118.98 & 0.032 & 130.77 & 118.17 \\
KNN Matting & - & - & 133.99 & 0.051 & 140.29 & 134.03 \\
DCNN & - & - & 122.40 & 0.030 & 129.57 & 121.80 \\ \hline \hline
DIM-Trimap-less & 130.55M & 443.56G & 103.07 & 0.108 & 73.76 & 106.64 \\
DIM & 130.55M & 443.56G & 33.64 & 0.008 & 30.23 & 31.92 \\
IndexNet & 5.72M & 71.41G & 14.97 & \uline{0.006} & 12.02 & 14.58 \\
AlphaGAN & 40.3M & 496.36G & 20.28 & 0.010 & 21.11 & 20.46 \\
GCA & 25.16M & 89.41G & \uline{13.54} & \uline{0.006} & \uline{11.39} & \uline{12.96} \\ \hline \hline
Late Fusion & 37.91M & 74.85G & 58.29 & 0.011 & 36.58 & 59.63 \\
HAttMatting & 59.2M & 159.7G & 44.01 & 0.007 & 29.26 & 46.41 \\
Ours & 344.82K & 0.93G & \textbf{19.59} & \textbf{0.006} & \textbf{18.70} & \textbf{20.34} \\ \hline
\end{tabular}}
\label{tab:total}
\end{wraptable}

As can be seen in Tab.\ref{tab:total}, our proposed method outperforms all the traditional image matting methods by a large margin. Compared to trimap-based deep-learning methods, our method is slightly inferior to GCA and IndexNet, and is better than DIM and AlphaGAN. However, trimaps are strongly required during their training and inference phase, which restricts their effectiveness in practical applications. Compared to trimap-free methods LateFusion and HAttMatting, our proposed method can already outperform these two methods. Moreover, our method achieves SOTA performance with fewer parameters ($0.3\% \thicksim 7.8\%$ of large models) and FLOPs ($0.2\% \thicksim 1\%$ of large models) than all of the above methods which will upgrade high practicality.

In Fig.\ref{fig:res}, compared to LateFusion, our method is able to find a more complete foreground of the matte objects since our network can exploit multi-scale features to better capture the semantics. Because of our proposed ENA, our method can regress more accurate alpha values in the unknown region.

\vspace{-0.3cm}
\subsection{Ablation Study}
\vspace{-0.3cm}

\begin{wraptable}{r}{0.55\linewidth}
\centering
\caption{Ablation study on CFM and ENA. We also replaced our backbone with the existing lightweight backbone.}
\resizebox{\linewidth}{11mm}{
\begin{tabular}{||c|cc|cccccc||}
\hline
\multirow{2}{*}{Backbone} & \multicolumn{2}{c|}{Configurations} & \multicolumn{2}{c}{Model} & \multicolumn{4}{c||}{Metrics} \\
 & CFM & ENA & \# Param. & FLOPs & SAD & MSE & Grad & Conn \\ \hline \hline
\multirow{4}{*}{Ours} & $\times$ & $\times$ & 285.70K & 0.63G & 56.86 & 0.019 & 36.81 & 48.16 \\
 & $\times$ & $\checkmark$ & 342.70K & 0.93G & 34.33 & 0.021 & 28.24 & 32.98 \\
 & $\checkmark$ & $\times$ & 287.83K & 0.63G & 28.90 & 0.013 & 24.27 & 26.35 \\
 & $\checkmark$ & $\checkmark$ & 344.93K & 0.93G & \textbf{19.59} & \textbf{0.006} & \textbf{18.70} & \textbf{20.34} \\ \hline \hline
MobileNetV2 & $\checkmark$ & $\checkmark$ & 37.76M & 27.02G & 47.21 & 0.014 & 30.28 & 36.46 \\
ShuffleNetV2 & $\checkmark$ & $\checkmark$ & 30.22M & 20.52G & 48.72 & 0.015 & 33.84 & 38.21 \\ \hline
\end{tabular}}
\label{tab:ab1}
\end{wraptable}

The core idea of our method contains three parts: light-weighted backbone, cross-level fusion module (CFM), and Efficient Non-Local Attention (ENA). To validate the effectiveness of our proposed light-weighted backbone, we replace it with other light-weighted backbones, including MobileNet and ShuffleNet, and results are shown in Tab.\ref{tab:ab1}. Compared to these two light-weighted backbones, our proposed backbone can achieve better performance with fewer parameters and FLOPs. Because our backbone can achieve multi-scale feature representations for better capturing sufficient semantics. Moreover, only using CFM or ENA can already heavily improve the performance. Better performance has been achieved through the combination of these two architectures.

To further verify the validity and reasonableness of our designed efficient non-local attention module. We trained our model with different types of attention: common non-local attention and efficient non-local attention model. As shown in Tab.\ref{tab:ab2}, compared to common-local attention, our proposed ENA is not only more efficient but also more usable for matting tasks. Using only short-range attention (S) or long-range attention (L) can improve model performance because they can model the relevance between different pixels, which can help regress alpha values. As can be seen in Tab.\ref{tab:ab2}, different sampling intervals $\sqrt{k}$ lead to different results. When $\sqrt{k}=1,2$, ENA cannot fully exploit its convenience and effectiveness, because the sampling is too intensive. And the long-range attention will lead to some artifacts because of irrelevant matches. Experimentally, we set $\sqrt{k}$ to 4 in all experiments. For more comprehensive analyses of our proposed method, please refer to the supplementary materials.
Fig.\ref{fig:real} shows the matting results for internet images for further testing of our method.

\begin{table}[t]
\centering
\caption{Ablation study for different modalities using attention. $S$ and $L$ means short- and long-range attentions respectively.}
\resizebox{\linewidth}{21mm}{
\begin{tabular}{||cc|cccccc||}
\hline
\multicolumn{2}{||c|}{\multirow{2}{*}{Configurations}} & \multicolumn{2}{c}{Model} & \multicolumn{4}{c||}{Metrics} \\
\multicolumn{2}{||c|}{} & \# Param. & FLOPs & SAD & MSE & Grad & Conn \\ \hline \hline
\multicolumn{2}{||c|}{Common Non-Local} & 421.69K & 1.33G & 25.13 & 0.010 & 20.89 & 22.67 \\ \hline \hline
\multirow{6}{*}{Efficient Non-local(Ours)} & \multicolumn{1}{l|}{S} & 325.15K & 0.82G & 25.98 & 0.011 & 21.93 & 23.02 \\
 & \multicolumn{1}{l|}{L($\sqrt k$=4)} & 292.19K & 0.72G & 27.01 & 0.013 & 23.94 & 24.31 \\
 & S+L($\sqrt k$=1) & 424.11K & 1.34G & 25.01 & 0.010 & 20.31 & 22.96 \\
 & S+L($\sqrt k$=2) & 371.43K & 1.02G & 23.27 & 0.009 & 19.37 & 21.61 \\
 & S+L($\sqrt k$=4) & 344.83K & 0.93G & \textbf{19.59} & \textbf{0.006} & \textbf{18.70} & \textbf{20.34} \\
 & S+L($\sqrt k$=8) & 340.27K & 0.91G & 19.90 & 0.008 & 19.05 & 21.35 \\ \hline
\end{tabular}}
\label{tab:ab2}
\vspace{-0.3cm}
\end{table}
\begin{figure}[!t]
    \centering
    \includegraphics[width=\textwidth]{./real.pdf}
    \caption{The results on real world images and the unknown region predicted by our method.}
    \label{fig:real}
    \vspace{-0.3cm}
\end{figure}
\vspace{-0.5cm}
\section{Conclusion}
\vspace{-0.3cm}
In this paper, we design a two-stage trimap-free light-weighted natural image matting model, which makes one of the earliest effort on this area. SN is designed to capture sufﬁcient semantics and classify the pixels into unknown, foreground and background regions. MRN aims at capturing detailed texture information and regressing accurate alpha values. With proposed CFM, SN can efficiently utilize multi-scale features with less computational cost. ENA in MRN can efficiently model the relevance between different pixels and help regress high-quality alpha values. Experiment results on Composition-1k testing set demonstrate that our method outperforms state-of-the-art (SOTA) approaches in natural image matting with ~1\% parameters (344k) of large models. We believe our proposed backbone, CFM and ENA can contribute to other light-weighted computer vision tasks in the future.

\section{Introduction}

This supplemental material contains 6 parts:

\begin{itemize}
    \item Section 2 gives more details about our proposed lightweight backbone.
    \item Section 3 gives more analysis of the proposed ENA.

\end{itemize}

We hope this supplemental material can help you get a better understanding of our work.

\section{More Details of Our Network}

\subsection{Details of Network Architecture}

\begin{figure}[ht!]
    \centering
    \includegraphics[width=\textwidth]{./supp/net.pdf}
    \caption{Different types of OCBlock.}
    \label{fig:net}
\end{figure}

When OCBolck has only one scale input, OCBlock will be built like Fig.\ref{fig:net}(b) to obtain multi-scale output. When OCBlock has only one scale output, it will be built like Fig. \ref{fig:net}(c) to fuse the multi-scale input. The general OCBlock is shown in Fig. \ref{fig:net}(a).

Specifically, in stage1 of the proposed network, we use another OCBlock like Fig.\ref{fig:net}(b) as En1-0 to decompose the input image into features at two resolutions (256 and 512). So the dimension of the input of our backbone is $[256, 512]$. The dimension output of each level is $[256, 512], [256], [128], [64]$ respectively. We only use the $512$ features from En1-1 as described in main text. Thus, the first OCBlock in En1-3 and En1-4 uses Fig. \ref{fig:net}(b) structure. The last OCBlock in En1-2, En1-3 and En1-4 uses Fig. \ref{fig:net}(c) structure. MRN uses a similar structure.

\section{More analysis of ENA}

\subsection{The Efficiency of ENA}

Given an input feature map of size $H\times W\times C$, we analyze the computation cost of both the common non-local attention mechanism in the whole image and our proposed ENA.

For common non-local attention mechanism, three $1\times 1$ convolution layers acting on $Q,K,V$ to change them from $\mathbb{R}^{C\times H \times W}$ to $\mathbb{R}^{\frac{C}{2}\times H \times W}$ and one $1\times 1$ convolution layers acting on the output feature to change it from $\mathbb{R}^{\frac{C}{2}\times H \times W}$ to $\mathbb{R}^{C\times H \times W}$. The complexity of this part is ${\cal{O}}(2C^2HW)$. And, the complexity of multiplying between features is ${\cal{O}}(\frac{3}{2}C(HW)^2)$. So, the complexity of self-attention mechanism is
\begin{equation}
    C_1 = {\cal{O}}(2C^2HW+\frac{3}{2}C(HW)^2).
\end{equation}

Considering our approach, we divide the height and width dimension to $\sqrt{k}$ groups in calculating long-range relations, and $\frac{H}{\sqrt{k}}$ and $\frac{W}{\sqrt{k}}$ groups in calculating short-range relations. So the complexity of our approach is
\begin{equation}
    C_2 = {{\cal{O}}(4C^2HW+\frac{3}{2}C(HW)^2(\frac{1}{k}+\frac{k}{HW}))}.
\end{equation}
Since the channel number $C$ is a smaller value in our approach, it is much smaller than $HW$. The increase in the first term is negligible compared to the decrease in the second term. The complexity of our approach can be minimized to
\begin{equation}
    min(C_2) = {{\cal{O}}(4C^2HW+3C(HW)^\frac{3}{2})}.
\end{equation}
when $k=\sqrt{HW}$. Although $\sqrt{k}$ will take a small value in the implementation (e.g. 4) and will not be able to minimize $C_2$, our approach is efficient enough compared to original self-attention.

For the numerical complexity, we consider the feature dimensions of 80 and $\sqrt{k}=4$. When the input size is $64\times 64$, the computation cost of the self-attention mechanism is 2.06 GFLOPs and ours ENA is 0.11 GFLOPs. When the input size is $128\times 128$, it becomes 32.42 GFLOPs and 0.45 GFLOPs. So, our proposed efficient non-local attention module not only guarantees a small amount of computation cost when the input size is small but also guarantees a slow growth when the input size increases.

\subsection{The location choice of ENA in our network}

CFM wants to avoid the gradual dilution of the high-level semantic information
during decoding and propagate it to the low-level. Multiple CFMs will lead to duplication of operations and produce redundant computations, so we only build CFM between En1-2 corresponding to the last decoder and En1-4. For the ENA module, our experiments show that putting it at En/De2-2 does not improve performance while increasing complexity.

\end{document}


\section{Introduction}
\input{./sections/introduction}

\section{The chiral Gross-Neveu model}
\input{./sections/model}

\section{Correlators and related quantities}
\input{./sections/results}

\newpage
\acknowledgments

We are indebted to Laurin Pannullo, Marc Wagner, Marc Winstel and
Andreas Wipf for numerous enlightening discussions and previous
collaborations on inhomogeneous condensates. We furthermore thank
Björn Wellegehausen for providing the simulation code base used in
this work. This work has been funded by the Deutsche
Forschungsgemeinschaft (DFG) under Grant No. 406116891 within the
Research Training Group RTG 2522/1. The numerical simulations were
performed on resources of the Friedrich Schiller University of Jena
supported in part by the DFG grants INST 275/334-1 FUGG and INST
275/363-1 FUGG.

\bibliographystyle{JHEP}

\section{Phase diagrams}

We study phase diagrams in the $(\mu,T)$ plane using
\eqref{eq:cshort} and \eqref{eq:clong}. In order to quantify finite-volume effects
we performed an infinite-volume extrapolation at fixed lattice spacing and show the
results in Fig.~\ref{fig:pd_infinite_volume}.

First of all, there is a region where  $C_{\mathrm{short}}>0$ and thus homogeneous
configurations dominate, i.e. the red region in Fig.~\ref{fig:pd_infinite_volume}.
Its extent in the $\mu$-direction shrinks in the infinite-volume limit, which can
be understood by recalling that the chiral spiral's wavelength is inversely
proportional to $\mu$. This means that for small but non-vanishing chemical potential
smaller lattices are simply not large enough to fit in a full wavelength, thus
leading to a suppression of inhomogeneous configurations. The large extent of the
red region in the $T$-direction on the smallest lattice can also be seen to be a
finite-size effect.

We furthermore observe that the boundary between the "short-ranged"
($C_{\mathrm{short}}\approx0$) and "inhomogeneous" ($C_{\mathrm{short}}<0$) regions remains essentially unchanged
in the infinite-volume extrapolation. This is actually expected because the
appearance of $C_{\mathrm{short}}<0$ only depends on the two scales introduced by the temperature
and the chemical potential respectively, neither of which changes in the infinite-volume limit.
More precisely, negative values of $C_{\mathrm{short}}$ occur whenever the preferred
wavelength of the oscillatory (quasi-)condensates is smaller than the thermally
induced finite correlation length. From their analytically known values in an
expansion in $1/N_{\mathrm{f}}$ (see \cite{LMW21}), it is easy to anticipate the shape of this
boundary -- at least on a qualitative level.
In contrast, $C_{\mathrm{long}}$ becomes smaller and smaller in the infinite-volume limit since it
probes the largest scales on which we do not expect correlations.
This is indeed strong evidence against spontaneous symmetry breaking
and fundamentally differs from what was observed in \cite{LPW20}.

Finally, we study discretization effects via a continuum extrapolation at roughly
fixed physical volume in Fig.~\ref{fig:pd_continuum}. We find remnant structural order
for three strongly decreasing lattice spacings. However, due to subtleties in the
scale-setting procedure (see \cite{LMW21}) we refrain from drawing
definite conclusions at this point and instead hope to revisit the continuum limit
in the future. In related models in higher dimensions the continuum limit turned out
to be crucial for removing misleading lattice artifacts \cite{BKW21}. We do emphasize,
however, that our results are consistent with the
persistence of remnant inhomogeneities as $a$ appproaches zero -- in accordance with
the observation that in lower dimensions the UV limit usually tends to be less problematic.

\begin{figure}
\begin{subfigure}{.333\linewidth}
	\includegraphics[scale=0.33]{pd_2_63_infVol}
\end{subfigure}
\hspace{-2.3cm}
\begin{subfigure}{.333\linewidth}
	\includegraphics[scale=0.33]{pd_2_127_infVol}
\end{subfigure}
\hspace{-2.8cm}
\begin{subfigure}{.333\linewidth}
	\includegraphics[scale=0.33]{pd_2_255_infVol}
\end{subfigure}
\begin{subfigure}{.333\linewidth}
	\includegraphics[scale=0.33]{pd_alt_2_63_infVol}
\end{subfigure}
\hspace{.11cm}
\begin{subfigure}{.333\linewidth}
	\includegraphics[scale=0.33]{pd_alt_2_127_infVol}
\end{subfigure}
\hspace{-.4cm}
\begin{subfigure}{.333\linewidth}
	\includegraphics[scale=0.33]{pd_alt_2_255_infVol}
\end{subfigure}
\caption{From left to right: infinite-volume extrapolation of the $(\mu,T)$ phase
	diagram at fixed lattice spacing; top row: using $C_{\mathrm{short}}$
	(Eq.~\eqref{eq:cshort}, taken from \cite{LMW21}) ; bottom row: using
	$C_{\mathrm{long}}$ (Eq.~\eqref{eq:clong}).}
\label{fig:pd_infinite_volume}
\end{figure}

\begin{figure}
\begin{subfigure}{.333\linewidth}
	\includegraphics[scale=0.33]{pd_2_63_cont}
\end{subfigure}
\hspace{-2.3cm}
\begin{subfigure}{.333\linewidth}
	\includegraphics[scale=0.33]{pd_2_127_cont}
\end{subfigure}
\hspace{-2.8cm}
\begin{subfigure}{.333\linewidth}
	\includegraphics[scale=0.33]{pd_2_255_cont}
\end{subfigure}
\begin{subfigure}{.333\linewidth}
	\includegraphics[scale=0.33]{pd_alt_2_63_cont}
\end{subfigure}
\hspace{.11cm}
\begin{subfigure}{.333\linewidth}
	\includegraphics[scale=0.33]{pd_alt_2_127_cont}
\end{subfigure}
\hspace{-.4cm}
\begin{subfigure}{.333\linewidth}
	\includegraphics[scale=0.33]{pd_alt_2_255_cont}
\end{subfigure}
\caption{From left to right: continuum extrapolation of the $(\mu,T)$ phase
	diagram at roughly fixed physical volume; top row: using $C_{\mathrm{short}}$
	(Eq.~\eqref{eq:cshort}, taken from \cite{LMW21}); bottom row: using
	$C_{\mathrm{long}}$ (Eq.~\eqref{eq:clong}).}
\label{fig:pd_continuum}
\end{figure}

\clearpage