\section{Introduction}\label{Introduction}

Transformer-based language models have achieved promising results on natural language understanding tasks, \emph{e.g.,}\xspace Q\&A~\citep{qu2019bert,yang2020bert}, relation extraction~\citep{xue2020embarrassingly, zhou2020document} and dialogue system~\citep{ni2021recent}. Recently, on vision tasks, transformers~\citep{dosovitskiy2020image,zhou2021deepvit, xue2021go, xue2022one} also outperform convolution-based models by a large margin. With sufficient training data,  transformer-based models can be scaled to trillions of trainable parameters~\citep{fedus2021switch, du2021glam}. Through scaling along width (\emph{i.e.,}\xspace hidden dimension) and depth (\emph{i.e.,}\xspace number of transformer blocks), these huge transformers show effectiveness across various tasks and even areas.

When we use transformer, we typically follow existing work to set width and depth. For instance, we usually set the width of transformer-base model as 768 and the depth as 12. An interesting question here is: \textit{Why do we select these fixed hyper-parameters, even for problems in different areas?} To answer this question, we revisit the conventional configurations from representative studies. For vision transformer~\citep{dosovitskiy2020image}, authors set the base ViT configuration according to those used in BERT~\citep{devlin2018bert}.  BERT selects such configuration following OpenAI GPT~\citep{radford2018improving}. OpenAI also follows the original transformer paper~\citep{vaswani2017attention}. In the original transformer paper, \citet{vaswani2017attention} conduct a set of ablation studies on machine translation task to find the optimal configurations. That is, for a good range of tasks, we have largely follow the transformer configuration based on an ablation study on machine translation task, \emph{i.e.,}\xspace a sequence-to-sequence task.

Nowadays, transformer-based models can be trained with various different training objectives or strategies. Taking vision transformer~\citep{dosovitskiy2020image} as an example, we can train transformer from scratch with a supervised learning setting for image classification.
In this straightforward image classification task, each image is modeled as a sequence of multiple tokens, and each token corresponds to a patch in the image. We use the  global information (from all tokens/patches of the image) to predict a single label, the category of the image. Here, as the training objective is to capture the global information of an image, the differences between token representations would not be considered directly. This image classification task is quite different from  machine translation task, which requests for strong understanding of each input token and generating another sequence. Hence, it is natural to assume that different optimal transformer configurations exist for these two different tasks.

Previous work has tried to train a deeper transformer from scratch. However, as reported in~\citep{zhou2021deepvit,gong2021vision}, training by classification task (\emph{i.e.,}\xspace using the global signal of the input sequence) has the over-smoothing problem. That means, at the deeper transformer layers, all token representations tend to be identical. Such an issue harms the scalability of training vision transformer, especially scaling along depth. When we scale to a larger model, we only get a slight improvement or even worse accuracy. Recently, \citet{zhou2021deepvit, gong2021vision} show that, when we add special-designed regularization to avoid the ``uniform tokens'' (\emph{i.e.,}\xspace the over-smoothing problem), it is possible to train deeper transformer on the sequence (image) classification setting.

Different from training from scratch, the masked autoencoder is a two-stage training framework, including pre-training and fine-tuning. Given partially masked input sequence, the pre-training stage aims to recover the original unmasked sequence. The fine-tuning stage is similar to the aforementioned training from scratch.
With masked autoencoder, recent studies~\citep{bao2021beit,he2021masked} successfully train large scale transformers,  even without using additional training data compared with supervised learning. This is counterintuitive because we usually assume more training data is the key in self-supervised pre-training to improve effectiveness. However, masked autoencoder is able to scale transformer to larger or deeper without the need of bigger dataset. This result motivates us to rethink the reason behind it.

We hypothesize that masked autoencoder training can alleviate the over-smoothing issue, which is an important reason why masked autoencoder can help to scale transformer up. The main reason behind this is the different training objectives.
In masked autoencoder frameworks (\emph{e.g.,}\xspace BERT, BEiT), the target is to recover the masked tokens according to the unmasked tokens. Compared with training transformer from scratch under supervised learning, whose target is a simple classification task, masked autoencoder framework is using a sequence labeling target. The sequence labeling task requires the model to learn semantic information from neighboring unmasked tokens. Since different masked tokens have different unmasked neighboring tokens, the unmasked token representations must carry sufficient semantics for accurate prediction of the masked tokens, which in turn prevents the token representations to become identical. In a word, we can inference the masked autoencoder's training objective helps to alleviate over-smoothing problem by its regularization on token differences. To justify the reasoning above, we conduct both theoretical analysis and experimental verification. The results show that the over-smoothing issue is indeed alleviated in masked autoencoder. Compared with training under masked autoencder, training transformer by a simple classification task (\emph{e.g.,}\xspace training vision transformer from scratch) does not have such strength.

If the masked autoencoder well alleviates the over-smoothing issue, which is a challenge for scaling transformer along depth, does this mean the masked autoencoder prefers deeper models than wider ones for some tasks? We argue that we need to re-visit the configurations for different training objectives, especially for masked autoencoder. To this end, we conduct experiments to investigate the masked autoencoder configurations and propose our idea, Bamboo. Bamboo is, when we train transformer with masked autoencoder, we suggest to use deeper and narrower configurations to achieve better effectiveness with comparable computation cost as a typical setting. To evaluate our new configurations, we conduct comprehensive experiments on computer vision and natural language processing tasks. On vision tasks, we evaluate our configuration on large-scale vision transformer training. With Bamboo configurations, the masked autoencoders outperforms baselines by a large margin. For instance, on ImageNet, with comparable number of trainable parameters and computation cost, our narrower and deeper base-scale masked autoencoder, Bamboo-B, outperform MAE-B by 0.9\% in terms of top-1 accuracy. On natural language processing tasks, we conduct experiments on BERT. Results show that our configurations can improve BERT-L by 1.1 points on GLUE datasets.

In summary, our main contributions are three folds:
\begin{itemize}
\item  We propose the insight that masked autoencoder is better at scaling along depth. Training objective of masked autoencoder helps to handle transformer over-smoothing. We justify this reasoning by both theoretical analysis and experimental verification.

\item  We argue that the existing transformer configurations cannot fully use the strength of masked autoencoder. To this end, we propose Bamboo, an idea to scale transformer along depth with training with masked autoencoder. We show that narrower and deeper version show better performance than existing configurations, in a plug-and-play manner.

\item  We further verify our Bamboo configurations on larger scale vision transformer pre-training and language tasks. Results show that our Bamboo achieves state-of-the-art top-1 accuracy on image classification, and outperforms original BERT configurations by 1.1 points on GLUE.

\end{itemize}

The following sections are organized based on the analysis process of this work. In Section~\ref{Over-smoothing in Masked Autoencoder}, we briefly review the over-smoothing problem in transformer and show the strength of masked autoencoder in handling this issue theoretically and experimentally. We then conduct experiments to investigate scaling masked autoencoder along depth in Section~\ref{Bamboo}. Based on the consist sweet depth across scales, we suggest the Bamboo idea,
using narrower and deeper configurations in masked autoencoder training. Then, we adapt the new configurations to larger scale, and conduct evaluations across different areas, vision tasks in Section~\ref{Evaluation Vision} and NLP tasks in Section~\ref{Evaluation Language}. Finally, we discuss the difference between this work and existing ones in Section~\ref{Discussion}.

\section{Over-smoothing in Masked Autoencoder}\label{Over-smoothing in Masked Autoencoder}

The over-smoothing issue is common in graph neural networks. When we stack many graph convolution networks, the node representations tend to be identical~\citep{chen2020measuring}. Recent studies show that transformer has a similar problem~\citep{shi2021revisiting}, known as ``uniform tokens'' in transformer models. In deep transformer, each token representation can be seen as a node in the graph, and each attention score is an edge. Therefore, different token representations  tend to be identical when we scale transformer to deeper. In this work, we mainly focus on the over-smoothing problem due to different training settings.

\subsection{Theoretical Analysis}

We do theoretical analysis to compare the over-smoothing issue on sequence-level supervised learning, and masked antoencoder-based self-supervised learning, respectively. We use supervised vision transformer (ViT)~\citep{dosovitskiy2020image} and vision masked autoencoder (MAE)~\citep{he2021masked} as two representative platforms to show our insights in both theoretical and experimental analysis.

Formally, transformer block's output (\emph{i.e.,}\xspace token representations) can be written as $h=\{h_0, h_1, ..., h_{T-1}\}$, where $h_t \in \mathbb{R}^{d}$, $d$ is the hidden dimension, $T$ is the number of tokens at different transformer blocks. Usually, $T$ is fixed across transformer blocks. For the token representations after $l^{\mathrm{th}}$ transformer block, we denote the token representations as $h^l$.

ViT is trained from scratch. The input is $x=\{x_0, x_1, ..., x_{T-1}\}$, and the label is $y \in \mathbb{R}^C$, where $C$ is the number of classes. We analysis the over-smoothing issue when we use either the [CLS] token or average token representations as the prediction head.

If we use [CLS] token to classify an input image, the training target is to minimize the distance between $f(h^L_0)$ and $y$, where $f$ is a linear projection. Observe that the final classification only relays on one token representation from the [CLS] token. More importantly, when over-smoothing happens, \emph{i.e.,}\xspace $h^L_0=h^L_1=...=h^L_{T-1}$, the prediction head remains meaningful for the task. If we use the average pooling prediction head, we reach the same conclusion. In this case, we are to minimize the distance between $f(\bar{h}^L)$ and $y$, where $\bar{h}^L$ is the average pooling vector from $L^\mathrm{th}$ layer token representations. If $h^L_0=h^L_1=...=h^L_{T-1}=\bar{h}^L$, the classification output remains the same. However, model effectiveness will be limited since less attention layers are effectively used.

Then, what would happen if we use masked autoencoder to pre-train? For masked autoencoder, the input is $x=\{x_0, x_1, ..., x_{T-1}\}$. If $x_t$ token is masked, we denote this masked token as $x^m_t$. The set of the masked tokens is $x^m$. The corresponding original token set before masking is $\tilde{x}^m$. During masked autoencoder pre-training, we are to minimize the distance between $h^m$ and $\tilde{x}^m$. We now state a key lemma to show that the masked autoencoder training target regularizes token representations. For simplicity, we treat each token representation as a scalar to analyze the variance $\mathrm{Var(\cdot)}$. The following results can be extended to the vector case directly.

\begin{lemma}[Variance of final token representations is bounded in masked autoencoder.]\label{lemma: variance_1}
{In masked autoencoder, given the variance of masked token representation $\sigma^2$, the variance of final token representations is bounded by the masked tokens:
\begin{equation}
    \mathrm{Var}({h^L}^m) \ge  \sigma^2
\end{equation}
}
\end{lemma}
The proof of the Lemma above is in Appendix~\ref{proof_1}. As the transformer can also be seen as a weighted multi-view graph, given the Lemma above, following the over-smoothing issue in graph neural network, we can also infer that the intermediate token representations are also bounded:

\begin{lemma}[Variance of intermediate token representation is bounded in masked autoencoder.]\label{lemma: variance_2}
{In masked autoencoder, given the  variance of intermediate token representations from $l^{\mathrm{th}}$ layer is bounded by the variance of  the token representation from ${l+1}^{\mathrm{th}}$ layer:
\begin{equation}
    \mathrm{Var}({h^l}^m) > \mathrm{Var}({h^{l+1}}^m)
\end{equation}
}
\end{lemma}
The proof of the Lemma above is in Appendix~\ref{proof_2}. Given the theoretical analysis above, we can find both the intermediate and final token representations are bounded by the variance of the masked tokens, \emph{i.e.,}\xspace $\mathrm{Var}({h^l}^m)>\mathrm{Var}({h^{l+1}}^m)\ge  \sigma^2$. Therefore, the over-smoothing issue is alleviated in masked autoencoder.

In fact, revisiting the training of ViT from scratch again, Lemma~\ref{lemma: variance_2} also holds for the image classification setting. However, the difference is, Lemma~\ref{lemma: variance_1} cannot work on training ViT from scratch. Without the masked token modeling target in MAE, there is no bound to push last layer's token representations to be different. Since the Lemma~\ref{lemma: variance_2} still holds, the variance of token representations decreases layer-by-layer. When we scale transformer to deeper, over-smoothing happens.

We do not take residual connection into consideration in the two lemmas above. In reality, the residual connection is a key module in transformer training to alleviate over-smoothing. The residual connection would keep variance increasing along depth, although over-smoothing may still be observed~\citep{dong2021attention}. We thus, based on the two lemmas above, state the following theorem to show the variance in masked autoencoder increases faster than the variance in training transformer by simple classification setting.

\begin{theorem}[Variance in masked autoencoder increases faster than the variance in training transformer by simple classification.]\label{lemma: variance_3}
{Given a converaged masked autoencoder $g_1(\cdot)$ and transformer trained by simple classification $g_2(\cdot)$ with residual connection, the increase in variance $\Delta \mathrm{MAE}=\mathrm{Var}(x+g_1(x))-\mathrm{Var}(x)$ is larger than, $\Delta \mathrm{ViT}=\mathrm{Var}(x+g_2(x))-\mathrm{Var}(x)$:

\begin{equation}
    \Delta \mathrm{MAE} > \Delta \mathrm{ViT}
\end{equation}
}
\end{theorem}
The proof of the theorem above is in Appendix~\ref{proof_3}.

In summary, when we train transformer on the supervised learning setting directly and treat the task as a simple classification task, the transformer does not need to consider the differences across token representations. Such behavior means there is no regularization to avoid over-smoothing. For masked autoencoder training framework, the training target is to recover the masked tokens base on the neighboring tokens.
Since the different masked tokens have different semantic-related neighbors, the different unmasked tokens must cover different semantics in deeper layers. Therefore, although we are not regularizing the transformer model in a straightforward way, the masked autoencoder training framework still pushes the token representations to be different.

\subsection{Experimental Verification}

\begin{figure}[t]
\begin{minipage}[b]{0.4\linewidth}
\centering
\includegraphics[width=1.0\textwidth]{standard_deviation.pdf}
\subcaption{Average standard deviation}
\label{fig:over-smoothing-std}
\end{minipage}\hspace{5mm}
\begin{minipage}[b]{0.4\linewidth}
\centering
\includegraphics[width=1.0\textwidth]{cosine_similarity.pdf}
\subcaption{Average patch-pair cosine similarity}
\label{fig:over-smoothing-cos}
\end{minipage}
\centering
\vspace{-0.1cm}
\caption{Over-smoothing analysis of ViT and MAE, at different transformer layers.}
\label{fig:over-smoothing}
\vspace{-0.8cm}
\end{figure}

\textbf{Standard deviation.} To verify our reasoning above, we conduct two sets of quantitative analysis. For the first set of quantitative analysis, we focus on the standard deviation of token representations at different transformer layers. To compare the over-smoothing issue between supervised learning and masked autoencoder, we calculate the mean standard deviation of the token representations at different transformer layers:
\begin{equation}
\small
    {ms}^{i} =  \sqrt{\frac{1}{T-1} \sum\limits_{t=0}^{T-1}  (h_t^i - \bar{h^i})^2 }
\end{equation}
where $\bar{h^i}=\frac{1}{T-1}{\sum\limits_{t=0}^{T-1} h_t^i}$ is the mean vector of token representations at $i^{\mathrm{th}}$ transformer block.
We use a deeper version of ViT-L and MAE-L as platforms to validate our analysis above.  Note that, the ViT-L is trained on the supervised learning setting and treats image classification as a simple classification task, by average pooling prediction head. The MAE-L is trained on the masked autoencoder setting, which means the pre-training target is closed to sequence labeling task. We then fine-tune the pre-trained model for image classification. We set the width as 768 and the depth as 48. The total number of trainable parameters and computation cost is comparable with the original configuration (\emph{i.e.,}\xspace width is 1024, depth is 24). The results are shown in Figure~\ref{fig:over-smoothing-std}.

Observe that mean standard deviations of ViT-L and MAE-L both increase along depth. Such a result matches well with our theoretical analysis. The residual connection within transformer indeed alleviates over-smoothing (to some extent)~\citep{dong2021attention}. Nevertheless, the over-smoothing issue still exists even if we have residual connection as stated in~\citep{dong2021attention,shi2021revisiting}. We can observe that the increasing on MAE-L is much faster than that on ViT-L, which matches well with our Theorem~\ref{lemma: variance_3}. That means the deeper transformer blocks can learn different semantics for different token representations in MAE. In other words, the over-smoothing issue is alleviated in the transformer model pre-trained with the masked autoencoder setting.

\textbf{Patch-pair cosine similarity.} To further verify that the over-smoothing issue can be alleviated in masked autoencoder, following \citet{gong2021vision}, we compare the patch-pair cosine similarity between ViT-L and MAE-L with deeper and narrower configuration. If there is an over-smoothing issue in the model, we should observe that the patch-pair cosine similarity increases along depth. The more serious the over-smoothing issue is, the faster cosine similarity increases. To remove the impact from input representations (residual connection), we use the zero-centered token representations $\tilde{h^i}= h^i - \bar{h^i}$ for evaluation. The results are shown in Figure~\ref{fig:over-smoothing-cos}. The cosine similarity of ViT increases along depth due to over-smoothing. However, for the model pre-trained by masked autoencoder framework, cosine similarity keeps constant along depth. This comparison is interesting, as we can barely observe over-smoothing issue on the model pre-trained by the masked autoencoder, even if we are using a deeper model than usual.

\section{Bamboo}\label{Bamboo}

Our analysis above shows that the masked antoencoder  helps to alleviate the over-smoothing issue, which is a main challenge in scaling transformer along depth. Based on this finding, we propose Bamboo, an idea of using deeper and narrower transformer configurations, for masked autoencoder pre-training. Certainly,  deeper and narrower configurations do not always improve performance, as there are a few other reasons hindering scaling transformer along depth. In this section, we devote to find a sweet point following Bamboo idea, to achieve effective masked autoencoder by experiments.

\begin{table}[t]
\small
\caption{The configurations we used to scale transformer along depth. For a fair comparison, we keep a comparable computation cost with the original transformer configurations (\emph{i.e.,}\xspace depth=12 for Base, depth=24 for Large).}
\vspace{5pt}
\centering
\begin{tabular}{l llll llll}
\toprule
Scale                  & \multicolumn{4}{l}{Base} & \multicolumn{4}{l}{Large} \\  \cmidrule(lr){2-5}\cmidrule(lr){6-9}
Depth             & 12   & 24   & 48   & 96  & 24    & 48   & 60   & 96  \\
Width             & 768  & 512  & 384  & 256 & 1024  & 768  & 640  & 512 \\
\#Attention Heads & 12   & 8    & 6    & 4   & 16    & 12   & 10   & 8   \\
Computation Cost  & 1$\times$    & 0.9$\times$  & 1$\times$    & 0.9$\times$ & 1$\times$    & 1.1$\times$  & 1$\times$    & 1$\times$  \\ \bottomrule
\end{tabular}
\vspace{-0.3cm}
\label{tbl:config-explore}
\end{table}

To validate our reasoning above, we conduct experiments on training transformer on ILSVRC-2012 ImageNet~\citep{deng2009imagenet} (ImageNet-1K), and report the top-1 accuracy. We use the original viT trained by supervised learning and MAE pre-trained masked image modeling as the test platform. For a fair comparison with the original transformer configurations, when we scale transformer to deeper, we also reduce the width to keep comparable number of trainable parameters and computation cost. The configurations we used are summarized in Table~\ref{tbl:config-explore}.

For supervised learning models (\emph{i.e.,}\xspace ViT), we train base scale models for 300 epochs and large scale models for 200 epochs following \citet{he2021masked}. We use RandAugment~\citep{cubuk2020randaugment}, drop path~\citep{huang2016deep}, mixup, cutmix~\citep{yun2019cutmix}, label smoothing~\citep{szegedy2016rethinking} for data augmentation.  Detailed hyper-parameters are summarized in Appendix~\ref{appendix:hyper-parameter}. For the masked autoencoder, we pre-train the base scale models for 1600 epochs and fine-tune for 100 epochs. For large models, we pre-train for 800 epochs and fine-tune for 50 epochs. We use the per-patch normalization following \citet{he2021masked} for better representations.

\begin{figure}[t]
\begin{minipage}[b]{0.4\linewidth}
\centering
\includegraphics[width=1.0\textwidth]{base-sweet.pdf}
\subcaption{Scaling supervised ViT along depth}
\label{fig:sweet-base}
\end{minipage}\hspace{5mm}
\begin{minipage}[b]{0.4\linewidth}
\centering
\includegraphics[width=1.0\textwidth]{large-sweet.pdf}
\subcaption{Scaling MAE along depth}
\label{fig:sweet-large}
\end{minipage}
\centering
\caption{Scaling ViT and MAE with comparable number of parameters and computation cost.}
\label{fig:sweet}
\vspace{-0.6cm}
\end{figure}

The results of scaling to deeper transformers are summarized in Figure~\ref{fig:sweet}. For both the base-scale and large-scale models, we report the top-1 accuracy on ImageNet-1K dataset. We can observe the models trained by supervised image classification directly (\emph{i.e.,}\xspace ViT-B and ViT-L) cannot improve accuracy when we use the deeper architectures. For ViT-B, the top-1 accuracy on ImageNet-1K keeps comparable when the depth is smaller than 50, but after that, there is a significant accuracy drop. For ViT-L, the accuracy decreases even faster than the ViT-B. There is a significant accuracy drop when we use the narrower and deeper models. However, for masked autoencoders, we can observe totally different patterns. Even if we keep the comparable trainable parameters and computation cost, with such a simple modification, the masked autoencoders gain significant improvements. More importantly, when we scale to 48 layers, both
MAE-B and MAE-L reach sweet points. When scaling to 96 layers, we observe the training is unstable compared with shallower models. This result matches well with our observation in Figure~\ref{fig:over-smoothing-cos}. At around 40th layer, the over-smoothing issue starts to happen slightly in masked autoencoder, which is much later than the ViT model. We suggest another reason of the unstable deep model training is the too large model updates~\citep{wang2022deepnet}. In this work, we focus on the over-smoothing issue of deep transformer training instead. The large model updates in deeper layers are out-of-scope. We leave that as our future work.

\begin{table}[t]
\small
\caption{Re-designed configurations under Bamboo idea.}
\vspace{5pt}
\centering
\begin{tabular}{l ll ll ll}
\toprule
Scale                  & \multicolumn{2}{l}{Base} & \multicolumn{2}{l}{Large} & \multicolumn{2}{l}{Huge} \\  \cmidrule(lr){2-3} \cmidrule(lr){4-5}\cmidrule(lr){6-7}
                 & Original & Bamboo & Original & Bamboo & Original & Bamboo  \\ \midrule
Depth             & 12   & 48   & 24   & 48    & 32   & 64    \\
Width             & 768  & 384  & 1024  & 768 & 1280  & 896 \\
\#Attention Heads & 12   & 6    & 16    & 12   & 16    & 14    \\
Computation Cost  & 1$\times$    & 1$\times$  & 1$\times$    & 1.1$\times$ & 1$\times$    & 1$\times$   \\ \bottomrule
\end{tabular}
\label{tbl:bamboo}
\end{table}

According to the experimental results above, we find that masked autoencoder can indeed scale transformer well along depth. Even if we keep comparable trainable parameters and computation cost with the original transformer, the model achieves better accuracy. Another observation is, both transformer-base and transformer-large reach their sweet points at around 50 layers. We thus recommend a new set of transformer configurations in Table~\ref{tbl:bamboo} following our Bamboo idea, which are deeper and narrower than the original transformer configurations.

\section{Evaluation on Vision Tasks}\label{Evaluation Vision}

\subsection{Settings}

We further evaluate our deeper and narrower transformer configurations on vision task. We train different models with more training epochs and compare with state-of-the-art vision models. We conduct experiments on ImageNet-1K and compare with recent supervised vision models \emph{e.g.,}\xspace DeepViT~\citep{zhou2021deepvit} and DeiT~\citep{touvron2021training},  and self-supervised vision models \emph{e.g.,}\xspace DINO~\citep{caron2021emerging}, MoCo v3~\citep{chen2021empirical}, BEiT~\citep{bao2021beit} and MAE~\citep{he2021masked}. Compared with the MAE, \textit{the only difference is the configurations}. That is, MAE uses original transformer configurations and we use our Bamboo configurations. The experiments are to verify that such a simple modification can improve model effectiveness.

We evaluate the models on three different scales, \emph{i.e.,}\xspace base, large, and huge. The data augmentation setting is exactly the same as MAE for a fair comparison. Again, the only difference is that we use the Bamboo configurations instead of the original transformer configurations. During fine-tuning, we use the same script with training ViT from scratch. We fine-tune base models for 100 epochs. For large and huge models, we fine-tune them for 50 epochs following existing work~\citep{bao2021beit, he2021masked}.

\begin{table}[t]
\vspace{-0.5cm}
\small
\caption{Top-1 accuracy on ImageNet-1K. We report two versions of ViT training from scratch. The first one is from original ViT paper~\citep{dosovitskiy2020image}, and the second one is from \citet{he2021masked}'s re-implementation with strong data augmentation. For MAE-B, we reproduce the results by running the official code and obtain a slightly different result (denoted by 83.3*). The original result is 83.6.}
\vspace{5pt}
\centering
\begin{tabular}{l llll}

\toprule
Method             & Pre-train Data    & Base & Large & Huge \\ \midrule
ViT from scratch~\citep{dosovitskiy2020image}            & - & 77.9 & 76.5 & - \\
DeepViT~\citep{zhou2021deepvit}             & - & 80.9 & - & - \\
DeiT~\citep{touvron2021training}             & - & 81.8 & - & - \\
ViT from scratch~\citep{he2021masked}            & - & 82.1 & 81.5 & 80.9  \\ \midrule
DINO~\citep{caron2021emerging}             & IN1K & 82.8 & - & -  \\
MoCo v3~\citep{chen2021empirical}             & IN1K & 83.2 & 84.1 & -  \\ \midrule
BEiT~\citep{bao2021beit}             & IN1K + DALL-E & 83.2 & 85.2 & -  \\
MaskFeat~\citep{wei2021masked}             & IN1K & 84.0 & 85.7 & -  \\
IBOT~\citep{zhou2021ibot}             & IN1K & 84.0 & 84.8 & -  \\ \midrule
MAE~\citep{he2021masked} (Direct baseline)           & IN1K & 83.3* & 85.9 & 86.9 \\
Bamboo (Ours)             & IN1K & \textbf{84.2} & \textbf{86.3} & \textbf{87.1} \\\bottomrule
\end{tabular}
\label{tbl:exp-cv-sota}
\end{table}

\subsection{Results}

Results on ImageNet-1K are reported in Table~\ref{tbl:exp-cv-sota}. For training ViT from scratch, we report the original results~\citep{dosovitskiy2020image} and the results with strong data augmentation~\citep{he2021masked}. A few recent work~\citep{zhou2021deepvit,touvron2021training} focusing on training ViT from scratch on ImageNet-1K are also included. In general, we can find the models pre-trained by self-supervised learning (\emph{e.g.,}\xspace DINO, MoCO v3, MAE) perform much better than training from scratch. If we only consider the self-supervised learning approaches, masked image modeling-based methods (\emph{e.g.,}\xspace BEiT, MaskFeat, IBOT, MAE) outperform the contrastive learning-based methods (\emph{e.g.,}\xspace DINO, MoCo v3) significantly, especially on larger scale.

Since we are focusing on the scalability of training transformer with masked autoencoder, we choose MAE as our direct baseline, recent work with better scalability. We train MAE with Bamboo configurations and report the results in Table~\ref{tbl:exp-cv-sota}. Observe that our Bamboo achieves the best top-1 accuracy on all scales. On the base scale, Bamboo achieves state-of-the-art performance, 84.2 top-1 accuracy, which is 0.9 (0.6) points higher than MAE-B. When we scale the model up to large scale and huge scale, Bamboo remains the best performer and achieves 86.3 and 87.1 top-1 accuracy respectively. Note that, compared to other scales, the improvement is not so significant on the huge scale. One reason is, the original huge configuration has been deep (\emph{i.e.,}\xspace 32 layers), which is closer to the sweet point. Similarly, this can also explain why the configurations designed under Bamboo can improve MAE-B significantly.

\section{Evaluation on Language Tasks}\label{Evaluation Language}

\subsection{Settings}

We further evaluate our Bamboo configurations on language tasks. We select BERT~\citep{devlin2018bert} as the platform to evaluate our Bamboo configuration because it is widely used on many language tasks. Note that BERT is a post-layer normalization transformer model. If our configuration shows better performance on BERT, that means it has good generalization performance.  We follow the BERT paper to use Wikipedia and bookscorpus to pre-train. The masked ratio is 15\% as by default. During pre-training, we use LAMB optimizer and set batch size and learning rate as 4096 and 1.76e-3, respectively, following~\citet{you2019large}.

During fine-tuning, we conduct experiments on General Language Understanding Evaluation (GLUE) benchmark.
The GLUE benchmark~\citep{wang-etal-2018-glue} is widely used in natural language understanding tasks, which include 8 tasks, \emph{i.e.,}\xspace CoLA, MNLI, MRPC, QNLI, QQP, RTE, SST-2 and STS-B. We set the learning rate as 1e-5 or 2e-5. Batch size is fixed as 32. For small datasets, \emph{i.e.,}\xspace CoLA, MRPC, RTE and STS-B, we fine-tune for 10 epochs. For larger datasets, \emph{i.e.,}\xspace MNLI, QNLI, QQP and SST-2, we fine-tune for 3 epochs. Matthew’s correlation is used as metric for CoLA. For MNLI, we report the average accuracy on MNLI-m and MNLI-mm. QNLI and RTE also adapt the accuracy as metric. The results on MRPC and QQP are reported with the average of F1 and accuracy. We use Spearman correlation on STS-B. For all fine-tuning results, we run the code 5 times and report the median.

\begin{table*}[t]
\small
\caption{Results of fine-tuning on GLUE benchmark. }
\vspace{-0.3cm}
\label{nlp:table:main}
\begin{center}
\begin{tabular}{l llllllll l}

\toprule
Method                   & CoLA & MNLI & MRPC & QNLI & QQP & RTE & SST-2  & STS-B & Avg \\ \midrule
BERT-B   & 59.6 & 83.7 & 88 & 90.4 & 88.8 &  68.6  & 91.5 & 89.4 & 82.5 \\
BERT-L   & 60.9 & 86.2 & 89.3 & 92.3 & \textbf{89.6} &  73.1  & 92.5 & 90.4 & 84.3 \\ \midrule
Bamboo-B   & 60.5 & 84.3 & 88.4 & 90.5 & 89.0 & 70.0 &  92.2  & 89.5 & 83.1 \\
Bamboo-L   & \textbf{62.9} & \textbf{87.1} & \textbf{89.8} & \textbf{92.4} & 89.4 &  \textbf{77.3}  & \textbf{93.8} & \textbf{90.6} & \textbf{85.4} \\
\bottomrule
\end{tabular}
\end{center}
\vspace{-0.6cm}
\end{table*}

\subsection{Results}

The results on language tasks are reported in Table~\ref{nlp:table:main}. Under the same pre-training and fine-tuning settings, models with Bamboo configurations outperform BERT by a large margin. Bamboo-L achieves the best performance in Table~\ref{nlp:table:main}. Compared with BERT-L, our Bamboo-L wins on 7 out of 8 datasets, and surpassed BERT-L by 1.1 points on average. It is also notable Bamboo configurations can outperform baselines on both large datasets (\emph{e.g.,}\xspace MNLI) and small datasets (\emph{e.g.,}\xspace CoLA).

\section{Discussion}\label{Discussion}

\textbf{Compared with simply scaling along depth}, this work keeps comparable computation cost and number of trainable parameters. When scaling along depth, we also make the deep transformer narrower. Under such setting, the deeper and narrower configurations re-designed under Bamboo idea can still outperform the baseline configurations, suggesting that we should consider narrower and deeper transformer when training by masked autoencoder. One related work is~\citep{tay2021scale}, an empirical study of practical scaling of transformer, which has a similar observation, deeper and narrower can improve accuracy. On this basis, our insight is, the insight in~\citep{tay2021scale} depends on different training objectives.

\textbf{Compared with brute-force hyper-parameter tuning}, we provide both theoretical analysis and experimental verification to justify our insight, \emph{i.e.,}\xspace masked autoencoder alleviates over-smoothing issue in transformer. Motivated by this, we propose Bamboo by experimental investigation. There is no guarantee to ensure the configurations are always the optimal ones as different tasks may require different configurations. We believe it is impossible in deep learning to know the optimal configurations before experiments. However, we justify the tendency that masked autoencoder gets more benefits from deeper configurations. Our insight may instruct future works to consider configurations according to the training objectives.

\textbf{Instead of proposing a new approach or a new set of configuration,} this work focuses on an existing but neglected problem. After revisiting, we find configurations should be re-designed for different training objectives. We then re-design a set of configurations for masked autoencoder, which can improve transformer on both vision and language tasks. On vision tasks, we even achieve SoTA top-1 accuracy on ImageNet. However, we highlight our main contribution is an insight instead of a SoTA model, and this insight is orthogonal to the advanced masked antoencoder modifications in the future.

\section{Conclusion}\label{Conclusion}

We first rethink the masked autoencoder's scalability in this work. Through theoretical analysis, we show that training transformer with masked autoencoder can alleviate the over-smoothing issue when scaling transformer along depth. We then verify this finding through experiments. The results show that, compared with supervised learning, model pre-trained by masked autoencoder can handle over-smoothing (\emph{i.e.,}\xspace uniform tokens) better. Under this insight, we rethink the widely used configurations in vision and language transformer, and suggest deeper transformer when training with masked autoencoder. We then explore this strategy on base and large scale vision transformer and re-design a set of vision transformer configurations for masked autoencoder empirically. To further verify the effectiveness of our configuration, we conduct experiments on larger scale vision transformer training and achieved SoTA top-1 accuracy with comparable computation cost on ImageNet-1K. To show our configurations are better than original transformer configurations, we further conduct experiments on language tasks. Compared with widely used language masked autoencoder, BERT, our model surpass it by 1.1 points on average on GLUE benchmark. In summary, our work provides an insight to explain why masked autoencoder can scale transformer along depth, and we propose Bamboo, a set of configurations to fully utilize such strength.

In the future, we plan to integrate the existing methods focusing on scaling along depth, \emph{e.g.,}\xspace DeepNorm~\citep{wang2022deepnet} to further scale transformer. Another future direction is to verify our Bamboo configurations on other masked autoencoder-based models (\emph{e.g.,}\xspace BEiT~\citep{bao2021beit}, RoBERTa~\citep{liu2019roberta}).

\printbibliography

\newpage

\section*{Checklist}

\begin{enumerate}

\item For all authors...
\begin{enumerate}
  \item Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

    \answerYes{}
  \item Did you describe the limitations of your work?

    \answerYes{}
  \item Did you discuss any potential negative societal impacts of your work?

    \answerNA{}
  \item Have you read the ethics review guidelines and ensured that your paper conforms to them?

    \answerYes{}
\end{enumerate}

\item If you are including theoretical results...
\begin{enumerate}
  \item Did you state the full set of assumptions of all theoretical results?
    \answerYes{}
  \item Did you include complete proofs of all theoretical results?
    \answerYes{}
\end{enumerate}

\item If you ran experiments...
\begin{enumerate}
  \item Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

    \answerYes{}
  \item Did you specify all the training details (\emph{e.g.,}\xspace, data splits, hyperparameters, how they were chosen)?
    \answerYes{}
    \item Did you report error bars (\emph{e.g.,}\xspace, with respect to the random seed after running experiments multiple times)?
    \answerYes{}
    \item Did you include the total amount of compute and the type of resources used (\emph{e.g.,}\xspace, type of GPUs, internal cluster, or cloud provider)?
    \answerYes{}
\end{enumerate}

\item If you are using existing assets (\emph{e.g.,}\xspace, code, data, models) or curating/releasing new assets...
\begin{enumerate}
  \item If your work uses existing assets, did you cite the creators?
    \answerYes{}
  \item Did you mention the license of the assets?
    \answerYes{}
  \item Did you include any new assets either in the supplemental material or as a URL?
    \answerYes{}
  \item Did you discuss whether and how consent was obtained from people whose data you're using/curating?
    \answerNA{}
  \item Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
    \answerYes{}
\end{enumerate}

\item If you used crowdsourcing or conducted research with human subjects...
\begin{enumerate}
  \item Did you include the full text of instructions given to participants and screenshots, if applicable?
    \answerNA{}
  \item Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
    \answerNA{}
  \item Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
    \answerNA{}
\end{enumerate}

\end{enumerate}

\newpage


\section{Introduction}\label{sec:one}

During the 2010s, microlensing entered an era of high-cadence surveys with the instrumental
upgrade of previously established experiments and the participation of a new experiment.
The new era started when the Microlensing Observations in Astrophysics survey
\citep[MOA:][]{Bond2001}, which had previously carried out a lensing survey using a 0.61~m telescope
and a camera with a 1.3~deg$^{2}$ field of view (FOV) during the early phase, launched its second
phase experiment with the employment of a 1.8~m telescope equipped with a wide-field camera yielding
a 2.2~deg$^{2}$ FOV. Since its first operation in 1992, the Optical Gravitational Lensing Experiment
(OGLE) has been upgraded multiple times, and it is in its fourth phase \citep[OGLE-IV:][]{Udalski2015}
with the employment of a 1.3~m telescope and a camera with a 1.4~deg$^{2}$ FOV. The Korea
Microlensing Telescope Network \citep[KMTNet:][]{Kim2016}, which was launched in 2016, is being
carried out with the use of three globally distributed 1.6~m telescopes, each of which is
mounted with a wide-field detector providing a 4~deg$^{2}$ FOV. With the use of wide-field cameras
mounted on multiple telescopes, the current lensing surveys can monitor lensing events with a
dramatically enhanced observational cadence.

\begin{table*}[t]
\small
\caption{Coordinates, fields, alert dates, and baseline magnitudes\label{table:one}}
\begin{tabular}{llllll}
\hline\hline
\multicolumn{1}{c}{Event}                     &
\multicolumn{1}{c}{(RA, decl)$_{\rm J2000}$}  &
\multicolumn{1}{c}{$(l, b)$}                  &
\multicolumn{1}{c}{Field}                     &
\multicolumn{1}{c}{Alert date}                &
\multicolumn{1}{c}{$I_{\rm base}$ (mag)}      \\
\hline
 OGLE-2017-BLG-1691  & (17:34:22.52, -29:17:05.39)    &  $(-1^\circ\hskip-2pt .601,  1^\circ\hskip-2pt .895)$   &  OGLE (BLG654.31)     &  2017 Sep 6  &  19.90  \\
 (KMT-2017-BLG-0752) &                                &                                                         &  KMT (BLG14)          &  postseason  &         \\
\hline
 KMT-2021-BLG-0320   & (17:57:33.15, -30:30:14.18)    &  $( 2^\circ\hskip-2pt .095, -4^\circ\hskip-2pt .299)$   &  KMT (BLG01, BLG41)   &  2021 Apr 9  &  20.08  \\
\hline
 KMT-2021-BLG-1303   & (18:07:27.33, -29:16:53.69)    &  $(-0^\circ\hskip-2pt .027, -3^\circ\hskip-2pt .028)$   &  KMT (BLG04)          &  2021 Jun 14 &  19.67  \\
 (MOA-2021-BLG-182)  &                                &                                                         &  MOA (gb13)           &  2021 Jun 17 &         \\
\hline
 KMT-2021-BLG-1554   & (17:51:12.82, -31:51:51.70)    &  $(-1^\circ\hskip-2pt .888, -2^\circ\hskip-2pt .543)$   &  KMT (BLG01, BLG41)   &  2021 Jul 1  &  22.39  \\
\hline
\end{tabular}
\tablefoot{ ${\rm HJD}^\prime = {\rm HJD}- 2450000$.  }
\end{table*}

The greatly increased observational cadence of the lensing surveys has brought out important
changes not only in the observational strategy but also in the outcome of planetary microlensing
searches.  First, being able to resolve short-lasting planetary signals from survey observations,
the high-cadence surveys can function without the survey+followup experiment mode, in which
low-cadence surveys mainly detect and alert lensing events and followup groups densely observe
the alerted events.  Second, the number of microlensing planets has substantially increased
with the operation of the high-cadence surveys.  Among the total 135 microlensing planets with
measured mass ratios\footnote{The sample includes 119 planets listed in the NASA Exoplanet
Archive (https://exoplanetarchive.ipac.caltech.edu/index.html) plus 16 planets that are not
included in the archive:
KMT-2020-BLG-0414Lb \citep{Zang2021a},
OGLE-2019-BLG-1053Lb \citep{Zang2021b},
KMT-2019-BLG-0253Lb, KMT-2019-BLG-0953Lb, OGLE-2018-BLG-0977Lb,
OGLE-2018-BLG-0506Lb, OGLE-2018-BLG-0516Lb, OGLE-2019-BLG-1492Lb \citep{Hwang2022},
KMT-2017-BLG-2509Lb,  OGLE-2017-BLG-1099Lb, OGLE-2019-BLG-0299Lb \citep{Han2021},
OGLE-2016-BLG-1093Lb \citep{Shin2022},
KMT-2018-BLG-1988Lb \citep{Han2022b},
KMT-2021-BLG-0912Lb \citep{Han2022a},
OGLE-2019-BLG-0468Lb, and OGLE-2019-BLG-0468Lc \citep{Han2022c}.},
80 (59\%) planets were found with the use of the data from the KMTNet survey.  This can be seen
in Figure~\ref{fig:one}, in which we present the histogram of microlensing planets as a function
of the planet-to-host mass ratio $q$.  The histogram also shows that the high-cadence surveys
contribute to the detections of planets with very low mass ratios, especially those with $q<10^{-4}$.

\begin{figure}[t]
\includegraphics[width=\columnwidth]{f1.eps}
\caption{
Histogram of published microlensing planets as a function of the planet-to-host mass ratio. The
histogram of planets detected with the data from KMTNet survey is separately marked in brown.
The solid dots with error bars indicate the four planets reported in this work.
}
\label{fig:one}
\end{figure}

In this paper, we report four planetary systems with low planet-to-host mass ratios detected
from the microlensing surveys. Despite the short durations, ranging from a few hours to a couple
of days, the planetary signals were clearly detected by the combined data of the lensing
surveys. It is found that all of the discovered planets have sub-Jovian masses lying in the range
of [0.05--0.38]~$M_{\rm J}$, illustrating the importance of the high-cadence surveys in detecting
low-mass planets.

We present the analysis of the planetary lensing events according to the following organization.
In Sect.~\ref{sec:two}, we describe the observations of the individual lensing events, instrument
used for observations, and the process of the data reduction. In Sect.~\ref{sec:three}, we explain
the procedure of analysis that is applied to each of the individual events. In the subsequent
subsections, we describe the detailed features of the planetary signals and present the results
of the analyses conducted for the individual lensing events.  In Sect.~\ref{sec:four}, we specify
the types of the source stars and estimate the angular Einstein radii of the lens systems. In
Sect.~\ref{sec:five}, we estimate the physical parameters of the planetary systems by conducting
Bayesian analyses. We summarize the results of the analyses and conclude in Sect.~\ref{sec:six}.

\section{Observations and data}\label{sec:two}

The four lensing events for which we present analyses are (1) OGLE-2017-BLG-1691 (KMT-2017-BLG-0752),
(2) KMT-2021-BLG-0320, (3) KMT-2021-BLG-1303 (MOA-2021-BLG-182), and (4) KMT-2021-BLG-1554.
The source stars of all events lie toward the Galactic bulge field. In Table~\ref{table:one},
we list the equatorial and galactic coordinates, observation fields of the surveys, alert
dates, and baseline magnitudes of the individual events.  The baseline magnitude $I_{\rm base}$
is approximately scaled to the OGLE-III photometry system.

The events were detected from the combined observations of the lensing surveys conducted by the
KMTNet, OGLE, and MOA groups. The KMTNet survey utilizes three identical 1.6~m telescopes that are
distributed in three countries of the Southern Hemisphere for the continuous coverage of lensing
events. The sites of the individual telescopes are the Siding Spring Observatory in Australia (KMTA),
the Cerro Tololo InterAmerican Observatory in Chile  (KMTC), and the South African Astronomical
Observatory in South Africa (KMTS).  Observations by the OGLE survey were conducted using the
1.3~m telescope located at Las Campanas Observatory in Chile.  The MOA survey uses the 1.8~m
telescope at the Mt.~John Observatory in New Zealand.  Observations were done mainly in the $I$
band for the KMTNet and OGLE surveys and in the customized MOA-$R$ band for the MOA survey. A subset
of images were acquired in the $V$ band for the purpose of estimating source colors. The detailed
procedure of the source color estimation will be discussed in Sect.~\ref{sec:four}.

\begin{table}[t]
\small
\caption{Error bar readjustment factors\label{table:two}}
\begin{tabular*}{\columnwidth}{@{\extracolsep{\fill}}llccc}
\hline\hline
\multicolumn{1}{c}{Event}                        &
\multicolumn{1}{c}{Data set}                     &
\multicolumn{1}{c}{$k$}                          &
\multicolumn{1}{c}{$\sigma_{\rm min}$ (mag)}     \\
\hline
 OGLE-2017-BLG-1691  & OGLE          & 1.010   & 0.010  \\
                     & KMTA          & 0.866   & 0.040  \\
                     & KMTC          & 1.170   & 0.010  \\
                     & KMTS          & 1.131   & 0.020  \\
\hline
 KMT-2021-BLG-0320   & KMTA (BLG01)  & 1.127   & 0.040  \\
                     & KMTA (BLG41)  & 1.389   & 0.025  \\
                     & KMTC (BLG01)  & 1.091   & 0.020  \\
                     & KMTC (BLG41)  & 1.175   & 0.020  \\
                     & KMTS (BLG01)  & 1.356   & 0.010  \\
                     & KMTS (BLG41)  & 1.319   & 0.010  \\
\hline
 KMT-2021-BLG-1303   & KMTA          & 1.304   & 0.020  \\
                     & KMTC          & 1.237   & 0.010  \\
                     & KMTS          & 1.364   & 0.010  \\
                     & MOA           & 1.034   & 0.020  \\
\hline
 KMT-2021-BLG-1554   & KMTA (BLG01)  & 1.371   & 0.020  \\
                     & KMTA (BLG41)  & 1.446   & 0.020  \\
                     & KMTC (BLG01)  & 1.418   & 0.020  \\
                     & KMTC (BLG41)  & 1.367   & 0.020  \\
                     & KMTS (BLG41)  & 1.364   & 0.020  \\
\hline
\end{tabular*}
\end{table}

Reduction and photometry of the data were done using the pipelines of the individual survey
groups: \citet{Albrow2009} for the KMTNet survey, \citet{Wozniak2000} for the OGLE survey, and
\citet{Bond2001} for the MOA survey. All these pipelines apply the difference image analysis
algorithm \citep{Tomaney1996, Alard1998} developed for the optimal photometry of stars lying in
very dense star fields.  For a subset of the KMTC data set, we carried out additional photometry
utilizing the pyDIA code \citep{Albrow2017} for the specification of the source colors.  Following
the routine described in \citet{Yee2012}, we rescale the error bars of data by $\sigma=k
(\sigma_{\rm min}^2+\sigma_0^2)^{1/2}$, where $\sigma_0$ represents the error estimated from
the photometry pipeline, $\sigma_{\rm min}$ is a factor used to make the data consistent with
the scatter of data, and the factor $k$ is used to make $\chi^2$ per degree of freedom for
each data set become unity.  In Table~\ref{table:two}, we list the factors $k$ and
$\sigma_{\rm min}$ of the individual data sets.

\section{Analyses}\label{sec:three}

The light curves of the analyzed events share a common characteristic that short-lived anomalies
appear on the otherwise smooth and symmetric form of a single-lens single-source (1L1S) event.
Such anomalies in lensing light curves can be produced by two channels, in which the first
channel is a perturbation induced by a low-mass companion, such as a planet, to the lens
\citep{Gould1992b}, and the second channel is an anomaly caused by a faint companion to the
source \citep{Gaudi1998}. Hereafter we denote events with a binary lens and a binary source as
2L1S and 1L2S events, respectively.  We examine the origins of the anomalies by modeling the
light curve of the individual events under these 2L1S and 1L2S interpretations.

Lensing light curves are described by the combination of various lensing parameters. For
a 1L1S event, the light curve is described by three parameters of $(t_0, u_0, t_{\rm E})$,
which denote the time of the closest approach of the source to the lens, the separation
(scaled to the angular Einstein radius $\theta_{\rm E}$) between the source and lens at that time,
and the time scale of the event, respectively. The event time scale is defined as the time
required for a source to transit $\theta_{\rm E}$.

In addition to these basic parameters, describing the light curve of a 2L1S event requires
one to include extra parameters of $(s, q, \alpha)$, which represent the separation (scaled
to $\theta_{\rm E}$) and mass ratio between the binary lens components, and the angle between the
binary-lens axis and the direction of the source motion (source trajectory angle), respectively.
Because planet-induced anomalies are usually generated by the crossings over or close approach
of the source to the planet-induced caustics, an additional parameter of the normalized source
radius $\rho$, which is defined as the ratio of the angular radius of the source
$\theta_*$ to $\theta_{\rm E}$, is needed to describe the deformation of the anomaly by finite-source
effects \citep{Bennett1996}.  In computing finite-source magnifications, we consider the
limb-darkening variation of the source.

One also needs extra parameters to describe the lensing light curve of a 1L2S event. These
parameters are $(t_{0,2}, u_{0,2}, q_F)$, which represent the closest approach time and
separation between the lens and the second source, and the flux ratio between the binary
source stars, respectively. In the 1L2S model, we designate the lensing parameters related
to the primary source, $S_1$, as $(t_{0,1}, u_{0,1})$ to distinguish them from those related
to the source companion, $S_2$.  In order to consider finite-source effects occurring when
the lens passes over either of the source stars, we add two additional parameters of the
normalized source radii $\rho_1$ and $\rho_2$ for the lens transits over $S_1$ and $S_2$,
respectively \citep{Dominik2019}.

In modeling 1L1S and 1L2S light curves,  for which the lensing magnification is smooth with
the variation of the lensing parameters, we search for the solution of the lensing parameters
via a downhill approach by minimizing $\chi^2$ using the Markow Chain Monte Carlo (MCMC)
algorithm. In modeling 2L1S light curves, for which the lensing magnification variation is
discontinuous due to the formation of caustics and there may exist multiple local solutions
caused by various types of degeneracy, we model the light curves in two steps. In the first step,
we conduct grid searches for the binary parameters $s$ and $q$, construct a $\chi^2$ map on
the $s$--$q$ parameter plane, and then identify local solutions appearing on the $\chi^2$ map.
In the second step, we polish the individual local solutions using the downhill approach. Below
we present the details of the modeling conducted for the individual lensing events.

\begin{figure}[t]
\includegraphics[width=\columnwidth]{f2.eps}
\caption{
Lensing light curve of OGLE-2017-BLG-1691. The bottom panel shows the whole view, and the
top panel displays the enlargement of the region around the anomaly (the region enclosed by
a box in the bottom panel). In the top panel, curves of the three tested models, 2L1S (outer),
1L2S, and 1L1S models, are drawn over the data points, and the residuals from the individual
models are presented in the three middle panels. The curve drawn in the bottom panel is the
best-fit model (outer 2L1S model). Colors of data points are set to match those of the
telescopes used for observations marked in the legend. The curves drawn in the 1L2S and 1L1S
residual panels represent the differences from the 2L1S model.
}
\label{fig:two}
\end{figure}

\subsection{OGLE-2017-BLG-1691 (KMT-2017-BLG-0752)}\label{sec:three-one}

The lensing event OGLE-2017-BLG-1691 occurred during the 2017 bulge season. It was first found
by the OGLE survey, and later confirmed by the KMTNet survey from the post-season analyses of
the data collected during the season.  The KMTNet group designated the event as KMT-2017-BLG-0752.
Hereafter, we use the nomenclatures of events by the ID references of the surveys who first found
the events  in accordance with the convention of the microlensing community.  The baseline magnitude
of the event was $I_{\rm base}=19.900\pm 0.004$.

The lensing light curve of OGLE-2017-BLG-1691 is shown in Figure~\ref{fig:two}, in which the
bottom panel shows the whole view and the top panel displays the enlargement of the region
around the anomaly. The event was alerted by the OGLE survey on 2017 September 6,
${\rm HJD}^\prime\equiv {\rm HJD}-2450000 \sim 8002.5$, which approximately corresponds to the
time near the peak.  An anomaly occurred about one day after the peak, but it was not noticed
during the progress of the event because it was covered by just a single OGLE data point, and
the KMTNet data were not released during the lensing magnification.  The existence of the anomaly
was identified five years after the event, shortly after the re-reduction of all 2017 KMT light
curves, from a project of reinvestigating the previous KMTNet data conducted to find unnoticed
planetary signals \citep{Han2022b}. The top panel of Figure~\ref{fig:two} shows that the anomaly,
which lasted for $\sim 5$~hours centered at ${\rm HJD}^\prime \sim 8003.6$, was additionally
covered by three KMTC data points, and this confirmed that the single anomalous OGLE point was real.

\begin{figure}[t]
\includegraphics[width=\columnwidth]{f3.eps}
\caption{
Lens system configurations of OGLE-2017-BLG-1691: upper panel (inner solution) and lower
panel (outer solution).  The inset in each panel shows the whole view of the lens system,
and the main panel presents the zoom-in view of the caustics.  The red cuspy figure is the
caustic and the line with an arrow is the source trajectory.  The orange circle on the
trajectory is drawn to represent the source scale with respect to the caustic.  The blue
solid dots marked by $M_1$ and $M_2$ indicate the positions of the host and planet,
respectively.  The dashed circle with unit radius in the inset represents the Einstein ring.
}
\label{fig:three}
\end{figure}

\begin{table*}[t]
\small
\caption{Lensing parameters of OGLE-2017-BLG-1691\label{table:three}}
\begin{tabular}{llll}
\hline\hline
\multicolumn{1}{c}{Parameter}           &
\multicolumn{1}{c}{2L1S (Inner)}        &
\multicolumn{1}{c}{2L1S (Outer)}        &
\multicolumn{1}{c}{1L2S}                \\
\hline
$\chi^2$                  &   866.9                    &   866.5                    &   880.4                   \\
$t_{0}$ (HJD$^\prime$)    &   $8002.540 \pm  0.010  $  &  $8002.531 \pm 0.010  $    &  $8002.499 \pm 0.012  $   \\
$u_{0}$ ($10^{-2}$)       &   $4.83 \pm  0.27       $  &  $4.95 \pm 0.26       $    &  $6.55 \pm 0.49       $   \\
$t_{\rm E}$ (days)              &   $19.42 \pm 0.83       $  &  $18.94 \pm 0.82      $    &  $18.77 \pm 0.83      $   \\
$s$                       &   $1.058 \pm 0.011      $  &  $1.003 \pm 0.014     $    &  --                       \\
$q$ ($10^{-4}$)           &   $0.708 \pm 0.23       $  &  $0.97 \pm 0.34       $    &  --                       \\
$\alpha$ (rad)            &   $3.873 \pm  0.012     $  &  $3.865 \pm 0.010     $    &  --                       \\
$\rho$ ($10^{-3}$)        &   $3.40 \pm 0.47        $  &  $3.54 \pm 0.52       $    &  --                       \\
$t_{0,2}$ (HJD$^\prime$)  &   --                       &  --                        &  $8003.587 \pm 0.015  $   \\
$u_{0,2}$ ($10^{-2}$)     &   --                       &  --                        &  $0.029 \pm 0.124     $   \\
$\rho_2$ ($10^{-3}$)      &   --                       &  --                        &  $3.39 \pm 0.93       $   \\
$q_F$ ($10^{-3}$)         &   --                       &  --                        &  $6.11 \pm 1.85       $   \\
\hline
\end{tabular}
\tablefoot{ ${\rm HJD}^\prime = {\rm HJD}- 2450000$.  }
\end{table*}

We model the light curve of the event under the 1L1S, 2L1S and 1L2S interpretations.  The
residuals of the individual tested models are compared in Figure~\ref{fig:two}.  It is found
that the 2L1S model best describes the anomaly, being favored over the 1L1S and 1L2S models
by $\Delta\chi^2=456.1$ and 13.9, respectively.  From the comparison of the residuals, it is
found that the 2L1S model is confirmed not only by the 4 data points (3 KMTC plus 1 OGLE points)
with positive deviations near the peak of the bump but also by the 3 additional points (2
KMTS and 1 KMTC points) before the major bump with slightly negative deviations.

We find two degenerate 2L1S solutions with binary parameters of $(s, q) \sim (1.058, 0.71
\times 10^{-4})$ and $\sim (1.003, 0.97 \times 10^{-4})$, which we designate as ``inner''
and ``outer'' solutions, respectively, for the reason to be mentioned below.  We list the
full lensing parameters of the two 2L1S solutions in Table~\ref{table:three},
together with the parameters of the 1L2S model.  The degeneracy is very
severe and the outer solution is favored only by $\Delta\chi^2=0.4$.  The lensing parameters
of the solutions indicate that the anomaly was produced by a very small mass-ratio planetary
companion to the lens lying close to the Einstein ring of the primary regardless of the solutions.
From the fact that both solutions have separations greater than unity and do not follow the
relation of $s_{\rm inner}\times s_{\rm outer}\simeq 1$, the degeneracy is different from the
``close-wide'' degeneracy \citep{Griest1998, Dominik1999, An2005}, which arises due to the
similarity between the central caustics of planetary lens systems with planet separations $s$
and $1/s$.  Instead, the planet separations of the two degenerate solutions follow the relation
\begin{equation}
\sqrt{s_{\rm in}\times s_{\rm out}}=s^\dagger;\qquad
s^\dagger = {\sqrt{u_{\rm anom}^2+4}+u_{\rm anom} \over 2},
\label{eq1}
\end{equation}
where $u_{\rm anom}^2=\tau_{\rm anom}^2+u_0^2$, $\tau_{\rm anom}=(t_{\rm anom}-t_0)/t_{\rm E}$, and
$t_{\rm anom}$ indicates the time of the planetary anomaly \citep{Hwang2022, Zhang2022, Ryu2022}.
With the values $t_0\simeq 8002.5$, $u_0 \simeq 4.85\times 10^{-2}$, $t_{\rm E}\simeq 19$~day and
$t_{\rm anom}\simeq 8003.6$, one finds that $s^\dagger\simeq 1.04$, which matches well
$\sqrt{s_{\rm in}\times s_{\rm out}}\simeq 1.03$.  This indicates that the similarity between
the light curves of the two solutions is caused by the degeneracy identified by \citet{Yee2021},
who first mentioned the continuous transition between the ``close-wide'' and ``inner-outer''
\citep{Gaudi1997} degeneracies.  Hereafter, we refer to this degeneracy as ``offset degeneracy''
following \citet{Zhang2022}.

Figure~\ref{fig:three} shows the lens systems configurations, in which the source trajectory
(line with an arrow) with respect to the caustic and positions of the lens components (blue
solid dots marked by $M_1$ and $M_2$) is presented: inner solution in the upper panel and
outer solution in the lower panel.  The source passed the inner side (with respect to $M_1$)
of the caustic according to the inner solution, while the source passes the outer side according
to the outer solution.  For both solutions, the source crossed the cusp of the caustic, and thus
the anomaly is affected by finite-source effects during the caustic crossing, allowing us to
precisely measure the normalized source radius.  For each solution, the source size relative
to the caustic is represented by an orange circle marked on the source trajectory. As will be
discussed in Sect.~\ref{sec:four}, the measurement of $\rho$ is important to estimate the
lensing observable of the angular Einstein radius $\theta_{\rm E}$, which can be used to constrain
the physical lens parameters.  However, the microlens parallax vector, $\pi_{\rm E}=(\pi_{\rm rel}/
\theta_{\rm E})/(\mbox{\boldmath $\mu$}/\mu)$, which is another observable constraining the physical lens parameters,
cannot be securely measured because the event time scale, $t_{\rm E}\sim 19$~days, is not long enough
to produce detectable deviations induced by the orbital motion of Earth around the Sun
\citep{Gould1992a}. Here $\mbox{\boldmath $\mu$}$ represents the vector of the relative
lens-source proper motion.

\subsection{KMT-2021-BLG-0320}\label{sec:three-two}

The lensing event KMT-2021-BLG-0320 was found on 2021 April 9 (${\rm HJD}^\prime \sim 9313$),
when the event had not yet reached its peak, with the employment of the KMTNet AlertFinder
system \citep{Kim2018}, which began full operation since the 2019 season.  Two days after the
detection, the event reached its peak with a magnification of $A_{\rm max}\sim 170$, and then
gradually declined until it reached its baseline of $I_{\rm base}=20.08$.  The source of the
event lies in the two KMTNet prime fields of BLG01 and BLG41, toward which observations were
conducted with a 30~min cadence for each field, and thus with a 15~min combined cadence. The
areas covered by the two fields overlap except for about 15\% of the area of each field filling
the gaps between the chips of the camera.  Because the data were taken from two fields of three
telescopes, there are 6 data sets: KMTA (BLG01), KMTA (BLG41), KMTC (BLG01), KMT (BLG41), KMTS
(BLG01), and KMTS (BLG41).

\begin{figure}[t]
\includegraphics[width=\columnwidth]{f4.eps}
\caption{
Lensing light curve of KMT-2021-BLG-0320. Notations are same as those in Fig.~\ref{fig:two}.
}
\label{fig:four}
\end{figure}

\begin{table}[t]
\small
\caption{Lensing parameters of KMT-2021-BLG-0320\label{table:four}}
\begin{tabular*}{\columnwidth}{@{\extracolsep{\fill}}llll}
\hline\hline
\multicolumn{1}{c}{Parameter}    &
\multicolumn{1}{c}{Close}        &
\multicolumn{1}{c}{Wide}         \\
\hline
$\chi^2$                &   4075.5                  &   4075.5                       \\
$t_{0}$ (HJD$^\prime$)  &  $9315.615 \pm 0.001   $  &  $9315.615 \pm 0.001   $       \\
$u_{0}$ ($10^{-3}$)     &  $5.86 \pm 0.21        $  &  $5.93 \pm 0.21        $       \\
$t_{\rm E}$ (days)            &  $13.12 \pm 0.39       $  &  $12.97 \pm 0.40       $       \\
$s$                     &  $0.771 \pm 0.014      $  &  $1.274 \pm 0.024      $       \\
$q$ ($10^{-4}$)         &  $3.02 \pm 0.42        $  &  $2.89 \pm 0.44        $       \\
$\alpha$ (rad)          &  $0.604 \pm 0.014      $  &  $0.602 \pm 0.014      $       \\
$\rho$ ($10^{-3}$)      &  --                       &  --                            \\
\hline
\end{tabular*}
\end{table}

In Figure~\ref{fig:four}, we present the light curve constructed from the combination of the
6 KMTNet data sets.  Although it would be difficult to notice the anomaly from a glimpse, we
inspected the light curve because the event reached a very high magnification at the peak, near
which the light curve is susceptible to perturbations induced by a planet \citep{Griest1998}.
From this inspection, it was found that the light curve exhibited an anomaly that lasted for
about 4 hours with a negative deviation with respect to a 1L1S model.  See the enlarged view
around the peak region of the light curve presented in the top panel of Figure~\ref{fig:four}.
It is known that a planetary companion to a lens can induce anomalies with both positive and
negative deviations, while a faint companion to a source can induce anomalies with only positive
deviations \citep{Gaudi1998}. We, therefore, modeled the light curve under the 2L1S interpretation.

It is found that the anomaly is well described by a 2L1S model, in which the mass ratio between
the binary lens components is very low. We found two sets of solutions with $(s, q)_{\rm close}
\sim (0.77, 3.0\times 10^{-4})$ and $(s, q)_{\rm wide} \sim (1.27, 2.9\times 10^{-4})$.  We
designate the individual solutions as ``close'' and ``wide'' solutions, because the former
solution has $s<1.0$ and the latter has $s>1.0$. In Table~\ref{table:four}, we list the full
lensing parameters for the two sets of solutions. The fits of the two solutions are nearly
identical with $\chi^2$ values that are equal to the first digit after the decimal point,
indicating that the degeneracy between the two solutions is very severe.  The fact that the
binary separations of the two solutions follow the relation of $s_{\rm close}\times s_{\rm wide}
\simeq 1.0$ indicates that the similarity between the solution stems from the close-wide degeneracy.
It is known that the relation between the planet separations of the solutions under the offset
degeneracy in Equation~(\ref{eq1}) applies to more general cases, including resonant case, than
the $s_{\rm close}\times s_{\rm wide} \simeq 1.0$ relation of the close-wide degeneracy, and
we confirm this.  In addition to the relation between $s_{\rm in}$ and $s_{\rm out}$,
\citet{Hwang2022} provided analytic formulas for the heuristic estimation of the source
trajectory angle and the mass ratio;
\begin{equation}
\alpha = \tan^{-1}\left( {u_0 \over \tau_{\rm anom}}\right);\qquad
q = \left( {\Delta t_{\rm anom}\over 4t_{\rm E}} \right)  \left( {s\over |u_0|}\right)
\left\vert \sin^3 \alpha\right\vert,
\label{eq2}
\end{equation}
where $\tau_{\rm anom}$ denotes the duration of the planet-induced anomaly.  We confirm that
the heuristically estimated values of $\alpha$ and $q$ match well those estimated from the
modeling.  It was found that the normalized source radius could not be constrained, not even
the upper limit, and thus the line for $\rho$ in Table~\ref{table:four} is left blank.

\begin{figure}[t]
\includegraphics[width=\columnwidth]{f5.eps}
\caption{
Lens system configurations of KMT-2021-BLG-0320.  The upper and lower panels show the configurations
of the close and wide solutions, respectively.
}
\label{fig:five}
\end{figure}

Figure~\ref{fig:five} shows the lensing configuration of KMT-2021-BLG-0320 for the close (upper
panel) and wide (lower panel) solutions. It shows that the source passed the back-end region of
the tiny central caustic induced by a planet, and this generated the negative deviation of the
observed anomaly. The central caustics of the close and wide solutions are very similar, resulting
in nearly identical deviations.  Because the source passed well outside the caustic, finite-source
effects could not be detected, and thus we do not mark an orange circle representing the source
size.  Microlens-parallax effects could not be securely detected because the event time scale,
$t_{\rm E} \sim 13$~days, is much shorter than the orbital period of Earth, that is, 1~year.

\begin{figure}[t]
\includegraphics[width=\columnwidth]{f6.eps}
\caption{
Lensing light curve of KMT-2021-BLG-1303. Notations are same as those in Fig.~\ref{fig:two}.
}
\label{fig:six}
\end{figure}

\subsection{KMT-2021-BLG-1303 (MOA-2021-BLG-182)}\label{sec:three-three}

The event KMT-2021-BLG-1303 was first found on 2021 June 14 (${\rm HJD}^\prime \sim 9379$) by the
KMTNet survey, and later identified by the MOA survey on 2021 June 17 (${\rm HJD}^\prime \sim 9382$).
The MOA survey designated the event as MOA-2021-BLG-182. A day after the first discovery, the event
displayed an anomaly that lasted for about 2~days. After the anomaly, the event reached peak at
${\rm HJD}^\prime \sim 9385.1$ with a magnification of $A_{\rm max}\sim 46$, and then gradually
declined to the baseline of $I_{\rm base}=19.67$.

The light curve of KMT-2021-BLG-1303 constructed with the combined KMTNet and MOA data is shown in
Figure~\ref{fig:six}. Compared to the previous two events, the light curve of this event displays
a very obvious anomaly with a maximum deviation of $\Delta I\sim 1.4$~mag from a 1L1S model. The
rapid brightening at ${\rm HJD}^\prime \sim 9381.0$ indicates that the event experienced a caustic
crossing. Because caustics form due to the multiplicity of lens components, we rule out the 1L2S
origin of the anomaly and test only
the 2L1S interpretation.

\begin{table}[t]
\small
\caption{Lensing parameters of KMT-2021-BLG-1303\label{table:five}}
\begin{tabular*}{\columnwidth}{@{\extracolsep{\fill}}llll}
\hline\hline
\multicolumn{1}{c}{Parameter}    &
\multicolumn{1}{c}{Value }        \\
\hline
$\chi^2$                &   1654.9                    \\
$t_{0}$ (HJD$^\prime$)  &  $9385.091 \pm 0.003  $     \\
$u_{0}$ ($10^{-2}$)     &  $2.17 \pm 0.06       $     \\
$t_{\rm E}$ (days)            &  $25.21 \pm 0.68      $     \\
$s$                     &  $1.029 \pm 0.001     $     \\
$q$ ($10^{-4}$)         &  $6.42 \pm 0.45       $     \\
$\alpha$ (rad)          &  $6.138 \pm 0.001     $     \\
$\rho$ ($10^{-3}$)      &  $0.95 \pm 0.08       $     \\
\hline
\end{tabular*}
\end{table}

We find that the anomaly is well explained by a 2L1S model with a planetary companion
lying close to the Einstein ring of the host.  We find a unique solution without any
degeneracy, and the binary parameters of the solution are $(s, q)\sim (1.029, 6.4\times
10^{-4})$.  Here again the planet-to-host mass ratio is of the order of $10^{-4}$ like
the two previous events.  We double checked the uniqueness of the solution by thoroughly
inspecting the region around the $s$ and $q$ parameters predicted by the relations in
Equations~(\ref{eq1}) and (\ref{eq2}), and confirmed that there is only a single solution.
The full lensing parameters of the event are listed in Table~\ref{table:five}.

\begin{figure}[t]
\includegraphics[width=\columnwidth]{f7.eps}
\caption{
Lens system configuration of KMT-2021-BLG-1303. Notations are same as those in Fig.~\ref{fig:three}.
}
\label{fig:seven}
\end{figure}

\begin{table*}[t]
\small
\caption{Lensing parameters of KMT-2021-BLG-1554\label{table:six}}
\begin{tabular}{llll}
\hline\hline
\multicolumn{1}{c}{Parameter}         &
\multicolumn{1}{c}{2L1S (Close)}      &
\multicolumn{1}{c}{2L1S (Wide)}       &
\multicolumn{1}{c}{1L2S}              \\
\hline
$\chi^2$                  &   3486.0               &   3485.7                    &   3496.1                  \\
$t_{0}$ (HJD$^\prime$)    &  $9394.737 \pm 0.010$  &  $9394.742 \pm 0.010 $      &  $9394.719 \pm 0.019  $   \\
$u_{0}$                   &  $0.051 \pm 0.009   $  &  $0.051 \pm 0.011    $      &  $0.003 \pm 0.054     $   \\
$t_{\rm E}$ (days)              &  $5.04 \pm 0.78     $  &  $5.15 \pm 0.72      $      &  $6.55 \pm 1.11       $   \\
$s$                       &  $0.888 \pm 0.039   $  &  $1.189 \pm 0.051    $      &  --                       \\
$q$ ($10^{-4}$)           &  $14.14 \pm 4.48    $  &  $14.94 \pm 5.41     $      &  --                       \\
$\alpha$ (rad)            &  $4.705 \pm 0.038   $  &  $4.727 \pm 0.037    $      &  --                       \\
$\rho$ ($10^{-3}$)        &  $6.98 \pm 1.18     $  &  $7.00 \pm 1.28      $      &  $98.8 \pm 24.2       $   \\
$t_{0,2}$ (HJD$^\prime$)  &   --                   &  --                         &  $9394.740 \pm 0.002  $   \\
$u_{0,2}$                 &   --                   &  --                         &  $0.002 \pm 0.002     $   \\
$\rho_2$ ($10^{-3}$)      &   --                   &  --                         &  --                       \\
$q_F$                     &   --                   &  --                         &  $0.048 \pm 0.008     $   \\
\hline
\end{tabular}
\end{table*}

The lensing configuration of the event is presented in Figure~\ref{fig:seven}, which shows
that the planet lies very close to the Einstein ring and induces a single six-sided resonant
caustic near the primary of the lens. The source entered the caustic at ${\rm HJD}^\prime
\sim 9381.0$, and this produced the sharp rise of the light curve at the corresponding time.
According to the model, the source exited the caustic at ${\rm HJD}^\prime \sim 9384.2$, but
the caustic-crossing feature at this epoch is not obvious in the lensing light curve due to
the combination of the weak caustic and the poor coverage of the region. The pattern of the
anomaly between the caustic entrance and exit deviates from a typical ``U''-shape pattern
because the source passed along the fold of the caustic.  The light curve during the caustic
entrance of the source was well resolved by the KMTA data, and thus the normalized source
radius, $\rho=(0.95\pm 0.08)\times 10^{-3}$, is  tightly constrained.  However, it was difficult
to constrain the microlens parallax because the time scale, $t_{\rm E} \sim 25$~days, is not long
enough and the photometric precision of the faint source with $I\sim 21$ is not high enough
to detect subtle deviations induced by microlens-parallax effects.

\begin{figure}[t]
\includegraphics[width=\columnwidth]{f8.eps}
\caption{
Lensing light curve of KMT-2021-BLG-1554. Notations are same as those in Fig.~\ref{fig:two}.
}
\label{fig:eight}
\end{figure}

\subsection{KMT-2021-BLG-1554}\label{sec:three-four}

This short event with a time scale of $t_{\rm E}\sim 5$~days and a faint baseline magnitude of
$I_{\rm base}=22.39$ was detected on 2021 Jul~1 (${\rm HJD}^\prime \sim 9396$), about two
days after the event peaked at $t_0\sim 9394.7$, by the KMTNet survey. The source lies in
the prime fields of BLG01 and BLG41, and thus there are 6 data sets from the three KMTNet
telescopes. Among these data sets, the data set of the BLG41 field acquired by the KMTS
telescope was not used in the analysis due to its poor photometric quality caused by bad
seeing during the lensing magnification.  Although the other data set from KMTS and those
from KMTA were included in the analysis, the results are mainly derived from the KMTC data
sets, which cover the peak region of the light curve.

The light curve constructed with the available KMTNet data sets is shown in Figure~\ref{fig:eight}.
In the peak region, it exhibits an anomaly that lasted for about 3~hours from a 1L1S model with
a positive deviation of $\Delta I\sim 0.5$~mag.  The anomaly appears both in the BLG01
(two data points) and BLG41 (3 points) data sets obtained from KMTC observations, confirming
that the signal is real.  Because a positive anomaly can be produced by a binary companion to
either a lens or a source, we test both 2L1S and 1L2S models.

\begin{table*}[t]
\caption{De-reddened colors, magnitudes, and spectral types of source stars\label{table:seven}}
\begin{tabular}{llllll}
\hline\hline
\multicolumn{1}{c}{Event}       &
\multicolumn{1}{c}{$(V-I)_0$}   &
\multicolumn{1}{c}{$I_0$}       &
\multicolumn{1}{c}{Spectral type}        \\
\hline
 OGLE-2017-BLG-1691  &  $0.714 \pm 0.065$ &  $17.397 \pm   0.039$  &  G1 turnoff or subgiant   \\
 KMT-2021-BLG-0320   &  $0.700 \pm 0.040$ &  $18.843 \pm   0.002$  &  G1V  \\
 KMT-2021-BLG-1303   &  $0.849 \pm 0.015$ &  $19.809 \pm   0.003$  &  G9V  \\
 KMT-2021-BLG-1554   &  $0.873 \pm 0.061$ &  $18.649 \pm   0.008$  &  K1V  \\
\hline
\end{tabular}
\end{table*}

\begin{figure}[t]
\includegraphics[width=\columnwidth]{f9.eps}
\caption{
Lens system configuration of KMT-2021-BLG-1554. Notations are same as those in Fig.~\ref{fig:five}.
}
\label{fig:nine}
\end{figure}

\begin{table*}[t]
\caption{Angular source radii, Einstein radii, and relative lens-source proper motions \label{table:eight}}
\begin{tabular}{llllll}
\hline\hline
\multicolumn{1}{c}{Event}                  &
\multicolumn{1}{c}{$\theta_*$ ($\mu$as)}   &
\multicolumn{1}{c}{$\theta_{\rm E}$ (mas)}        &
\multicolumn{1}{c}{$\mu$ (mas yr$^{-1}$)}  \\
\hline
 OGLE-2017-BLG-1691  &  $1.043 \pm  0.100$ &  $0.30 \pm 0.05$  &  $5.68 \pm 1.01$  \\
 KMT-2021-BLG-0320   &  $0.528 \pm  0.042$ &  --               &  --               \\
 KMT-2021-BLG-1303   &  $0.401 \pm  0.029$ &  $0.42 \pm 0.05$  &  $6.13 \pm 0.70$  \\
 KMT-2021-BLG-1554   &  $0.703 \pm  0.065$ &  $0.10 \pm 0.02$  &  $7.30 \pm 1.50$  \\
\hline
\end{tabular}
\end{table*}

Detailed analysis of the light curve indicates that the anomaly was produced by a planetary
companion to the lens.  We find two sets of solutions with planet parameters of $(s, q)_{\rm close}
\sim (0.89, 1.4\times 10^{-3})$ and $(s, q)_{\rm wide}\sim (1.12, 1.5\times 10^{-3})$ resulting
from the close-wide degeneracy. The full lensing parameters of both solutions are provided in
Table~\ref{table:six}.  It is found that the degeneracy between the two 2L1S solution is severe
with $\Delta\chi^2=0.3$, and the 2L1S solutions are favored over the model under the 1L2S
interpretation by $\Delta\chi^2\sim 11$.  The lensing parameters of the 1L2S model are listed
in Table~\ref{table:six}.  We compare the residuals of the 2L1S and 1L2S models in the two middle
panels of Figure~\ref{fig:eight}.  It turns out that the source crossed a caustic, and thus we
are able to measure the normalized source radius, $\rho = (6.98\pm 1.18)\times 10^{-3}$.

The lensing configurations of the event according to the close and wide solutions are shown in the
upper and lower panels of Figure~\ref{fig:nine}, respectively.  The figure shows that the anomaly
was produced by the crossing of the source over the sharp tip of the central caustic induced by a
planet lying near the Einstein ring of the planet host.  The gap between the caustic entrance and
exit is much smaller than the source size, and thus the individual caustic-crossing features do not
show up in the anomaly and instead the anomaly appears as a single bump.  The incidence angle of the
source on the binary axis is nearly $90^\circ$, and thus the features on the rising and falling sides
of the anomaly are symmetric.

\section{Source stars and angular Einstein radii}\label{sec:four}

Among the four analyzed planetary events, the anomalies of the three events (KMT-2017-BLG-0752,
KMT-2021-BLG-1303, and KMT-2021-BLG-1554) were affected by finite-source effects, and thus the
normalized source radii can be measured. In this section, we estimate the angular Einstein
radii for these events. Estimating $\theta_{\rm E}$ from a measured $\rho$ value requires one to specify
the source type, from which the angular source radius $\theta_*$ is deduced and the Einstein
radius is determined by
\begin{equation}
\theta_{\rm E} = {\theta_*\over \rho}.
\label{eq3}
\end{equation}
Although the normalized source radius and thus $\theta_{\rm E}$ cannot be measured for the event
KMT-2021-BLG-0320, we specify the source type for the sake  of completeness.

The specification of the source type is done by estimating the color and brightness of the
source.  We measure the $I$ and $V$-band magnitudes of the source from the regression of
the data processed using the pyDIA photometry code \citep{Albrow2017} with the variation
of the lensing magnification. We then place the source on the instrumental color-magnitude
diagram (CMD) of stars lying around the source constructed using the same photometry code.
Following the method of \citet{Yoo2004}, the instrumental source color and magnitude,
$(V-I, I)$, are then calibrated using the centroid of red giant clump (RGC), for which its
reddening and extinction-corrected (de-reddened) color and magnitude, $(V-I, I)_{\rm RGC,0}$,
are known, on the CMD as a reference.

Figure~\ref{fig:ten} shows the positions of the source stars (marked by blue dots) with
respect to those of the RGC centroids (red dots) on the CMDs of the individual events.  For
OGLE-2017-BLG-1691, the source color could not be securely measured not only because the
$V$-band data sparsely covered the light curve during the lensing magnification but also
because the photometry quality of these data is low, although the $I$-band magnitude was
relatively well measured.  In this case, we estimated the source color as the median
color of stars with the same $I$-band magnitude offset from the RGC centroid in the CMD constructed
from the images of Baade's window taken from the observations using the {\it Hubble Space Telescope}
\citep{Holtzman1998}.  From the offsets in color and magnitudes between the source and RGC centroid,
$\Delta (V-I, I)$, together with the known de-reddened color and magnitude of the RGC centroid,
$(V-I, I)_{\rm RGC,0}$ \citep{Bensby2013, Nataf2013}, we estimate the calibrated source color and
magnitude as $(V-I, I)_0=(V-I, I)_{\rm RGC,0}+\Delta (V-I, I)$.  In Table~\ref{table:seven}, we list
the source colors and magnitudes of the individual events estimated from this procedure along with
the spectral types of the source stars.  It is found that the source of OGLE-2017-BLG-1691 is a
turnoff star or a subgiant of a G spectral type, while the source stars of the other events
are main-sequence stars with spectral types ranging from G to K.

\begin{figure}[t]
\includegraphics[width=\columnwidth]{f10.eps}
\caption{
Source locations with respect to the centroids of the red giant clump (RGC) on the instrumental
color-magnitude diagrams of stars lying around the source stars of the events OGLE-2017-BLG-1691,
KMT-2021-BLG-0320, KMT-2021-BLG-1303, and KMT-2021-BLG-1554.  For OGLE-2017-BLG-1691, the CMD is
constructed by combining the two CMDs from KMTC (grey dots) and {\it HST} (brown dots) observations.
For each event, the locations of the source and RGC centroid are marked by blue and red solid dots,
respectively.
}
\label{fig:ten}
\end{figure}

For each event, the angular source radius and Einstein radius were estimated from the measured
color and magnitude of the source.  For this, the measured $V-I$ color was converted into $V-K$
color using the color-color relation of \citet{Bessell1988}, $\theta_*$ value was interpolated
from the ($V-K$)--$\theta_*$ relation of \citet{Kervella2004}, and then the angular Einstein
radius was estimated using the relation in Equation~(\ref{eq3}).  With the measured Einstein
radius together with the event time scale, the value of the relative lens-source proper motion
was assessed as
\begin{equation}
\mu = {\theta_{\rm E}\over t_{\rm E}}.
\label{eq4}
\end{equation}
In Table~\ref{table:eight}, we list the estimated values of $\theta_*$, $\theta_{\rm E}$, and $\mu$
for the individual events. The $\theta_{\rm E}$ and $\mu$ values for the event KMT-2021-BLG-0320 are
left blank because finite-source effects in the lensing light curve were not detected and
subsequently the values of $\rho$, $\theta_{\rm E}$, and $\mu$ could not be measured. We note that
the angular Einstein radius of KMT-2021-BLG-1554, $\theta_{\rm E}\sim 0.10$~mas, is substantially
smaller than those of the other events, and this together with its short time scale,
$t_{\rm E} \sim 5$~days, suggests that the mass of the lens would be very low.  KMT-2021-BLG-1554 is
the seventh shortest microlensing event with a bound planet. See Table~3 of \citet{Ryu2021}.

\begin{table*}[t]
\caption{Physical lens parameters\label{table:nine}}
\begin{tabular}{llllll}
\hline\hline
\multicolumn{1}{c}{Event}                           &
\multicolumn{1}{c}{$M_{\rm host}$ ($M_\odot$)}      &
\multicolumn{1}{c}{$M_{\rm planet}$ ($M_{\rm J}$)}  &
\multicolumn{1}{c}{$D_{\rm L}$ (kpc)}                     &
\multicolumn{1}{c}{$a_\perp$ (AU)}                  \\
\hline
 OGLE-2017-BLG-1691  &  $0.45^{+0.36}_{-0.25}$  &  $0.046^{+0.037}_{-0.025}$    &  $7.29^{+1.01}_{-1.33}$  &  $2.41^{+0.34}_{-0.44}$ (inner)   \\  [0.7ex]
                     &  --                      &  --                           &  --                      &  $2.54^{+0.36}_{-0.46}$ (outer)   \\  [0.7ex]
 KMT-2021-BLG-0320   &  $0.32^{+0.39}_{-0.21}$  &  $0.10 ^{+0.13 }_{-0.07 }$    &  $6.95^{+1.07}_{-1.33}$  &  $1.54^{+0.24}_{-0.30}$ (close)   \\  [0.7ex]
                     &  --                      &  --                           &  --                      &  $2.55^{+0.39}_{-0.49}$ (wide)    \\  [0.7ex]
 KMT-2021-BLG-1303   &  $0.57^{+0.32}_{-0.29}$  &  $0.38 ^{+0.22 }_{-0.20 }$    &  $6.28^{+0.95}_{-1.43}$  &  $2.89^{+0.44}_{-0.66}$           \\  [0.7ex]
 KMT-2021-BLG-1554   &  $0.08^{+0.13}_{-0.04}$  &  $0.12 ^{+0.20}_{-0.07}$      &  $7.68^{+1.04}_{-1.10}$  &  $0.72^{+0.13}_{-0.14}$ (close)   \\  [0.7ex]
                     &   --                     &  --                           &  --                      &  $0.96^{+0.15}_{-0.16}$ (wide)    \\  [0.7ex]
\hline
\end{tabular}
\end{table*}

\section{Physical parameters}\label{sec:five}

In this section, we estimate the physical parameters of the planetary systems.  For the unique
constraint of the physical lens parameters, one must simultaneously measure the extra observables
of $\theta_{\rm E}$ and $\pi_{\rm E}$, from which the mass $M$ and distance to the lens, $D_{\rm L}$, are determined as
\begin{equation}
M={\theta_{\rm E} \over \kappa \pi_{\rm E}};\qquad
D_{\rm L} = { {\rm AU} \over  \pi_{\rm E}\theta_{\rm E} + \pi_{\rm S}}.
\label{eq5}
\end{equation}
Here $\kappa=4G/(c^2{\rm AU})$, $\pi_{\rm S}={\rm AU}/D_{\rm S}$, and $D_{\rm S}$ denotes the
distance to the source \citep{Gould1992a, Gould2000}.  The values of the microlens parallax
could not be measured for any of the events, and thus the physical parameters cannot be
unambiguously determined from the relation in Equation~(\ref{eq5}).  However, one can still
constrain $M$ and $D_{\rm L}$ using the other measured observables of $t_{\rm E}$ and $\theta_{\rm E}$, which are
related to the physical lens parameters as
\begin{equation}
t_{\rm E} = {\theta_{\rm E} \over \mu};\qquad
\theta_{\rm E} = (\kappa M \pi_{\rm rel})^{1/2},
\label{eq6}
\end{equation}
where $\pi_{\rm rel}={\rm AU}(D_{\rm L}^{-1}-D_{\rm S}^{-1})$ denotes the relative parallax
between the lens and source.  The event time scales were well measured for all events,
and the Einstein radii were assessed for three of the four events.  In order to estimate the
physical lens parameters with the constraints provided by the measured observables of the
individual events, we conduct Bayesian analyses using a Galactic model.

\begin{figure}[t]
\includegraphics[width=\columnwidth]{f11.eps}
\caption{
Bayesian posteriors of the host masses of the planetary systems. For each distribution, the
solid vertical line indicates the median value, and the dotted lines represent 1$\sigma$ range
of the distribution.  The blue and red curves represent the distributions contributed by disk
and bulge populations of lenses, respectively, and the black curve is the sum from the two lens
populations.
}
\label{fig:eleven}
\end{figure}

In the Bayesian analysis, we started by producing many lensing events from a Monte Carlo
simulation conducted with the use of a Galactic model, which defines the matter-density and
kinetic distributions, and mass function of Galactic objects.  We adopted the Galactic model
constructed by \citet{Jung2021}.  In this model, the density distributions of disk and bulge
objects are described by the \citet{Robin2003} and \citet{Han2003} models, respectively.  The
kinematic distribution of disk objects is based on the modified version of the \citet{Han1995}
model, in which the original version is modified to reconcile the changed density model, that
is, the \citet{Robin2003} model.  The bulge kinematic distribution is modeled based on proper
motions of stars in the Gaia catalog \citep{Gaia2016, Gaia2018}.  For the details of the density
and kinematic distributions, see \citet{Jung2021}.  In the Galactic model, the mass function is
constructed with the adoption of the initial mass function and the present-day mass function of
\citet{Chabrier2003} for the bulge and disk lens populations, respectively.

\begin{figure}[t]
\includegraphics[width=\columnwidth]{f12.eps}
\caption{
Bayesian posteriors of the distances to the
planetary systems. Notations are same as those in
Fig.~\ref{fig:eleven}.
}
\label{fig:twelve}
\end{figure}

In the second step, we computed the time scales and Einstein radii of the
simulated events using the relations in Equation~(\ref{eq6}), and then constructed posterior
distributions of $M$ and $D_{\rm L}$ for the simulated events with the $t_{\rm E}$ and $\theta_{\rm E}$ values that
are consistent with the observables of the individual lensing events. For the three events
OGLE-2017-BLG-1691, KMT-2021-BLG-1303, and KMT-2021-BLG-1554, we use the constraints of both
$t_{\rm E}$ and $\theta_{\rm E}$, and for KMT-2021-BLG-0320, we apply the constraint of only $t_{\rm E}$ because
$\theta_{\rm E}$ is not measured for the event.

Figures~\ref{fig:eleven} and \ref{fig:twelve} show the Bayesian posteriors of $M$ and $D_{\rm L}$
of the individual events, respectively.  In Table~\ref{table:nine}, we summarize the estimated
values of the host and planet masses, $M_{\rm host}$ and $M_{\rm planet}$, distance, and projected
host-planet separation, $a_\perp=sD_{\rm L} \theta_{\rm E}$, for all of which medians are presented as
representative values and the uncertainties are estimated as 16\% and 84\% of the Bayesian
posterior distributions.  The median values and uncertainty ranges of the individual events
are marked by solid and dotted vertical lines in the Bayesian posteriors presented in
Figures~\ref{fig:eleven} and \ref{fig:twelve}, respectively.  For the events with degenerate
solutions, OGLE-2017-BLG-1691, KMT-2021-BLG-0320 and KMT-2021-BLG-1554, we present the planetary
separations $a_\perp$ corresponding to both the close and wide (or inner and outer) solutions.
We note that the uncertainties of the estimated physical parameters are big because of the
intrinsically weak Bayesian constraint.  For example, from the comparison of the posterior
distributions of OGLE-2017-BLG-1691 and KMT-2021-BLG-0320, one finds that the difference
between the two posterior distributions is not significant, although the mean lens mass
for the former event is slightly bigger than that of the latter event due to the longer
time scale and the uncertainty is smaller due to the additional constrain of $\theta_{\rm E}$.

The estimated masses indicate that all of the discovered planets have sub-Jovian masses.  The
planet masses KMT-2021-BLG-0320L, KMT-2021-BLG-1303L, and KMT-2021-BLG-1554L correspond to
$\sim 0.10$, $\sim 0.38$, and $\sim 0.12$ times of the mass of the Jupiter, and the planet mass
of OGLE-2017-BLG-1691L corresponds to that of Uranus.  To be noted among the planet hosts is that
the estimated mass of the planetary system KMT-2021-BLG-1554L,  $M_{\rm host}\sim 0.08~M_\odot$,
corresponds to the boundary between a star and a brown dwarf (BD).  Together with the previously
detected planetary systems with very low-mass hosts\footnote{MOA-2011-BLG-262 \citep{Bennett2014},
OGLE-2012-BLG-0358 \citep{Han2013},
MOA-2015-BLG-337 \citep{Miyazaki2018},
OGLE-2015-BLG-1771 \citep{Zhang2020},
KMT-2016-BLG-1820 \citep{Jung2018a},
KMT-2016-BLG-2605 \citep{Ryu2021},
OGLE-2017-BLG-1522 \citep{Jung2018b},
OGLE-2018-BLG-0677 \citep{Herrera2020},
KMT-2018-BLG-0748 \citep{Han2020b}, and
KMT-2019-BLG-1339L \citep{Han2020a}},
the discovery of the system demonstrates the microlensing capability of detecting planetary systems
with very faint or substellar hosts.  Besides KMT-2021-BLG-1554L, the host stars of the other planetary
systems are low-mass stars with masses lying in the range of $\sim [0.3$--$0.6]~M_\odot$.  Under the
approximation that the snow line distance scales with the host mass as $a_{\rm sl}\sim 2.7(M/M_\odot)$~AU,
all discovered planets lie beyond the snow lines of the hosts regardless of the solutions, demonstrating
the high microlensing sensitivity to cold planets.  The planetary systems lie in the distance range of
$\sim [6.3$--$7.6]$~kpc, demonstrating the usefulness of the microlensing method in detecting remote
planets.

\section{Summary and conclusion}\label{sec:six}

We presented the analyses of four planetary microlensing events OGLE-2017-BLG-1691,
KMT-2021-BLG-0320, KMT-2021-BLG-1303, and KMT-2021-BLG-1554.  The events share a common
characteristic that the planetary signals appeared as anomalies with very short durations,
ranging from a few hours to a couple of days, and they were clearly detected solely by the
combined data of the high-cadence lensing surveys without additional data from followup
observations.

From the detailed analyses of the events, it was found that the signals were generated by
planets with low planet-to-host mass ratios: three of the planetary systems with mass ratios
of the order of $10^{-4}$ and the other with a mass ratio slightly greater than $10^{-3}$.
In the histogram of microlensing planets presented in Figure~\ref{fig:one}, we mark the
positions of the four planets discovered in this work.  The estimated masses indicated that
all discovered planets have sub-Jovian masses, in which the planet masses of KMT-2021-BLG-0320Lb,
KMT-2021-BLG-1303Lb, and KMT-2021-BLG-1554Lb correspond to $\sim 0.10$, $\sim 0.38$, and $\sim 0.12$
times of the mass of the Jupiter, and the mass of OGLE-2017-BLG-1691Lb corresponds to that of the
Uranus.  It was found that the host of the planetary system KMT-2021-BLG-1554L has a mass at around
the boundary between a star and a brown dwarf.  Besides this system, it was found that the host
stars of the other planetary systems are low-mass stars with masses in the range of
$\sim [0.3$--$0.6]~M_\odot$.  The discoveries of the planets well demonstrate the capability of
the current high-cadence microlensing surveys in detecting low-mass planets.

\begin{acknowledgements}
Work by C.H. was supported by the grants  of National Research Foundation of Korea
(2020R1A4A2002885 and 2019R1A2C2085965).
This research has made use of the KMTNet system operated by the Korea Astronomy and Space
Science Institute (KASI) and the data were obtained at three host sites of CTIO in Chile,
SAAO in South Africa, and SSO in Australia.
The OGLE project has received funding from the National Science Centre, Poland, grant
MAESTRO 2014/14/A/ST9/00121 to AU.
he MOA project is supported by JSPS KAKENHI grant Nos. JSPS24253004,
JSPS26247023, JSPS23340064, JSPS15H00781, JP16H06287, JP17H02871, and JP19KK0082.
\end{acknowledgements}