\section{Introduction}
Knowing the generalization ability of a classifier is important for its applications in fields like autonomous driving, medical diagnosis and product recommendation. Many well-established benchmarks in the community, \emph{e.g.,} ImageNet \cite{deng2009imagenet} and CIFAR \cite{krizhevsky2009learning}, provide held-out labeled test sets that allow empirical results to be computed.
However, when applying the classifier in real-world scenarios, we often encounter novel distributions and unknown cases, and empirical results calculated from existing benchmarks do not accurately reflect its generalization ability during deployment.
To evaluate classifier performance under novel environments, the standard way is to collect new labeled test sets. However, it is prohibitively expensive to label sufficient images in every novel scenario.

This work investigates an interesting problem: how to estimate classifier accuracy on unlabeled test sets? \textcolor{black}{Some existing research studies classifier generalization on \emph{fixed} test sets by developing complexity measurements on model parameters \cite{arora2018stronger,corneanu2020computing,jiang2018predicting}.
In this work, we focus on \emph{varying test domains}.
In the absence of test labels, we aim to find an unsupervised evaluation criterion that can reflect classifier accuracy. \textcolor{black}{A self-supervised task produces ground truths from data without human annotations. Thus, we can always calculate its accuracy on an unlabeled test set.} As such we raise a question: given a network that is good at a self-supervised task on an unlabeled test set, can it also address object recognition task well under the same environment?
If such a relationship exists, we can infer classifier performance on an unlabeled test set from the accuracy of the self-supervised task. In light of this question, we study the statistical correlation between semantic classification and rotation prediction under varying environments.
}

\textcolor{black}{In the community, self-supervised tasks are usually used to learn effective representations for downstream tasks \cite{he2020momentum,noroozi2016unsupervised,gidaris2018unsupervised}. Different from them, we explore the feasibility of using self-supervision to predict classifier accuracy on novel test sets.} We pay particular attention to the pretext task of \emph{rotation prediction}, where we classify the rotation angle of an image into \{0\degree, 90\degree, 180\degree, 270\degree\}.
Specifically, we train a network jointly for the main classification task and pretext task. To study the accuracy correlation between the two tasks, we construct different setups, including CIFAR-10, MNIST, Tiny-ImageNet, and COCO. Each setup contains many labeled test sets for calculating network performance on the two tasks. By plotting classification accuracy against rotation prediction accuracy, we consistently observe a strong correlation between them (Pearson's Correlation $r >0.88$) across the four setups.
This shows that if the network can solve the rotation prediction task well, it is likely to achieve good classification accuracy, and vice versa.

Based on this, we learn a linear regression model to predict classification accuracy on unseen test sets. Given an unlabeled test set, we obtain rotation ground truths by manually rotating the images. Then, we use the multi-task network to predict rotation degrees of the test images and calculate rotation prediction accuracy. Finally, the linear regression model predicts the semantic classification performance from rotation prediction accuracy.
Empirical results on three different setups show that our regression model achieves very promising estimation results on unseen test sets.

In summary, this work investigates the problem of unsupervised classifier prediction, \emph{i.e.}, estimating classifier performance on unseen test sets without any human-annotated label. We observe that the accuracy of a simple self-supervised task, rotation prediction \cite{gidaris2018unsupervised}, is highly correlated with the classification accuracy under various testing environments.

\section{Related Work} \label{sec:related}
\textbf{Model generalization.} Estimating model generalization error on unseen images is a crucial problem.
Some works develop \emph{complexity measurements} on trained models and training sets to predict the generalization gap between a certain pair of training-testing set \cite{eilertsen2020classifying,unterthiner2020predicting,arora2018stronger,corneanu2020computing,jiang2018predicting,neyshabur2017exploring,jiang2019fantastic}.
The authors of \cite{corneanu2020computing} use persistent topology measures to predict the performance gap between training and testing error.
Jiang \emph{et al. } \cite{jiang2018predicting} introduce a measurement of layer-wise margin distributions for generalization ability. Neyshabur \emph{et al. } \cite{neyshabur2017exploring} develop bounds on the generalization gap based on the product of norms of the weights across layers.
 Our work differs significantly. Instead of studying the statistics of model weights, we use rotation prediction accuracy to characterize models. Moreover, we are concerned with various test distributions that have domain gaps with the training one, while the aforementioned works typically assume a fixed test set without a domain gap.

In addition, some methods try to predict accuracy on unlabeled test samples based on the agreement score of several classifies' predictions \cite{madani2004co,platanios2016estimating,platanios2017estimating,donmez2010unsupervised}. Platanios \emph{et al. } \cite{platanios2017estimating} use a probabilistic soft logic (PSL) model to infer the errors of the classifiers.
Recent work also considers the classifier accuracy estimation and proposes to predict accuracy based on the distribution difference \cite{deng2020labels}.
In this work, we attempt to estimate classifier accuracy on unlabeled test sets by leveraging self-supervision.

\textbf{Self-Supervised learning.} Self-supervised learning aims to learn good representations on unlabeled data for the downstream tasks \cite{he2020momentum,noroozi2016unsupervised,gidaris2018unsupervised,caron2018deep,bojanowski2017unsupervised,kolesnikov2019revisiting,doersch2017multi,lee2019making,ren2018cross}. To achieve this, several pretext tasks are proposed, such as solving jigsaw puzzles \cite{noroozi2016unsupervised}, colorization \cite{larsson2017colorization}, rotation prediction \cite{gidaris2018unsupervised}, denoising auto-encoders \cite{vincent2008extracting}, tracking \cite{Wang_2015_ICCV}, and clustering \cite{caron2019unsupervised, caron2018deep}.
\textcolor{black}{While these works use self-supervision for learning unsupervised representations, this work investigates its utility for unsupervised accuracy estimation.}

Reed \emph{et al. } \cite{metzger2020evaluating} use rotation prediction is used to automatically select augmentation policies for unsupervised contrastive learning method (\emph{e.g.}, MoCo \cite{he2020momentum}). Liu \cite{liu2020labels} leverage self-supervised pretext tasks for unsupervised neural architecture search.
Self-supervised methods can also be used to improve supervised classifier learning \cite{hendrycks2019using,sun2020test,carlucci2019domain,sun2019unsupervised}. Hendrycks \emph{et al. } \cite{hendrycks2019using} show jointly training classification and self-supervised task improves classifier robustness and uncertainty estimation. Sun \emph{et al. } \cite{sun2020test} achieve test-time training by using self-supervised task on unlabeled test images. Moreover, the supervised tasks can be used to alleviate distribution shifts \cite{carlucci2019domain, sun2019unsupervised,bucci2021self}.
\textcolor{black}{This work does not aim to improve supervised classification by using self-supervised tasks. Instead, we study the feasibility of using the rotation prediction task for estimating classifier accuracy.}

\textbf{Out-of-distribution detection.} The overall goal of out-of-distribution is to detect test samples from a novel distribution (a distribution different from the training distribution) \cite{hsu2020generalized,hendrycks2016baseline,lee2018simple,liang2017enhancing,guo2017calibration,ovadia2019can,ren2019likelihood}. Hendrycks \emph{et al. } \cite{hendrycks2016baseline} use probability outputs from a softmax classifier as an indicator to find out-of-distribution samples. Liang \emph{et al. } \cite{liang2017enhancing} use temperature scaling \cite{guo2017calibration} and input pre-processing to improve detection accuracy.
Ren \emph{et al. } \cite{ren2019likelihood} investigate deep generative model-based methods for the sample detection.
While the above methods aim to capture abnormal samples during the testing, our work focuses on predicting overall accuracy that involves all test samples.

\section{Classifier Accuracy Estimation}
\subsection{Problem Definition}
In the problem of unsupervised accuracy estimation, we are provided with a labeled training set and a set of unlabeled datasets for testing.
We denote the training set as $\mathcal{D}_{train} = \{({\bf m}{x}_i,{{\bf m} y}_i)\}_{i=1}^{n}$, where ${\bf m}{x}_i$ is the $i$-th training image, ${{\bf m} y}_i \in \{0, 1, ..., K-1\}$ is its label, and $n$ is the total number of images.
Similarly, we denote each test dataset as ${{\cal D}_{test}^{t}} = \{ {{\bf m}{x}}_j^t\} _{j = 1}^{{n_{t}}}$, where $t \in \{1,2, ..., M\}$ indicates the $t$-th test set, ${\bf m}{x}_j^t$ is the $j$-th test image, and $n_{t}$ is the total number of images in ${{\cal D}_{test}^{t}}$.
Given a classifier trained on $\mathcal{D}_{train}$, our goal is to estimate its accuracy on {every unlabeled test set ${{\cal D}_{test}^{t}}, t=1,2,...,M$.}

\subsection{Correlation Study} \label{correlation}

\begin{figure}[t]
\begin{center}
\includegraphics[width=1.0\linewidth]{Figs/fig2.pdf}
\caption{Our multi-task network structure for studying the statistical correlation between the accuracy of rotation prediction and classification under different test sets. The network contains a shared backbone (\emph{e.g.}, ResNet-18) for the two tasks, a fully-connected layer for 4-way rotation prediction, and another fully-connected layer for $K$-way classification ($K$ is the number of classes). The shared backbone receives losses from both tasks.}
\label{fig:2}
\end{center}
\end{figure}

\subsubsection{Network Description} \label{multi-task net}
We learn a multi-task neural network for semantic classification along with an auxiliary self-supervised task. The self-supervised task \emph{should}
1) introduce no learning complexity for the main classification,
2) require minimal network structure change, and
3) not degrade classification accuracy.
We choose \emph{rotation prediction} \cite{gidaris2018unsupervised} as the auxiliary task because it meets the above requirements. In the experiment, we observe using Jigsaw also has a strong correlation, but it decreases the classification task accuracy.

\textbf{Rotation prediction} is an effective pretext task initially designed for unsupervised representation learning \cite{gidaris2018unsupervised}. It is shown that predicting rotation degrees helps learn effective features for various downstream tasks, such as recognition, detection and segmentation.

Following the practice in \cite{gidaris2018unsupervised}, we define the set of rotation transformations $G= \{R_{r}({\bf m}{x})$\}, where $R_{r}$ is rotation transformation and $r\in\{0 \degree, 90 \degree, 180 \degree, 270 \degree \}$.
Given an image, the goal of rotation prediction is to tell which one of the four rotations this image undergoes; it can be formulated as a 4-way classification task.

\textbf{Network structure.} We use a convolutional neural network (ConvNet), \emph{e.g.}, ResNet-50. We only add one fully-connected (FC) layer for rotation prediction and do not modify any other component. The overall structure is very simple, shown in Fig. \ref{fig:2}. The ConvNet has three parts: a shared backbone parameterized by ${\bf m}{\theta}_{s}$, a FC layer for classification parameterized by ${\bf m}{\theta}_{c}$, and another FC layer for rotation prediction parameterized by ${\bf m}{\theta}_{r}$.

\textbf{Loss function.} Our network is a kind of multi-task model that jointly learns the self-supervised pretext task and semantic classification task. Specifically, the model is trained to simultaneously 1) estimate the geometric transformation applied to an image and 2) classify the semantic category of this image. For rotation prediction, its loss function is written as,
\begin{equation}\small
\mathcal{L}_{\textup{rot}} = \frac{1}{4} \left[ \sum_{  r \in \{ 0^{\circ}, 90^{\circ}, 180^{\circ}, 270^{\circ} \} } \mathcal{L}_{\textup{CE}}({\bf m}{y}_{r}, {\bf m}{\theta}_{r}({\bf m}\theta_{s}(R_{r}({\bf m}{x})))\right],
\end{equation}
where ${\bf m}{y}_{r}$ is the one-hot label of $ r\in\{ 0^{\circ}, 90^{\circ}, 180^{\circ}, 270^{\circ} \}$.

For semantic classification, the loss function is:
\begin{equation}
\mathcal{L}_{\textup{cls}} = \mathcal{L}_{\textup{CE}}({\bf m}{y}_{c}, {\bf m}{\theta}_{c}({\bf m}{\theta}_{s}({\bf m}{x}))),
\end{equation}
where ${\bf m}{y}_{c}$ is the one-hot class label of image ${\bf m}{x}$. The overall loss function of our multi-task network is: $\mathcal{L} = \mathcal{L}_{\textup{cls}} + \mathcal{L}_{\textup{rot}}$.

\textbf{Network inference.}
\textcolor{black}{After training is complete, we test the multi-task network on unlabeled test images. For the classification task, the network predicts the semantic class for each test image.
{For rotation prediction task, we apply \emph{all} four rotation transformations $G$ to each test image. The network predicts rotations of all transformed images, from which the accuracy of rotation prediction is calculated.}}

\begin{figure*}[t]
\begin{center}
\includegraphics[width=1.0\linewidth]{Figs/fig3.pdf}
\caption{Examples of synthetic datasets used in the correlation study. \textbf{Left:}  datasets under the COCO setup. Given an original set (labeled, top left), we use geometric and photometric transforms to synthesize more datasets. For example, we adopt corruption methods \emph{zoom blur} and \emph{impulse noise} to generate ``synthetic set 1'' and ``synthetic set 2'', respectively. We combine several image transformations (\emph{e.g.}, sharpness, ColorTemperature) to produce other synthetic sets.
\textbf{Right:} datasets under the MNIST setup. We change the image background of the original set to synthesize various datasets.
The synthetic datasets exhibit distinct visual characteristics, and yet inherit the foreground objects from images in the original set, and thus are fully labeled.
}\label{fig:3}
\end{center}
\end{figure*}

\subsubsection{Dataset Synthesis} \label{classifers}
With the network introduced above, we now study the accuracy relationship between rotation prediction and semantic classification under various testing environments.

To this end, we need many test sets that 1) have various distributions, 2) contain the same classes with the training set, and 3) have sufficient samples. However, there are very few datasets meeting these requirements. Following \cite{deng2020labels}, we synthesize test sets by using various geometric and photometric transforms on top of
MNIST, CIFAR-10 \cite{krizhevsky2009learning} and COCO \cite{lin2014microsoft}, creating the corresponding setups.

\textbf{MNIST setup.} MNIST contains 50K training image and 10K test image from 10 categories. We use MNIST training set to train a ConvNet based on LeNet-5.
To generate various test sets, we change the backgrounds of MNIST test set (named \emph{original set}). Specifically, we use COCO training images as backgrounds and synthesize 1,000 test sets. For all the setups in this section, because the transformations do not change the main image content, the synthesized test sets \emph{inherit the semantic labels} from the original set.

\textbf{CIFAR-10 setup.} We use DenseNet-40-12 (40 layers with growth rate 12) \cite{huang2017densely} as the backbone. We train our network on CIFAR-10 training set (50K images), and test it on CIFAR-10-C \cite{hendrycks2019benchmarking}. CIFAR-10-C was proposed to study the model robustness to common corruptions. It contains 15 types of corruption and each corruption type comes at 5 levels of severity, resulting in 75 corruption datasets. These corruptions include noise, blur, weather, and digital categories. We refer readers to \cite{hendrycks2019benchmarking} for more details.
To synthesize test sets, we apply image transformations. For each dataset, we randomly select three transformations from \{Sharpness, Equalize, ColorTemperature, Solarize, Autocontrast, Brightness, Rotate\}. Then, we apply three transformations with random magnitudes on images. This practice generates another 425 datasets.

\textbf{COCO setup.} We use 12 classes, \emph{i.e.}, aeroplane, bike, bird, boat, bottle, bus, car, dog, horse, monitor, motorbike, and person, following \cite{peng2017visda}. We build the training and test sets for the classification task based on COCO training and validation sets, respectively. Each class has 600 images. We also use the 15 common corruptions on the test set and synthesize 75 corrupted test sets. Moreover, we randomly use three transformations to generate another 425 datasets. Examples of synthetic sets are shown in Fig. \ref{fig:3}.

\begin{figure*}[t]
\begin{center}
\includegraphics[width=1.0\linewidth]{Figs/fig4_1.pdf}
\vspace{-1em}
\caption{
Correlation between classifier accuracy and rotation prediction accuracy on three setups. From left to right: CIFAR-10 with DenseNet-40-12, MNIST with LeNet-5, and COCO with ResNet-50, respectively. Each point in the figure represents a synthetic test set. Across the three setups, we consistently observe a strong linear relationship (Pearson Correlation $r > 0.88$) between the two accuracy numbers.
The straight lines are calculated by robust linear regression \cite{huber2004robust}.
}\label{fig:4}
\end{center}
\end{figure*}

\subsubsection{Empirical Analysis}
For each setup introduced above, we compute the accuracy numbers of semantic classification and 4-way rotation prediction on every synthetic test set. We empirically study the statistical relationship between the two tasks' performance using Pearson's Correlation Coefficient ($r$).

\textbf{Linear correlation.} In Fig.~\ref{fig:4}, we plot semantic classification accuracy against rotation prediction accuracy on these synthetic test sets.
We have an interesting observation: \emph{given a multi-task model, when the testing environment undergoes changes (i.e., testing on different synthesized test sets), the performance of semantic classification has a {linear} relationship with that of rotation prediction.}
This observation is consistent across three different setups. For example, on the CIFAR-10 and COCO, we find the Pearson's Correlation Coefficient ($r$) is 0.977 and 0.883, respectively, indicating strong linearity. In other words, if the multi-task network is good at predicting rotations, it is most likely to achieve good object recognition accuracy under the same environment, and vice versa.

\subsection{Estimation with Linear Regression}\label{linear_reg}
Based on the strong linear relationship between classification and rotation prediction under changing environments, we train a linear regression model to predict classification accuracy. Specifically, we use $N$ synthetic sets as training samples. The $i$-th ($i=1,2,...,N$) synthetic set is denoted as $(a_i^{rot}, a_i^{cls})$, where $a_i^{rot}$ and $a_i^{cls}$ denote the rotation prediction accuracy and semantic classification accuracy on this set, respectively. The regression model is:
\begin{equation}\label{linear}
    a^{cls} = w_1 a^{rot} + w_0,
\end{equation}
where $w_1, w_0 \in \mathbb{R}$ are linear regression parameters.
In practice, we use robust linear regression \cite{huber2004robust} with Huber loss to learn the regression model.
In Section \ref{sec:experiment}, we will present results of using the learned linear regression models for accuracy prediction on unseen test sets.

\section{Experiment}\label{sec:experiment}
\subsection{Task Setting and Baseline}
\setlength{\tabcolsep}{5pt}
\begin{table*}[t]
    \caption{Results of classifier accuracy prediction using various methods on three groups of unseen test sets: 1) SVHN and USPS; 2) CIFAR-10.1; 3) Pascal, Caltech and ImageNet. The training set for each group is MNIST, CIFAR-10, and COCO, respectively.
For each test set, we report the estimated classification accuracy (\%) and the ground-truth recognition accuracy (\%). RMSE (\%) is reported to measure the estimation precision. For ``prediction'' and ``entropy'' methods, images with high prediction scores (greater than $\tau_1$) or low entropy scores (less than $\tau_2$) are regarded as being classified correctly, respectively.}
    \begin{center}
    \begin{tabular}{l|cc|c|c|c|ccc|c}
  			\toprule
     Train Set&\multicolumn{3}{c|}{MNIST}&\multicolumn{2}{c|}{CIFAR-10}&\multicolumn{4}{c}{COCO}\\
    \hline
    Unseen Test Set & SVHN &  USPS & RMSE$\downarrow$&  CIFAR-10.1 & RMSE$\downarrow$ & Caltech & Pascal & ImageNet & RMSE$\downarrow$\\
    \hline
    Ground-truth Accuracy & 23.06 & 65.52 & - & 88.15 & - & 92.61 & 86.43 & 87.83 &-\\
    \hline
    Prediction ($\tau_1 =0.8$) & 33.64 & 44.34& 16.74 & 91.15 & 3.00 & 89.36 & 83.98 & 85.17 & 2.81\\
    Prediction ($\tau_1 =0.9$) & 22.07 & 30.39 & 24.85 & 86.85 & 1.30 & 84.30 & 78.00 & 79.83 & 8.25\\
    \hline
    Entropy ($\tau_2 =0.2$) & 26.63& 33.23 & 22.97& 89.20 & 1.05 & 86.80 & 80.14 & 82.50 & 5.82\\
    Entropy ($\tau_2 =0.3$) & 40.35& 46.87 & 17.98& 93.80 & 5.65 & 92.49 & 86.21 & 88.50 & 0.41\\
    \hline
    \rowcolor{weijian} Linear Regression & 24.84  & 53.10 &8.87 & 91.89 & 3.74 & 90.70 & 89.29 & 90.98 & 2.68\\
			\bottomrule
    \end{tabular}
    \end{center}
    \label{tab:estimate}
\end{table*}
\textbf{Task setting.} For the problem of unsupervised classifier evaluation, we experiment on two semantic classification settings, \emph{i.e.,} digit classification and natural image classification.
In each setting, we estimate the semantic classification accuracy on \emph{unseen test sets}.

For digit classification, we test LeNet-5 (trained on MNIST) on two unseen real datasets, \emph{i.e.}, USPS \cite{hull1994database} and SVHN \cite{netzer2011reading}, both with 10 classes. For natural image classification, we use ResNet-50 trained on COCO and estimate classification accuracy on three datasets: PASCAL \cite{everingham2007pascal}, Caltech \cite{griffinHolubPerona} and ImageNet \cite{deng2009imagenet}, all with 12 classes.
In addition, we use CIFAR-10.1 \cite{recht2018cifar} for CIFAR-10 setup: we estimate semantic classification accuracy of DenseNet-40-12 on this unseen test set through ConvNet trained on CIFAR-10.

\textbf{Metric.} To evaluate the correctness of performance estimation, we use root mean squared error (RMSE). RMSE measures
the average squared distance between the estimated accuracy and
ground-truth accuracy. Small RMSE corresponds to good estimation and vice versa.

\textbf{Baselines.} There are very few studies on the unsupervised accuracy estimation problem. Thus, we introduce two intuitive methods for comparison, which are inspired by the uncertainty estimation methods \cite{hendrycks2016baseline,kendall2017uncertainties,liang2017enhancing}.

\emph{Prediction score based.} Given an input image, its score is obtained as the maximum softmax output. If the score is greater than a threshold $\tau_1 \in [0,1]$, this image is regarded as being correctly classified.

\emph{Entropy score based.} The score is given by the entropy of softmax outputs, normalized by $log(K)$, where $K$ is the number of classes. Predictions with scores less than a threshold $\tau_2 \in [0, 1]$ are considered correct.

\subsection{Experimental Results}
In Table~\ref{tab:estimate}, we report the performance of our linear regression models on the six real-world test sets described above. We have the following observations.

\textbf{Score-based methods are sensitive to thresholds.} As shown in Table~\ref{tab:estimate}, the two intuitive score-based methods could produce good prediction results when using proper thresholds. However, they are sensitive to the threshold value. For example, using $\tau_1=0.9$ and $\tau_1=0.8$ will lead to a 5.44\% difference in RMSE under the COCO setup.
Furthermore, different datasets need different thresholds. For instance, for the prediction score based method, $\tau_1=0.9$ is suitable at CIFAR-10.1, but is not good for three natural image classification datasets.
Notably, it is hard to select the optimal threshold because 1) test labels are unavailable and 2) the testing environment keeps changing. That said, designing a strategy to select thresholds might be a potential way to improve the practicability of two intuitive methods.

\textbf{Our linear regression model is feasible to predict classifier performance.} The linear regression model (Section~\ref{linear_reg}) based on rotation prediction accuracy could achieve promising predictions across three setups, shown in Table~\ref{tab:estimate}.
For example, when predicting the accuracy of a classifier on three test sets under the COCO setup, our regression model has a 2.68\% RMSE. In addition, it estimates the accuracy to be 91.89\% on CIFAR-10.1 (the ground truth is 88.15\%).
Since the training samples (synthetic datasets) of the regression model are diverse enough (\emph{e.g.}, diverse distributions and various visual characteristics), it can learn to achieve reasonably accurate estimations on unseen test sets.

\textbf{Impact of the number of training samples (synthetic sets).}
In Fig.~\ref{fig:sample_size}, we present the RMSE (\%) obtained by linear regressors trained using different numbers of synthetic sets. Since the linear regression model only has two parameters, it does not require too many training samples \cite{huber2004robust}. In our problem, when using more than 200 synthetic sets, RMSE is relatively stable. In practice, we speculate that using sufficient and diverse synthetic sets is necessary to improve regression robustness.
\begin{figure}[t]
\begin{center}
\includegraphics[width=1.0\linewidth]{Figs/fig_sample_size.pdf}
\vspace{-1.5em}
\caption{
Impact of the number of training samples (synthetic sets) for linear regression. \textbf{Left:} RMSE (\%) on CIFAR10.1. \textbf{Right:} RMSE (\%) on Caltech, Pascal, and ImageNet.
We observe that the regression model becomes stable when using more than 200 training samples.
}
\label{fig:sample_size}
\end{center}
\vspace{-10pt}
\end{figure}
\begin{figure}[t]
\begin{center}
\includegraphics[width=1.0\linewidth]{Figs/fig6.pdf}
\caption{Visual examples of transformed natural ImageNet images. For ImageNet-A, we use \emph{Cutout} and \emph{Posterize} to edit original images. For ImageNet-B, we use \emph{Pepper} and \emph{FilterSmooth} to introduce visual changes.
}
\label{fig:6}
\end{center}
\vspace{-10pt}
\end{figure}

\textbf{Robustness of linear regressor on transformed test sets.} We apply various image transformations to edit three natural image datasets (ImageNet, Pascal and Caltach), and test the regression model on them.  Specifically, we use \emph{Cutout} and \emph{Posterize} to produce ImageNet-A, Pascal-A and Caltach-A, and use \emph{Pepper} and \emph{FilterSmooth} to produce ImageNet-B, Pascal-B and Caltach-B. Visual examples of these datasets are shown in Fig.~\ref{fig:6}.
Note that these transformations \emph{were not used} when synthesizing datasets to train the regression model.
Thus, testing on them allows us to understand the robustness of the regression model.

We have the following observations in Fig.~\ref{fig:7}. First, the transformations introduce extra visual changes, so the classifier accuracy decreases on the edited dataset. Second, the regression model still makes good estimated results on the edited test sets. For example, it makes good predictions on Caltech-A and Caltech-B, as the absolute errors are 0.01\% and 2.95\%, respectively.

\textcolor{black}{Although the absolute error on Pascal-A is 6.35\%, the estimated accuracy still helps us gain a general understanding of classifier behaviour on this dataset.}
We think the regression precision can be further improved by using other information, such as distribution shift \cite{deng2020labels}.
Moreover, \textcolor{black}{if we are provided with a small number of labeled test images, it could be helpful for adjusting regression predictions.}

\begin{figure}[t]
\begin{center}
\includegraphics[width=1.0\linewidth]{Figs/fig7.pdf}
\caption{Linear regression on transformed test sets (ImageNet, Pascal, and Caltech). The transformed datasets are denoted by ``-A" and ``-B".
The absolute error (\%) between estimated classier accuracy and the ground truth accuracy (\%) (shown below each dataset) is reported. (-) / (+) denotes the predicted accuracy is lower and higher than the ground-truth accuracy, respectively.
}
\label{fig:7}
\end{center}
\end{figure}

\textbf{Generalization ability of the regression model.}  To further validate this, we test on \emph{four natural test sets} with different types of domain shifts \cite{taori2020measuring} and report estimations in Fig. \ref{fig:8}. We observe that our method overall gains good estimations, and its the average RMSE of the four datasets is 3.07\%.

Based on the above experimental analysis, we think self-supervised rotation prediction provides a feasible way to predict classifier accuracy on varying testing environments without annotated category labels.

\begin{figure}[t]
\begin{center}
\includegraphics[width=0.88\linewidth]{Figs/fig_imagenet.pdf}
\caption{The absolute error (\%) of predictions on \emph{four natural} test sets under COCO setup. The ground truth (\%) is shown under each dataset. (-) / (+) means the predicted accuracy is lower / higher than the ground-truth accuracy, respectively.
}
\label{fig:8}
\end{center}
\end{figure}

\section{Discussion}
\textbf{When trained in a multi-task ConvNet, does rotation prediction affect semantic classification accuracy?} To answer this question, we conduct experiments to study whether classification is compromised by the introduction of the rotation prediction head.
In Table~\ref{tab2}, we use three types of networks (ResNet-26, DenseNet-40-12, and VGG-19). We observe that auxiliary rotation prediction task \emph{does not} degrade main classification on the CIFAR-10 test set.

In fact, rotation prediction requires the neural network to capture the rotation-related representations of an image (\emph{e.g.}, particular structures and salient object-level patterns), which are also useful for objection recognition \cite{gidaris2018unsupervised}.
Moreover, while rotation prediction does not improve accuracy on original or clean testing image, it is reported to improve classifier robustness to input corruptions \cite{hendrycks2019using}. In addition, recent works \cite{sun2019unsupervised,bucci2021self} have shown that using rotation prediction for training can improve the domain adaptation ability of the classifier.
Based on the above discussion, introducing auxiliary rotation prediction task \emph{will not} negatively impact the main classification task.

\setlength{\tabcolsep}{7pt}
\begin{table}[t]
\caption{Impact of the auxiliary rotation prediction (Rot.) task on the main classification task. We conduct experiment on CIFAR-10 with ResNet26, Dense40-12, and VGG19. Top-1 accuracy (\%) is reported. We show that the rotation prediction task \emph{does not} affect the classifier accuracy.}
\small
\begin{center}
\begin{tabular}{c|ccc}
		\toprule
{\multirow{2}{*}{Train w/ Rot.?}}&\multicolumn{3}{c}{CIFAR-10}\\
\cline{2-4}
&ResNet26&Dense40&VGG19\\
\hline
&92.86$\pm$0.10&94.65$\pm$0.15&92.31$\pm$0.13\\
\hline
\checkmark&92.84$\pm$0.10&94.75$\pm$0.15&92.51$\pm$0.20\\
	\bottomrule
\end{tabular}
\end{center}
\label{tab2}
\end{table}
\begin{figure}[t]
\begin{center}
\includegraphics[width=1.0\linewidth]{Figs/fig5.pdf}
\caption{Correlation between semantic classification accuracy and rotation prediction accuracy when using difference types of network structures. On CIFAR-10, we use ResNet-26 and VGG-19. The Pearson Correlation $r$ is 0.975 and 0.987 with ResNet-26 and VGGNet-19, respectively.
On Tiny ImageNet, the correlation $r$ is 0.921 and 0.919 using VGG-19 and ResNet-50, respectively.
}
\label{fig:5}
\end{center}
\end{figure}

\textbf{Does the linear relationship hold when using other backbone structures?}
To study this, we use different types of backbones for correlation study and report results in Fig.~\ref{fig:5}.
We also include Tiny ImageNet for the correlation study. It has 200 classes, and each class has 500 training images. We follow the COCO setup to synthesize 500 datasets.

For CIFAR-10, when using ResNet-26 and VGGNet-19 as backbones, we observe a strong linear correlation between the two task performance. Specifically, Pearson's Correlation ($r$) is 0.975 and 0.987 with ResNet-26 and VGGNet-19, respectively.
When using VGG-19 and ResNet-50 on Tiny ImageNet, we also observe the correlation ($r$) is 0.921 and 0.919, respectively. It also indicates the strong linear correlation between two task accuracy on varying test sets is maintained when using different backbones.

In addition, we show the correlation values on CIFAR-10 setup using different backbones in Table~\ref{tab3}. In the experiment, we use five backbones: VGG11, VGG19, ResNet26, ResNet44, and Dense40-12.
\textcolor{black}{We have two observations. First, the correlation values are very high ($r>0.975$) when using different backbones. Another interesting finding is: the performance of two tasks is varying with different backbones, but their correlation on different test sets remains at the same level: \emph{the ``mean $\pm$ std" of Pearson's Correlation ($r$) is $0.983 \pm 0.005$}}.

\setlength{\tabcolsep}{3pt}
\begin{table}[t]
\caption{
Linear correlation ($r$) on CIFAR-10 with different backbone structures, including VGG11, ResNet26, VGG19, ResNet44, and Dense40-12. We also include the accuracy (\%) of classification (Class. Acc) and rotation prediction (Rot. Acc) on the CIFAR-10 test set. Using different backbones, the two task performance is varying, but their correlation remains at the same level: {the ``mean $\pm$ std" of correlation ($r$) is $0.983 \pm 0.005$}.
}
\begin{center}
\small
\begin{tabular}{c|c|c|c|c|c}
	\toprule
              & VGG11   & VGG19 & ResNet26 & ResNet44 & Dense40 \\ \hline
Class. Acc.   & 92.53   & 92.51 & 92.84 & 93.73    & 94.75         \\ \hline
Rot. Acc.     & 91.32   & 92.07 & 87.84 & 88.81    & 91.28         \\ \hline
Cor. ($r$) & 0.990   & 0.987 & 0.975 & 0.981   & 0.981          \\
\bottomrule
\end{tabular}
\end{center}
\vspace{-10pt}
\label{tab3}
\end{table}

\textbf{Impact of the number of categories on correlation.} For rotation prediction, it distinguishes between four different classes, but for image classification, the number of categories can be much greater. In Fig. \ref{fig:4} and Fig. \ref{fig:5}, we observe the correlation is high ($r >0.88$) across different datasets with various numbers of categories. For example, the correlation is  $r>0.90$ on Tiny-ImageNet with 200 categories.

We further study on CIFAR-100 setup and report results in Table \ref{tab:4}. We follow the CIFAR-10 setup to synthesize 500 datasets for CIFAR-100 setup. Although the correlation $r$ is lower than that on CIFAR-10 (0.918 vs. o.975 with ResNet26), the value $r=0.918$ still indicates strong linear correlation under the CIFAR-100 setup. Based on the above analysis, we speculate when the number of categories is huge (\emph{e.g.}, 10K \cite{deng2010does}), the correlation $r$ might decrease but it will still have a high value.

\textcolor{black}{\textbf{Why is rotation prediction performance closely correlated with classification performance?} Our multi-task network jointly learns both tasks. By doing so, they are coupled together and share the feature representations. When testing on images from the novel distribution, the dataset shift will make the presentations not suitable for the test set. Thus, the network performance on both tasks drops simultaneously.
That is, different test sets introduce different dataset shifts, leading to varying degrees of performance drop.
This might be the reason why their performance shows a positive correlation trend on varying test domains.}

\setlength{\tabcolsep}{6pt}
\begin{table}[]
\caption{Effect of the number of semantic categories on the correlation. Although the rank correlation ($\rho$) on CIFAR-100 is slightly lower than that on CIFAR-10, it still indicates that the two tasks have a strong linear correlation ($\rho>0.9$).
The accuracy (\%) of classification (Class. Acc) and rotation prediction (Rot. Acc) on CIFAR-100 test set is also reported. }
\begin{center}
\small
\begin{tabular}{c|c|ccc}
	\toprule
\multirow{2}{*}{Backbone}
         & \multicolumn{1}{c|}{CIFAR-10} & \multicolumn{3}{c}{CIFAR-100} \\
          \cline{2-5}
         & Cor. ($r$)       &  Cor. ($r$)         &  Class Acc.  & Rot. Acc.    \\ \hline
ResNet26 & 0.975               &     0.918              &   69.31      &   73.18      \\ \hline
ResNet44 & 0.981               &     0.910              &   71.38      &   75.60      \\ \hline
Dense40 & 0.981                &     0.950              &   74.55      &   75.20     \\ \bottomrule
\end{tabular}
\end{center}
\vspace{-10pt}
\label{tab:4}
\end{table}

\textbf{Difference from those using self-supervision for classification \cite{sun2020test,hendrycks2019using,carlucci2019domain,zhai2019s4l}.} These works present that the \emph{self-supervised} task can help \emph{supervised} classification task in different perspectives, such as robustness \cite{hendrycks2019using} and domain adaptive ability \cite{sun2020test}. This work differs significantly:
our work does not aim to improve supervised classification by using self-supervised tasks. Rather we offer a new and interesting perspective into the relationship between the two tasks, a finding that allows estimation of generalization performance for semantic classification models from easily obtained self-supervised rotation prediction results.

\textbf{Limitation.} We might encounter corner cases (\emph{e.g.}, balls and airplanes) where they are even impossible for a human to tell the image rotation angle. The proposed method, because of the intrinsic limitation of rotation prediction, would fail in such cases. In fact, as an underlying assumption, the self-supervised task is required to be well-defined and non-trivial \cite{sun2020test}. Moreover, we might encounter images from unseen classes in the open world. To improve the estimation under this scenario, a feasible solution is using out-of-distribution detection techniques \cite{hsu2020generalized,hendrycks2016baseline,lee2018simple,liang2017enhancing} to detect and reject such novel images.

\textbf{Feasibility of other self-supervised tasks than rotation prediction.} There are many other self-supervised tasks, such as jigsaw puzzles \cite{noroozi2016unsupervised} and colorization \cite{larsson2017colorization}. Since our network jointly learns semantic classification and pretext tasks, we select the pretext task based on the criteria listed at the beginning of Section \ref{multi-task net}.
In the preliminary experiment, we found that \emph{colorization} discards the color information of images and thus makes the learning of semantic classification unstable. In addition, we observe that \emph{Jigsaw} and semantic classification exhibit a linear correlation ($r > 0.950$) under the CIFAR-10 setup (Table \ref{tab5}). Nevertheless, in our multi-task network, Jigsaw compromises the classification accuracy by about $2\%$, while rotation prediction does not affect semantic classification (shown in Table \ref{tab2}).
Moreover, our network needs to learn the semantic classification and pretext task together. It would be interesting to study other potential ways to use pretext tasks for estimating classifier accuracy.

\setlength{\tabcolsep}{6pt}
\begin{table}[]
\caption{The extent of linear correlation ($r$) on CIFAR-10 when using Jigsaw or rotation prediction as the pretext task. The accuracy (\%) of classification (Class. Acc) on the CIFAR-10 test set is also reported. Jigsaw and semantic classification also have a strong linear correlation ($r>0.95$). However, it degrades the classification accuracy when trained in the multi-task network.
}
\vspace{1.5em}
\begin{center}
\small
\begin{tabular}{c|cc|cc}
	\toprule
\multirow{2}{*}{Backbone}
         & \multicolumn{2}{c}{Rotation} & \multicolumn{2}{c}{Jigsaw}\\
          \cline{2-5}
          &  Cor. ($r$)       &  Class Acc.  & Cor. ($r$)  & Class Acc.  \\ \hline
ResNet26  &     0.975         &   92.84      &   0.953     & 90.90 \\ \hline
ResNet44  &     0.981         &   93.73      &   0.958     & 91.81 \\
\bottomrule
\end{tabular}
\end{center}
\label{tab5}
\end{table}

\section{Conclusion}
In this paper, we try to estimate classifier accuracy under varying testing environments without human-annotated labels. We report a strong linear correlation ($r>0.88$) between semantic classification accuracy and rotation prediction accuracy through empirical studies. This interesting finding makes it feasible to estimate classifier accuracy on unlabeled test sets using rotation prediction performance which can be easily obtained. The extensive experiments provide supports and insights for using such a pretext task to predict the generalization ability of the classifier.

\section*{Acknowledge}
{This work was supported in part by the Australian Research Council (ARC) Discovery Early Career Researcher Award (DE200101283), ARC Centre of Excellence in Robotic Vision (CE140100016), and the ARC Discovery Project (DP210102801). We thank all anonymous reviewers for their constructive comments in improving this paper.}


\section{Introduction}

Consider the bimanual robot manipulation tasks such as rearranging multiple objects to their target locations in Figure~\ref{fig:teaser} (a). This complex and compositional task is very challenging as the agents will first need to reduce it to several sub-tasks (pushing or grasping each object), and then the two agents will need to figure out how to allocate each sub-task to each other (which object each robot should operate on) for better collaboration. Importantly, two robots should avoid collision in a narrow space for safety concerns. While training a single RL agent that can solve such compositional tasks has caught research attention recently \citep{acrt, mcp, cpv,ladrl, sir, richardLi}, there are still two main challenges that are barely touched when it comes to tackle bimanual manipulation: (i) \emph{domination}, \textit{i.e.}, one robot may tend to solve all the sub-tasks while the other robot remains idle, which hurts the task solving efficiency; (ii) \emph{conflict}, \textit{i.e.}, two robots may try to solve the same sub-task simultaneously, which result in unsafe conflicts and interruptions on shared workspace.

One possible solution is to design a task-allocation reward function to encourage better coordination. However, it is particularly non-trivial and often sub-optimal to manually design such a reward function for complex problems that contain a large continuous sub-task space, such as the rearrangement task in Figure~\ref{fig:teaser} (a). Moreover, even with the reward function described above in hand, it remains unclear how to reduce collisions, particularly for the tasks that require two robots to act simultaneously and safely. For example, in the task shown in Figure~\ref{fig:teaser} (d), one robot needs to push the green door to make space for the other robot to move the blue box to the goal position. However, these two robots can easily interrupt and collide with each other when they perform these coordination actions.

\begin{figure}[htpb]
	\centering
	\includegraphics[width=0.99 \textwidth]{figures/teaser.pdf}
	\vspace{-0.1in}
	\caption{{\small{Five bimanual manipulation tasks. (a) Rearrange the blocks to goal positions. (b) Stack the blocks into a tower. (c) Open the box and put the block inside. (d) Open the green door and push the block through the wall to the goal position. There are springs on both the box in (c) and the door in (d), which will close automatically without external force. Thus it requires one robot to hold the box cover and the door. (e) Lift up and rotate a bar with two arms to a target configuration, where the gripper is locked as closed, so it cannot be done by one arm. More details about our environments are in Appendix~\ref{sec:app1}.}}
	}
	\label{fig:teaser}
	\vspace{-0.15in}
\end{figure}

We consider an alternative setting using sparse rewards without explicitly assigning sub-tasks to the robots. However, this leads to another challenge: How to encourage the robots to explore collaborative and safe behaviors with limited positive feedbacks?  For bimanual manipulation, an intrinsic motivation is introduced by \cite{rohan-iclr}, leveraging the difference between the actual effect of an action (taken by two robots) and the composition of individual predicted effect from each agent using a forward model. While this intrinsic reward encourages the two robots to collaborate for tasks that are hard to achieve by a single robot, it does not address the \emph{domination} and \emph{conflict} problems for efficient and safe manipulation.

In this paper, we present \textbf{DAIR}: \textbf{D}isentangled \textbf{A}ttention \textbf{I}ntrinsic \textbf{R}egularization which encourages the two robots to safely and efficiently collaborate on different sub-tasks during bimanual manipulation. Instead of designing a new intrinsic reward function, we introduce a simple regularization term for representation learning, which encourages the robots to \emph{attend} to different interaction regions. Specifically, we adopt the attention mechanism~\citep{self-attention} in both our policy and value networks, where we compute the dot-product between each robot representation and the object interaction region representations to obtain a probability distribution. Each robot has its own probability distribution to represent which interaction region it is focusing on. We define our intrinsic regularization as minimizing the dot product between the two probability distributions between two robots (\textit{i.e.}, to be orthogonal) in each time step. By adding this loss function, different robots will be regularized to attend to different interaction points within their policy representation. This forces the policies to tackle sub-tasks over disjoint working space without interfering with each other. We remark that disentangled attention can be generalized to environments with multiple agents.

In our experiments, we focus on five diverse manipulation tasks in simulation environments with two Fetch robots as shown in Figure~\ref{fig:teaser}. These tasks not only require the robots to manipulate multiple objects (more than two, up to \emph{eight}) with each object offering one interaction region, but also a single heavy object with multiple interaction regions (Figure~\ref{fig:teaser} (e)). In our experiments, we show that our approach not only improves performance and sample efficiency in learning, but also helps avoiding the domination problem and largely reducing the conflicts for safe coordination between two robots. Moreover, the learned policies can also solve the task in fewer steps, which is the significance of bimanual cooperation compared to single-arm manipulation. We highlight our main contributions as:
\begin{itemize}
\vspace{-0.1in}
    \item Observation for two important problems (\emph{domination} and \emph{conflict}) in training RL agents for safe bimanual manipulation, and a new robotics task set with one to eight objects.
    \item  We propose DAIR, a novel and general intrinsic regularization. It not only improves the success rate in bimanual manipulation, solves the tasks more efficiently, but also reduces the conflicts between robots. This allows the robots to collaborate and coordinate more safely.
\end{itemize}
\section{Related Work}
\vspace{-0.1in}
{\bf Intrinsic motivation in reinforcement learning.} To train RL agents with sparse rewards, \cite{schmidhuber1991possibility} first proposed to motivate the agent to reach state space giving a large model prediction error, which indicates the state is currently unexplored and unseen by the model. Such a mechanism is also called intrinsic motivation, which provides a reward for agents to explore what makes it curious~\citep{oudeyer2007intrinsic,barto2013intrinsic,bellemare2016unifying,ostrovski2017count,huang2019learning}. Recently, it has also been shown that the agents can explore with such an intrinsic motivation without extrinsic rewards~\citep{curiosity,burda2018exploration,large-scale-curiosity}. Besides using prediction error, diverse skills can also be discovered by maximizing the mutual information between skills and states as the intrinsic motivation~\citep{diayn,dads}. While these approaches have achieved encouraging results in single agent cases, they are not directly applicable to environments with multiple agents. In our paper, we propose a novel intrinsic regularization for helping two robots work actively on different sub-tasks.

{\bf Multi-agent collaboration.} Cooperative multi-agent reinforcement learning has exhibited progress over the recent years~\citep{foerster2016learning,he2016opponent,peng2017multiagent,maddpg,coma,vdn,q-mix,q-tran,roma}. For example, \cite{maddpg} proposed to extend the DDPG~\citep{ddpg} algorithm to the multi-agent setting with decentralized policies and centralized Q functions, which implicitly encourages the agents to cooperate. However, the problem of exploration still remains as a bottleneck, and in fact even more severe in multi-agent RL. Motivated by the previous success on a single agent, intrinsic motivation is also introduced to help multiple agents explore and collaborate~\citep{foerster2016learning,strouse2018learning,hughes2018inequity,iqbal2019coordinated,jaques2019social,wang2019influence}. For example, \cite{jaques2019social} proposed to use social motivation to provide intrinsic rewards which model the influence of one agent on another agent's decision making. The work that is most related to ours is by \cite{rohan-iclr} on the intrinsic motivation for synergistic behaviors, which encourages the robots to collaborate for a task that is hard to solve by a single robot. As this paper has not focused on the domination and conflict problems, our work on disentangled attention is a complementary technique to the previous work.

{\bf Bimanual manipulation.} The field of bimanual manipulation has been long studied as a problem involving both hardware design and control~\citep{raibert1981hybrid,hsu1993coordinated,xi1996intelligent,smith2012dual}. In recent years, researchers applied learning based approach to bimanual manipulation using imitation learning from demonstrations~\citep{zollner2004programming,gribovskaya2008combining,tung2020learning,xie2020deep} and reinforcement learning~\citep{kroemer2015towards,amadio2019exploiting,chitnis2020efficient,rohan-iclr,ha2020learning}. For example, \cite{amadio2019exploiting} proposed to leverage probabilistic movement primitives from human demonstrations. \cite{chitnis2020efficient} further introduced a high-level planning policy to combine a set of parameterized primitives to solve complex manipulation tasks. In contrast to these works, our approach does not assume access to pre-defined primitives. Both robots will learn how to perform each sub-task and how to collaborate without conflicts in an end-to-end manner, which makes the approach more general.

{\bf Attention mechanism.} Our intrinsic motivation is built upon the attention mechanism which has been widely applied in natural language processing~\citep{self-attention} and computer vision~\citep{wang2018non,vit}. Recently, the attention mechanism is also utilized in multi-agent RL to model the communication and collaboration between agents~\citep{zambaldi2018relational,jiang2018learning,malysheva2018deep,iqbal2019actor,epc}. For example, \cite{epc} proposed to utilize attention to flexibly increase the number of agents and perform curriculum learning for large-scale multi-agent interactions. \cite{richardLi} adopt the attention mechanism to generalize multi-object stacking with a single arm. In our paper, instead of simply using attention for interaction among hand and a variable number of objects, we propose DAIR to encourage the agents to attend on different sub-tasks for better collaboration.
\section{Preliminaries}
\label{Preliminaries}
\vspace{-0.08in}
We consider a multi-agent Markov decision process (MDP)~\cite{littman1994markov} with $N$ agents, which can be represented by $(S, A, P, R, H, \gamma)$. The state $s \in S$ and the action $a_i \in A$ for agent $i$ are continuous. $P(s^{t+1}|s^t, a^t_1,...,a^t_N)$ represents the stochastic transition dynamics. $R_i(s^t, a^t_i)$ represents the reward function for agent $i$. $H$ is the horizon and  $\gamma$ is the discount factor. The policy $\pi_{\theta_i} (a^t|s^t)$ for agent $i$ is parameterized by ${\theta_i}$. The goal is to learn multi-agent policies maximizing the  return. In this paper, we tackle a two-agent collaboration problem ($N=2$), but our method can  generalize to more agents.

\vspace{-0.1in}
\subsection{Reinforcement Learning with Soft Actor-Critic}
\vspace{-0.1in}

We adopt the Soft Actor-Critic (SAC)~\cite{sac} for reinforcement learning (RL) training in this paper. It is an off-policy RL method using the actor-critic framework. The soft Q-function for agent $i$ is $Q_{\theta_i}(s^t, a^t_i)$ parameterized by $\theta_i$. For  agent $i$, there are three types of parameters to learn in SAC: (i) the policy parameters ${\phi_i}$; (ii) a temperature $\tau_i$; (iii) the soft Q-function parameters $\theta_i$. We can represent the policy optimization objective for agent $i$ as,
\begin{equation}
J_\pi(\phi_i) = \mathbb{E}_{s^t\sim\mathcal{D}}\left[\mathbb{E}_{a^t_i\sim\pi_{\phi_i}}[\tau_i \log \pi_{\phi_i}(a^t_i|s^t) - Q_{\theta_i}(s^t, a^t_i)]\right],
\label{eq:reparam_objective}
\end{equation}
where $\tau_i$ is a learnable temperature coefficient for agent $i$, and $D$ is the replay buffer. It can be learned to maintain the entropy level of the policy:
\begin{equation}
J(\tau_i)  = \mathbb{E}_{a^t_i \sim \pi_{\phi_i}} \left[ - \tau_i\log\pi_{\phi_i}(a^t_i|s^t) - \tau_i \bar{\mathcal{H}}\right],
\label{eq:ecsac:tau_objective}
\end{equation}
where $\bar{\mathcal{H}}$ is a desired minimum expected entropy. The soft Q-function parameters $\theta_i$ for agent $i$ can be trained by minimizing the soft Bellman residual as,
\begin{align}
J_Q(\theta_i) = \mathbb{E}_{(s^t,a^t_i)\sim\mathcal{D}}\left[ \frac{1}{2} (Q_{\theta_i}(s^t,a^t_i)-\hat{Q}(s^t,a^t_i))^2\right], \label{eq:Qfunc} \\
\hat{Q}(s^t,a^t_i) = R_i(s^t,a^t_i) +\gamma \mathbb{E}\left[ \max\limits_{{a^{t+1}_i} \sim \pi_{\phi_i}} Q_{\theta_i}(s^{t+1},a^{t+1}_i)\right].
\label{eq:Qfunc2}
\end{align}
Since we focus on collaborative robotics manipulation tasks, the reward is always shared and synchronized among the agents. That is, if one agent is able to finish a goal and obtain a reward, the other agents will receive the same reward.

\vspace{-0.05in}
\subsection{Criteria for Measuring Efficiency and Safety}
\vspace{-0.05in}
\label{sec:criteria}
Our core contribution is to ensure the manipulating efficiency and safety by reducing the problems of domination and conflict in the process of manipulation, which also improves the speed of finishing the task. We define three criteria which are all lower the better, and evaluate our approach using them in our experiments:
(i) \textbf{Domination Rate:} We count how many steps an arm is interacting with an object in one episode as the manipulating steps. We compute the ratio of an agent's manipulating steps over the two agents' total manipulating steps. We select the maximum ratio as the Domination Rate. Ideally, we hope the Domination Rate to be close to $50\%$ which indicates both robots are actively interacting with the objects.
(ii) \textbf{Conflict Rate:} It counts the percentage of the ``conflict step'' over all steps. We consider it a  conflict step when the distance between two grippers is smaller than a small threshold. This means two robots are interrupting each other's action.
(iii) \textbf{Finish Steps:} How many steps do two agents take to finish the task successfully. The maximum episode length in both environments is 100. Note that it's reasonable only when we consider all the metrics together, since robots can use trivial solution to reduce one of them, for example, only complete the task with one robot (extreme domination behavior) to avoid conflicts.

\section{Method}
\vspace{-0.05in}
\label{method}
Our goal is to design a model and introduce a novel intrinsic regularization to better train the policies for bimanual manipulation tasks. We hope the agents can learn to allocate the workload efficiently and safely. In this section, we will first introduce our base network architecture with the attention mechanism motivated by~\citep{self-attention}. Based on this architecture, we will then introduce DAIR and how to perform reinforcement learning with it.

\vspace{-0.05in}
\subsection{Network Architecture}
\vspace{-0.05in}
Policy and Q-function networks share the same architecture as follow. For simplicity, we omit superscript time $t$ when there is no ambiguity. We can then represent the state as $s=[s_1,\ldots,s_N, s_{N+1},\ldots,s_{N+M}]$ where the first $N$ entities represent the state of the robot arms, the next $M$ entities represent the states of the interaction regions. Here we define an \emph{interaction region} as a space on object that robot can manipulate on (grasp or push, etc) to handle the task. In Figure~\ref{fig:teaser}, for the first four tasks with smaller objects such as cubes, door, and cover, we define one interaction region located around each object. For the last task of adjusting a large heavy bar, we define it with two interaction regions with one region corresponding to one face. Note that the interaction regions can be specified according to the tasks flexibly, and our method can be generalized to any numbers of interaction regions.

For agent $i$, we have a set of \emph{state encoder} functions $\{f_{i,1}(\cdot),\ldots,f_{i,N}(\cdot), f_{i,N+1}(\cdot),\ldots,f_{i,N+M}(\cdot)\}$ corresponding to the input states, as shown in Figure~\ref{fig:pipeline}. Each state encoder function $f_{i,j}(\cdot)$  takes the state $s_j$ as the input and outputs a representation (512-D) for the state in our policy network. We use a 2-layer multilayer perceptron (MLP) to model $f_{i,j}(\cdot)$. While there are $N\!+\!M$ state encoder functions, there are only three sets of parameters (visualized by three different colors in Figure~\ref{fig:pipeline}): (i) the parameters of the state encoder for agent $i$ itself $f_{i,i}(\cdot)$; (ii) the parameters of the other agents $f_{i,j}(\cdot), (1\leq j \leq N, i \neq j)$ (shared); (iii) the parameters of all the interaction regions $f_{i,j}(\cdot), (N+1\leq j \leq N+M)$ (shared). In this way, our model can be extended to environments with different number of interaction regions and agents.  We represent the policy for agent $i$ as,
\begin{equation}
\pi_{\phi_i}(a_i|s) =h_i (f_{i,i}(s_i) \!+\! \textnormal{LayerNorm}(g_i( v_i))),
\label{eq:policyfun}
\end{equation}
where $g_i(\cdot)$ is one fully connected layer to further process the  \emph{attention embedding} $v_i$, which encodes the relationship between agent $i$ and all the state entities (including all agents and interaction regions). Adding attention embedding to $f_{i,i}(s_i) $ with a LayerNorm operator serves as a residual module to retain agent $i$'s own state information. The combined features are fed to a 2-layer MLP $h_i(\cdot)$. The output of $h_i(\cdot)$ is the action distribution. Note that the parameters of $h_i(\cdot),g_i(\cdot)$ are not shared across the agents.

Motivated by~\cite{self-attention}, we further define the attention embedding $v_i$ for agent $i$ as,
\begin{equation}
v_i = \sum_{j=1}^{N+M} \alpha_{i,j} f_{i,j} (s_j),\;\;\; \alpha_{i,j} = \frac{\exp{(\beta_{i,j})}}{\sum \exp{(\beta_{i,j})}}, \;\;\;
\beta_{i,j} = \frac{f_{i,i}^T(s_i)W^T_{q}W_{k}f_{i,j}(s_j)}{\sqrt{d_q}},
\label{eq:vfun}
\end{equation}
where $W_{q}$ represents one fully connected layer to encode the
\emph{query} representation $f_{i,i}(s_i)$ and $W_{q}$ represents another fully connected layer to encode the \emph{key} representation $f_{i,j}(s_j)$.  $d_q$ is the dimension of the query representation. $\beta_{i,j}$ represents the correlation between agent $i$ and all the other entities.

It is then normalized by a softmax function to $\alpha_{i,j}$ as the probability value, which indicates where agent $i$ is ``attending'' or focusing on in the current time step and $\alpha_i \in \mathbb{R}^{N+M}$. $v_i$ is computed via a weighted sum over all the state encoder representations. For Q-function, there are two modifications: (i) The state encoder $f_{i,i}^Q(s_i, a_i)$ for the agent $i$ also takes in the action as inputs; (ii) The final layer of the Q-function network outputs a scalar value instead of an action distribution as $Q_{\theta_i}(s^t,a_i^t)$, which is used in Eq.~\ref{eq:reparam_objective} and Eq~\ref{eq:Qfunc}.

\subsection{Disentangled Attention as Intrinsic Regularization}
\label{subsec:DAIR}
\begin{wrapfigure}{r}{0.55\textwidth}\label{pipeline}
\vspace{-0.15in}
	\centering
	\includegraphics[width= 0.55\textwidth]{figures/pipeline.pdf}
		\vspace{-0.25in}
	\caption{\small{Our model framework. We use attention mechanism to combine all embedded representations from agents and interaction regions. The output of attention module, together with another embedded vector from $s_i$ are summed together with $\bigoplus$. The combined feature is fed into a 2-layer MLP $h_i$ to output $a_i$. The intrinsic loss is computed from the attention probability $\alpha_i$ and encourages the agents to attend to different sub-tasks.}}
	\label{fig:pipeline}
	\vspace{-0.5in}
\end{wrapfigure}
We propose \emph{disentangled attention as intrinsic regularization (DAIR)} to improve the state encoder representations for solving the problem of \emph{domination} of a single agent and the \emph{conflict} between the agents. Each manipulation task can be decomposed to multiple sub-tasks, each with a different interaction regions. Our key insight is to encourage different robots to attend or focus on different interaction regions, and consequently to work on different sub-tasks.

Specifically, we look into the softmax probability $\alpha_{i,j} \in [0,1]$ from the attention mechanism in Eq.~\ref{eq:vfun}. This variable represents how much attention agent $i$ is putting on interaction region/agent $j$. To encourage the agents to focus on different entities, we propose the following loss function for agent $i$ as,
\begin{equation}
L_{\textrm{attn}}(\phi_i) = \sum\limits_{\mbox{\tiny
$\begin{array}{c}
j=1, j\neq i\end{array}$
}}^{N} <\alpha_i,\alpha_j> ^ 2,
\label{eq:attenobj}
\end{equation}
where $<\cdot,\cdot>$ denotes dot product of two vectors. This loss forces the dot product between two attention probability vector to be small, which encourages different agents to attend on different entities (including agents and interaction regions). We call this particular  attention maps regulated by the orthogonal constraint as \emph{disentangled attention}.

Recall that $\alpha_i$ is predicted via the state encoder functions, parameterized by a part of $\phi_i$. Instead of proposing a new reward function, our disentangled attention regularization is directly applied on learning the state encoder representation itself. The training objective for the policy network and Q-function can be represented as,
\begin{equation}\label{eq_joint}
\begin{aligned}
\min\limits_{\phi_i}J_\pi(\phi_i) + \lambda L_{\textrm{attn}}(\phi_i),\ \min_{\theta_i} J_Q(\theta_i) + \lambda L_{\textrm{attn}}(\theta_i).
\end{aligned}
\end{equation}
where $\lambda=0.05$ is a constant to balance the reinforcement learning objective and our regularization.

\textbf{Implementation Details.} The robot state $s_i (1 \leq i \leq N)$ contains the joint positions and velocities and the end-effector positions. Thus each robot can reason the other robot's joint state and avoid conflicts. Each interaction region $s_i (N+1 \leq i \leq N+M)$ contains the target interacting position, velocity, pose and its goal position, which are all in $(x,y,z)$-coordinates. The action representation contains the positional control and the gripper motion information.
\section{Experiments}
\label{Experiments}
\textbf{Environment and setting.} We perform our experiments on complex bimanual manipulation tasks ($N=2$) in the MuJoCo simulator~\citep{todorov2012mujoco}. By further leveraging curriculum learning, we successfully complete the scenarios with up to eight objects with sparse rewards. We evaluate the sample efficiency of training, conflict rate, domination rate, and completion steps across approaches to demonstrate that DAIR can (i) help discover efficient collaboration strategies; (ii) improve efficiency and safety by avoiding domination and conflict;  (iii) bring adaptation capability with learned task decomposition knowledge; (iv) retain learning capability of synergistic skills.

\textbf{Baselines.} We compare our approach with three baselines: (i) The same architecture as our model with the attention mechanism, but without the intrinsic regularization ({\bf Attention}); (ii) SAC with Multi-layer Perceptron ({\bf MLP}) neural network; (iii) Multi-Agent Deep Deterministic Policy Gradient~\citep{maddpg} with MLP ({\bf MADDPG + MLP}). We also tried replacing DDPG with SAC in MADDPG but observed minor differences. Thus we only report results with {\bf MADDPG + MLP} for simplicity.

{\bf Training details.} All networks and learnable parameters are trained with Adam optimizer \citep{adam} with learning rate $0.0001$, $\beta_1\!=\!0.9,\beta_2\!=\!0.999$. We set the discount factor as $\gamma=0.98$, buffer size as 1M, and batch size as 512 for all tasks. We follow the replay $k$ setting in HER~\citep{her}, and set $k=4$ with the future-replace strategy. We update the network parameters after every two episodes. The episode length equals 50 times the object number for each environment. We train all the methods with 3 seeds and report both the mean and standard derivation for the success rate. More details are in Appendix~\ref{sec:app2}.

\vspace{-0.1in}
\subsection{Sub-Task Allocation for Collaboration}
We first perform our experiments on two tasks that requires alternately operating different parts in a certain order to show DAIR balancedly and safely allocates the sub-tasks for solving one task. The first task is {\it Open Box and Place} (Figure~\ref{fig:teaser} (c)): The robots need to put the blue block object inside the box with a sliding cover, which requires one robot arm to open the sliding cover for the other robot arm to put the object inside. The second task is {\it Push with Door} (Figure~\ref{fig:teaser} (d)): The robots need to push the blue object to the goal on the other side of a sliding green door that requires one robot arm to open it and clear the way for the pushing arm (with the grasping function disabled). In both cases, we also apply a force on the sliding cover/door for it to bounce back to its original position if there is no outside forces.
The following results show that DAIR not only helps achieve better sample efficiency and performance, but more importantly, addresses the problems of \emph{domination} and \emph{conflict}, thus further reduces the steps to finish the task at the same time.

\textbf{Reward setting.} We consider two different reward settings: (i) a \emph{sparse} reward setting where the agents only obtain a reward $1.0$ when the block is on the target position; (ii) a \emph{informative} reward setting which gives a reward $1.0$ when the box/door is open and another reward $1.0$ when the block reaches to the goal. If the block reaches its goal in a trial, we count it as a successful trial.

\begin{figure*}[t]
	\centering
	\includegraphics[width=0.94 \textwidth]{figures/manipulation_comparison.pdf}
	\vspace{-0.15in}
	\caption{\small{Performances of different methods on two bimanual manipulation tasks, \emph{Open Box and Place} (2 on the left) and \emph{Push with Door} (2 on the right).
	We consider two reward settings for each task, (i) a \emph{sparse} reward (right in each group), where agents only receive a success reward when all the goals are reached; (ii) an \emph{informative} reward (left in each group), where agent will additionally receive a reward for reaching each individual goal in addition to the final success reward.}
	}
	\label{fig:manipulation_comparison}
	\vspace{-0.2in}
\end{figure*}
\begin{figure*}[t]
	\centering
	\includegraphics[width=0.94 \textwidth]{figures/lambda_ablation.pdf}
	\vspace{-0.15in}
	\caption{\small{Ablation studies on the value of $\lambda$. Our method is generally robust to the choice of $\lambda$, when even $\large$ is large (e.g., $\lambda$=0.2). In our practice, we choose $\lambda$=0.05 for all the experiments.}}
	\label{fig:lambda_ablation}
		\vspace{-0.25in}
\end{figure*}

\textbf{Comparison on success rate.} We plot the success rate of all the methods over the environment steps in Figure~\ref{fig:manipulation_comparison}. We can observe that DAIR achieves better sample efficiency and better success rate than the baselines in most cases. We also observe that in both environments, DAIR and Attention achieve better success rate in the sparse reward setting than using informative reward. The reason is that sparse reward offers more flexibility for the agents to collaborate under the guidance with intrinsic disentangled attention, while the explicit informative reward can lead to local minimum more easily.

\vspace{-0.05in}
\input{tables/statistics}
\textbf{Ablation on $\lambda$.} We set the hyperparameter  $\lambda=0.05$ (defined in Equation \ref{eq_joint} for balancing the regularization) in all our experiments. To study the stability of DAIR, we perform ablation on different values of $\lambda$ in Figure~\ref{fig:lambda_ablation}. We observe that our method is robust to the change of $\lambda$ from $0.02$ to $0.2$.

\textbf{Comparison on the three criteria.} Table~\ref{statistics}  shows the comparison on the three criteria defined in Section~\ref{sec:criteria}. We observe significant improvements over all the settings using intrinsic regularization, which proves that using disentangled attention can lead to better collaboration. For example, in the task of Open Box and Place with informative reward, our approach achieves almost half less conflict rate, $24\%$ less domination rate, and $12$ fewer steps comparing to the Attention baseline without the intrinsic regularization. For Push with Door using sparse reward, we reduce more than half the conflict rate.

We perform ablation by introducing an extra collision penalty during training: Two robots will receive $-1.0$ reward if their grippers collide to each other. Note that such a reward is \emph{not realistic} in practice since we do not hope the robots to collide to get the reward. We show the \emph{Conflict rate} for both Attention baseline and our approach training with this collision penalty in Table~\ref{statistics_cp}. DAIR shows consistent improvements and remains to be an effective way to reduce conflicts even with the collision penalty.

\textbf{Visualization on attention probability $\alpha$.} We visualize the two tasks in Figure~\ref{fig:heat_map}. In each task, we visualize the attention $\alpha_1$ and $\alpha_2$ for each robot in two rows. Each attention vector $\alpha_i$ contains four items that correspond to left arm (1st column), right arm (2nd column), and the two task-specific interaction regions (last 2 columns). Figure~\ref{fig:heat_map} (a) shows the Push with Door task: The left arm is interacting with the object block, so it has a high value in the corresponding probability $\alpha_{1,3}$ (1st row and 3rd column); the right arm is interacting with the door, it also has a high value in the corresponding probability $\alpha_{2,4}$ (2nd row and 4th column). Similarly in Figure~\ref{fig:heat_map} (b) for the Open Box and Place task, a high probability with $\alpha_{i,j}$ indicates the $i$th arm is interacting with object $j$. The two interaction regions here are the block object (3rd column) and the box cover (4th column).
\begin{wrapfigure}{r}{0.4\textwidth}
\vspace{-0.15in}
    \centering
\includegraphics[width=0.4\textwidth]{figures/heat_map_case.pdf}
	\captionof{figure}{\small{Visualization of attention $\alpha_i$. Each row corresponds to one robot arm attending to four items. (a) \emph{Push with Door}: one robot holds the door while the other pushes the block; (b) \emph{Open Box and Place}: one robot opens the box while the other picks the block.}}
	\label{fig:heat_map}
	\vspace{-0.2in}
\end{wrapfigure}
\vspace{-0.25in}
\subsection{Generalizing to More Objects with Curriculum Learning}
We increase the number of objects and train with curriculum to show that the regularized representation gains better learning capacity when task gradually becomes harder. We test on \emph{Stack Tower} (Figure~\ref{fig:teaser} (b)), where the robots need to stack objects as a tower with indicated goal positions; and \emph{Rearrangement} (Figure~\ref{fig:teaser} (a)), where the robots need to rearrange the objects to their own goal locations on the table. When manipulating one object in these environments, it is easy for the arm to perturb other objects without intention. We train RL agents for both tasks in the \emph{informative reward} setting: the agents will receive a reward $1.0$ when each object reaches its goal. We leverage curriculum learning to start training the agents to manipulate one object and then gradually increase the objects to three to the end.

We evaluate our approach on two aspects: (i) How does the approach perform in each curriculum stage; (ii) How does the approach generalize to object numbers that exceed its training number (up to \textbf{eight} objects in the Rearrangement task). DAIR achieves better results in both aspects, especially in generalization to multiple objects.

{\bf Results on each curriculum stage.} We first compare DAIR to Attention on 3-block Rearrangement and Stack Tower tasks. Note that both \textbf{MLP} and \textbf{MADDPG+MLP} baselines cannot handle a flexible number of objects due to fixed dimensions of inputs. Thus it is not applicable in these two tasks with curriculum learning, and directly training both of them with 3 objects leads to zero success rate. This also suggests that DAIR and Attention have the flexibility to handle a variant number of input objects. DAIR gains significant improvements over Attention in all different training stages. Our gain over Attention even becomes larger as the number of objects increases.

\input{tables/stack_comparison}

{\bf Results on generalization.} We conduct generalization experiments on both tasks where we test the policies trained with $i$ objects on the same environment with $i+k$ objects. We show the results of generalization success rate in Table \ref{curriculum-stack} and \ref{curriculum-rearrange} with the columns labeled by $i\rightarrow i\!+\!k$. For Stack Tower, DAIR trained with 2-block stacking generalizes to stacking 2 towers each with 2 blocks (last column in Table \ref{curriculum-stack}), while Attention completely fails.  For Rearrangement, DAIR can rearrange 8 objects even we only train it to rearrange 3 objects (last column in Table \ref{curriculum-rearrange}), while Attention fails to generalize to even 4 objects. We conjecture that the reason for improvement is similar to conclusions in continual learning area~\citep{delange2021clsurvey}: regularizing parameters with simple $L2$ loss avoids overfit to specific stage of task and gains better generalization ability in continually shifted tasks.

\vspace{-0.11in}
\paragraph{Visualization on stacking and rearrangement.} We visualize the three demonstrations for our approach in Figure~\ref{fig:visualization}: (a) stacking 3 blocks; (b) stacking 2 towers each with 2 blocks using the policy trained with 2 block-stacking; (c) rearranging 8 blocks to their target positions using the rearrangement policy trained with 3 blocks. For stacking tasks, both robots are able to pick up different objects without interrupting the other robot and the stacked tower. For the rearrangement task, the policy is transferred to rearrange 8 objects, far beyond the training object number 3. The two robots are still able to collaborate without conflicts to solve the tasks. Please refer to our project page for more policy visualization in videos.
\begin{figure*}[t]
	\centering
	\includegraphics[width= 0.99\textwidth]{figures/vis.pdf}
	\vspace{-0.1in}
	\caption{\small{Visualization of bimanual manipulation. For each object, we represents its goal as a transparent dot in the same color. (a) Both arms are picking up objects and alternatively stacking them into a tower; (b) To stack two tower, each arm is working on one tower that is close to it; (c) We show the two arms can collaborate without conflict to pick up the 8 objects to their target locations.} }
	\label{fig:visualization}
\vspace{-0.2in}
\end{figure*}
\vspace{-0.1in}
\subsection{Synergistic Behaviors Discovery}
\input{tables/bar_statistics}
Besides manipulation tasks with multiple objects, DAIR retains the ability to help two robots collaboratively manipulate one object with multiple interaction regions. To analyze the ability on such synergistic skills learning, we conduct experiment on the \emph{Adjust Bar} task, as visualized in Figure~\ref{fig:teaser} (e). This task requires the two robots to lift and rotate a heavy bar to target height and orientation. We lock the gripper to the closed state and set the bar's mass and size large so the two robots have to collaborate to finish the task. The interaction region state inputs are represented by the 3D position of two sides of the bar. The goal is defined by the target positions of the two sides of the bar. The reward is sparse that only an accurate adjustment gives 1.0 to two robots.
DAIR maintains advantage on performance over the Attention baseline as shown in Table~\ref{table:adjust_bar} and Figure~\ref{fig:adjust_bar}. Two robots can successfully clamp up the bar with the opposite forces with very consistent movements. Such results suggest that DAIR is not harmful for learning synergistic skills when it encourages the robots to look at different interaction regions. DAIR also serves as a mechanism to avoid overfitting to domination and help robots to finish the task efficiently with smaller finish steps.

\section{Conclusion}
\label{Conclusion}
 While previous works consider how to learn collaborative skills like synergistic behavior, we notice two main limitations in complex bimanual manipulation tasks: {\it domination} and {\it conflict}, corresponding to the efficiency and safety of control. We propose a simple and effective DAIR to solve these problems. We validate our approach on challenging bimanual manipulation tasks with multiple objects (up to 8 objects) or one objects with multiple interaction regions. We demonstrate that DAIR not only reduces the domination and conflict problems but also improves the generalization ability of the policies to manipulate much more objects than in the training environments. We hope our work contributes as a step towards safe robotics.

\section{Task Descriptions and Details}
\label{sec:app1}
\begin{figure*}[htb]
	\centering
	\includegraphics[width= \textwidth]{figures/task_fig.pdf}
 	\vspace{-0.15in}
 	\caption{The environments used in our experiments}
	\label{fig:task_fig}
\end{figure*}

\subsection{Environment Descriptions}

\paragraph{Push with Door, Figure~\ref{fig:task_fig}(a).}
The two robots are placed on both sides of a 100cm × 70cm table, opposite each other (all robot manipulation environments are same for this setting).
The goal is to push a block through a sliding door and make it reach the target position on the other side of the door. We put a spring on the sliding door, such that it will close automatically in the absence of external force.  The initial positions of the projections of the two grippers onto the table plane are sampled in a 40cm $\times$ 40cm square on the table  (all robot manipulation environments are same for this setting). The initial position of block and the goal position are sampled from a circle with radius 20cm around the table center. We fix the initial height of the two grippers  (all robot manipulation environments are same for this setting), and set the initial position of door at the center of the table.

\paragraph{Open box and Place, Figure~\ref{fig:task_fig}(b).}
The task is to pick up the block on the table and place it into the box in table center. We put a spring on the sliding lid of the box, so it will close automatically in the absence of external force. The initial positions of the block and goal are sampled from a circle with radius 20cm at the center of the table, outside the box.

\paragraph{Stack Tower,  Figure~\ref{fig:task_fig}(c).}
The task is to stack several blocks into a tower. All blocks are randomly sampled from a circle with radius 20cm around the center of the table. We perform curriculum learning in this environment, with one more block sampled in each stage. In the first stage in the curriculum, we have one block, we sample the corresponding goal with the height randomly from 0cm to 30cm. In each following stage, we sample one more object block and goal. After the first stage, the goals will form into a tower.

\paragraph{Rearrangement, Figure~\ref{fig:task_fig}(d).}
The task is to push multiple blocks to their corresponding target positions on the table. We perform curriculum learning in this environment, with one more block sampled in each stage. All blocks and goals are randomly sampled from a circle with radius 20cm around the table center. We train our method up to 3 blocks (3 curriculum stages) and generalize the approach to up to 8 blocks.

\paragraph{Adjust Bar, Figure~\ref{fig:task_fig}(e).}
The task is to adjust a heavy bar to the state that the two sides of it match with the two blue goal positions. That requires the height and the orientation are matched. We randomly sample the two goal positions but make sure the distance between them equals to the length of bar. We fix the gripper to force two robots synergistically use the force in opposite direction to pinch up the bar.

\subsection{Observation space}
The observation vector consists of object states, robot states, and the goals for the objects. Specifically, the object states consist of the position and velocity of all the objects. The robot stages consist of the position and velocity of the gripper and the robot joints. The goal vector consists of the target position coordinates.

\subsection{Action space}
For robot tasks, the action is a 8-dimensional vector, which is the concatenation of two 4-dimensional action vectors for each robot. For each robot, the first 3 elements indicates the desired position shift
of the end-effector and the last element controls the gripper fingers (locked in push with door scenario). For mass point tasks, the action is a 4-dimensional vector, similarly combined with two 2-dimensional vectors for each mass point. The 2-dimensional action vector only controls the the desired position shift
of the end-effector in a plane.

\section{Training Sample Number}
\label{sec:app2}
For {\it Push with Door} and {\it Open Box and Place}, we use 10M samples to train; For {\it Tower Stack} and {\it Rearrangement}, we leverage curriculum learning and increase the number of blocks by one in each stage. Specifically, we list the number of samples for each stage of training in Table~\ref{table:training_runs}.
\input{tables/training_runs}

\textbf{Computation.}
In our experiments, we use a single GPU and 8 CPU cores for all the method on each task.

\section{Video results}
Please refer to our project page: \url{https://mehooz.github.io/bimanual-attention/}.

\section{Reproducibility Statement}

To ensure the reproducibility of our work, we provide the following illustrations in our paper and appendix:
\begin{itemize}
    \item \textbf{Environment}: We provide the detailed description of the environment in  Appendix \ref{sec:app1}.
    \item \textbf{Evaluation Criteria}: We provide the detailed evaluation criteria for domination, conflict and finsh steps in Section \ref{sec:criteria}.
    \item \textbf{Implementation Details}: We provide all implementation details and related hyperparameters in the end of  Section \ref{subsec:DAIR}, the beginning of \ref{Experiments} and Appendix \ref{sec:app2}.
\end{itemize}

We are committed to releasing the code for our approach, the baselines, and the simulation environment. We believe the open source of our code, the task sets, and evaluation code will be an important contribution to the robotics community. We have released our videos in project page: \url{https://mehooz.github.io/bimanual-attention/}, and we will release the code and environment on the same website upon publication.